spring-boot集成spark并使用spark-sql

需要注意的是依赖中排除掉的日志模块，以及特殊的打包方式

定义配置类:

SparkContextBean.class

  @Configuration
 public   class  SparkContextBean {

     private String appName =  "sparkExp";

     private String master =  "local";

     @Bean
     @ConditionalOnMissingBean(SparkConf.class)
      public SparkConf  sparkConf ()  throws Exception {
        SparkConf conf =  new SparkConf().setAppName(appName).setMaster(master);
         return conf;
    }

     @Bean
     @ConditionalOnMissingBean
      public JavaSparkContext  javaSparkContext ()  throws Exception {
         return  new JavaSparkContext(sparkConf());
    }

     @Bean
     @ConditionalOnMissingBean
      public HiveContext  hiveContext ()  throws Exception {
         return  new HiveContext(javaSparkContext());
    }
    ......
}
                                     

启动类：

StartApplication.class

  @SpringBootApplication
 public   class  StartApplication  implements  CommandLineRunner {

     @Autowired
     private HiveContext hc;

      public  static  void  main (String[] args) {
        SpringApplication.run(StartApplication.class, args);
    }

     @Override
      public  void  run (String... args)  throws Exception {
        DataFrame df = hc.sql( "select count(1) from LCS_DB.STAFF_INFO");
        List<Long> result = df.javaRDD().map((Function<Row, Long>) row -> {
             return row.getLong( 0);
        }).collect();
        result.stream().forEach(System.out::println);
    }
}
                          

执行方式：

  spark-submit      --class com.cord.StartApplication       --executor-memory 4G      --num-executors 8      --master yarn-client /  data/cord/spark-example-1.0- SNAPSHOT.jar
         

参考链接：

https://stackoverflow.com/questions/45189701/submitting-spring-boot-application-jar-to-spark-submit

https://my.oschina.net/woter/blog/1843755

 

 转自：https://www.cnblogs.com/cord/p/9530404.html

spring-boot集成spark并使用spark-sql

标签：appname   local   特殊   slf4j   pil   transform   注意   reduce   overflow   

查看更多关于spring-boot集成spark并使用spark-sql的详细内容...