spring-boot集成spark并使用spark-sql

需要注意的是依赖中排除掉的日志模块，以及特殊的打包方式

定义配置类:

SparkContextBean.class

@Configuration
public class SparkContextBean {

    private String appName = "sparkExp";

    private String master = "local";

    @Bean
    @ConditionalOnMissingBean(SparkConf.class)
    public SparkConf sparkConf() throws Exception {
        SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
        return conf;
    }

    @Bean
    @ConditionalOnMissingBean
    public JavaSparkContext javaSparkContext() throws Exception {
        return new JavaSparkContext(sparkConf());
    }

    @Bean
    @ConditionalOnMissingBean
    public HiveContext hiveContext() throws Exception {
        return new HiveContext(javaSparkContext());
    }
    ......
}

启动类：

StartApplication.class

@SpringBootApplication
public class StartApplication implements CommandLineRunner {

    @Autowired
    private HiveContext hc;

    public static void main(String[] args) {
        SpringApplication.run(StartApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {
        DataFrame df = hc.sql("select count(1) from LCS_DB.STAFF_INFO");
        List<Long> result = df.javaRDD().map((Function<Row, Long>) row -> {
            return row.getLong(0);
        }).collect();
        result.stream().forEach(System.out::println);
    }
}

执行方式：

spark-submit     --class com.cord.StartApplication      --executor-memory 4G     --num-executors 8     --master yarn-client /data/cord/spark-example-1.0-SNAPSHOT.jar

参考链接：

https://stackoverflow.com/questions/45189701/submitting-spring-boot-application-jar-to-spark-submit

https://my.oschina.net/woter/blog/1843755

 

 转自：https://www.cnblogs.com/cord/p/9530404.html

spring-boot集成spark并使用spark-sql

标签：appname   local   特殊   slf4j   pil   transform   注意   reduce   overflow