Starrocks-SparkDpp

SparkEtlJob

新版

processData()

RollupTree

LOAD

硬编码int aggregateConcurrency = 200;

initSparkConfigs(fileGroup.hiveTableProperties)

并发

原版

ETL

串行分层遍历RollupTree

isHive？

doDpp()

hdfs://127.0.0.1:10000/jobconfig.json

SparkDpp

data quality don't check for orc/parquet load

EtlJobConfig.SourceType.FILE

SparkLoadPendingTask

submit job

init()

processDpp()

EtlJobConfig.SourceType.HIVE

USE starrocks_demo;LOAD LABEL starrocks_demo.label1(    DATA INFILE(\"hdfs://127.0.0.1:1000/starrocks-demo/data/demo3_data1*\")    INTO TABLE demo3_spark_tb1    COLUMNS TERMINATED BY \"\\t\