Doris BrokerLoad报错 Scan bytes per broker scanner exceed limit: 3221225472

brokerLoad语句

LOAD
LABEL gaofeng_broker_load_HDD
(DATA INFILE("hdfs://eoop/user/coue_data/hive_db/couta_test/ader_lal_offline_0813_1/*")INTO TABLE ads_user
)WITH BROKER "hdfs_broker"
("dfs.nameservices"="eadhadoop","dfs.ha.namenodes.eadhadoop" = "nn1,nn2","dfs.namenode.rpc-address.eadhadoop.nn1" = "h4:8000","dfs.namenode.rpc-address.eadhadoop.nn2" = "z7:8000","dfs.client.failover.proxy.provider.eadhadoop" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider","hadoop.security.authentication" = "kerberos","kerberos_principal" = "ou3.CN","kerberos_keytab_content" = "BQ8uMTYzLkNPTQALY291cnNlXgAAAAFfVyLbAQABAAgCtp0qmxxP8QAAAAE="
);

报错

任务Cancelled

type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Scan bytes per broker scanner exceed limit: 3221225472

解决

该Doris测试环境为3个BE节点，而FE的配置max_bytes_per_broker_scanner默认为3G，所要导入的文件共有大约13GB
需要修改参数
FE执行如下动态修改参数命令
ADMIN SET FRONTEND CONFIG ("max_bytes_per_broker_scanner" = "5368709120");
修改为5G，这样一样来，集群最大可导入的文件大小为5G * 3(个BE)=15GB
再次执行即可