打开日志之后，任务老是被 kill

Question

之前的任务提交到 yarn 都成功了。但是，刚才配置 spark log 之后，每次提交都被杀死了，我看日志里面也没有特别的错误

每次都报已杀死

下面是在 yarn 里面查看到的日志，只是看的 driver 退出了，也没有报错

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-pain/nm-local-dir/usercache/pain/filecache/49/__spark_libs__7350116270074248892.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pain/app/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/06/20 23:03:06 INFO util.SignalUtils: Registered signal handler for TERM
20/06/20 23:03:06 INFO util.SignalUtils: Registered signal handler for HUP
20/06/20 23:03:06 INFO util.SignalUtils: Registered signal handler for INT
20/06/20 23:03:06 INFO spark.SecurityManager: Changing view acls to: pain
20/06/20 23:03:06 INFO spark.SecurityManager: Changing modify acls to: pain
20/06/20 23:03:06 INFO spark.SecurityManager: Changing view acls groups to: 
20/06/20 23:03:06 INFO spark.SecurityManager: Changing modify acls groups to: 
20/06/20 23:03:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(pain); groups with view permissions: Set(); users  with modify permissions: Set(pain); groups with modify permissions: Set()
20/06/20 23:03:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/06/20 23:03:07 INFO yarn.ApplicationMaster: Preparing Local resources
20/06/20 23:03:10 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1591835801011_0012_000001
20/06/20 23:03:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
20/06/20 23:03:10 INFO yarn.YarnRMClient: Registering the ApplicationMaster
20/06/20 23:03:11 INFO client.TransportClientFactory: Successfully created connection to hadoop000/172.19.32.157:45390 after 82 ms (0 ms spent in bootstraps)
20/06/20 23:03:12 INFO yarn.ApplicationMaster: 
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
    SPARK_YARN_STAGING_DIR -> hdfs://hadoop000:8020/user/pain/.sparkStaging/application_1591835801011_0012
    SPARK_USER -> pain
    PYTHONPATH -> {{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.10.7-src.zip

  command:
    {{JAVA_HOME}}/bin/java \ 
      -server \ 
      -Xmx1024m \ 
      -Djava.io.tmpdir={{PWD}}/tmp \ 
      '-Dspark.driver.port=45390' \ 
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ 
      -XX:OnOutOfMemoryError='kill %p' \ 
      org.apache.spark.executor.CoarseGrainedExecutorBackend \ 
      --driver-url \ 
      spark://CoarseGrainedScheduler@hadoop000:45390 \ 
      --executor-id \ 
      <executorId> \ 
      --hostname \ 
      <hostname> \ 
      --cores \ 
      1 \ 
      --app-id \ 
      application_1591835801011_0012 \ 
      --user-class-path \ 
      file:$PWD/__app__.jar \ 
      1><LOG_DIR>/stdout \ 
      2><LOG_DIR>/stderr

  resources:
    pyspark.zip -> resource { scheme: "hdfs" host: "hadoop000" port: 8020 file: "/user/pain/.sparkStaging/application_1591835801011_0012/pyspark.zip" } size: 591289 timestamp: 1592665365552 type: FILE visibility: PRIVATE
    py4j-0.10.7-src.zip -> resource { scheme: "hdfs" host: "hadoop000" port: 8020 file: "/user/pain/.sparkStaging/application_1591835801011_0012/py4j-0.10.7-src.zip" } size: 42437 timestamp: 1592665365635 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "hadoop000" port: 8020 file: "/user/pain/.sparkStaging/application_1591835801011_0012/__spark_libs__7350116270074248892.zip" } size: 285982907 timestamp: 1592665365006 type: ARCHIVE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "hadoop000" port: 8020 file: "/user/pain/.sparkStaging/application_1591835801011_0012/__spark_conf__.zip" } size: 220717 timestamp: 1592665366166 type: ARCHIVE visibility: PRIVATE

===============================================================================
20/06/20 23:03:12 INFO yarn.YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
20/06/20 23:03:12 INFO yarn.YarnAllocator: Submitted 2 unlocalized container requests.
20/06/20 23:03:13 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
20/06/20 23:03:13 INFO yarn.YarnAllocator: Launching container container_1591835801011_0012_01_000002 on host hadoop000 for executor with ID 1
20/06/20 23:03:13 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
20/06/20 23:03:14 INFO yarn.YarnAllocator: Launching container container_1591835801011_0012_01_000003 on host hadoop000 for executor with ID 2
20/06/20 23:03:14 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
20/06/20 23:03:32 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. hadoop000:45390
20/06/20 23:03:32 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
20/06/20 23:03:33 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
20/06/20 23:03:33 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
20/06/20 23:03:33 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://hadoop000:8020/user/pain/.sparkStaging/application_1591835801011_0012
20/06/20 23:03:34 INFO util.ShutdownHookManager: Shutdown hook called

Answer 1

Michael_PK 2020-06-20 23:50:23

1） 刚才配置 spark log  确定配置的是否OK，我估计应该不会配置错

2） 你的机器是什么配置？你启动了一个Spark HistoryServer的进程，是耗费资源的，如果你的机器资源很低，那么有可能就受到影响了，进而导致你的作业由于资源受限，被kill掉了

• 提问者 pain7 #1

  1. 配置感觉没有问题
  2. Spark HistoryServer 有时候会突然挂掉。我用的阿里云，1 核 2g  的
  3. 我刚才又提交了，用的是 cluster 模式，这次没有出现“已杀死”三个字了。但是界面好想没有我的打印输出，然后我在 yarn 的 ui 界面打开应用记录，又 stderr, stdout 两项，里面都有输出，stuout 就是之前能在界面上看到我在打印的输出日志

• Michael_PK 回复 提问者 pain7 #2

  1c2g。。。。只能玩玩Linux还差不多，进程多点就不够的，你这问题基本可以确定出在这里

• Michael_PK 回复 提问者 pain7 #3

  我一般上课的云主机都是8c16g的，不然全流程大数据的都够呛

点开查看后面3条评论

Answer 2

提问者 pain7 2020-06-21 00:01:38

两个里面都有内容