执行日志清理时退出

Question

老师下面是，打出的错误是指信息，是由于资源不够吗？  使用的是阿里云的服务器2核4G的， hadoop后台显示的Memory Total是8G

17/11/09 23:57:33 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job

17/11/09 23:57:33 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 

17/11/09 23:57:33 INFO storage.BlockManagerMaster: Removal of executor 4 requested

17/11/09 23:57:33 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 4

17/11/09 23:57:33 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.

17/11/09 23:57:33 INFO cluster.YarnScheduler: Cancelling stage 1

17/11/09 23:57:33 INFO scheduler.DAGScheduler: ResultStage 1 (save at SparkStatCleanJobYARN.scala:29) failed in 82.394 s due to Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, hadoop001, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container marked as failed: container_1510122812790_0009_02_000002 on host: hadoop001. Exit status: 1. Diagnostics: Exception from container-launch.

Container id: container_1510122812790_0009_02_000002

Exit code: 1

Stack trace: ExitCodeException exitCode=1: 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)

at org.apache.hadoop.util.Shell.run(Shell.java:478)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 1

Answer 1

Michael_PK 2017-11-10 02:07:45

从这日志看应该是yarn有问题，你到失败作业的yarn页面上，仔细查看下各个链接的日志信息

• 提问者 慕粉2110073833 #1

  java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.  日志 里是出现这么个异常

• Michael_PK 回复 提问者 慕粉2110073833 #2

  这个错你搜索下看看