Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

Question

我在YARN上运行第5章的MapReduce任务时发生了以下的报错：
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:392)
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:307)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:366)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)
请问大家这个问题要怎么解决呢？

Answer 1

纪墨 2019-03-01 12:56:15

补充一下

hdfs-site.xml中hadoop.tmp.dir配的什么目录，在yarn-site.xml的yarn.nodemanager.local-dirs中就配这个目录加上nm-local-dir

比如hdfs-site.xml中内容：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/export/servers/tmp</value>
</property>
</configuration>

yarn-site.xml中就配置

<?xml version="1.0"?>

<configuration>
 
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/export/servers/tmp/nm-local-dir</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

• yst997 #1

  我也遇到了同样的问题，按你的方法已解决

• 雷精 #2

  修改了yarn 的配置文件，需要重启yarn

Answer 2

提问者 WuCh1k1n 2019-02-07 17:30:52

# yarn.nodemanager.local-dirs对应的路径必须在hdfs-site.xml中的hadoop.tmp.dir路径下，否则yarn会报错找不到缓存文件

<property>

    <name>yarn.nodemanager.local-dirs</name>

    <value>hadoop.tmp.dir/nm-local-dir</value>

</property>

Answer 3

Michael_PK 2019-02-07 03:27:39

给点提示：这个错误是不同版本之间配置文件中的属性值兼容性不够导致的。有些属性必须要写在指定的配置文件中。乳沟我没记错的话，应该是HDFS的属性配置错文件了