请稍等 ...
×

采纳答案成功！

向帮助你的同学说点啥吧！感谢那些助人为乐的人

java.lang.NoSuchMethodError错误

老师您好,按照您说的配置了一遍,4-9中的TestApp能够通过,但是在这一章的时候出现了这个错误,找了很长时间不知道是哪里的问题,只能过来求助了…
Exception in thread “dag-scheduler-event-loop” java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;

20/03/26 12:38:49 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
This is Imooc Spark Project... 
20/03/26 12:38:50 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 165.8 KB, free 1948.9 MB)
20/03/26 12:38:50 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 16.7 KB, free 1948.9 MB)
20/03/26 12:38:50 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoop000:42985 (size: 16.7 KB, free: 1949.1 MB)
20/03/26 12:38:50 INFO spark.SparkContext: Created broadcast 0 from wholeTextFiles at CustomDatasourceRelation.scala:81
20/03/26 12:38:50 INFO codegen.CodeGenerator: Code generated in 157.36685 ms
20/03/26 12:38:51 INFO codegen.CodeGenerator: Code generated in 14.523292 ms
20/03/26 12:38:51 INFO input.FileInputFormat: Total input paths to process : 1
20/03/26 12:38:51 INFO input.FileInputFormat: Total input paths to process : 1
20/03/26 12:38:51 INFO spark.SparkContext: Starting job: show at Demo.scala:13
20/03/26 12:38:51 INFO scheduler.DAGScheduler: Got job 0 (show at Demo.scala:13) with 1 output partitions
20/03/26 12:38:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (show at Demo.scala:13)
20/03/26 12:38:51 INFO scheduler.DAGScheduler: Parents of final stage: List()
20/03/26 12:38:51 INFO scheduler.DAGScheduler: Missing parents: List()
20/03/26 12:38:51 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[8] at show at Demo.scala:13), which has no missing parents
Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;
	at org.apache.spark.rdd.NewHadoopRDD.getPreferredLocations(NewHadoopRDD.scala:310)
	at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:275)
	at org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:275)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:274)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2005)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:2016)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:2015)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2015)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:2013)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:2013)
	at org.apache.spark.scheduler.DAGScheduler.getPreferredLocs(DAGScheduler.scala:1979)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$17.apply(DAGScheduler.scala:1112)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$17.apply(DAGScheduler.scala:1110)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1110)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:1069)
	at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1013)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)

另外,我本地的maven仓库变成这样了
图片描述
羁书贵公子 2020-03-26 12:46:03
源自：4-10  开发环境依赖使用说明
1321
收起

正在回答 回答被采纳积分+3

4回答

哆啦A梦爱学习吖 2020-09-10 22:46:51

版本都按照老师提供的，还是出现了这个问题，怎么解决的？

0 回复 有任何疑惑可以回复我~
收起回答
  • Michael_PK #1 
    这个是依赖包冲突了，你需要把hbase中依赖的hadoop相关的依赖都execlude掉，就可以了。你看报错信息，那明显依赖的hadoop版本是老的，getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;。学过hadoop的都知道mapred这个包是老的，在不用这个的。
    回复 有任何疑惑可以回复我~ 2020-09-10 22:55:04
  • 哆啦A梦爱学习吖 回复 Michael_PK #2 
    谢谢老师
    回复 有任何疑惑可以回复我~ 2020-09-11 00:09:24
  • Michael_PK 回复 哆啦A梦爱学习吖 #3 
    如果搞不定，QQ群里加我QQ，明晚找个时间给你远程搞定
    回复 有任何疑惑可以回复我~ 2020-09-11 00:10:05
点开查看后面1条评论
取消 回复
Michael_PK 2020-08-19 10:39:52

NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;这个错看起来应该是hadoop版本不兼容，你用的是Hadoop什么版本？

0 回复 有任何疑惑可以回复我~
收起回答
取消 回复
每天努力一点 2020-08-19 07:54:10

请问以下解决了么？我也遇到同样的问题

0 回复 有任何疑惑可以回复我~
收起回答
取消 回复
Michael_PK 2020-03-26 13:37:00

从这个错：java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo() 来看，可能是依赖的hadoop版本有不兼容。你这个spark是什么版本？对应的hadoop是什么版本

0 回复 有任何疑惑可以回复我~
收起回答
  • 提问者 羁书贵公子 #1 
    Spark 版本:2.4.2(用的老师给的安装包)
Scala 版本:2.11.12
Hadoop 版本:Hadoop 2.6.0-cdh5.15.1
    回复 有任何疑惑可以回复我~ 2020-03-26 13:40:15
  • Michael_PK 回复 提问者 羁书贵公子 #2 
    你操作的过程详细描述下看看，是在idea还是在spark-shell 以及执行了什么代码
    回复 有任何疑惑可以回复我~ 2020-03-26 13:49:22
  • 提问者 羁书贵公子 回复 Michael_PK #3 
    马上，我现在看一下
    回复 有任何疑惑可以回复我~ 2020-03-26 13:57:49
点开查看后面10条评论
取消 回复

相似问题

jmeter执行线程报错 ，找不到方法
3523 0 2
java.lang.NoSuchMethodError 500错误
570 0 3
500错误
696 0 2
错误
673 0 4
java.lang.NoSuchMethodError 500错误
436 0 2

登录后可查看更多问答，登录/注册

问题已解决，确定采纳
还有疑问，暂不采纳
Spark进阶 大数据离线与实时项目实战
  • 参与学习       698    人
  • 解答问题       190    个

大数据生态圈实用框架(Spark/Hbase/Redis/Hadoop)整合应用及调优

了解课程

本课精华内容

问答 作业

java.lang.NoSuchMethodError错误

1.3k 21

java.lang.OutOfMemoryError: Java heap space错误

836 11

IDEA 无法访问远程虚拟机的datanode拿到数据

1.7k 11

关于虚拟机设置机器名

1.1k 11

编译支持hadoop2.6.0-cdh5.15.1的spark3.0.0失败

1.4k 11

查看更多本课问答
意见反馈 帮助中心 APP下载
官方微信