使用spark-submit时报以下错误

Question

spark-submit \
--class com.imooc.spark.FlumePushWordCount \
--master local[2] \
--packages org.apache.spark:spark-streaming-flume_2.11:2.2.0 \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000 41414

java.lang.NoClassDefFoundError: org/apache/flume/source/avro/AvroSourceProtocol
	at org.apache.spark.streaming.flume.FlumeReceiver.responder$lzycompute(FlumeInputDStream.scala:146)
	at org.apache.spark.streaming.flume.FlumeReceiver.responder(FlumeInputDStream.scala:145)
	at org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:162)
	at org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:169)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:607)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

从头到尾一直循环报这个错误，最后内存一处自动关闭了

Answer 1

提问者 96年的nash 2019-11-05 16:17:38

本地环境联调时没毛病，在服务器上就不行了

• Michael_PK #1

  找不到那个包，需要去maven仓库找到那个包，submit时提交上去

• 提问者 96年的nash 回复 Michael_PK #2

  --packages里面不是spark-streaming-flume_2.11:2.2.0把这个包提交上去了吗？还是说只提交这个不够？

• Michael_PK 回复 提问者 96年的nash #3

  不一定的我，有些时候会有缺失。没有一定的说法

点开查看后面2条评论