报错

Question

首先报错是：
[hadoop@hadoop000 lib]$ ./streaming.sh
Ivy Default Cache set to: /home/hadoop/.ivy2/cache
The jars for the packages stored in: /home/hadoop/.ivy2/jars
:: loading settings :: url = jar:file:/home/hadoop/app/spark-3.0.0-bin-2.6.0-cdh5.16.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-streaming_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-77e344fc-074a-4eeb-9448-ec0e19644377;1.0
confs: [default]
:: resolution report :: resolve 157ms :: artifacts dl 1ms
:: modules in use:
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 0 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-77e344fc-074a-4eeb-9448-ec0e19644377
confs: [default]
0 artifacts copied, 0 already retrieved (0kB/10ms)
17:47:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Error: Failed to load class com.meng.bigdata.project.stream.StreamingApp.
17:47:20 INFO ShutdownHookManager: Shutdown hook called
17:47:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-7b229ce8-9090-4810-888d-1d062ce11549
[hadoop@hadoop000 lib]$

配置如下：
vi streaming.sh

idea代码中的

我clear掉了所有的缓存，重新package生成的jar包，我看配置什么都没错误，老师求解

Answer 1

Michael_PK 2021-02-13 17:58:37

Error: Failed to load class com.meng.bigdata.project.stream.StreamingApp

很明显，要么你的jar里面没有这个class，要么你的jar的路径不对

• 提问者 慕九州3016327 #1

  确实有这个类，也对比了。这个应该怎么搞嘞

• 提问者 慕九州3016327 #2

  老师可以了，报了个错，我也加了序列化参数可还是报序列化错误。
  Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer
          at com.meng.bigdata.project.stream.StreamingApp$.main(StreamingApp.scala:24)
          at com.meng.bigdata.project.stream.StreamingApp.main(StreamingApp.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
          at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)

• Michael_PK 回复 提问者 慕九州3016327 #3

  你是整合了kafka了吧，这里少了ss和kafka整合的那个jar没有。要么package加上，要么jars加上

点开查看后面8条评论