Kafka sink启动失败

Question

老师您好，按照视频中的配置，flume总是启动失败。pom文件中的kafka依赖已经注释掉了。Flume和Kafka下的lib文件夹中的jar包都对应上了，但是还是会报下面的错误：

2019-06-18 11:17:19,907 (lifecycleSupervisor-1-3) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6f3f9da3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoClassDefFoundError: org/apache/kafka/common/utils/Utils
	at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:221)
	at kafka.producer.Producer.<init>(Producer.scala:61)
	at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
	at org.apache.flume.sink.kafka.KafkaSink.start(KafkaSink.java:164)
	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.Utils
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more
2019-06-18 11:17:19,909 (lifecycleSupervisor-1-3) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:264)] Unsuccessful attempt to shutdown component: {} due to missing dependencies. Please shutdown the agentor disable this component, or the agent will bein an undefined state.
java.lang.NullPointerException
	at org.apache.flume.sink.kafka.KafkaSink.stop(KafkaSink.java:171)
	at org.apache.flume.sink.DefaultSinkProcessor.stop(DefaultSinkProcessor.java:53)
	at org.apache.flume.SinkRunner.stop(SinkRunner.java:115)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:259)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

这个还是jar包冲突的问题么？

Answer 1

Michael_PK 2019-06-18 11:39:30

这个看起来是版本的兼容问题，你的版本和上课一样吗？版本不兼容导致里面的依赖的东西有冲突

• 提问者 Clay_mu #1

  scala是2.11.12，其他都是一样的。问题出在scala版本上了么？

• Michael_PK 回复 提问者 Clay_mu #2

  异常信息里有： by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.Utils，这个看起来就是整合的时候版本不兼容

• 提问者 Clay_mu 回复 Michael_PK #3

  好的，谢谢老师。我再查一下。换个scala试试