Flume配置文件报错

Question

老师您好，我在本地联调的时候Flume是正常启动的，但是在SparkStreaming应用程序启动以后Flume就会报错。
错误信息：

2019-05-30 00:27:28,705 (Spark Sink Processor Thread - 10) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:80)] Error while processing transaction.
java.lang.IllegalStateException: begin() called when transaction is OPEN!
	at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
	at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:114)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:113)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.populateEvents(TransactionProcessor.scala:113)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:243)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:43)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-05-30 00:27:28,706 (Spark Sink Processor Thread - 10) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Spark was unable to successfully process the events. Transaction is being rolled back.
2019-05-30 00:27:28,706 (New I/O  worker #1) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Received an error batch - no events were received from channel! 

我使用的是2.3.3版本的Spark，但是maven的依赖中没有2.3.3的依赖，所以用的是2.3.2的依赖，在Push style中是可以运行的。会不会是因为Spark的版本问题啊？

Answer 1

Michael_PK 2019-05-30 11:11:16

这些都是本地一起测试是吗？启动顺序OK的不？

• 提问者 Clay_mu #1

  是在本地。启动顺序是：启动Flume，启动telnet，启动SparkStreaming应用程序。
  
  所有的配置都是根据课程内容配置的。

• Michael_PK 回复 提问者 Clay_mu #2

  你不同版本我没测试过，个人觉得版本之间应该问题不大，还是怀疑配置的问题。你找一个软件对比工具，对比下上课的试试

• 提问者 Clay_mu 回复 Michael_PK #3

  配置文件都是一样的T^T没有差异，很费解为什么会出现这种情况。

点开查看后面1条评论

Answer 2

Michael_PK 2019-05-30 11:11:48

看样子还是对接的sink配置有点问题

• 提问者 Clay_mu #1

  老师，问题解决了！！！
  https://stackoverflow.com/questions/34657391/unable-to-pull-events-in-spark-streaming-application-from-flume
  参考的是这个链接里的解决方案。根据第三个回答，我去Flume的lib文件中看了一下，lib文件中的scala-library jar包的版本是2.10.1，我们课程里使用的是2.11.x的scala。所以我先把2.10.1的jar包删掉，然后下载对应scala版本的jar包放进去就解决掉这个问题了。