spark 写ES报错

Question

PK哥

1. 确认了ES可以写入， 通过kibana console 可以写入
2. 网络通信正常。
   请问什么原因呢，
   raceback (most recent call last):
   File “/data/spark/script/spark032302.py”, line 39, in
   result2014.selectExpr(“Grade as grade”,“count”, “percent”).write.format(“org.elasticsearch.spark.sql”).option(“es.nodes”, “172.20.224.90:9200”).mode(“overwrite”).save(“index_weaes/weather”)
   File “/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/pyspark.zip/pyspark/sql/readwriter.py”, line 705, in save
   File “/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1257, in call
   File “/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/pyspark.zip/pyspark/sql/utils.py”, line 63, in deco
   File “/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”, line 328, in get_return_value

Answer 1

提问者 爬行的蜗牛2019 2020-03-27 19:07:21

将log4j.rootCategory=INFO, console改为log4j.rootCategory=WARN, console

输出信息：

./spark-submit --master local[2] --jars /data/es/elasticsearch-spark-13_2.11-7.4.2.jar /data/spark/script/spark032302.py 

20/03/27 18:57:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Traceback (most recent call last):

  File "/data/spark/script/spark032302.py", line 39, in <module>

    result2014.selectExpr("Grade as grade","count", "percent").write.format("org.elasticsearch.spark.sql").option("es.nodes", "172.20.224.90:9200").mode("overwrite").save("index_weaes/weather")

  File "/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 705, in save

  File "/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__

  File "/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco

  File "/root/app/spark/spark-2.3.1-bin-2.6.0-cdh5.7.0/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o80.save.

: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;

        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)

        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)

        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)

        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)

        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)

        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)

        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)

        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)

        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)

        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)

        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)

        at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)

        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)

        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)

        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)

        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)

        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:225)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

        at py4j.Gateway.invoke(Gateway.java:282)

        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

        at py4j.commands.CallCommand.execute(CallCommand.java:79)

        at py4j.GatewayConnection.run(GatewayConnection.java:238)

        at java.lang.Thread.run(Thread.java:748)

• Michael_PK #1

  : org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;
  
  这个es的spark外部数据源的依赖加到环境里面了吗

• 提问者 爬行的蜗牛2019 回复 Michael_PK #2

  有的， 我的ES 是7.2 的
  root@spark:bin]$  cd /data/es/
  [root@spark:es]$ ll
  total 1572
  -rw-r--r-- 1 root root 852461 Mar 24 09:10 elasticsearch-spark-13_2.11-7.4.2.jar
  -rw-r--r-- 1 root root 753388 Mar 21 17:45 elasticsearch-spark-20_2.11-6.0.0.jar

• Michael_PK 回复 提问者 爬行的蜗牛2019 #3

  这两的版本对不上把

点开查看后面2条评论

Answer 2

Michael_PK 2020-03-24 18:24:23

这个异常信息看不出什么问题，降低下日志级别看看，有没有更详细的日志

• 提问者 爬行的蜗牛2019 #1

  可否抽空帮看看呢，

• Michael_PK 回复 提问者 爬行的蜗牛2019 #2

  降低日志级别看看，没有具体日志不好分析

• 提问者 爬行的蜗牛2019 回复 Michael_PK #3

  好的， 我试试