spark-submit无法跑在yarn上

Question

[hadoop@hadoop000 spark-3.0.0-bin-2.6.0-cdh5.16.2]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn examples/jars/spark-examples*.jar 2

Exception in thread "main" org.apache.spark.SparkException: Could not load YARN classes. This copy of Spark may not have been compiled with YARN support.

at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:936)

at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:255)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

Michael_PK 2020-09-24 21:06:19

你这个spark是自己编译的还是？

This copy of Spark may not have been compiled with YARN support.

不支持yarn呢

• 提问者 慕田峪0177977 #1

  自己编译完成的，我在github上下载的spark 3.0.0,我之前下过spark2.4.3，不知是否冲突？

• Michael_PK 回复 提问者 慕田峪0177977 #2

  你编译的时候有-Pyarn吗