编译支持hadoop2.6.0-cdh5.15.1的spark3.0.0失败

Question

pk哥你好，
在编译支持hadoop2.6.0-cdh5.15.1的spark3.0.0时，
依照官方文档使用maven3.6.3，jdk8，scala2.12;

对pom.xml文件进行了以下2处修改
1.默认的hadoop版本由2.7.4改成2.6.0

<hadoop.version>2.6.0</hadoop.version>

2.模仿hadoop-3.2,新增了2.6.0相关的profile.

hadoop-2.6

报如下错误：
+++ dirname /Users/meilei/imooc/source/spark-3.0.0/core/…/R/install-dev.sh
++ cd /Users/meilei/imooc/source/spark-3.0.0/core/…/R
++ pwd

• FWDIR=/Users/meilei/imooc/source/spark-3.0.0/R
• LIB_DIR=/Users/meilei/imooc/source/spark-3.0.0/R/lib
• mkdir -p /Users/meilei/imooc/source/spark-3.0.0/R/lib
• pushd /Users/meilei/imooc/source/spark-3.0.0/R
• . /Users/meilei/imooc/source/spark-3.0.0/R/find-r.sh
  ++ ‘[’ -z ‘’ ‘]’
  ++ ‘[’ ‘!’ -z ‘’ ‘]’
  +++ command -v R
  ++ ‘[’ ‘!’ /usr/local/bin/R ‘]’
  ++++ which R
  +++ dirname /usr/local/bin/R
  ++ R_SCRIPT_PATH=/usr/local/bin
  ++ echo 'Using R_SCRIPT_PATH = /usr/local/bin’
  Using R_SCRIPT_PATH = /usr/local/bin
• . /Users/meilei/imooc/source/spark-3.0.0/R/create-rd.sh
  ++ set -o pipefail
  ++ set -e
  ++++ dirname /Users/meilei/imooc/source/spark-3.0.0/R/create-rd.sh
  +++ cd /Users/meilei/imooc/source/spark-3.0.0/R
  +++ pwd
  ++ FWDIR=/Users/meilei/imooc/source/spark-3.0.0/R
  ++ pushd /Users/meilei/imooc/source/spark-3.0.0/R
  ++ . /Users/meilei/imooc/source/spark-3.0.0/R/find-r.sh
  +++ ‘[’ -z /usr/local/bin ‘]’
  ++ /usr/local/bin/Rscript -e ’ if(“devtools” %in% rownames(installed.packages())) { library(devtools); setwd("/Users/meilei/imooc/source/spark-3.0.0/R"); devtools::document(pkg="./pkg", roclets=c(“rd”)) }'
  错误: ‘roxygen2’ >= 5.0.0 must be installed for this functionality.
  停止执行
  [ERROR] Command execution failed.
  org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
  at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
  at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
  at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
  at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
  at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
  at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
  at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
  at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
  at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
  at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
  at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
  at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
  at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
  at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
  at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
  at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
  at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
  at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke (Method.java:498)
  at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
  at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
  at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
  at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
  [INFO] ------------------------------------------------------------------------
  [INFO] Reactor Summary for Spark Project Parent POM 3.0.0:
  [INFO]
  [INFO] Spark Project Parent POM … SUCCESS [ 6.615 s]
  [INFO] Spark Project Tags … SUCCESS [ 13.204 s]
  [INFO] Spark Project Sketch … SUCCESS [ 9.185 s]
  [INFO] Spark Project Local DB … SUCCESS [ 2.885 s]
  [INFO] Spark Project Networking … SUCCESS [ 5.786 s]
  [INFO] Spark Project Shuffle Streaming Service … SUCCESS [ 2.078 s]
  [INFO] Spark Project Unsafe … SUCCESS [ 11.326 s]
  [INFO] Spark Project Launcher … SUCCESS [ 2.979 s]
  [INFO] Spark Project Core … FAILURE [01:41 min]
  [INFO] Spark Project ML Local Library … SKIPPED
  [INFO] Spark Project GraphX … SKIPPED
  [INFO] Spark Project Streaming … SKIPPED
  [INFO] Spark Project Catalyst … SKIPPED
  [INFO] Spark Project SQL … SKIPPED
  [INFO] Spark Project ML Library … SKIPPED
  [INFO] Spark Project Tools … SKIPPED
  [INFO] Spark Project Hive … SKIPPED
  [INFO] Spark Project REPL … SKIPPED
  [INFO] Spark Project YARN Shuffle Service … SKIPPED
  [INFO] Spark Project YARN … SKIPPED
  [INFO] Spark Project Mesos … SKIPPED
  [INFO] Spark Project Kubernetes … SKIPPED
  [INFO] Spark Project Hive Thrift Server … SKIPPED
  [INFO] Spark Project Assembly … SKIPPED
  [INFO] Kafka 0.10+ Token Provider for Streaming … SKIPPED
  [INFO] Spark Integration for Kafka 0.10 … SKIPPED
  [INFO] Kafka 0.10+ Source for Structured Streaming … SKIPPED
  [INFO] Spark Project Examples … SKIPPED
  [INFO] Spark Integration for Kafka 0.10 Assembly … SKIPPED
  [INFO] Spark Avro … SKIPPED
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD FAILURE
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time: 02:37 min
  [INFO] Finished at: 2020-08-25T22:02:35+08:00
  [INFO] ------------------------------------------------------------------------
  [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (sparkr-pkg) on project spark-core_2.12: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
  [ERROR]
  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  [ERROR]
  [ERROR] For more information about the errors and possible solutions, please read the following articles:
  [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
  [ERROR]
  [ERROR] After correcting the problems, you can resume the build with the command
  [ERROR] mvn -rf :spark-core_2.12

异常截图如下：

问下pk哥，
pom.xml中哪里的配置设置错误了，还是遗漏了修改的点？
maven的settings.xml中是否有额外的配置？
期待pk哥的答复。

备注：
1.spark3.0.0源码下载地址：https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0.tgz

2.maven3.6.2，jdk8，scala2.12 ，编译支持hadoop2.6.0-cdh5.15.1的spark3.0.0-preview版本时，也是报同样的错误。

Answer 1

Michael_PK 2020-08-29 17:41:01

如果issue和pr找不到的话，我晚上发一个，你先找找

Answer 2

Michael_PK 2020-08-25 22:38:05

你的maven版本就不行了呢，Spark官网的编译部分明确告诉我们了Maven的版本要求是什么。

不同的Spark版本需要的maven版本是不一样的，必须要严格按照官网上的来

这就是我为什么上课一再强调要看文档的原因

不是一个命令就能走遍天的

• 提问者 qq_梦也_1 #1

  spark-3.0.0-preview官网要求的maven版本是3.6.2，spark-3.0.0正式版本官网要求的maven版本是3.6.3。maven版本确定没问题。

• Michael_PK 回复 提问者 qq_梦也_1 #2

  哦哦你是preview啊，没注意。现在已经发布3了，直接用3。还有就是你那个报错要求是大于几的，你先把那个按照报错的来，先版本对了先

• 提问者 qq_梦也_1 回复 Michael_PK #3

  我看出正式版了，就用3.6.3编译了下3.0.0的正式版，编译失败；就又尝试了3.6.2编译下3.0.0的预览版1，结果也失败了。都是同样的错误: ‘roxygen2’ >= 5.0.0 must be installed for this functionality. google了下，也没找出好办法，所以就来问问老师，是不是其他的环节出了问题。

点开查看后面6条评论