Spark local模式启动报错

Question

18/02/15 23:05:48 ERROR SparkContext: Error initializing SparkContext.

java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:444)

at sun.nio.ch.Net.bind(Net.java:436)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)

at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)

at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)

at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)

at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)

at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)

at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)

at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)

at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

at java.lang.Thread.run(Thread.java:744)

java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

  at sun.nio.ch.Net.bind0(Native Method)

  at sun.nio.ch.Net.bind(Net.java:444)

  at sun.nio.ch.Net.bind(Net.java:436)

  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)

  at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)

  at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)

  at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)

  at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)

  at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)

  at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)

  at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)

  at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)

  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)

  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)

  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)

  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

  at java.lang.Thread.run(Thread.java:744)

<console>:14: error: not found: value spark

       import spark.implicits._

              ^

<console>:14: error: not found: value spark

       import spark.sql

              ^

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0

      /_/

         

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)

Type in expressions to have them evaluated.

Type :help for more information.

Answer 1

Michael_PK 2018-02-16 18:02:41

这个默认的端口被占用了，从4040开始，它会自动加1，但是这个默认一个机器最多只能启动16个。你可以jps看下，是否已经有16个spark submit进程了。你可以调大那个参数即可