在windows环境下，IDEA写json文件报错，麻烦帮忙看看，谢谢！

Question

在windows环境下，IDEA写json文件报错，麻烦帮忙看看，谢谢！
object ParquetApp {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName(“ParquetApp”).
master(“local[2]”).getOrCreate()

val path = "E:/scalaproject/users.parquet"
val userDF = spark.read.format("parquet").load(path)
userDF.printSchema()
userDF.show()

//未指定format，默认为parquet
val userDF3 = spark.read.load(path)
userDF3.show()

val path2 = "E:/scalaproject/jsonout"
val userDF2 = userDF.filter("favorite_color != 'null'").
  select("name","favorite_color")
userDF2.show()
userDF2.write.format("json").save(path2)

spark.stop()

}

19/10/16 09:44:18 ERROR Utils: Aborting task
java.lang.NullPointerException
19/10/16 09:44:18 ERROR FileFormatWriter: Job job_20191016094417_0005 aborted.
19/10/16 09:44:18 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
19/10/16 09:44:18 ERROR FileFormatWriter: Aborting job null.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows

Answer 1

Michael_PK 2019-10-16 10:27:11

默认就是parquet。你把你的parquet数据读出来看看有没有你的字段吧。还有你本地盘符那么写？确认下是否能读得到

• 提问者 慕斯8493011 #1

  谢谢~本地盘符偷懒了，是能读到，IDEA运行结果也能看到读出的DF。现在是在WINDOWS环境下写文件报错，放在LINUX上执行是没问题的。这个错误不知道怎么解决呢？

• Michael_PK 回复 提问者 慕斯8493011 #2

  win写操作就和win开发mr写出去数据一个道理，默认是报错的，是win系统的问题。需要找一个win的hadoop common的插件才可以。直接使用spark shell测试不就行了，或者打包到服务器执行

• 提问者 慕斯8493011 #3

  非常感谢！