com.github.mrpowers.spark.daria.sql.SparkSessionExt
Creates a DataFrame, similar to createDataFrame, but with better syntax
spark-daria defined a createDF method that allows for the terse syntax of toDF
and the control of createDataFrame
.
Creates a DataFrame, similar to createDataFrame, but with better syntax
spark-daria defined a createDF method that allows for the terse syntax of toDF
and the control of createDataFrame
.
spark.createDF( List( ("bob", 45), ("liz", 25), ("freeman", 32) ), List( ("name", StringType, true), ("age", IntegerType, false) ) )
The createDF
method can also be used with lists of Row
and StructField
objects.
spark.createDF( List( Row("bob", 45), Row("liz", 25), Row("freeman", 32) ), List( StructField("name", StringType, true), StructField("age", IntegerType, false) ) )
Creates an empty DataFrame given schema fields
Creates an empty DataFrame given schema fields
This is a handy fallback when you fail to read from a data source
val schema = List(StructField("col1", IntegerType)) val df = Try { spark.read.parquet("non-existent-path") }.getOrElse(spark.createEmptyDf(schema))