sql

Type Members

1. case class CustomTransform(transform: (DataFrame) ⇒ DataFrame, requiredColumns: Seq[String] = Seq.empty[String], addedColumns: Seq[String] = Seq.empty[String], removedColumns: Seq[String] = Seq.empty[String], skipWhenPossible: Boolean = true) extends Product with Serializable

   

2. case class DariaValidationError(smth: String) extends Exception with Product with Serializable

   

3. case class DataFrameColumnsException(smth: String) extends Exception with Product with Serializable

   

4. trait DataFrameValidator extends AnyRef

   

5. case class EtlDefinition(sourceDF: DataFrame, transform: (DataFrame) ⇒ DataFrame, write: (DataFrame) ⇒ Unit, metadata: Map[String, Any] = ...) extends Product with Serializable

   

   spark-daria can be used as a lightweight framework for running ETL analyses in Spark.

   spark-daria can be used as a lightweight framework for running ETL analyses in Spark.

   You can define EtlDefinitions, group them in a collection, and run the etls via jobs.

   Components of an ETL

   An ETL starts with a DataFrame, runs a series of transformations (filter, custom transformations, repartition), and writes out data.

   The EtlDefinition class is generic and can be molded to suit all ETL situations. For example, it can read a CSV file from S3, run transformations, and write out Parquet files on your local filesystem.

6. case class InvalidColumnSortOrderException(smth: String) extends Exception with Product with Serializable

   

7. case class InvalidDataFrameSchemaException(smth: String) extends Exception with Product with Serializable

   

8. case class MissingDataFrameColumnsException(smth: String) extends Exception with Product with Serializable

   

9. class ParquetCompactor extends AnyRef

   

10. case class ProhibitedDataFrameColumnsException(smth: String) extends Exception with Product with Serializable