Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free

sdf_schema

Read the Schema of a Spark DataFrame


Description

Read the schema of a Spark DataFrame.

Usage

sdf_schema(x, expand_nested_cols = FALSE, expand_struct_cols = FALSE)

Arguments

x

A spark_connection, ml_pipeline, or a tbl_spark.

expand_nested_cols

Whether to expand columns containing nested array of structs (which are usually created by tidyr::nest on a Spark data frame)

expand_struct_cols

Whether to expand columns containing structs

Details

The type column returned gives the string representation of the underlying Spark type for that column; for example, a vector of numeric values would be returned with the type "DoubleType". Please see the Spark Scala API Documentation for information on what types are available and exposed by Spark.

Value

An R list, with each list element describing the name and type of a column.


sparklyr

R Interface to Apache Spark

v1.6.2
Apache License 2.0 | file LICENSE
Authors
Javier Luraschi [aut], Kevin Kuo [aut] (<https://orcid.org/0000-0001-7803-7901>), Kevin Ushey [aut], JJ Allaire [aut], Samuel Macedo [ctb], Hossein Falaki [aut], Lu Wang [aut], Andy Zhang [aut], Yitao Li [aut, cre] (<https://orcid.org/0000-0002-1261-905X>), Jozef Hajnala [ctb], Maciej Szymkiewicz [ctb] (<https://orcid.org/0000-0003-1469-9396>), Wil Davis [ctb], RStudio [cph], The Apache Software Foundation [aut, cph]
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.