Read file(s) into a Spark DataFrame using a custom reader
Run a custom R function on Spark workers to ingest data from one or more files into a Spark DataFrame, assuming all files follow the same schema.
spark_read(sc, paths, reader, columns, packages = TRUE, ...)
sc |
A |
paths |
A character vector of one or more file URIs (e.g., c("hdfs://localhost:9000/file.txt", "hdfs://localhost:9000/file2.txt")) |
reader |
A self-contained R function that takes a single file URI as argument and returns the data read from that file as a data frame. |
columns |
a named list of column names and column types of the resulting data frame (e.g., list(column_1 = "integer", column_2 = "character")), or a list of column names only if column types should be inferred from the data (e.g., list("column_1", "column_2"), or NULL if column types should be inferred and resulting data frame can have arbitrary column names |
packages |
A list of R packages to distribute to Spark workers |
... |
Optional arguments; currently unused. |
Other Spark serialization routines:
collect_from_rds(),
spark_load_table(),
spark_read_avro(),
spark_read_csv(),
spark_read_delta(),
spark_read_jdbc(),
spark_read_json(),
spark_read_libsvm(),
spark_read_orc(),
spark_read_parquet(),
spark_read_source(),
spark_read_table(),
spark_read_text(),
spark_save_table(),
spark_write_avro(),
spark_write_csv(),
spark_write_delta(),
spark_write_jdbc(),
spark_write_json(),
spark_write_orc(),
spark_write_parquet(),
spark_write_source(),
spark_write_table(),
spark_write_text()
## Not run:
library(sparklyr)
sc <- spark_connect(
master = "yarn",
spark_home = "~/spark/spark-2.4.5-bin-hadoop2.7"
)
# This is a contrived example to show reader tasks will be distributed across
# all Spark worker nodes
spark_read(
sc,
rep("/dev/null", 10),
reader = function(path) system("hostname", intern = TRUE),
columns = c(hostname = "string")
) %>% sdf_collect()
## End(Not run)Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.