sparklyr: hof_filter – R documentation

Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!

hof_filter

Filter Array Column

Description

Apply an element-wise filtering function to an array column (this is essentially a dplyr wrapper for the filter(array<T>, function<T, Boolean>): array<T> built-in Spark SQL functions)

Usage

hof_filter(x, func, expr = NULL, dest_col = NULL, ...)

Arguments

`x`	The Spark data frame to filter
`func`	The filtering function
`expr`	The array being filtered, could be any SQL expression evaluating to an array (default: the last column of the Spark data frame)
`dest_col`	Column to store the filtered result (default: expr)
`...`	Additional params to dplyr::mutate

Examples

## Not run: 

library(sparklyr)
sc <- spark_connect(master = "local")
# only keep odd elements in each array in `array_column`
copy_to(sc, tibble::tibble(array_column = list(1:5, 21:25))) %>%
  hof_filter(~ .x %% 2 == 1)

## End(Not run)

sparklyr

R Interface to Apache Spark

v1.6.2

Apache License 2.0 | file LICENSE

Authors

Javier Luraschi [aut], Kevin Kuo [aut] (<https://orcid.org/0000-0001-7803-7901>), Kevin Ushey [aut], JJ Allaire [aut], Samuel Macedo [ctb], Hossein Falaki [aut], Lu Wang [aut], Andy Zhang [aut], Yitao Li [aut, cre] (<https://orcid.org/0000-0002-1261-905X>), Jozef Hajnala [ctb], Maciej Szymkiewicz [ctb] (<https://orcid.org/0000-0003-1469-9396>), Wil Davis [ctb], RStudio [cph], The Apache Software Foundation [aut, cph]

Initial release