Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free

tokenize

Obtain token table from text


Description

utils::getParseData() is used to obtain a flat parse table from text.

Usage

tokenize(text)

Arguments

text

The text to parse.

Details

Apart from the columns provided by utils::getParseData(), the following columns are added:

  • A column "short" with the first five characters of "text".

  • A column "pos_id" for (positional id) which can be used for sorting (because "id" cannot be used in general). Note that the nth value of this column corresponds to n as long as no tokens are inserted.

  • A column "child" that contains nests.

Value

A flat parse table


styler

Non-Invasive Pretty Printing of R Code

v1.4.1
MIT + file LICENSE
Authors
Kirill Müller [aut], Lorenz Walthert [cre, aut]
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.