12 packages returned for Tags:"Tokenization"

YAML grammar based on Lexepars parser lib.

Package Description

JSON grammar based on Lexepars parser lib.

SDK for server-side integration with CyberSource Flex API

Tokenization of raw text is a standard pre-processing step for many NLP tasks. For English, tokenization usually involves punctuation splitting and separation of some affixes like possessives. Other languages... More information

Extract tokens from a string of text for use with NLP tools or statistical analysis.

Concise monadic parser combinator library with separate lexer/parser phases and big-size input support.

Text tokenization based on Unicode grapheme clustering, and the XID_Start and XID_Continue binary properties.

TextMatch is a library for searching inside texts using Lucene query expressions. Supports all types of Lucene query expressions - boolean, wildcard, fuzzy. Options are available for tweaking tokenization, such... More information

Package Description

Class library for working with general language constructs.

Tokenize strings into custom tokens using ordered regex operations.