Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
46 packages returned for Tags:"Tokenizer"
Sort by
Relevance
Downloads
Recently updated
TokenCs
by:
zeplar_exe
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,209 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
Brainfuck-
Runner
by:
nikolayresh
.NET 5.0
9,706 total downloads
last updated
11/9/2021
Latest version:
1.0.14
brainfuck
brain
brainfuck
interpreter
brainfuck
tokenizer
brainfuck
validator
brainfuck
engine
More tags
Powerful Brainfuck language interpreter with ability to tokenize & validate the Brainfuck code
LLMSharp.
OpenAi.
Tokenizer
by:
veerash-ayyagari
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
4,081 total downloads
last updated
10/4/2023
Latest version:
2.0.3
openai
gpt
3.5
gpt
4
tokenizer
dotnet
tiktoken
Open AI Chat Completion Models (GPT 3.5/GPT 4) BPE Tokenizer unofficial implementation
NReco.
NLQuery
by:
nreco
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.5
9,810 total downloads
last updated
8/29/2022
Latest version:
1.1.1
NLP
NER
NLQ
search
search-interface
natural
language
query
named
entity
More tags
NLQuery: natural language query parser recognizes entities in context of structured sources (like tabular dataset). Can be used for building natural language interface to SQL database or OLAP cube, implementing...
More information
GParse
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
2,206 total downloads
last updated
6/15/2021
Latest version:
5.0.0-alpha.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Parsing and lexing utilities to create your own parser and lexer
XLemmatizer
by:
rchristen
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
841 total downloads
last updated
5/29/2020
Latest version:
0.1.0
nlp
lemmatizer
tokenizer
Pre-release version. API might change later. A lemma is the canonical form of the word. For example, the words "run", "runs", "ran" and "running" can be lemmatized to "run" XLemmatizer tokenizes and lemmatizes...
More information
GParse.
Extensions.
StateMachines
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
1,030 total downloads
last updated
6/7/2021
Latest version:
5.0.0-alpha09
parser
lexer
parsing
lexing
tokenizer
tokenizing
Extension methods to integrate GParse with Tsu.StateMachines
VisualFA.
SourceGenerator
by:
honey.the.codewitch
427 total downloads
last updated
4/25/2024
Latest version:
1.3.1
lexer
tokenizer
dfa
regex
Generates fast DFA lexers and matchers in C# during the build process
Virastyar
by:
senobari
.NET Framework
5,407 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library. This package includes both library and it's required data files.
Tweekenizer
by:
cristipufu
.NET 5.0
439 total downloads
last updated
11/30/2020
Latest version:
1.0.0
tokenizer
hashtag
nlp
emoji
Tokenizer for social media posts and comments
Noise
by:
m-wantia
.NET 6.0
249 total downloads
last updated
6/19/2022
Latest version:
0.0.0.1-alpha
lexer
lexical
tokenizer
Package Description
Cynic-
Magnit.
Tokenization
by:
catapart
.NET 6.0
1,040 total downloads
last updated
10/20/2022
Latest version:
1.0.1
string
parse
parsing
token
tokenizer
tokenization
Tokenize strings into custom tokens using ordered regex operations.
GBertTokenizer
by:
GeorgeTer
.NET 6.0
285 total downloads
last updated
2/4/2024
Latest version:
1.1.1
BERT
Tokenizer
charp
dotnet
Package Description
PrismSharp
by:
tkubec
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
192 total downloads
last updated
2/18/2022
Latest version:
1.0.0-beta
syntax
highlighting
highlighter
tokenizer
PrismSharp is a syntax highlighting library based on an excellent javascript library PrismJS, fully written in C#. It currently supports over 270 programming languages and has 44 built-in visual themes, also...
More information
Virastyar.
Data
by:
senobari
3,817 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
Required data files for Virastyar library.
APCSharp
by:
Dotch
.NET 5.0
.NET Core 3.1
3,571 total downloads
last updated
4/11/2021
Latest version:
1.0.8
lexer
parser
parsing
language
tokenizer
lexical
semantic
analysis
compiler
combinator
More tags
Another Parser Combinator for C#. A library for building optimized and flexible parsers.
Leeax.
Parsing.
CSS
by:
leeax
.NET 5.0
199 total downloads
last updated
1/12/2021
Latest version:
1.0.0-beta
css
css3
tokenizer
parser
A lightweight CSS tokenizer/parser with no dependencies.
TKN
by:
Gulg
.NET Framework 4.7.2
193 total downloads
last updated
7/22/2022
Latest version:
1.0.0
Tokenizer
Lexer
Parser
Regex
An advanced tokenizer by GulgDevs
Virastyar.
Lib
by:
senobari
.NET Framework
3,754 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library.
AllMiniLmL6V2Sharp
by:
ksanman
.NET 5.0
.NET Core 3.0
.NET Standard 2.1
528 total downloads
last updated
4/15/2024
Latest version:
0.0.2
embeddings
all-mini-lm-l6-v2
tokenizer
BERT
Sentence
Transfomers
onnx
NET Standard 2.1 library to produces embeddings using C# Bert Tokenizer and Onnx All-Mini-LM-L6-v2 model.
Previous
Next