NipahTokenizer 3.1.1

.NET 6.0 This package targets .NET 6.0. The package is compatible with this framework or higher.
There is a newer version of this package available.
See the version list below for details. 
dotnet add package NipahTokenizer --version 3.1.1                

                    
                

            
NuGet\Install-Package NipahTokenizer -Version 3.1.1
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="NipahTokenizer" Version="3.1.1" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
paket add NipahTokenizer --version 3.1.1                

                    
                

            
#r "nuget: NipahTokenizer, 3.1.1"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
// Install NipahTokenizer as a Cake Addin
#addin nuget:?package=NipahTokenizer&version=3.1.1

// Install NipahTokenizer as a Cake Tool
#tool nuget:?package=NipahTokenizer&version=3.1.1

Nipah Tokenizer

NuGet version (NipahTokenizer)

Installing

dotnet add package NipahTokenizer

Using

First setup the tokenizer options as needed, you can use the default as follows:

var tokenizerOptions = TokenizerOptions.Default;

Then just create a new instance of tokenizer and call "Tokenize" passing tokenizer options as arg:

var tokenizer = new Tokenizer();
var tokens = tokenizer.Tokenize("any-text-here", tokenizerOptions);

You can now just iterate over tokens as needed.

Product Compatible and additional computed target framework versions.
.NET net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

  • net6.0

    • No dependencies.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on NipahTokenizer:

Package Downloads
Nipah.Markdown

A library to make simple parses of markdown

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
3.1.9 181 9/9/2023
3.1.8 151 9/5/2023
3.1.7 133 9/3/2023
3.1.6 508 12/18/2022
3.1.5 413 10/22/2022
3.1.3 421 10/1/2022
3.1.1 421 10/1/2022
3.1.0 418 10/1/2022
3.0.0 454 9/24/2022
1.0.1 412 10/9/2021
1.0.0 355 10/9/2021

Refactoring code again.
More optimizations and bugfixes.
Adding option to use parallelism to tokenize larger texts (can reach +50% speed), results in less accurate tokenization but good.
Switching to struct split item (less heap memory usage).
Switching to use chars directly instead Regex (+80% speed).
- fast fix