BpeTokenizer 1.0.2
Heavy refactoring of the internal types. Breaking changes from previous versions. This deprecates the prior versions for that reason.
See the version list below for details.
dotnet add package BpeTokenizer --version 1.0.2
NuGet\Install-Package BpeTokenizer -Version 1.0.2
<PackageReference Include="BpeTokenizer" Version="1.0.2" />
paket add BpeTokenizer --version 1.0.2
#r "nuget: BpeTokenizer, 1.0.2"
// Install BpeTokenizer as a Cake Addin #addin nuget:?package=BpeTokenizer&version=1.0.2 // Install BpeTokenizer as a Cake Tool #tool nuget:?package=BpeTokenizer&version=1.0.2
BpeTokenizer
BpeTokenizer is a C# implementation of tiktoken written by OpenAI. It is a byte pair encoding tokenizer that can be used to tokenize text into subword units.
As a BpeTokenizer derived from tiktoken, it can be used as a token counter. Useful to ensure that when streaming tokens from the OpenAI API for GPT Chat Completions, you could keep track of the cost related to the software calling the API.
This library is built for x64 architectures.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
-
net7.0
- Newtonsoft.Json (>= 13.0.3)
NuGet packages (1)
Showing the top 1 NuGet packages that depend on BpeTokenizer:
Package | Downloads |
---|---|
BpeChatAI
Package Description |
GitHub repositories
This package is not used by any popular GitHub repositories.
Altered how Decode method handles errors. Previously, "strict" did not behave as intended. Converted method to use an enum for its parameter vs a string. Added tags and documentation comments.