Tiktoken 0.9.6
See the version list below for details.
dotnet add package Tiktoken --version 0.9.6
NuGet\Install-Package Tiktoken -Version 0.9.6
<PackageReference Include="Tiktoken" Version="0.9.6" />
paket add Tiktoken --version 0.9.6
#r "nuget: Tiktoken, 0.9.6"
// Install Tiktoken as a Cake Addin #addin nuget:?package=Tiktoken&version=0.9.6 // Install Tiktoken as a Cake Tool #tool nuget:?package=Tiktoken&version=0.9.6
Tiktoken
Implemented encodings
cl100k_base
r50k_base
p50k_base
p50k_edit
Usage
var encoding = Tiktoken.Encoding.ForModel("gpt-4");
var tokens = encoding.Encode("hello world"); // [15339, 1917]
var text = encoding.Decode(tokens); // hello world
var numberOfTokens = encoding.CountTokens(text); // 2
var encoding = Tiktoken.Encoding.Get("p50k_base");
var tokens = encoding.Encode("hello world"); // [31373, 995]
var text = encoding.Decode(tokens); // hello world
Benchmarks
You can view the reports for each version here
BenchmarkDotNet=v0.13.5, OS=macOS Ventura 13.3.1 (a) (22E772610a) [Darwin 22.4.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK=7.0.203
[Host] : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD
Job-TGQYUW : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD DEBUG
BuildConfiguration=Debug
Method | Data | Mean | Error | StdDev | Ratio | Gen0 | Gen1 | Gen2 | Allocated | Alloc Ratio |
---|---|---|---|---|---|---|---|---|---|---|
SharpTokenV1_0_28 | 1. (...)57. [19866] | 5,316.826 μs | 55.8088 μs | 52.2036 μs | 1.00 | 601.5625 | 296.8750 | - | 3716.57 KB | 1.00 |
TiktokenSharpV1_0_5 | 1. (...)57. [19866] | 1,677.066 μs | 11.4936 μs | 10.1888 μs | 0.32 | 253.9063 | 128.9063 | 3.9063 | 1534.33 KB | 0.41 |
Tiktoken | 1. (...)57. [19866] | 894.654 μs | 5.6507 μs | 5.2857 μs | 0.17 | 84.9609 | 29.2969 | - | 525.11 KB | 0.14 |
SharpTokenV1_0_28 | Hello, World! | 3.269 μs | 0.0157 μs | 0.0131 μs | 1.00 | 0.6752 | 0.0038 | - | 4.14 KB | 1.00 |
TiktokenSharpV1_0_5 | Hello, World! | 6.651 μs | 0.0364 μs | 0.0322 μs | 2.03 | 2.1820 | 0.0458 | - | 13.41 KB | 3.24 |
Tiktoken | Hello, World! | 1.230 μs | 0.0040 μs | 0.0034 μs | 0.38 | 0.3109 | 0.0019 | - | 1.91 KB | 0.46 |
SharpTokenV1_0_28 | King(...)edy. [275] | 62.255 μs | 0.1989 μs | 0.1861 μs | 1.00 | 8.5449 | 0.4883 | - | 52.89 KB | 1.00 |
TiktokenSharpV1_0_5 | King(...)edy. [275] | 21.834 μs | 0.0833 μs | 0.0739 μs | 0.35 | 5.0964 | 0.3052 | - | 31.34 KB | 0.59 |
Tiktoken | King(...)edy. [275] | 11.433 μs | 0.0265 μs | 0.0235 μs | 0.18 | 1.3580 | 0.0305 | - | 8.34 KB | 0.16 |
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 was computed. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.0 is compatible. netstandard2.1 is compatible. |
.NET Framework | net461 is compatible. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen40 was computed. tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETFramework 4.6.1
- System.ValueTuple (>= 4.5.0)
-
.NETStandard 2.0
- No dependencies.
-
.NETStandard 2.1
- No dependencies.
-
net6.0
- No dependencies.
-
net7.0
- No dependencies.
NuGet packages (8)
Showing the top 5 NuGet packages that depend on Tiktoken:
Package | Downloads |
---|---|
tryAGI.OpenAI
Generated C# SDK based on official OpenAI OpenAPI specification. Includes C# Source Generator which allows you to define functions natively through a C# interface, and also provides extensions that make it easier to call this interface later |
|
HuggingFace
Generated C# SDK based on HuggingFace OpenAPI specification. |
|
Anthropic
Generated C# SDK based on official Anthropic OpenAPI specification. |
|
drittich.SemanticSlicer
A library for slicing text data into smaller chunks while attempting to preserve context. |
|
Aeolin.AwosFramework.OpenAIFunctionCalling
Package Description |
GitHub repositories (1)
Showing the top 1 popular GitHub repositories that depend on Tiktoken:
Repository | Stars |
---|---|
Richasy/Rodel.Agent
支持主流在线 AI 服务的应用
|
Version | Downloads | Last updated |
---|---|---|
2.0.3 | 14,518 | 7/10/2024 |
2.0.2 | 12,499 | 5/19/2024 |
2.0.1 | 101 | 5/18/2024 |
2.0.0 | 1,077 | 5/15/2024 |
1.2.0 | 73,870 | 2/18/2024 |
1.1.3 | 135,943 | 10/11/2023 |
1.1.2 | 93,879 | 8/28/2023 |
1.1.1 | 84,545 | 7/12/2023 |
1.1.0 | 1,866 | 6/28/2023 |
1.0.2 | 245 | 6/21/2023 |
1.0.1 | 296 | 6/16/2023 |
1.0.0 | 536 | 6/12/2023 |
0.9.7 | 225 | 6/11/2023 |
0.9.6 | 295 | 6/9/2023 |
0.9.5 | 213 | 6/9/2023 |
0.9.4 | 146 | 6/9/2023 |
0.9.3 | 135 | 6/9/2023 |
0.9.2 | 195 | 6/9/2023 |