Microsoft.ML.OnnxRuntimeGenAI.Cuda 0.3.0-rc2

Prefix Reserved
This is a prerelease version of Microsoft.ML.OnnxRuntimeGenAI.Cuda.
There is a newer version of this package available.
See the version list below for details.
dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda --version 0.3.0-rc2                
NuGet\Install-Package Microsoft.ML.OnnxRuntimeGenAI.Cuda -Version 0.3.0-rc2                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Microsoft.ML.OnnxRuntimeGenAI.Cuda" Version="0.3.0-rc2" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Microsoft.ML.OnnxRuntimeGenAI.Cuda --version 0.3.0-rc2                
#r "nuget: Microsoft.ML.OnnxRuntimeGenAI.Cuda, 0.3.0-rc2"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Microsoft.ML.OnnxRuntimeGenAI.Cuda as a Cake Addin
#addin nuget:?package=Microsoft.ML.OnnxRuntimeGenAI.Cuda&version=0.3.0-rc2&prerelease

// Install Microsoft.ML.OnnxRuntimeGenAI.Cuda as a Cake Tool
#tool nuget:?package=Microsoft.ML.OnnxRuntimeGenAI.Cuda&version=0.3.0-rc2&prerelease                

ONNX Runtime generate() API

Latest version

Run Llama, Phi, Gemma, Mistral with ONNX Runtime.

This API gives you an easy, flexible and performant way of running LLMs on device.

It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, and KV cache management.

You can call a high level generate() method to generate all of the output at once, or stream the output one token at a time.

See documentation at

Support matrix Supported now Under development On the roadmap
Model architectures Gemma <br/> Llama * <br/> Mistral + <br/>Phi <br/> Whisper Stable diffusion
API Python <br/>C# <br/>C/C++ Java Objective-C
Platform Linux <br/> Windows Android Mac <br/> iOS
Architecture x86 <br/> x64 <br/> Arm64
Hardware Acceleration CUDA<br/>DirectML<br/> QNN <br/> ROCm OpenVINO

* The Llama model architecture supports similar model families such as CodeLlama, Vicuna, Yi, and more.

+ The Mistral model architecture supports similar model families such as Zephyr.



Sample code for Phi-3 in Python

  1. Download the model

    huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .
  2. Install the API

    pip install numpy
    pip install --pre onnxruntime-genai
  3. Run the model

    import onnxruntime_genai as og
    model = og.Model('cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4')
    tokenizer = og.Tokenizer(model)
    tokenizer_stream = tokenizer.create_stream()
    # Set the max length to something sensible by default,
    # since otherwise it will be set to the entire context length
    search_options = {}
    search_options['max_length'] = 2048
    chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>'
    text = input("Input: ")
    if not text:
       print("Error, input cannot be empty")
    prompt = f'{chat_template.format(input=text)}'
    input_tokens = tokenizer.encode(prompt)
    params = og.GeneratorParams(model)
    params.input_ids = input_tokens
    generator = og.Generator(model, params)
    print("Output: ", end='', flush=True)
       while not generator.is_done():
         new_token = generator.get_next_tokens()[0]
         print(tokenizer_stream.decode(new_token), end='', flush=True)
    except KeyboardInterrupt:
        print("  --control+c pressed, aborting generation--")
    del generator


This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact with any additional questions or comments.


This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.1 is compatible. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
native native is compatible. 
Tizen tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on Microsoft.ML.OnnxRuntimeGenAI.Cuda:

Package Downloads

Semantic Kernel connector for Microsoft.ML.OnnxRuntimeGenAI.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.3.0 2,166 6/21/2024
0.3.0-rc2 3,234 5/29/2024
0.3.0-rc1 552 5/22/2024
0.2.0 1,451 5/20/2024
0.2.0-rc7 932 5/14/2024
0.2.0-rc6 1,102 5/4/2024
0.2.0-rc4 1,582 4/25/2024
0.2.0-rc3 206 4/24/2024
0.1.0 296 4/8/2024
0.1.0-rc4 235 3/27/2024

Introducing the ONNX Runtime GenAI Library.