ksqlDb.RestApi.Client.ProtoBuf 2.0.0

This package generates KSQL push and pull queries from your .NET C# LINQ queries. You can filter, project, limit, etc. your push notifications server side with ksqlDB push queries. You can continually process computations over unbounded (theoretically never-ending) streams of data. It also allows you to execute SQL statements via the REST API such as inserting records into streams and creating tables, types, etc. or execute admin operations such as listing streams.

ksqlDB.RestApi.Client is a contribution to Confluent ksqldb-clients

Install with NuGet package manager:

Install-Package ksqlDB.RestApi.Client

or with .NET CLI

dotnet add package ksqlDB.RestApi.Client

This adds a <PackageReference> to your csproj file, similar to the following:

<PackageReference Include="ksqlDB.RestApi.Client" Version="3.0.0" />

Alternative option is to use Protobuf content type:

dotnet add package ksqlDB.RestApi.Client.ProtoBuf

The following example can be tried out with a .NET interactive Notebook:

using ksqlDB.RestApi.Client.KSql.Linq;
using ksqlDB.RestApi.Client.KSql.Query;
using ksqlDB.RestApi.Client.KSql.Query.Context;
using ksqlDB.RestApi.Client.KSql.Query.Options;

var ksqlDbUrl = @"http:\\localhost:8088";

var contextOptions = new KSqlDBContextOptions(ksqlDbUrl)
{
  ShouldPluralizeFromItemName = true
};

await using var context = new KSqlDBContext(contextOptions);

using var disposable = context.CreateQueryStream<Tweet>()
  .WithOffsetResetPolicy(AutoOffsetReset.Latest)
  .Where(p => p.Message != "Hello world" || p.Id == 1)
  .Select(l => new { l.Message, l.Id })
  .Take(2)
  .Subscribe(tweetMessage =>
  {
    Console.WriteLine($"{nameof(Tweet)}: {tweetMessage.Id} - {tweetMessage.Message}");
  }, error => { Console.WriteLine($"Exception: {error.Message}"); }, () => Console.WriteLine("Completed"));

Console.WriteLine("Press any key to stop the subscription");

Console.ReadKey();

public class Tweet : Record
{
  public int Id { get; set; }

  public string Message { get; set; }
}

An entity class in ksqlDB.RestApi.Client represents the structure of a table or stream. An instance of the class represents a record in that stream while properties are mapped to columns respectively.

LINQ code written in C# from the sample is equivalent to this KSQL query:

SELECT Message, Id
  FROM Tweets
 WHERE Message != 'Hello world' OR Id = 1 
  EMIT CHANGES 
 LIMIT 2;

In the above mentioned code snippet everything runs server side except of the IQbservable<TEntity>.Subscribe method. It subscribes to your ksqlDB stream created in the following manner:

using ksqlDB.RestApi.Client.KSql.RestApi.Http;
using ksqlDB.RestApi.Client.KSql.RestApi.Statements;
using ksqlDB.RestApi.Client.KSql.RestApi;
using ksqlDB.Api.Client.Samples.Models;

EntityCreationMetadata metadata = new()
{
  KafkaTopic = nameof(Tweet),
  Partitions = 3,
  Replicas = 3
};

var httpClient = new HttpClient()
{
  BaseAddress = new Uri(@"http:\\localhost:8088")
};

var httpClientFactory = new HttpClientFactory(httpClient);
var restApiClient = new KSqlDbRestApiClient(httpClientFactory);

var httpResponseMessage = await restApiClient.CreateOrReplaceStreamAsync<Tweet>(metadata);

CreateOrReplaceStreamAsync executes the following statement:

CREATE OR REPLACE STREAM Tweets (
	Id INT,
	Message VARCHAR
) WITH ( KAFKA_TOPIC='Tweet', VALUE_FORMAT='Json', PARTITIONS='3', REPLICAS='3' );

Run the following insert statements to publish some messages with your ksqldb-cli

docker exec -it $(docker ps -q -f name=ksqldb-cli) ksql http://ksqldb-server:8088

INSERT INTO tweets (id, message) VALUES (1, 'Hello world');
INSERT INTO tweets (id, message) VALUES (2, 'ksqlDB rulez!');

or insert a record from C#:

var responseMessage = await new KSqlDbRestApiClient(httpClientFactory)
  .InsertIntoAsync(new Tweet { Id = 2, Message = "ksqlDB rulez!" });

or with KSqlDbContext:

await using var context = new KSqlDBContext(ksqlDbUrl);

context.Add(new Tweet { Id = 1, Message = "Hello world" });
context.Add(new Tweet { Id = 2, Message = "ksqlDB rulez!" });

var saveChangesResponse = await context.SaveChangesAsync();

Sample projects can be found under Samples solution folder in ksqlDB.RestApi.Client.sln

External dependencies:

• kafka broker and ksqlDB-server 0.14.0
• the solution requires Docker desktop and Visual Studio 2019
• .NET 6.0

Clone the repository

git clone https://github.com/tomasfabian/ksqlDB.RestApi.Client-DotNet.git

CD to Samples

CD Samples\ksqlDB.RestApi.Client.Sample\

run in command line:

docker compose up -d

AspNet Blazor server side sample:

• set docker-compose.csproj as startup project in InsideOut.sln for an embedded Kafka connect integration and stream processing examples.

IQbservable<T> extension methods

As depicted bellow IObservable<T> is the dual of IEnumerable<T> and IQbservable<T> is the dual of IQueryable<T>. In all four cases LINQ providers are using deferred execution. While the first two are executed locally the latter two are executed server side. The server side execution is possible thanks to traversing ASTs (Abstract Syntax Trees) with visitors. The KSqlDbProvider will create the KSQL syntax for you from expression trees and pass it along to ksqlDB.

<img src="https://www.codeproject.com/KB/cs/646361/WhatHowWhere.jpg" />

List of supported push query extension methods:

• Select

• Where

• Take (LIMIT)

• Subscribe

• ToObservable

• ToAsyncEnumerable

• ToQueryString

• ExplainAsync

• SubscribeAsync

• SubscribeOn

• ObserveOn

• ToStatementString

• WithOffsetResetPolicy

• Window bounds

• Raw string KSQL query execution

• IKSqlGrouping.Source

Setting query parameters

Default settings: 'auto.offset.reset' is set to 'earliest' by default. New parameters could be added or existing ones changed in the following manner:

var contextOptions = new KSqlDBContextOptions(@"http:\\localhost:8088");

contextOptions.QueryStreamParameters["auto.offset.reset"] = "latest";

Overriding stream names

Stream names are generated based on the generic record types. They are pluralized with Pluralize.NET package.

By default the generated from item names such as stream and table names are pluralized. This behavior could be switched off with the following ShouldPluralizeStreamName configuration.

context.CreateQueryStream<Person>();

FROM People

This can be disabled:

var contextOptions = new KSqlDBContextOptions(@"http:\\localhost:8088")
{
  ShouldPluralizeFromItemName = false
};

new KSqlDBContext(contextOptions).CreateQueryStream<Person>();

FROM Person

Setting an arbitrary stream name (from_item name):

context.CreateQueryStream<Tweet>("custom_topic_name");

FROM custom_topic_name

Register the KSqlDbContext

IKSqlDBContext and IKSqlDbRestApiClient can be provided with dependency injection. These services can be registered during app startup and components that require these services, are provided with these services via constructor parameters.

To register KsqlDbContext as a service, open Program.cs, and add the lines to the ConfigureServices method shown bellow or see some more details in the workshop:

using ksqlDB.RestApi.Client.Sensors;
using ksqlDB.RestApi.Client.KSql.Query.Options;
using ksqlDb.RestApi.Client.DependencyInjection;
using ksqlDB.RestApi.Client.Sensors.KSqlDb;

IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
      var ksqlDbUrl = @"http:\\localhost:8088";

      services.AddDbContext<ISensorsKSqlDbContext, SensorsKSqlDbContext>(
        options =>
        {
          var setupParameters = options.UseKSqlDb(ksqlDbUrl);

          setupParameters.SetAutoOffsetReset(AutoOffsetReset.Earliest);

        }, ServiceLifetime.Transient, restApiLifetime: ServiceLifetime.Transient);
    })
    .Build();

await host.RunAsync();

Aggregation functions

List of supported ksqldb aggregation functions:

• GROUP BY

• MIN, MAX

• AVG

• COUNT

• COLLECT_LIST, COLLECT_SET, EARLIEST_BY_OFFSET, LATEST_BY_OFFSET

• SUM

• COUNT_DISTINCT

• HISTOGRAM

• TOPK,TOPKDISTINCT

• COLLECTSET, COLLECTLIST, COUNT_DISTINCT

• HAVING

• TimeWindows - EMIT FINAL

• WindowedBy

  • Session Window
  • Tumbling Window
  • Hopping Window

Rest API reference

Some KSql function examples can be found here

List of supported data types:

• Supported data types mapping
• Structs
• Maps
• Time types DATE, TIME AND TIMESTAMP
• System.GUID as ksqldb VARCHAR type

List of supported Joins:

• RightJoin
• Full Outer Join
• Left Join
• Inner Joins
• Multiple joins with query comprehension syntax (GroupJoin, SelectMany, DefaultIfEmpty)

List of supported pull query extension methods:

• Take (LIMIT)
• FirstOrDefaultAsync
• GetManyAsync
• CreatePullQuery
• ExecutePullQuery

List of supported ksqlDB SQL statements:

• Pause and resume persistent qeries
• Added support for extracting field names and values (for insert and select statements)
• Assert topics
• Assert schemas
• Rename stream or table column names with the JsonPropertyNameAttribute
• Create source streams and tables
• InsertIntoAsync
• Connectors
• Drop a stream
• Drop type
• Creating types
• Execute statement async
• PartitionBy
• Terminate push queries
• Drop a table
• Creating connectors
• Get topics
• Getting queries and termination of persistent queries
• Execute statements
• Create or replace table statements
• Creating streams and tables
• Get streams
• Get tables

KSqlDbContext

• Dependency injection with ServicesCollection
• Creating query streams
• Creating queries
• AddDbContext and AddDbContextFactory
• Logging info and ConfigureKSqlDb
• Basic auth
• Add and SaveChangesAsync
• KSqlDbContextOptionsBuilder

Config

• Bearer token authentication
• Replacing HttpClient
• Processing guarantees

Operators

• Operator LIKE
• Operator IN
• Operator BETWEEN
• Operator CASE
• Arithmetic operations on columns
• Lexical precedence
• WHERE IS NULL, IS NOT NULL

Data definitions

• Headers

Miscelenaous

• Change data capture
• List of breaking changes
• Operators
• Invocation functions
• Setting JsonSerializerOptions
• Kafka stream processing example

Functions

• String functions
• Numeric functions
• Date and time functions
• Lambda functions (Invocation functions) - Maps
  • Transform maps
  • Filter maps
  • Reduce maps

LinqPad samples

Push Query

Pull Query

Nuget

https://www.nuget.org/packages/ksqlDB.RestApi.Client/

ksqldb links

Scalar functions

Aggregation functions

Push query

Acknowledgements:

• ksql

• Microsoft.Extensions.DependencyInjection

• Pluralize.NET

• System.Interactive.Async

• System.Reactive

• System.Text.Json