Contrib.KafkaFlow.CryptoShredding.Avro
3.1.6
See the version list below for details.
dotnet add package Contrib.KafkaFlow.CryptoShredding.Avro --version 3.1.6
NuGet\Install-Package Contrib.KafkaFlow.CryptoShredding.Avro -Version 3.1.6
<PackageReference Include="Contrib.KafkaFlow.CryptoShredding.Avro" Version="3.1.6" />
paket add Contrib.KafkaFlow.CryptoShredding.Avro --version 3.1.6
#r "nuget: Contrib.KafkaFlow.CryptoShredding.Avro, 3.1.6"
// Install Contrib.KafkaFlow.CryptoShredding.Avro as a Cake Addin #addin nuget:?package=Contrib.KafkaFlow.CryptoShredding.Avro&version=3.1.6 // Install Contrib.KafkaFlow.CryptoShredding.Avro as a Cake Tool #tool nuget:?package=Contrib.KafkaFlow.CryptoShredding.Avro&version=3.1.6
Contrib.KafkaFlow.CryptoShredding.Avro
Contrib.KafkaFlow.CryptoShredding.Avro
is a .NET package designed for secure handling of sensitive data in Kafka messages
using Avro serialization. The package introduces a string-based logical type called "encrypted-string"
that ensures sensitive information is encrypted and securely managed throughout the message lifecycle.
Overview
In many systems, particularly multi-tenant architectures, messages may contain sensitive data that must be protected. This package enables consumers to use tenant-specific encryption keys, ensuring that once a key is dropped, the secrets for a given tenant cannot be decrypted, effectively rendering the data as deleted.
Key Features
- Sensitive Data Protection: Supports the use of an "encrypted-string" logical type for handling sensitive data.
- Tenant-Specific Encryption: Allows each message to have its own encryption key, ideal for multi-tenant systems.
- Safe Serialization: Ensures that plain text strings cannot be accidentally serialized by throwing exceptions if attempted.
EncryptedString Type
When C# code is generated for messages that include sensitive data, these strings are represented by the EncryptedString type. The EncryptedString has two cases:
EncryptedString.Encrypted
: Represents the encrypted state of the data.EncryptedString.Plain
: Represents the plain text version of the data.
To set a value in plain text, use:
sensitiveData = EncryptedString.FromPlain("secret-value");
To prevent accidental leakage of secrets, EncryptedString.Plain
cannot be serialized to Avro. An attempt to do so will result in an
exception.
Generating Avro types
Here is an example of an Avro schema that uses encrypted secrets:
{
"type": "record",
"name": "EncryptedMessage",
"namespace": "TestContract",
"fields": [
{ "name": "secret",
"type": { "type": "string", "logicalType": "encrypted-string" } }
]
}
C# types can be generated using the avrogen tool
or by using Contrib.KafkaFlow.CryptoShredding.Avro.Analyzers
package.
In both cases, make sure to register EncryptedString
as a logical type:
LogicalTypeFactory.Instance.Register(new EncryptedStringLogicalType());
Using avrogen
Install avrogen:
$ dotnet tool install Apache.Avro.Tools
Then generate C# types:
$ dotnet avrogen -s <schema-file> <output-directory>
Using Avro Analyzers
Reference the package in your .csproj
file:
<PackageReference Include="Contrib.KafkaFlow.CryptoShredding.Avro.Analyzers"
Version="x.x.x"
OutputItemType="Analyzer"
ReferenceOutputAssembly="false" />
and add *.avsc
files:
<ItemGroup>
<AdditionalFiles Include="avro\*.avsc" />
</ItemGroup>
C# classes will be generated automatically for all schemas in the avro
directory.
Encryption Process
For serialization to occur, the secrets must be encrypted first, converting EncryptedString.Plain
into EncryptedString.Encrypted
. The
encryption process requires an encryption key, which is retrieved using an IEncryptionKeyProvider
service:
public interface IEncryptionKeyProvider
{
Task<string> GetKey(string keyId);
}
Application developers are responsible for securely storing and retrieving encryption keys. The key ID for encryption should be specified in
the Crypto-Shredding-Key-Id
header when publishing the message to the KafkaFlow producer.
Enabling Encryption
Encryption can be enabled transparently by adding the AvroEncryptionProducerMiddleware
to the producer pipeline:
services
.AddSingleton<IEncryptionKeyProvider, BigOrgSecureEncryptionKeyProvider>()
.AddKafkaFlowHostedService(kafka => kafka
.AddCluster(cluster => cluster
.AddProducer<IMyMessagesProducer>(producer => producer
.AddMiddlewares(middlewares => middlewares
// Encrypt sensitive data before serialization
.Add<AvroEncryptionProducerMiddleware>()
.AddSchemaRegistryAvroSerializer()
The middleware will:
- Check the
Crypto-Shredding-Key-Id
header - Retrieve the encryption key from the registered
IEncryptionKeyProvider
- Encrypt the sensitive parts of the message before serialization to Avro
Deserialization Process
Deserialization is handled by the AvroEncryptionConsumerMiddleware
, which can be registered as follows:
.AddConsumer(consumer => consumer
.AddMiddlewares(middlewares => middlewares
.AddSchemaRegistryAvroDeserializer()
// Decrypt sensitive data before consumption
.Add<AvroEncryptionConsumerMiddleware>()
The middleware will:
- Check the
Crypto-Shredding-Key-Id
header - Retrieve the encryption key from the registered
IEncryptionKeyProvider
- Decrypt the sensitive parts of the message
Optional Decryption
The decryption step is optional. If a consumer does not need access to the encrypted parts of the message, the decryption process can be omitted. The message will still be deserialized and processed, but the secrets will remain encrypted.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 was computed. net5.0-windows was computed. net6.0 was computed. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. |
.NET Core | netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.0 is compatible. netstandard2.1 is compatible. |
.NET Framework | net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen40 was computed. tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 2.0
- KafkaFlow.Abstractions (>= 3.0.10)
-
.NETStandard 2.1
- KafkaFlow.Abstractions (>= 3.0.10)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.