Aiursoft.ArrayDb.WriteBuffer 1.0.6

Aiursoft ArrayDb

Aiursoft ArrayDb is a lightweight, efficient database engine optimized for storing fixed-length data with constant-time indexing performance (O(1)). ArrayDb is ideal for scenarios where fast, reliable storage and access to time-sequenced or resource-utilization data are essential, making it a strong choice for logging, telemetry, and performance tracking use cases.

Key Design Principles of ArrayDb

Unlike traditional databases, which can struggle with high-frequency data storage requirements, ArrayDb is purpose-built for append-only, fixed-length data storage. It stores entries as continuous, fixed-length data blocks, optimized for minimal read/write operations, ensuring high performance on both SSDs and HDDs.

ArrayDb organizes data into two categories:

1. Fixed-Length Attributes: These attributes, such as integers, DateTime, and booleans, are stored in a fixed-length array, facilitating fast O(1) access by index.
2. Variable-Length Attributes: For data like strings, ArrayDb maintains a separate variable-length array, where each entry contains a pointer in the fixed-length array for fast access.

Advantages of ArrayDb

1. High-Speed Access: ArrayDb stores each element sequentially on disk. By calculating an element’s exact location based on its index, ArrayDb can retrieve or count elements in constant time, O(1).
2. Optimized Append Performance: New entries are simply appended, which maintains data consistency and reduces disk fragmentation. This approach also ensures sustained high-speed write performance, even under heavy load.
3. Efficient Reads: Reading from ArrayDb only requires a single disk operation to load all columns in an entry, which contrasts with traditional columnar storage that performs multiple reads for each column.

Limitations

ArrayDb is designed for simplicity and speed but with limited data manipulation:

• No Structural Modifications: The structure of stored data cannot be edited after creation.
• Append-Only: Supports only appending new entries; no item deletions or mid-array insertions are allowed.
• Limited Updates: Modifying variable-length data (e.g., resizing strings) is not supported.

Best Use Cases for ArrayDb

• Fixed-Length Data: Suitable for storing time-series data, such as CPU or memory usage metrics.
• Time-Based Indexing: Ideal for sequential logging or telemetry data, where entries are naturally appended and queried by timestamp.

Read-Write Performance Difference

Large-scale writes are significantly faster than reads because ArrayDb optimizes writes by pre-arranging data (including strings) in memory. This enables sequential, continuous writes to disk, minimizing disk-seeking time to O(1).

In contrast, reads require accessing each string or variable-length attribute individually, creating random access patterns due to potential data fragmentation. As a result, reading incurs a higher O(n) disk-seeking time, where n is the element count. ArrayDb uses an LRU cache to reduce physical disk reads, but in multi-threaded reads, this cache introduces high CPU load.

How to use ArrayDb

Unlike MySQL, working as a process, ArrayDb works as a library. You can use ArrayDb in your C# project by adding the ArrayDb NuGet package to your project.

mkdir LearnArrayDb
cd LearnArrayDb
dotnet new console
dotnet add package Aiursoft.ArrayDb.Partitions

That's it. Now you have ArrayDb in your project.

Building the module

You can start using it by creating a new entity with type: PartitionedBucketEntity<T>, where T is the partition key type.

Supported property types are:

• int
• bool
• string
• DateTime
• long
• float
• double
• TimeSpan
• Guid

public class MyLogItem : PartitionedBucketEntity<string>
{
    [PartitionKey] 
    public string ApplicationName { get; set; } = string.Empty;

    [PartitionKey]
    public override string PartitionId
    {
        get => ApplicationName;
        set => ApplicationName = value;
    }
    
    public DateTime HappenTime { get; set; } = DateTime.UtcNow;

    public string LogMessage { get; set; } = string.Empty;

    public int HttpResponseCode { get; set; }

    public string RequestPath { get; set; } = string.Empty;
    
    public TimeSpan ResponseTime { get; set; }
}

Then you can start using ArrayDb by creating a new PartitionedBucket<T> instance.

var databaseName = "my-db";
var databaseFilePath = "/tmp/my-db";
Directory.CreateDirectory(databaseFilePath);
        
var db = new PartitionedObjectBucket<MyLogItem, string>(databaseName, databaseFilePath);

Writing data

Now you can start using the db instance to write some data.

// Write to the database.
db.Add(new MyLogItem
{
    ApplicationName = "NextCloud",
    LogMessage = "A user logged in.",
    HttpResponseCode = 200,
    RequestPath = "/account/login",
    ResponseTime = TimeSpan.FromMilliseconds(100)
});

db.Add(new MyLogItem
{
    ApplicationName = "NextCloud",
    LogMessage = "A user logged out.",
    HttpResponseCode = 200,
    RequestPath = "/account/logout",
    ResponseTime = TimeSpan.FromMilliseconds(50)
});

db.Add(new MyLogItem
{
    ApplicationName = "GitLab",
    LogMessage = "A user created a new project.",
    HttpResponseCode = 201,
    RequestPath = "/projects/new",
    ResponseTime = TimeSpan.FromMilliseconds(200)
});

db.Add(new MyLogItem
{
    ApplicationName = "Jellyfin",
    LogMessage = "Server crashed when playing a video.",
    HttpResponseCode = 500,
    RequestPath = "/play/video",
    ResponseTime = TimeSpan.FromMilliseconds(500)
});

And you can use bulk write to improve performance.

var logs = new List<MyLogItem>();
for (var i = 0; i < 100; i++)
{
    logs.Add(new MyLogItem
    {
        ApplicationName = "HomeAssistant",
        LogMessage = $"A human was detected by the camera {i}.",
        HttpResponseCode = 200,
        RequestPath = $"camera/{i}/detect",
        ResponseTime = TimeSpan.FromMilliseconds(100)
    });
}
db.AddBulk(logs.ToArray());

Calling SyncAsync() is optional. It will block current thread and flush the data to the disk. However, if you don't call it, the data will also be archived very soon. Only call this to ensure the data is written to the disk.

await db.SyncAsync();

Reading data

You can read data from the database by using the db instance. For example, if you want to read from a specific partition and index, you can simply call Read with the partition key and index.

// Read a specific item.
var specificLog = db.Read(partitionKey: "NextCloud", index: 1);
Console.WriteLine($"[{specificLog.HappenTime}] {specificLog.LogMessage}");

Calling Read has low performance when you need to read a large amount of data. You can use ReadBulk to read bulk data.

// Bulk read logs from a specific partition.
var nextCloudLogs = db.ReadBulk(
    partitionKey: "NextCloud",
    indexFrom: 0,
    count: 2);

foreach (var log in nextCloudLogs)
{
    Console.WriteLine($"[{log.HappenTime}] {log.LogMessage}");
}

You may also want to know how many logs are there in a specific partition. You can use Count to get the count of logs in a specific partition.

var nextCloudLogsCount = db.GetPartitionById("NextCloud")
    .InnerBucket
    .ArchivedItemsCount;
Console.WriteLine("NextCloud logs count: " + nextCloudLogsCount);

If you want to get all data from all partitions, you can use ReadAll to get all data.

// Read all from the database. (Not recommended for large data)
var allLogs = db.ReadAll();
Console.WriteLine("All logs count: " + allLogs.Length);

How to contribute

There are many ways to contribute to the project: logging bugs, submitting pull requests, reporting issues, and creating suggestions.

Even if you with push rights on the repository, you should create a personal fork and create feature branches there when you need them. This keeps the main repository clean and your workflow cruft out of sight.

We're also interested in your feedback on the future of this project. You can submit a suggestion or feature request through the issue tracker. To make this process more effective, we're asking that these include more information to help define them more clearly.