TableStorage.Abstractions.POCO.SecondaryIndexes 1.4.0-beta

This is a prerelease version of TableStorage.Abstractions.POCO.SecondaryIndexes.
There is a newer version of this package available.
See the version list below for details.
dotnet add package TableStorage.Abstractions.POCO.SecondaryIndexes --version 1.4.0-beta
NuGet\Install-Package TableStorage.Abstractions.POCO.SecondaryIndexes -Version 1.4.0-beta
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="TableStorage.Abstractions.POCO.SecondaryIndexes" Version="1.4.0-beta" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add TableStorage.Abstractions.POCO.SecondaryIndexes --version 1.4.0-beta
#r "nuget: TableStorage.Abstractions.POCO.SecondaryIndexes, 1.4.0-beta"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install TableStorage.Abstractions.POCO.SecondaryIndexes as a Cake Addin
#addin nuget:?package=TableStorage.Abstractions.POCO.SecondaryIndexes&version=1.4.0-beta&prerelease

// Install TableStorage.Abstractions.POCO.SecondaryIndexes as a Cake Tool
#tool nuget:?package=TableStorage.Abstractions.POCO.SecondaryIndexes&version=1.4.0-beta&prerelease


Codacy Badge Build status Nuget

This project builds on top of TableStorage.Abstractions.POCO to introduce "secondary indexes" to Azure Table Storage. Internally this library uses an intra/inter partition (or table) secondary index pattern. When data gets mutated on your table store, the library takes care of reflecting the change in your secondary indexes.

Caveats And Notes

  1. Indexes are managed through a library, not Table Storage, thus data mutated outside of the library will not automatically be reflected in your indexes.
  2. Though Azure Table Storage does offer transactions within partitions, this library does not leverage this at this time.
  3. This library is intended for Azure Table Storage, not CosmosDB, which offers an Azure Table Storage API. CosmosDB does offer secondary indexes, so this library may not be as useful there.


Note that it may be useful to read about TableStorage.Abstractions.POCO to better understand the examples below.

All of the examples will use the following classes:

public class Employee
	public int CompanyId { get; set; }
	public int Id { get; set; }
	public string Name { get; set; }
	public Department Department { get; set; }
	public bool IsActive {get; set;} = true;

public class Department
	public int Id { get; set; }
	public string Name { get; set; }


Indexes are just regular PocoTableStores so you instantiate them like any other PocoTableStore. Here we instantiate the entity store and an index store. The PocoTableStore named TableStore will store records using CompanyId as a partition key, and Id as the row key. The PocoTableStore named IndexStore will store records using CompanyId as the partition key, and Name as the row key. In this example they use different tables.

TableStore = new PocoTableStore<Employee, int, int>("IXTestEmployee", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Id);

IndexStore = new PocoTableStore<Employee, int, string>("IXTestEmployeeNameIndex", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Name);

Next we tie them together by using AddIndex(). Indexes must be given a name so that you can specify which index to use when querying. Here we name our index "Name."

TableStore.AddIndex("Name", IndexStore);

After adding the index, mutations that happen on TableStore will result in mutations in IndexStore. For instance, if we insert a record as seen below, we can expect to find a corresponding record in IndexStore.

var employee = new Employee
	Name = "Test",
	CompanyId = 99,
	Id = 99,
	Department = new Department { Id = 5, Name = "Test" }

Conditional Indexes

Introduced in 1.1, you can now easily utilize conditional indexes. Conditional indexes allow you to add data to table storage only when a certain condition is true. Effectively this lets you easily place data into "buckets" that you can efficiently query later.

For example, suppose we want to quickly query only active employees. We can add a new index as described below:

TableStore.AddIndex("ActiveEmployee", new PocoTableStore<Employee, 
int, int>("IXActiveEmployees", "UseDevelopmentStorage=true", 
e => e.CompanyId, e => e.Id), e => e.IsActive);

Getting all active employees is now as easy as

var activeEmployees = TableStore.GetByIndexPartitionKey("ActiveEmployee", 99);

This query would yield all active employees for company 99, without penalty of an expensive partition scan at the server.

Note that conditional indexes are kept up to date, such that if a record were to no longer meet the condition (or later meet the condition), they will be removed or added to the index accordingly.

Fetching Data

To fetch a single data point from the index, we use the GetRecordByIndex (or GetRecordByIndexAsync) extension method on the entity PocoTableStore (note that we are doing this on the main data store, not on the index, as a convenience):

var e = TableStore.GetRecordByIndex("Name", 99, "Test");

Sometimes it may be useful to fetch all of the records from a partition for an index, such as historical data (described later). Example:

var records = await TableStore.GetByIndexPartitionKeyAsync("Name", 99);

One use of this pattern can be to store the current entity in the main entity store, and to keep historical data in a separate table. Here is an example of this pattern:

var pKeyMapper = new KeyMapper<Employee, int>(e => e.Id.ToString(), int.Parse, e => e.Id, id => id.ToString());

var rKeyMapper = new SequentialKeyMapper<Employee, int>(true);

var keysConverter = new CalculatedKeysConverter<Employee, int, int>(pKeyMapper, rKeyMapper);

var logStore = new PocoTableStore<Employee, int, int>("IXLogIndex", "UseDevelopmentStorage=true", keysConverter);

TableStore.AddIndex("Log", logStore);

In the example above we create an index called "Log", which will use Id as the partition key and a decreasing sequence number for row key (so that the most recent record is always on top).

If we want to fetch the history for employee 99, we do the following:

var records = TableStore.GetByPartitionKey(99);

Removing An Index

To remove an index without deleting data, use the Reindex() or ReindexAsync() extension method.

Dropping An Index

To remove and drop an index without deleting data, use the DropIndex() or DropIndexAsync() extension method. Deleting the original table will also drop all indexes on that table.

Seeding Or Reindexing

If you are adding an index to an existing table that already has data, or if for some reason data gets out of sync, you can use the Reindex() extension method, shown below. Note that this method is not yet optimized (for instance no batching is currently used). On my machine home internet connection, and data size, it took 22 minutes to index 1 million rows.

await TableStore.ReindexAsync("Name", maxDegreeOfParallelism: 20, recordsIndexedCallback: i=>count = i);

Call backs are available to get status updates and errors.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.0 is compatible.  netstandard2.1 was computed. 
.NET Framework net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen40 was computed.  tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.6.0 116 8/30/2023
1.5.0 517 4/2/2022
1.4.0 1,014 1/23/2022
1.4.0-beta 384 1/20/2022
1.3.2 378 11/17/2021
1.3.1 416 11/3/2021
1.3.0 405 10/5/2021
1.2.0 702 5/11/2020
1.1.0 499 5/7/2020
1.0.1 542 3/27/2020
1.0.0 511 3/26/2020

Updated underlying table storage