Clapsode.DataLayerTS.WeatherDataLoader 0.9.0

Ingestion Manager for Loading Weather Forecasts from the DWD into Data Layer TS

This library can be used to ingest weather forecasts from the DWD into a Data Layer TS instance. The data can be used for forecasting any time series, which is dependent on the local weather (e.g. photovoltaic panel or wind turbine power production).

Purpose

The DWD (Deutscher Wetterdienst - german national weather forecasting service) offers weather forecasts for free (fully commercial use allowed), but overwrites the data every 24 hours. The scheduler currently supports loading data from the ICON-D2 (scope Germany) and the ICON-EU (scope Europe) grids of the ICON model. The ICON-D2 and the ICON-EU grid/model forecasts are estimated and published every 3 hours. There are more than one hundred different weather variables available and some of them are available on different vertical levels. The data is stored in a way where each (GRIB2) file contains all available grid-points (hundreds of thousands of grid-points) and only 1 hour of time steps. If the data should be used to train models, run forecasts or analyze historical data, the data is needed in different form - many time steps (also more than 24 hours) are needed for just one or a few grid-points. Data Layer TS is a high-performance time-series database technology which enables storing this data and making it available in the right form using a simple RESTful HTTP API. Currently it is available in the Microsoft Azure Marketplace exclusively.

Requirements

The library needs .Net 7 or .Net 8 to work and the license only grants use, if the library is used with an own Data Layer TS instance.

Configuration

The IngestionManager needs a SchedulerConfiguration to be initialized. The most important part of this configuration is the target Data Layer TS instance URL and credentials. The IngestionManager can either load continuously until it is canceled or just load data until no more data can be found and then stop (although new data is available every three hours). It can be started using a SchedulingRequest with two main parameters: The weather variables (ContentConfiguration), which should be loaded and the coordinates that should be extracted from the weather variable files. Instead of giving this information to the IngestionManager directly, it can also be configured to load all the stored variables and coordinates from the configured ILoadingConfigurationRepository, which is the target Data Layer TS instance per default and extend the configuration to continue loading these series. It will then load all the coordinates for all the variables - if any new variable is created in the target Data Layer TS instance, a background job will pick up the changes and start to load the data without any restart. The state of the processing scheduler is stored using the IProcessingHistoryRepository, which per default uses local files on disk, but since Data Layer TS write operations do not really depend on the number of time steps ingested, but rather on the number of time series ingested, a cold-start (no state stored) is not very expensive. The target Data Layer TS instance IOPS will most likely be the bottleneck and the scheduler should be configured to run as few parallel ingestions as possible while keeping the Data Layer TS IOPS at nearly 100%. This ensures maximum ingestion-throughput while keeping the read-throughput capability high.

Example Usage

In this usage example an ILogger and a TraceProvider is added and one example coordinate is loaded from both the ICON-EU and the ICON-D2 model:

using ILoggerFactory loggerFactory = LoggerFactory.Create(builder =>
{
	builder.AddFilter("Microsoft", LogLevel.Warning)
			.AddFilter("System", LogLevel.Warning)
			.AddFilter("IconLoaderDLTS", LogLevel.Information)
			.AddSimpleConsole(c => c.TimestampFormat = "[HH:mm:ss]");
});
ILogger logger = loggerFactory.CreateLogger("IconLoaderDLTS");

using TracerProvider tracerProvider = Sdk.CreateTracerProviderBuilder()
			.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("IconLoaderDLTS"))
			.AddSource("Clapsode.DataLayerTS.WeatherDataLoader")
			.AddConsoleExporter()
			.Build();

SchedulerConfiguration baseConfiguration = new("http://datalayerts.com/", "Basic ABC123abc==")
{
	EnableExtendingContentConfigurationFromRepository = true,
	EnableExtendingCoordinatesFromRepository = true,
	ParallelIngestionsPerScheduling = 4,
	CoordinatesPerBatchIngestionRequest = 64,
	MaxFilesToProcessPerBatch = 256
};
await using LoadingScheduler ingestionScheduler = new(baseConfiguration, logger);

Dictionary<NumericalWeatherPredictionModel, HashSet<Coordinate>> coordsToLoad = new();
coordsToLoad.Add(NumericalWeatherPredictionModel.ICON_D2, new HashSet<Coordinate>() { new Coordinate(51.5, 6.75) });
coordsToLoad.Add(NumericalWeatherPredictionModel.ICON_EU, new HashSet<Coordinate>() { new Coordinate(51.5, 6.75) });
SchedulingRequest request = new()
{
	RunContinuously = true,
	ConfigurationsToLoad = ContentConfiguration.PhotovoltaicPowerProductionDefault,
	CoordinatesToLoad = coordsToLoad,
	CoordinateMapping = CoordinateMapping.MAP_TO_SURROUNDING_FOUR,
	ExtendCoordinatesWithRepository = false,
	ExtendConfigurationsWithRepository = false,
};

await ingestionScheduler.RunAsync(request).ConfigureAwait(false);