Ramin.GrokParser 1.0.0

dotnet add package Ramin.GrokParser --version 1.0.0
NuGet\Install-Package Ramin.GrokParser -Version 1.0.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Ramin.GrokParser" Version="1.0.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Ramin.GrokParser --version 1.0.0
#r "nuget: Ramin.GrokParser, 1.0.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Ramin.GrokParser as a Cake Addin
#addin nuget:?package=Ramin.GrokParser&version=1.0.0

// Install Ramin.GrokParser as a Cake Tool
#tool nuget:?package=Ramin.GrokParser&version=1.0.0

GrokParser

A dotnet library to parse and process Grok patterns in .NET

Build License

NuGet version Nuget

Install

Install from Nuget:

GrokParser

PM> dotnet add package Ramin.GrokParser

What is a Grok

A grok is a way to match a line of log data with a regular expression and extract structured fields from the line. It is commonly used in the context of log processing, where log data is stored in text files and it is necessary to extract specific information from the logs in order to analyze and understand them.

here is a sample grok pattern

%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:level}\] \[%{DATA:service}.%{DATA:module}\] %{GREEDYDATA:message}

this grok pattern matches the following string

2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input

In this example, the grok pattern is using named capture groups (e.g. TIMESTAMP_ISO8601, LOGLEVEL, DATA, and GREEDYDATA) to identify specific pieces of information in the log line. The named capture groups are enclosed in %{} and are separated by : from the field names (e.g. timestamp, level, service, module, and message) that will be used to represent the extracted information.

When the log line is processed using this grok pattern, the following field-value pairs will be extracted:

timestamp: 2022-12-27T10:15:30.456789Z
level: ERROR
service: my_service
module: my_module
message: An error occurred: invalid input

Usage

Simple pattern

var grokPattern = @"%{TIMESTAMP_ISO8601:timestamp:datetime} \[%{LOGLEVEL:level}\] \[%{DATA:service}.%{DATA:module}\] %{GREEDYDATA:message}";
var logs = @"2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input";
var grokBuilder = new GrokBuilder(grokPattern);
var grokParser = grokBuilder.Build();
var grokResult = grokParser.Parse(logs);

Adding custom patterns

you can add custom grok patterns to complement or override the default patterns

grok builder accepts any variable that implements the IDictionary<string,string> Interface

var customPatterns = new Dictionary<string,string>();
customPatterns.Add("TIMESTAMP_ISO8601", @"(?<timestamp>(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})T(?<hour>\d{2}):(?<minute>\d{2}):(?<second>\d{2})(\.(?<millisecond>\d{1,9}))?(?<timezone>Z|[+-]\d{2}:\d{2}))");
var grokPattern = @"%{TIMESTAMP_ISO8601:timestamp:datetime} \[%{LOGLEVEL:level}\] \[%{DATA:service}.%{DATA:module}\] %{GREEDYDATA:message}";
var logs = @"2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input";
var grokBuilder = new GrokBuilder(grokPattern,customPatterns);
var grokParser = grokBuilder.Build();
var grokResult = grokParser.Parse(logs);

Postprocessors

you can further match extracted fields with with additional grok patterns

the postProcessor argument accepts any variable that implements IEnumerable<KeyValuePair<string, string>>

after processing the main grok pattern any postProcess patterns will run on already extracted fields

var postProcessor = new List<KeyValuePair<string,string>>();
postProcessor.Add(new KeyValuePair<string, string>("timestamp", "postProcessPattern"));
var logs = @"2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input";
var grokBuilder = new GrokBuilder(grokPattern,postProcessors:postProcessor);
var grokParser = grokBuilder.Build();
var grokResult = grokParser.Parse(logs);

Filters

you can remove some of the extracted fields using the filters argument

in this example timestamp will not be in extracted fields

var filters = new List<string>();
filters.Add("timestamp");
var logs = @"2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input";
var grokBuilder = new GrokBuilder(grokPattern,filters:filters);
var grokParser = grokBuilder.Build();
var grokResult = grokParser.Parse(logs);

Timeout

since grok patterns are usually very complex regex patterns it is possible that matching a string will take a long time or worse never complete

you can use the timeout argument to ensure the match completes or fails in required time

the default timeout value is 1 second

var filters = new List<string>();
filters.Add("timestamp");
var logs = @"2022-12-27T10:15:30.456789Z [ERROR] [my_service.my_module] An error occurred: invalid input";
var grokBuilder = new GrokBuilder(grokPattern,timeout: TimeSpan.FromSeconds(30));
var grokParser = grokBuilder.Build();
var grokResult = grokParser.Parse(logs);

Types

you can specify the type of the field using the following syntax

%{TIMESTAMP_ISO8601:timestamp:datetime}

the last part specifies the type this filed needs to be converted to

currently the following types are supported

  • int
  • double
  • float
  • bool
  • datetime
  • long
  • datetimeoffset
Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.1 is compatible. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.0 203 12/27/2022