grok.net 2.0.0

dotnet add package grok.net --version 2.0.0
NuGet\Install-Package grok.net -Version 2.0.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="grok.net" Version="2.0.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add grok.net --version 2.0.0
#r "nuget: grok.net, 2.0.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install grok.net as a Cake Addin
#addin nuget:?package=grok.net&version=2.0.0

// Install grok.net as a Cake Tool
#tool nuget:?package=grok.net&version=2.0.0

Grok

Cross platform .NET grok implementation as a NuGet package

Build GitHub release) License contributions welcome

NuGet version Nuget PowerShell Gallery Version PowerShell Gallery

Code Coverage

Coverage Status

How to Install

Install as a library from Nuget:

Grok.Net

PM> Install-Package Grok.Net

Install as a PowerShell module from PowershellGallery:

Grok

Install-Module -Name Grok

What is grok

Grok is a great way to parse unstructured log data into something structured and queryable. It sits on top of Regular Expression (regex) and uses text patterns to match lines in log files.

A great way to get started with building yours grok filters is this grok debug tool: https://grokdebug.herokuapp.com/

What can I use Grok for?

  • reporting errors and other patterns from logs and processes
  • parsing complex text output and converting it to json for external processing
  • apply 'write-once use-everywhere' to regular expressions
  • automatically providing patterns for unknown text inputs (logs you want patterns generated for future matching)

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The SYNTAX is the name of the pattern that will match your text. SEMANTIC is the key.

For example, 3.44 will be matched by the NUMBER pattern and 55.3.244.1 will be matched by the IP pattern. 3.44 could be the duration of an event, so you could call it simply duration. Further, a string 55.3.244.1 might identify the client making a request. For the above example, your grok filter would look something like this:

%{NUMBER:duration} %{IP:client}

Examples: With that idea of a syntax and semantic, we can pull out useful fields from a sample log like this fictional http request log:

55.3.244.1 GET /index.html 15824 0.043

The pattern for this could be:

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

More about grok

How to use

Create a new instance with grok pattern:

Grok grok = new Grok("%{MONTHDAY:month}-%{MONTHDAY:day}-%{MONTHDAY:year} %{TIME:timestamp};%{WORD:id};%{LOGLEVEL:loglevel};%{WORD:func};%{GREEDYDATA:msg}");

then prepare some logs to parse

string logs = @"06-21-19 21:00:13:589241;15;INFO;main;DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165
                06-22-19 22:00:13:589265;156;WARN;main;DECODED: 775233900043 EMPTY DISTANCE: --------";

You are ready to parse and print result

var grokResult = grok.Parse(logs);
foreach (var item in grokResult)
{
    Console.WriteLine($"{item.Key} : {item.Value}");
}

output:

month : 06
day : 21
year : 19
timestamp : 21:00:13:589241
id : 15
loglevel : INFO
func : main
msg : DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165
month : 06
day : 22
year : 19
timestamp : 22:00:13:589265
id : 156
loglevel : WARN
func : main
msg : DECODED: 775233900043 EMPTY DISTANCE: --------

or use ToDictionary() on grokResult to get the result as IReadOnlyDictionary<string, IEnumerable<object>>

Custom grok patterns

There is the possibility to add your own patterns.

using file

Create a file and write the pattern you need as the pattern name, space, then the regexp for that pattern.

For example, Patterns\grok-custom-patterns:

ZIPCODE [1-9]{1}[0-9]{2}\s{0,1}[0-9]{3}

then load the file and pass the stream to Grok:

FileStream customPatterns = System.IO.File.OpenRead(@"Patterns\grok-custom-patterns");
Grok grok = new Grok("%{ZIPCODE:zipcode}:%{EMAILADDRESS:email}", customPatterns);
var grokResult = grok.Parse($"122001:Bob.Davis@microsoft.com");

using in-memory

Define a collection of patterns

var custom = new Dictionary<string, string>
{
    {"BASE64", "(?=(.{4})*$)[A-Za-z0-9+/]*={0,2}$"}
};

and use it as follows

var grok = new Grok("Basic %{BASE64:credentials}", custom);
GrokResult grokResult = grok.Parse("Basic YWRtaW46cGEkJHdvcmQ=");

PowerShell Module

Install and use the Grok as a PowerShell module

grok -i "06-21-19 21:00:13:589241;15;INFO;main;DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165" -g "%{MONTHDAY:month}-%{MONTHDAY:day}-%{MONTHDAY:year} %{TIME:timestamp};%{WORD:id};%{LOGLEVEL:loglevel};%{WORD:func};%{GREEDYDATA:msg}"

To get help use help grok command

Build

On Windows:

build.ps1

On Linux/Mac:

build.sh

Contributing

Would you like to help make grok.net even better? We keep a list of issues that are approachable for newcomers under the good-first-issue label.

Also. please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Thanks to @martinjt. The project is based on martinjt/grokdotnet.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 is compatible.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.0 is compatible.  netstandard2.1 is compatible. 
.NET Framework net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen40 was computed.  tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories (1)

Showing the top 1 popular GitHub repositories that depend on grok.net:

Repository Stars
sandreas/tone
tone is a cross platform audio tagger and metadata editor to dump and modify metadata for a wide variety of formats, including mp3, m4b, flac and more. It has no dependencies and can be downloaded as single binary for Windows, macOS, Linux and other common platforms.
Version Downloads Last updated
2.0.0 4,437 9/1/2023
1.2.0 179 8/28/2023
1.1.0 10,992 1/2/2022
1.0.1 11,146 7/13/2019