Net.RafaelEstevam.Spider.Simple.Lib 0.4.116

.NET Core 3.1
There is a newer version of this package available.
See the version list below for details.
dotnet add package Net.RafaelEstevam.Spider.Simple.Lib --version 0.4.116
NuGet\Install-Package Net.RafaelEstevam.Spider.Simple.Lib -Version 0.4.116
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Net.RafaelEstevam.Spider.Simple.Lib" Version="0.4.116" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Net.RafaelEstevam.Spider.Simple.Lib --version 0.4.116
#r "nuget: Net.RafaelEstevam.Spider.Simple.Lib, 0.4.116"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Net.RafaelEstevam.Spider.Simple.Lib as a Cake Addin
#addin nuget:?package=Net.RafaelEstevam.Spider.Simple.Lib&version=0.4.116

// Install Net.RafaelEstevam.Spider.Simple.Lib as a Cake Tool
#tool nuget:?package=Net.RafaelEstevam.Spider.Simple.Lib&version=0.4.116


[!] Work in Progress

A simple and modular web spider writen in C# .Net Core

Some advantages

  • Very simple to use and operate, ideal to personal or one of projects
  • Internal conversion from html to XElement, no need to external tools on use
  • Automatic Json parser to JObject
  • Automatic Json deserialize <T>
  • Modular Parser engine (you can add your own parsers!)
  • Modular Caching engine (you can add your own!)
  • Modular Downloader engine (you can add your own!)

Easy import with NuGet


Inside the Simple.Tests folders are various samples, these are some of them:

Use Json to parse Quotes

Json response? Get a event with your data already deserialized

( yes, these few lines below are full functional exemples! )

void run()
    var spider = new SimpleSpider("QuotesToScrape", new Uri(""));
    // create a json parser for our QuotesObject class
    spider.Parsers.Add(new JsonDeserializeParser<QuotesObject>(parsedResult_event))
    // add first
    spider.AddPage(buildPageUri(1), spider.BaseUri);
    // execute
void parsedResult_event(object sender, ParserEventArgs<QuotesObject> args)
    // add next
    if (args.ParsedData.has_next)
        int currPage =;
        ((SimpleSpider)sender).AddPage(buildPageUri(currPage + 1), args.FetchInfo.Link);
    // process data (show on console)
    foreach (var j in args.ParsedData.quotes)
        Console.WriteLine($"{ }: { j.text }");

see full source

Use XPath to select content

Use XPath to select elements and filter data

void run()
    var spider = new SimpleSpider("BooksToScrape", new Uri(""));
    // callback to gather items
    spider.FetchCompleted += fetchCompleted_items;
    // Ignore (cancel) the pages containing "/reviews/" 
    spider.ShouldFetch += (s, a) => { a.Cancel = a.Link.Uri.ToString().Contains("/reviews/"); };
    // execute from first page
void fetchCompleted_items(object Sender, FetchCompleteEventArgs args)
    // Colect new links
    (Sender as SimpleSpider).AddPage(AnchorHelper.GetAnchors(args.Link.Uri, args.Html), args.Link);

    // ignore all pages except the catalogue
    if (!args.Link.ToString().Contains("/catalogue/")) return;

    var XElement = HtmlToEXelement.Parse(args.Html);
    // collect book data
    var articleProd = XElement.XPathSelectElement("//article[@class=\"product_page\"]");
    if (articleProd == null) return; // not a book
    // Book info
    string sTitle = articleProd.XPathSelectElement("//h1").Value;
    string sPrice = articleProd.XPathSelectElement("//p[@class=\"price_color\"]").Value;
    string sStock = articleProd.XPathSelectElement("//p[@class=\"instock availability\"]").Value.Trim();
    string sDesc = articleProd.XPathSelectElement("p")?.Value; // books can be descriptionless

see full source

Easy initialization with chaining

Initialzie your spider easly with chaining

void run()
    var init = new InitializationParams()
        .SetCacher(new ContentCacher())
        .SetDownloader(new WebClientDownloader())
        .SetSpiderStarupDirectory(@"D:\spiders\") // Default directory
        // create a json parser for our QuotesObject class
        .AddParser(new JsonDeserializeParser<QuotesObject>(parsedResult_event))
        .SetConfig(c => c.Enable_Caching()

    var spider = new SimpleSpider("QuotesToScrape", new Uri(""), init);

    // add first
    spider.AddPage(buildPageUri(1), spider.BaseUri);
    // execute

see full source

Easy single resource fetch

Easy API pooling for updates

void run()
    var uri = new Uri("");
    var quotes = FetchHelper.FetchResourceJson<QuotesObject>(uri);
    // show the quotes deserialized
    foreach (var quote in quotes.quotes)
        Console.WriteLine($"Quote: {quote.text}");
        Console.WriteLine($"       - {}");

see full source

Some Helpers

  • FormsHelper: Deserialize html forms to easy manipulate data and create new requests
  • XmlSerializerHelper: Generic class to serialize and deserialzie stuff using Xml, easy way to save what you collected without any database
  • CSV Helper: Read csv files even compressed without exernal libraries
  • XElement to Stuff: Extract tables from page in DataTable

Giants' shoulders

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed. 
.NET Core netcoreapp3.1 is compatible. 
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.6.366 384 11/30/2020
0.6.365 618 10/20/2020
0.5.347 516 9/23/2020
0.5.320 356 9/17/2020
0.5.272 411 8/22/2020
0.5.239 423 8/10/2020
0.5.192 426 8/3/2020
0.5.164 479 7/30/2020
0.5.145 556 7/29/2020
0.4.116 449 7/26/2020
0.4.104 369 7/24/2020
0.4.76 414 7/21/2020
0.4.45 400 7/19/2020

Work in progress. See examples and documentation on GitHub page
Commit 03d5645