Net.RafaelEstevam.Spider.Simple.Lib 0.4.76

.NET Core 3.1
There is a newer version of this package available.
See the version list below for details.
dotnet add package Net.RafaelEstevam.Spider.Simple.Lib --version 0.4.76
NuGet\Install-Package Net.RafaelEstevam.Spider.Simple.Lib -Version 0.4.76
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Net.RafaelEstevam.Spider.Simple.Lib" Version="0.4.76" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Net.RafaelEstevam.Spider.Simple.Lib --version 0.4.76
#r "nuget: Net.RafaelEstevam.Spider.Simple.Lib, 0.4.76"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Net.RafaelEstevam.Spider.Simple.Lib as a Cake Addin
#addin nuget:?package=Net.RafaelEstevam.Spider.Simple.Lib&version=0.4.76

// Install Net.RafaelEstevam.Spider.Simple.Lib as a Cake Tool
#tool nuget:?package=Net.RafaelEstevam.Spider.Simple.Lib&version=0.4.76


[!] Work in Progress

A simple and modular web spider writen in C# .Net Core

Some advantages

  • Very simple to use and operate, ideal to personal or one of projects
  • Internal conversion from html to XElement, no need to external tools on use
  • Automatic Json parser to JObject
  • Automatic Json deserialize <T>
  • Modular Parser engine (you can add your own parsers!)
  • Modular Caching engine (you can add your own!)
  • Modular Downloader engine (you can add your own!)

Easy import with NuGet


Inside Simple.Tests are some spiders to show to crawl and collect data

Use Json to parse Quotes

see full source

void run()
    var spider = new SimpleSpider("QuotesToScrape", new Uri(""));
    // create a json parser for our QuotesObject class
    spider.Parsers.Add(new Parsers.JsonDeserializeParser<QuotesObject>(parsedResult_event))
    // add first
    spider.AddPage( buildPageUri(1), spider.BaseUri);
    // execute
void parsedResult_event(object sender, Interfaces.ParserEventArgs<QuotesObject> args)
    // add next
    if (args.ParsedData.has_next)
        int currPage =;
        ((SimpleSpider)sender).AddPage(buildPageUri(currPage + 1), args.FetchInfo.Link);
    // process data (show on console)
    foreach (var j in args.ParsedData.quotes)
        Console.WriteLine($"{ }: { j.text }");

Use XPath to select content

see full source

void run()
    var spider = new SimpleSpider("BooksToScrape", new Uri(""));
    // callback to gather items
    spider.FetchCompleted += fetchCompleted_items;
    // Ignore (cancel) the pages containing "/reviews/" 
    spider.ShouldFetch += (s, a) => { a.Cancel = a.Link.Uri.ToString().Contains("/reviews/"); };
    // execute from first page
void fetchCompleted_items(object Sender, FetchCompleteEventArgs args)
    // Colect new links
    (Sender as SimpleSpider).AddPage(AnchorHelper.GetAnchors(args.Link.Uri, args.Html), args.Link);

    // ignore all pages except the catalogue
    if (!args.Link.ToString().Contains("/catalogue/")) return;

    var XElement = HtmlToEXelement.Parse(args.Html);
    // collect book data
    var articleProd = XElement.XPathSelectElement("//article[@class=\"product_page\"]");
    if (articleProd == null) return; // not a book
    // Book info
    string sTitle = articleProd.XPathSelectElement("//h1").Value;
    string sPrice = articleProd.XPathSelectElement("//p[@class=\"price_color\"]").Value;
    string sStock = articleProd.XPathSelectElement("//p[@class=\"instock availability\"]").Value.Trim();
    string sDesc = articleProd.XPathSelectElement("p")?.Value; // books can be descriptionless

Easy initialization with chaining

see full source

void run()
    var init = new InitializationParams()
        .SetCacher(new ContentCacher())
        .SetDownloader(new WebClientDownloader())
        .SetSpiderStarupDirectory(@"D:\spiders\") // Default directory
        // create a json parser for our QuotesObject class
        .AddParser(new Parsers.JsonDeserializeParser<QuotesObject>(parsedResult_event))
        .SetConfig(c => c.Enable_Caching()

    var spider = new SimpleSpider("QuotesToScrape", new Uri(""), init);

    // add first
    spider.AddPage(buildPageUri(1), spider.BaseUri);
    // execute

Some Helpers

  • XmlSerializer helper: Generic class to serialize and deserialzie stuff using Xml, easy way to save what you collected without any database
  • CSV helper: Read csv files even compressed without exernal libraries
  • XElement to Stuff: Extract tables from page in DataTable

Giants' shoulders

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed. 
.NET Core netcoreapp3.1 is compatible. 
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.6.366 384 11/30/2020
0.6.365 618 10/20/2020
0.5.347 523 9/23/2020
0.5.320 362 9/17/2020
0.5.272 417 8/22/2020
0.5.239 424 8/10/2020
0.5.192 427 8/3/2020
0.5.164 479 7/30/2020
0.5.145 557 7/29/2020
0.4.116 455 7/26/2020
0.4.104 376 7/24/2020
0.4.76 414 7/21/2020
0.4.45 400 7/19/2020

Work in progress
Commit 61a64a6