Spidey 6.0.6

dotnet add package Spidey --version 6.0.6
                    
NuGet\Install-Package Spidey -Version 6.0.6
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Spidey" Version="6.0.6" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Spidey" Version="6.0.6" />
                    
Directory.Packages.props
<PackageReference Include="Spidey" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Spidey --version 6.0.6
                    
#r "nuget: Spidey, 6.0.6"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Spidey@6.0.6
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Spidey&version=6.0.6
                    
Install as a Cake Addin
#tool nuget:?package=Spidey&version=6.0.6
                    
Install as a Cake Tool

<img src="https://jacraig.github.io/Spidey/images/icon.png" style="height:25px" alt="Spidey Icon" /> Spidey

.NET Publish NuGet

Spidey is a flexible and extensible .NET library for crawling web content. It is designed for .NET Core applications and provides a modular architecture, allowing you to customize or extend any part of the crawling pipeline.

Features

  • Simple API for crawling websites
  • Highly configurable via the Options class
  • Dependency injection support (IoC/DI)
  • Easily replaceable subsystems (engine, parser, scheduler, etc.)
  • Callback-based result handling
  • NuGet package available

Quick Start

Install the NuGet package:

dotnet add package Spidey

Setting up the Library

Register Spidey in your app's service collection using the RegisterSpidey extension method:

using Microsoft.Extensions.DependencyInjection;
using Spidey;

var services = new ServiceCollection();
services.RegisterSpidey();

// Optionally, register your Options configuration
services.AddSingleton(new Options
{
    ItemFound = result => Console.WriteLine($"Found: {result.Url}"),
    Allow = new List<string> { "http://mywebsite", "http://mywebsite2" },
    FollowOnly = new List<string> { /* regex patterns */ },
    Ignore = new List<string> { /* regex patterns */ },
    StartLocations = new List<string> { "http://mywebsite", "http://mywebsite2" },
    UrlReplacements = new Dictionary<string, string> { /* { "old", "new" } */ },
    // Other options as needed
});

var provider = services.BuildServiceProvider();
var crawler = provider.GetRequiredService<Crawler>();

Alternatively, you can instantiate Crawler and Options directly without DI:

var options = new Options
{
    ItemFound = result => Console.WriteLine($"Found: {result.Url}"),
    // ...other options
};
var crawler = new Crawler(options);

Options Configuration

The Options class configures the crawler's behavior. Key properties include:

  • ItemFound (Action<ResultFile>): Callback invoked when a new page is discovered.
  • Allow (List<string>): Regex patterns for URLs allowed to be crawled.
  • FollowOnly (List<string>): Regex patterns for pages whose links should be followed.
  • Ignore (List<string>): Regex patterns for URLs to ignore.
  • StartLocations (List<string>): Initial URLs to start crawling from.
  • UrlReplacements (Dictionary<string, string>): URL replacements during crawling.
  • NetworkCredentials (NetworkCredential): Optional credentials for authentication.
  • UseDefaultCredentials (bool): Use default system credentials.
  • Proxy (IWebProxy): Optional proxy settings.

Example callback method:

void OnItemFound(ResultFile result)
{
    Console.WriteLine($"Discovered: {result.Url} (Status: {result.StatusCode})");
    // Additional processing...
}

Basic Usage

Once configured, start the crawl process:

crawler.StartCrawl();

The library will handle link discovery, content downloading, and result parsing. Your callback will be invoked for each discovered item.

Customization

Spidey is built with extensibility in mind. The system is divided into the following subsystems, each replaceable via DI:

  1. Content Parser (IContentParser) – Parses downloaded data into ResultFile objects.
  2. Engine (IEngine) – Handles HTTP requests and content downloading.
  3. Link Discoverer (ILinkDiscoverer) – Extracts links from content.
  4. Processor (IProcessor) – Processes parsed content (default: invokes your callback).
  5. Scheduler (IScheduler) – Manages work distribution.
  6. Pipeline (IPipeline) – Orchestrates the crawling process.

To customize, implement the relevant interface from Spidey.Engines.Interfaces and register your implementation in the service provider. Note that if you call RegisterSpidey(), the registration is handled for you automatically. If you instantiate Crawler directly, you must compose the pipeline manually.

FAQ

Q: Can I run the crawler on multiple nodes?

A: The default scheduler is single-node only. For distributed crawling, implement a custom scheduler (e.g., using a database or message queue) to coordinate work between instances.

Build Process

Requirements:

  • Visual Studio 2022

Clone the project and open the solution (Spidey.sln) in Visual Studio to build.

License

See LICENSE for details.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
6.0.6 204 8/28/2025
6.0.5 147 8/19/2025
6.0.4 164 7/16/2025
6.0.3 174 6/27/2025
6.0.2 181 6/25/2025
6.0.1 201 12/9/2024
6.0.0 175 11/25/2024
5.0.131 178 11/12/2024
5.0.130 160 11/11/2024
5.0.129 158 11/6/2024
5.0.128 150 11/5/2024
5.0.127 157 11/4/2024
5.0.126 156 10/31/2024
5.0.125 151 10/30/2024
5.0.124 165 10/29/2024
5.0.123 177 10/11/2024
5.0.122 169 10/10/2024
5.0.121 156 10/9/2024
5.0.120 166 10/2/2024
5.0.119 157 10/1/2024
5.0.118 171 9/24/2024
5.0.117 177 9/17/2024
5.0.116 205 9/10/2024
5.0.115 181 9/3/2024
5.0.114 172 8/30/2024
5.0.113 171 8/27/2024
5.0.112 184 8/26/2024
5.0.111 187 8/23/2024
5.0.110 193 8/21/2024
5.0.109 182 8/20/2024
5.0.108 195 8/16/2024
5.0.107 191 8/15/2024
5.0.106 161 8/5/2024
5.0.105 148 8/2/2024
5.0.104 157 8/1/2024
5.0.103 157 7/26/2024
5.0.102 176 7/11/2024
5.0.101 181 7/2/2024
5.0.100 181 6/27/2024
5.0.99 173 6/26/2024
5.0.98 185 6/19/2024
5.0.97 180 6/18/2024
5.0.96 191 6/17/2024
5.0.95 176 6/14/2024
5.0.94 159 6/13/2024
5.0.93 162 6/12/2024
5.0.92 176 5/31/2024
5.0.91 170 5/30/2024
5.0.90 178 5/17/2024
5.0.89 179 5/16/2024
5.0.88 199 5/8/2024
5.0.87 199 5/7/2024
5.0.86 211 5/6/2024
5.0.85 151 5/3/2024
5.0.84 163 5/2/2024
5.0.83 177 5/1/2024
5.0.82 189 4/30/2024
5.0.81 173 4/16/2024
5.0.80 180 4/12/2024
5.0.79 177 4/11/2024
5.0.78 197 4/1/2024
5.0.77 175 3/29/2024
5.0.76 196 3/18/2024
5.0.75 178 3/15/2024
5.0.74 187 3/14/2024
5.0.73 183 3/11/2024
5.0.72 173 3/8/2024
5.0.71 182 3/7/2024
5.0.70 202 3/6/2024
5.0.69 198 3/5/2024
5.0.68 174 3/4/2024
5.0.67 185 2/29/2024
5.0.66 181 2/28/2024
5.0.65 178 2/26/2024
5.0.64 178 2/23/2024
5.0.63 189 2/22/2024
5.0.62 188 2/21/2024
5.0.61 191 2/16/2024
5.0.60 197 2/15/2024
5.0.59 194 2/12/2024
5.0.58 185 2/8/2024
5.0.57 174 2/7/2024
5.0.56 181 2/6/2024
5.0.55 171 2/1/2024
5.0.54 187 1/31/2024
5.0.53 184 1/30/2024
5.0.52 167 1/24/2024
5.0.51 183 1/23/2024
5.0.50 185 1/12/2024
5.0.49 189 1/11/2024
5.0.48 196 12/26/2023
5.0.47 184 12/22/2023
5.0.46 181 12/18/2023
5.0.45 171 12/15/2023
5.0.44 169 12/14/2023
5.0.43 177 12/13/2023
5.0.42 176 12/12/2023
5.0.41 188 11/24/2023
5.0.40 187 11/21/2023
5.0.39 160 11/20/2023
5.0.38 160 11/17/2023
5.0.37 168 11/16/2023
5.0.36 164 11/14/2023
5.0.35 154 11/8/2023
5.0.34 151 11/7/2023
5.0.33 160 11/6/2023
5.0.32 158 11/1/2023
5.0.31 161 10/31/2023
5.0.30 180 10/30/2023
5.0.29 161 10/26/2023
5.0.28 206 10/12/2023
5.0.27 206 10/5/2023
5.0.26 189 9/26/2023
5.0.25 164 9/20/2023
5.0.24 164 9/19/2023
5.0.23 204 9/18/2023
5.0.22 180 9/14/2023
5.0.21 161 9/13/2023
5.0.20 191 9/11/2023
5.0.19 190 9/7/2023
5.0.18 192 9/6/2023
5.0.17 188 9/5/2023
5.0.16 199 9/4/2023
5.0.15 187 9/1/2023
5.0.14 194 8/31/2023
5.0.13 213 8/30/2023
5.0.12 219 8/29/2023
5.0.11 205 8/25/2023
5.0.10 219 8/23/2023
5.0.9 205 8/18/2023
5.0.8 228 8/10/2023
5.0.7 226 8/8/2023
5.0.6 226 8/8/2023
5.0.5 251 8/7/2023
5.0.4 231 8/3/2023
5.0.3 237 7/26/2023
5.0.2 217 7/20/2023
5.0.1 239 7/14/2023
5.0.0 400 12/12/2022
4.0.5 636 6/10/2022
4.0.2 599 1/11/2022
4.0.1 574 7/19/2021
3.0.9 635 1/7/2021
3.0.7 714 9/13/2020
3.0.6 692 6/26/2020
3.0.5 672 6/26/2020
3.0.3 673 3/25/2020
3.0.2 736 3/1/2020
3.0.1 756 1/1/2020
3.0.0 716 12/23/2019
2.0.15 760 11/22/2019
2.0.14 722 11/22/2019
2.0.13 702 11/22/2019
2.0.12 678 11/21/2019
2.0.11 689 11/21/2019
2.0.10 711 11/21/2019
2.0.9 705 11/21/2019
2.0.8 863 3/3/2019
2.0.7 796 3/3/2019
2.0.6 819 3/3/2019
2.0.5 804 3/3/2019
2.0.4 873 2/7/2019
2.0.3 1,364 6/1/2018
2.0.2 1,369 5/22/2018
2.0.1 1,498 1/2/2018
1.0.12 1,263 11/2/2017
1.0.11 1,262 10/30/2017
1.0.10 1,269 10/26/2017
1.0.9 1,260 10/26/2017
1.0.8 1,284 10/26/2017
1.0.7 1,241 10/25/2017
1.0.6 1,233 10/25/2017
1.0.5 1,268 10/24/2017
1.0.4 1,201 10/24/2017
1.0.3 1,192 10/19/2017
1.0.2 1,338 10/18/2017
1.0.1 1,247 9/29/2017