Encodeous.DirtyProxy 1.0.3

.NET 5.0
dotnet add package Encodeous.DirtyProxy --version 1.0.3
NuGet\Install-Package Encodeous.DirtyProxy -Version 1.0.3
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Encodeous.DirtyProxy" Version="1.0.3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Encodeous.DirtyProxy --version 1.0.3
#r "nuget: Encodeous.DirtyProxy, 1.0.3"
#r directive can be used in F# Interactive, C# scripting and .NET Interactive. Copy this into the interactive tool or source code of the script to reference the package.
// Install Encodeous.DirtyProxy as a Cake Addin
#addin nuget:?package=Encodeous.DirtyProxy&version=1.0.3

// Install Encodeous.DirtyProxy as a Cake Tool
#tool nuget:?package=Encodeous.DirtyProxy&version=1.0.3

DirtyProxy

A quick and easy proxy scraper!

That does not mean it lacks features 😉

Features

  • High-Performance Asynchronous Scraping
  • Highly Configurable
  • Comes with proxy checker
  • Comes with a default list of proxy sites
  • Lightweight (~6% CPU usage on a 4-core i7)
  • Filter unique proxies

Usage

Please note, enabling proxy checking (on by default) will take MUCH longer!

Using Default Parameters

var scraper = new ProxyScraper(ProxyScraper.DefaultList);
var proxies = await scraper.ScrapeAsync();

await File.WriteAllLinesAsync("validProxies.txt", proxies.ValidProxies.Select(x=>x.ToString()));
await File.WriteAllLinesAsync("validSources.txt", proxies.ValidSources.Select(x=>x.Trim()));
await File.WriteAllLinesAsync("allProxies.txt", proxies.Proxies.Select(x=>x.ToString()));

Using Custom Proxy Source List

var sources = new[]
{
    "https://source.proxy.list",
    "https://other.source.proxy.list"
};
// You can use your own list, or the list included by default!
var scraper = new ProxyScraper(sources);

...

Using Custom Proxy Checker

var scraper = new ProxyScraper(ProxyScraper.DefaultList, async proxy =>
{
    try
    {
        var wc = new WebClient();
        wc.Proxy = new WebProxy(proxy.ToString());
        wc.Headers[HttpRequestHeader.UserAgent] = ProxyScraper.DefaultAgent;
        // timeout in 10 seconds
        var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
        cts.Token.Register(wc.CancelAsync);
        await wc.OpenReadTaskAsync("https://google.com");
        cts.Dispose();
        return true;
    }
    catch
    {
        return false;
    }
});

...

Using Custom User Agent

// You can use any user agent you want!
var scraper = new ProxyScraper(ProxyScraper.DefaultList, "Your user agent");
var proxies = await scraper.ScrapeAsync();

...

Using Custom Request Timeouts

var scraper = new ProxyScraper(ProxyScraper.DefaultList, checkTimeout: 5, scrapeTimeout: 2);
var proxies = await scraper.ScrapeAsync();

...

Fast Scraping (Without proxy validation)

// Disable proxy checking
var scraper = new ProxyScraper(ProxyScraper.DefaultList, checkProxies: false);
var proxies = await scraper.ScrapeAsync();

...

Custom Proxy Check URL

// Make sure the proxies can successfully connect to a url
var scraper = new ProxyScraper(ProxyScraper.DefaultList, checkUrl: "https://google.ca");
var proxies = await scraper.ScrapeAsync();

...

Misc Configuration

// number of tasks for proxy checking (mainly waiting)
ProxyScraper.CheckTasks = 300;
Product Versions
.NET net5.0 net5.0-windows net6.0 net6.0-android net6.0-ios net6.0-maccatalyst net6.0-macos net6.0-tvos net6.0-windows net7.0 net7.0-android net7.0-ios net7.0-maccatalyst net7.0-macos net7.0-tvos net7.0-windows
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.
  • net5.0

    • No dependencies.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.3 238 5/17/2021
1.0.2 293 5/16/2021
1.0.1 264 5/15/2021
1.0.0 277 5/15/2021