- 21,883 total downloads
- last updated 10/26/2016
- Latest version: 1.1.2
.NET Core port of sjdirect/nrobots. A robots.txt parser written in c#. This is an unofficial fork of NRobots.txt on codeplex with extended functionality. For full documentation see https://nrobots.codeplex.com/.
- 8,785 total downloads
- last updated 11/21/2016
- Latest version: 0.1.13-beta (prerelease)
.NET Core port of sjdirect/abot. Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own... More information