• 207,827 total downloads
  • last updated 11/2/2020
  • Latest version: 1.1.3
  • utilities common
.NET Core port of sjdirect/nrobots. A robots.txt parser written in c#. This is an unofficial fork of NRobots.txt on codeplex with extended functionality. For full documentation see
  • 17,363 total downloads
  • last updated 11/21/2016
  • Latest version: 0.1.13-beta (prerelease)
  • crawler robot spider
.NET Core port of sjdirect/abot. Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own... More information