sjdirect 


Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to... More information
  • 78,681 total downloads
  • last updated 1/19/2016
  • Latest version: 1.0.8
  • utilities common
A robots.txt parser written in c#. This is an unofficial fork of NRobots.txt on codeplex with extended functionality. For full documentation see https://nrobots.codeplex.com/.
A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon the open source Abot C# Web Crawler by providing a powerful set of wrappers and extensions.
  • 11,415 total downloads
  • last updated 9/27/2017
  • Latest version: 1.0.9
Search engines and social networks are always trying to crawl your pages, but they only see the javascript tags... We render your javascript in a browser, save the static HTML, and you return that to the crawlers!