This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to... More information

FSharp Web Crawler

Simple web crawler that return IObservable using Reactive Extension(Rx) and async await.

Web scraper / crawler / spider. Supports robots protocol and user agent.

A library that includes a HttpHandler and HttpModule to let spiders know not to index your ASP.Net site