returned for Tags:"robots"
SimpleSitemap is a lite library that helps you create web sitemaps for collections or lists of items.
These sitemaps follow the Sitemap Protocol. Both sitemapindex and urlset links are generated, based upon the data collection size and 'page size'.
Examples of this could be a list of your users,...
Fornece todos os recursos necessários para o spidering de aplicações web.
A robots.txt parser for .NET
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.
See https://bitbucket.org/cagdas/robotstxt for usage examples.
19,166 total downloads
robots.txt middleware for .NET core
Configurable robots.txt handler supporting web.config based configuration. Icon supplied by Freepik (http://www.flaticon.com/authors/freepik)
Adds the X-Robots-Tag header to all responses.
IDeliverable.Seo is module for the Orchard CMS that allows site owners robust and granular control over SEO aspects. The module lets site owners control page titles, meta keywords and descriptions, robots.txt and sitemap.xml. Additionally, it allows 301/302 redirects to be configured for changed...
Sitecore Multisite Http module, allows mulitsite handling of 404, & 500 errors, along with custom robots.txt generation
A Simple and Powerful Library to Deal with Web Robots Control Strategy.
The Robots Handler package provides editors with the ability to dynamically change the contents of a site's robots file. Instead of storing the contents of a robots file on the file system, an editor can specify its contents in an Umbraco content page. The property content is then served via a Http...
- It adds a robots.txt and a robots_closed.txt to the root of the website
- It adds a rewrite rule to the web.config that rewrites the robots.txt to robots_closed.txt for all urls ending with tamtam.nl
Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.
380 total downloads
robots.text implementation for EPiServer
AngleSharp IHtmlParser Extension for Mechanize.NET
A tool for accessing and querying the contents of a robots.txt file.
A tool for loading and reading XmlSitemaps.
Sitemap and robots dynamic generator core classes
Sitemap and robots dynamic generator with gadget and admin plugins for EPiserver
456 total downloads
Robots.txt parsing and querying utility