34 packages returned for Tags:"Robots"

SimpleSitemap is a lite library that helps you create web sitemaps for collections or lists of items. These sitemaps follow the Sitemap Protocol. Both sitemapindex and urlset links are generated, based upon the data collection size and 'page size'. Examples of this could be a list of your users,... More information

Fornece todos os recursos necessários para o spidering de aplicações web.

A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards. See https://bitbucket.org/cagdas/robotstxt for usage examples.

robots.txt middleware for .NET core

Configurable robots.txt handler supporting web.config based configuration. Icon supplied by Freepik (http://www.flaticon.com/authors/freepik)

Adds the X-Robots-Tag header to all responses.

IDeliverable.Seo is module for the Orchard CMS that allows site owners robust and granular control over SEO aspects. The module lets site owners control page titles, meta keywords and descriptions, robots.txt and sitemap.xml. Additionally, it allows 301/302 redirects to be configured for changed... More information

Sitecore Multisite Http module, allows mulitsite handling of 404, & 500 errors, along with custom robots.txt generation

A Simple and Powerful Library to Deal with Web Robots Control Strategy.

Louw.SitemapParser

The Robots Handler package provides editors with the ability to dynamically change the contents of a site's robots file. Instead of storing the contents of a robots file on the file system, an editor can specify its contents in an Umbraco content page. The property content is then served via a Http... More information

- It adds a robots.txt and a robots_closed.txt to the root of the website - It adds a rewrite rule to the web.config that rewrites the robots.txt to robots_closed.txt for all urls ending with tamtam.nl

Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.

robots.text implementation for EPiServer

AngleSharp IHtmlParser Extension for Mechanize.NET

Robots.txt parsing and querying utility

A tool for accessing and querying the contents of a robots.txt file.

A tool for loading and reading XmlSitemaps.

Sitemap and robots dynamic generator core classes

Sitemap and robots dynamic generator with gadget and admin plugins for EPiserver