RobotsTxt 2014.2.19

RobotsTxt : A robots.txt parser for .NET

A robots.txt parser for .NET

Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.

See https://bitbucket.org/cagdas/robotstxt for usage examples.

Install-Package RobotsTxt -Version 2014.2.19
dotnet add package RobotsTxt --version 2014.2.19
<PackageReference Include="RobotsTxt" Version="2014.2.19" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxt --version 2014.2.19
The NuGet Team does not provide support for this client. Please contact its maintainers for support.

Dependencies

This package has no dependencies.

Version History

Version Downloads Last updated
2014.2.19 20,267 2/19/2014