Robots.AspNet 1.0.0

Robots.AspNet

A lightweight ASP.NET Core library that serves a dynamic robots.txt file via middleware, with a fluent builder API and appsettings.json support.

Quick start

1. Install

dotnet add package Robots.AspNet

2. Configure

In Program.cs, use the fluent builder to define your rules:

using Robots.AspNet;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRobotsTxt()
    .Disallow("GPTBot", "/")
    .Disallow("*", "/admin/", "/login/")
    .Allow("/login/blabla")
    .Sitemap("https://example.com/sitemap.xml");

var app = builder.Build();

app.UseRobotsTxt();
app.Run();

That's it. A GET /robots.txt now returns:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /login/blabla

Sitemap: https://example.com/sitemap.xml

API

AddRobotsTxt() returns IRobotsTxtBuilder

services.AddRobotsTxt()

Three overloads:

IRobotsTxtBuilder fluent methods:

IRobotsTxtBuilder Disallow(string userAgent, params string[] paths)
IRobotsTxtBuilder Allow(string path)
IRobotsTxtBuilder Allow(string userAgent, string path)
IRobotsTxtBuilder Sitemap(string url)

• Allow(path) applies to the user-agent from the preceding Disallow() call
• Allow(userAgent, path) explicitly targets a user-agent
• Duplicate paths and sitemaps are silently ignored

Middleware

app.UseRobotsTxt();

Place it anywhere in the pipeline. Requests to /robots.txt (case-insensitive) return the generated content as text/plain; charset=utf-8. All other requests pass through.

appsettings.json

{
  "RobotsTxt": {
    "Rules": [
      {
        "UserAgent": "GPTBot",
        "Disallow": ["/"]
      },
      {
        "UserAgent": "*",
        "Disallow": ["/admin/", "/login/"],
        "Allow": ["/login/blabla"]
      }
    ],
    "Sitemaps": ["https://example.com/sitemap.xml"]
  }
}

Then in Program.cs:

builder.Services.AddRobotsTxt(builder.Configuration.GetSection("RobotsTxt"))
    .Disallow("BadBot", "/"); // merge additional rules

Output format

Follows RFC 9309 — Robots Exclusion Protocol:

• UTF-8 encoded, text/plain content type
• Groups separated by blank lines
• Rules within a group are output in registration order

NuGet

dotnet pack src/Robots.AspNet

The package includes symbols (snupkg) and requires no NuGet dependencies beyond the ASP.NET Core shared framework.

Project structure

Robots.AspNet/
├── README.md
├── Robots.AspNet.slnx
└── src/
    ├── Robots.AspNet/           # Library
    │   ├── IRobotsTxtBuilder.cs
    │   ├── RobotsTxtBuilder.cs
    │   ├── RobotsTxtMiddleware.cs
    │   ├── RobotsTxtOptions.cs
    │   ├── RobotsTxtRule.cs
    │   ├── ServiceCollectionExtensions.cs
    │   └── ApplicationBuilderExtensions.cs
    └── Robots.AspNet.Tests/     # NUnit tests
        ├── RobotsTxtBuilderTests.cs
        ├── RobotsTxtMiddlewareTests.cs
        └── ConfigurationBindingTests.cs