RobotsTxtCore 2.0.0-preview1

.NET 5.0 .NET Core 3.1
This is a prerelease version of RobotsTxtCore.
There is a newer version of this package available.
See the version list below for details.
dotnet add package RobotsTxtCore --version 2.0.0-preview1
NuGet\Install-Package RobotsTxtCore -Version 2.0.0-preview1
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="RobotsTxtCore" Version="2.0.0-preview1" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxtCore --version 2.0.0-preview1
#r "nuget: RobotsTxtCore, 2.0.0-preview1"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install RobotsTxtCore as a Cake Addin
#addin nuget:?package=RobotsTxtCore&version=2.0.0-preview1&prerelease

// Install RobotsTxtCore as a Cake Tool
#tool nuget:?package=RobotsTxtCore&version=2.0.0-preview1&prerelease


From version 2.0 I've dropped support for .NET Core versions prior to 3.1 and made a huge breaking change by adding a IRobotsTxtProvider interface seperate from the middleware to handle configuration.

A Robots.txt middleware for ASP.NET Core. Why is this needed you ask? Because if you need to add dynamic values (such as a configured url from your CMS) you'll need some sort of code to handle that, and this makes it easy.



PM> Install-Package RobotsTxtCore

.Net CLI

> dotnet add package RobotsTxtCore


To specify multiple rules with the fluent interface makes it really easy.

public void ConfigureServices(IServiceCollection services) {
    services.AddStaticRobotsTxt(builder =>
            .AddSection(section =>
                    .AddComment("Allow Googlebot")
            .AddSection(section =>
                    .AddComment("Disallow the rest")

public void Configure(IApplicationBuilder app) {


# Allow Googlebot
User-agent: Googlebot
Allow: /

# Disallow the rest
User-agent: *
Disallow: /
Crawl-delay: 10


Or if you just want to deny everyone.

public void ConfigureServices(IServiceCollection services) {
    services.AddStaticRobotsTxt(builder =>

public void Configure(IApplicationBuilder app) {


User-agent: *
Disallow: /
Product Compatible and additional computed target framework versions.
.NET net5.0 is compatible.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp3.1 is compatible. 
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on RobotsTxtCore:

Package Downloads

Package Description


Package Description

GitHub repositories (1)

Showing the top 1 popular GitHub repositories that depend on RobotsTxtCore:

Repository Stars
☀☀支持Markdown导入的博客。后端基于最新的.Net6和Asp.Net Core框架,遵循RESTFul接口规范,前端基于Vue+ElementUI开发,可作为 .Net Core 入门项目学习~
Version Downloads Last updated
3.1.0-preview1 235 10/16/2023
3.0.0 17,736 10/1/2023
2.2.0 72,263 12/24/2022
2.1.1 34,233 6/24/2022
2.0.0 54,424 11/18/2021
2.0.0-preview3 338 9/11/2021
2.0.0-preview2 265 9/9/2021
2.0.0-preview1 246 9/9/2021
1.1.0 246,287 8/11/2019
1.0.1 6,342 2/27/2018
1.0.0 969 2/24/2018
1.0.0-beta-2 912 1/2/2018
1.0.0-beta-1 872 12/29/2017