RobotsTxtCore 2.0.0-preview2

This is a prerelease version of RobotsTxtCore.
There is a newer version of this package available.
See the version list below for details.
dotnet add package RobotsTxtCore --version 2.0.0-preview2
                    
NuGet\Install-Package RobotsTxtCore -Version 2.0.0-preview2
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="RobotsTxtCore" Version="2.0.0-preview2" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="RobotsTxtCore" Version="2.0.0-preview2" />
                    
Directory.Packages.props
<PackageReference Include="RobotsTxtCore" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add RobotsTxtCore --version 2.0.0-preview2
                    
#r "nuget: RobotsTxtCore, 2.0.0-preview2"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=RobotsTxtCore&version=2.0.0-preview2&prerelease
                    
Install RobotsTxtCore as a Cake Addin
#tool nuget:?package=RobotsTxtCore&version=2.0.0-preview2&prerelease
                    
Install RobotsTxtCore as a Cake Tool

RobotsTxtMiddleware

From version 2.0 I've dropped support for .NET Core versions prior to 3.1 and made a huge breaking change by adding a IRobotsTxtProvider interface seperate from the middleware to handle configuration.

A Robots.txt middleware for ASP.NET Core. Why is this needed you ask? Because if you need to add dynamic values (such as a configured url from your CMS) you'll need some sort of code to handle that, and this makes it easy.

Installation

NuGet

PM> Install-Package RobotsTxtCore

.Net CLI

> dotnet add package RobotsTxtCore

Usage

To specify multiple rules with the fluent interface makes it really easy.

public void ConfigureServices(IServiceCollection services) {
    services.AddStaticRobotsTxt(builder =>
        builder
            .AddSection(section =>
                section
                    .AddComment("Allow Googlebot")
                    .AddUserAgent("Googlebot")
                    .Allow("/")
                )
            .AddSection(section =>
                section
                    .AddComment("Disallow the rest")
                    .AddUserAgent("*")
                    .AddCrawlDelay(TimeSpan.FromSeconds(10))
                    .Disallow("/")
                )
            .AddSitemap("https://example.com/sitemap.xml")
    );
}

public void Configure(IApplicationBuilder app) {
    app.UseRobotsTxt();
}

Output

# Allow Googlebot
User-agent: Googlebot
Allow: /

# Disallow the rest
User-agent: *
Disallow: /
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Or if you just want to deny everyone.

public void ConfigureServices(IServiceCollection services) {
    services.AddStaticRobotsTxt(builder =>
        builder
            .DenyAll()
    );
}

public void Configure(IApplicationBuilder app) {
    app.UseRobotsTxt();
}

Output

User-agent: *
Disallow: /
Product Compatible and additional computed target framework versions.
.NET net5.0 is compatible.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed. 
.NET Core netcoreapp3.1 is compatible. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on RobotsTxtCore:

Package Downloads
Apprio.Enablement.Web.Api

Package Description

Kros.CqrsTemplate

Package Description

GitHub repositories (1)

Showing the top 1 popular GitHub repositories that depend on RobotsTxtCore:

Repository Stars
Deali-Axy/StarBlog
☀StarBlog 是一个基于 .NET Core 开发的现代博客系统,支持 Markdown 文章导入,遵循 RESTful 接口规范。前端基于 Vue + ElementUI 开发,可作为 .NET Core 入门学习项目,同时配套了一系列开发笔记,记录了从零开始构建这个博客系统的全过程,可以帮助学习理解 .Net Core 项目的开发流程。
Version Downloads Last updated
3.1.0-preview1 8,975 10/16/2023
3.0.0 162,359 10/1/2023
2.2.0 108,270 12/24/2022
2.1.1 47,067 6/24/2022
2.0.0 63,684 11/18/2021
2.0.0-preview3 405 9/11/2021
2.0.0-preview2 352 9/9/2021
2.0.0-preview1 316 9/9/2021
1.1.0 275,935 8/11/2019
1.0.1 6,805 2/27/2018
1.0.0 1,231 2/24/2018
1.0.0-beta-2 1,158 1/2/2018
1.0.0-beta-1 1,112 12/29/2017