RobotsTxtCore 2.0.0-preview2
See the version list below for details.
dotnet add package RobotsTxtCore --version 2.0.0-preview2
NuGet\Install-Package RobotsTxtCore -Version 2.0.0-preview2
<PackageReference Include="RobotsTxtCore" Version="2.0.0-preview2" />
<PackageVersion Include="RobotsTxtCore" Version="2.0.0-preview2" />
<PackageReference Include="RobotsTxtCore" />
paket add RobotsTxtCore --version 2.0.0-preview2
#r "nuget: RobotsTxtCore, 2.0.0-preview2"
#addin nuget:?package=RobotsTxtCore&version=2.0.0-preview2&prerelease
#tool nuget:?package=RobotsTxtCore&version=2.0.0-preview2&prerelease
RobotsTxtMiddleware
From version 2.0 I've dropped support for .NET Core versions prior to 3.1 and made a huge breaking change by adding a
IRobotsTxtProvider
interface seperate from the middleware to handle configuration.
A Robots.txt middleware for ASP.NET Core. Why is this needed you ask? Because if you need to add dynamic values (such as a configured url from your CMS) you'll need some sort of code to handle that, and this makes it easy.
Installation
NuGet
PM> Install-Package RobotsTxtCore
.Net CLI
> dotnet add package RobotsTxtCore
Usage
To specify multiple rules with the fluent interface makes it really easy.
public void ConfigureServices(IServiceCollection services) {
services.AddStaticRobotsTxt(builder =>
builder
.AddSection(section =>
section
.AddComment("Allow Googlebot")
.AddUserAgent("Googlebot")
.Allow("/")
)
.AddSection(section =>
section
.AddComment("Disallow the rest")
.AddUserAgent("*")
.AddCrawlDelay(TimeSpan.FromSeconds(10))
.Disallow("/")
)
.AddSitemap("https://example.com/sitemap.xml")
);
}
public void Configure(IApplicationBuilder app) {
app.UseRobotsTxt();
}
Output
# Allow Googlebot
User-agent: Googlebot
Allow: /
# Disallow the rest
User-agent: *
Disallow: /
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml
Or if you just want to deny everyone.
public void ConfigureServices(IServiceCollection services) {
services.AddStaticRobotsTxt(builder =>
builder
.DenyAll()
);
}
public void Configure(IApplicationBuilder app) {
app.UseRobotsTxt();
}
Output
User-agent: *
Disallow: /
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 is compatible. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. |
.NET Core | netcoreapp3.1 is compatible. |
-
.NETCoreApp 3.1
- Microsoft.CodeAnalysis.CSharp (>= 3.11.0)
-
net5.0
- Microsoft.CodeAnalysis.CSharp (>= 3.11.0)
-
net6.0
- Microsoft.CodeAnalysis.CSharp (>= 3.11.0)
NuGet packages (2)
Showing the top 2 NuGet packages that depend on RobotsTxtCore:
Package | Downloads |
---|---|
Apprio.Enablement.Web.Api
Package Description |
|
Kros.CqrsTemplate
Package Description |
GitHub repositories (1)
Showing the top 1 popular GitHub repositories that depend on RobotsTxtCore:
Repository | Stars |
---|---|
Deali-Axy/StarBlog
☀StarBlog 是一个基于 .NET Core 开发的现代博客系统,支持 Markdown 文章导入,遵循 RESTful 接口规范。前端基于 Vue + ElementUI 开发,可作为 .NET Core 入门学习项目,同时配套了一系列开发笔记,记录了从零开始构建这个博客系统的全过程,可以帮助学习理解 .Net Core 项目的开发流程。
|
Version | Downloads | Last updated |
---|---|---|
3.1.0-preview1 | 8,975 | 10/16/2023 |
3.0.0 | 162,359 | 10/1/2023 |
2.2.0 | 108,270 | 12/24/2022 |
2.1.1 | 47,067 | 6/24/2022 |
2.0.0 | 63,684 | 11/18/2021 |
2.0.0-preview3 | 405 | 9/11/2021 |
2.0.0-preview2 | 352 | 9/9/2021 |
2.0.0-preview1 | 316 | 9/9/2021 |
1.1.0 | 275,935 | 8/11/2019 |
1.0.1 | 6,805 | 2/27/2018 |
1.0.0 | 1,231 | 2/24/2018 |
1.0.0-beta-2 | 1,158 | 1/2/2018 |
1.0.0-beta-1 | 1,112 | 12/29/2017 |