21 packages returned for Tags:"robots.txt"

Search Engine management tools. This library helps developers to create routing for robots.txt and sitemap.xml. Also provides facility for creating sitemap.xml file and load existing sitemap.xml file.

A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards. See https://bitbucket.org/cagdas/robotstxt for usage examples.

An Umbraco package to edit your robots.txt file

SimpleSitemap is a lite library that helps you create web sitemaps for collections or lists of items. These sitemaps follow the Sitemap Protocol. Both sitemapindex and urlset links are generated, based upon... More information

Allows to serve a robots instruction file.

Small utility, used to check robots.txt on web app and make sure that it is valid. RobotsTxt check is based on https://www.nuget.org/packages/RobotsTxt, but this project is no longer maintained, so I copied it... More information

A Simple and Powerful Library to Deal with Web Robots Control Strategy.

A simple middleware built on reflection designed to support your Search Engine Optimized (SEO) app by dynamically creating a Sitemap.xml and Robots.txt. This package is designed to be very simple for simple use... More information

A tool for accessing and querying the contents of a robots.txt file.

A tool for loading and reading XmlSitemaps.

- It adds a robots.txt and a robots_closed.txt to the root of the website - It adds a rewrite rule to the web.config that rewrites the robots.txt to robots_closed.txt for all urls ending with tamtam.nl

This client library enables working with Robots.txt. Key Features: - Parse robots.txt into Typed object. - Lookup Allowed/Disallowed/Crawldelay based on User-Agent. - Traverse... More information

Louw.SitemapParser

Robots.txt parser and generator which I'm using on www.sunamo.cz and other sites.

Automatic generation of sitemap.xml and robots.txt based on ASP.NET routes. Extensible engine allows custom URL providers to get URLs from any source. Includes exclude from sitemap feature and video sitemap... More information

A simple lightweight .Net solution for websites which need easy management of multiple robots.txt

SeoTags create all SEO tags you need such as meta, link, twitter card (twitter:), open graph (og:), and JSON-LD schema (structred data).

Parsers for Robots Exclusion Standard (aka robots.txt), Robots Meta Tag, and X-Robots-Tag. Visit Project Site for documentation.

Parses robots.txt files and sitemaps. Supports the proposed RFC9309 Robots Exclusion Protocol standard.

https://github.com/nicholasbergesen/RobotsParser/blob/master/README.md