Verndale.Sitemap.Robots.Generator 5.0.0.19

------ Sitemap and Robots dynamic generator Readme -----------

!!!!!!!! RELEASE NOTES 5.0.0.19 !!!!!!!!

Version 5.0.0.19 for Optimizely/Episerver CMS 12

Migrate plugin to dot net core 8

Added options to update sitemap when content is published using Update_Sitemap_On_Published property and language namespace override using Language_Namespace_Override property

"Siro": { "Update_Sitemap_On_Published": "true", "Language_Namespace_Override": "http://www.w3.org/1999/xhtml", ... }

!!!!!!!! RELEASE NOTES 5.0.0.18 !!!!!!!!

Version 5.0.0.18 for Optimizely/Episerver CMS 12

Filter not published, archived and deleted pages from sitemap

!!!!!!!! RELEASE NOTES 5.0.0.17 !!!!!!!!

Version 5.0.0.17 for Optimizely/Episerver CMS 12

Added option to override default href lang values using SIRO configuration, Add Lang option

"Siro": { "Show_Localized_URLS": "true", ... "Lang": { "en": "en-US" } }

!!!!!!!! RELEASE NOTES 5.0.0.16 !!!!!!!!

Version 5.0.0.16 for Optimizely/Episerver CMS 12

Remove disallow /episerver line from robots.txt

!!!!!!!! RELEASE NOTES 5.0.0.15 !!!!!!!!

Version 5.0.0.15 for Optimizely/Episerver CMS 12

Improve UI and remove not used search engines option

!!!!!!!! RELEASE NOTES 5.0.0.13-14 !!!!!!!!

Version 5.0.0.13-14 for Optimizely/Episerver CMS 12

Detect properties like ExcludeFromSiteMap and ExcludeFromSitemap in base classes

!!!!!!!! RELEASE NOTES 5.0.0.12 !!!!!!!!

Version 5.0.0.12 for Optimizely/Episerver CMS 12

Fix issue with big xml files decompression

!!!!!!!! RELEASE NOTES 5.0.0.11 !!!!!!!!

Version 5.0.0.11 for Optimizely/Episerver CMS 12

Fix style issues with newer versions of CMS 12

!!!!!!!! RELEASE NOTES 5.0.0.10 !!!!!!!!

Version 5.0.0.10 for Optimizely/Episerver CMS 12

Added dependencies required for the package: EPiServer.CMS.AspNetCore and EPiServer.Framework.AspNetCore

!!!!!!!! RELEASE NOTES 5.0.0.9 !!!!!!!!

Version 5.0.0.9 for Optimizely/Episerver CMS 12

Fix issue with page url resolver which is returning already the primary domain, in CMS 11, url resolver returned just the relative path

!!!!!!!! RELEASE NOTES 5.0.0.8 !!!!!!!!

Version 5.0.0.8 for Optimizely/Episerver CMS 12

After installation add the following line to the Startup.cs class or where you initialize your site features

public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { app.UseSitemapRobotsMiddleware(); // Line to add }

After installation please add this configuration to your appsettings.json file:

"Siro": { "Show_Localized_URLS": "true", "Display_Localized_URLS_Separated": "false", "Show_Trailing_Slash": "false", "SiteMap_OutPutRobots_txt": "1" }

File sitemap.config has been removed, namespaces for sitemap generation now are burned in code. These namespaces were not changed at all anyway so it should not affect the normal generation of your sitemap.

Fix to properly use sitemap and robots handler in IIS express and Kestrel servers.

!!!!!!!! RELEASE NOTES 1.0.0.8 !!!!!!!!

Fixed an issue with the robots data which was replaced everytime the sitemap job was executed. Now it only updates the field if is empty.

An interface called ISitemapFilter can be added to your project to filter IContent elements using custom logic. There is an example at the end of this file.

!!!!!!!! RELEASE NOTES 1.0.0.7 !!!!!!!!

Fix for restricted types when using newer versions of Episerver CMS.

Quick Note on using the Add-On with Active Directory Membership Provider: A virtual role named SIRO has been set for accessing the add-on. So your new virtual role in Web.config should look like this:

<add name="SIRO" type="EPiServer.Security.MappedRole, EPiServer.Framework" roles="[AD_ROLES]" mode="Any" />

[AD_ROLES] should be replaced by a comma-separated list of the AD Roles you want to authorize for the SIRO Add-On.

!!!!!!!! RELEASE NOTES 1.0.0.6 !!!!!!!!

Improve code that generates the sitemap in the http handler.

Fix issue with href lang. Items were generated using link instead of xhtml:link

Added different namespace for localized items. Replace xmlNsLangTpl namespace in the sitemap.config file from http://www.w3.org/1999/xhtml to this http://www.w3.org/TR/xhtml11/xhtml11_schema.html

!!!!!!!! RELEASE NOTES 1.0.0.5 !!!!!!!!

Added two selection factories Sitemap Frequency and Sitemap Priority which will avoid editors to set random values (This selection factories are optional but recommended).

SiteMap_MAX_URLS app config setting no longer needed.

An interface called ISitemapExtend can be added to your project to add other IContent elements which are not pages, like catalog items, add images to those items and also modify how we get the change frequency and priority properties in case they are not being set in the page. There is an example at the end of this file.

Removed sitemap gadget, it was having issues with the new version, and it is not adding value to the plugin.

!!!!!!!! BREAKING CHANGES FROM VERSION 1.0.0.4 OR GREATER !!!!!!!!

Sitemap Administration Plugin (Configuration section in admin) has been renamed to Sitemap and Robots Manager.

Robots content is set by site inside the Sitemap and Robots Manager admin configuration Sites tab, field in start page is not longer valid.

Sitemaps are not generated as files anymore, instead we saved to Episerver directly. This avoids to have issues in balanced environments where the files cannot be synchronized.

!!!!!!!! IMPORTANT !!!!!!!!

Go to Admin --> Config --> Sitemap and Robots Manager and click on the "Update restricted types" button. This creates the necessary tables that the module uses.

Every inherit field must have their respective Display and/or UIHint (Optional) attributes in order to work correctly on EPiServer. For instance:

-- Common Page --

[Display(GroupName = "Navigation", Name = "Exclude from Xml Site Map", Order = 400)] public virtual bool ExcludeFromSiteMap { get; set; }

[SelectOne(SelectionFactoryType = typeof(SitemapPrioritySelectionFactory))] [Display(GroupName = "Navigation", Name = "Priority in Site Map", Order = 500)] public virtual string Priority { get; set; }

[SelectOne(SelectionFactoryType = typeof(SitemapFrequencySelectionFactory))] [Display(GroupName = "Navigation", Name = "Change Frequency of Page in Site Map", Order = 600)] public virtual string ChangeFrequency { get; set; }

These fields allow the editor to decide if a page can be individually included in the sitemap even when the page type is not restricted.

-- Global Asax --

If you are using web api in your project, please add this lines in Global.asax.cs

protected override void RegisterRoutes(System.Web.Routing.RouteCollection routes) { base.RegisterRoutes(routes); routes.Ignore("sitemap.xml"); routes.Ignore("robots.txt"); }

-- FEATURES Sitemap--

• Multi site
• Multi language (https://support.google.com/webmasters/answer/2620865?hl=en)
• Filter by page (Attribute ExcludeFromSiteMap)
• Sort by priority and frequency (Attributes Priority and ChangeFrequency)
• Filter by page type
• Filter by site
• Notification to search engines in production environment
• Automatic update of robots.txt file
• Small tweaks configurations inside web.config and sitemap.config files such as: ** Output to robots.txt file (Default true:1) - web.config ** Show a trailing slash at the end of every link (Default false) - web.config ** Show localized urls using lang href attributes with only one loc tag (Default true) - web.config ** Display localized urls separated, it will not use lang href attributes, instead it will show them as separate loc tags (Default false) - web.config ** Sitemap file required namespaces - sitemap.config ** Production environment (Default false) - sitemap.config
• Admin plugin which allows to configure sites, page types and search engines
• Admin job to generate sitemaps using the admin sitemap plugin
• Web editor gadget which generates sitemaps with restricted sites and page types
• Http handler to intercept calls to sitemap.xml per domain/site

-- FEATURES Robots --

• Multi site
• Customizable string field to edit robots.txt content inside Sitemap and Robots Manager admin configurations
• Http handler to intercept calls to robots.txt file per domain/site

!!!!!!!! EXTENDER EXAMPLE !!!!!!!!

-- SET EXTENDER IN A STRUCTUREMAP INITILIZATION MODULE AND REPLACE THE DEFAULT ONE -- if (context.Services.Contains(typeof(ISitemapExtend))) { context.Services.RemoveAll(typeof(ISitemapExtend)); context.Services.AddTransient<ISitemapExtend, CommerceSitemapExtend>(); }

-- EXTENDER CLASS --

public class CommerceSitemapExtend : ISitemapExtend { private readonly ReferenceConverter _referenceConverter; private readonly IContentRepository _contentRepository; private readonly IContentLoader _contentLoader;

public CommerceSitemapExtend(ReferenceConverter referenceConverter,
    IContentRepository contentRepository,
    IContentLoader contentLoader)
{
    _referenceConverter = referenceConverter;
    _contentRepository = contentRepository;
    _contentLoader = contentLoader;
}

// Adding commerce pages here
public List<IContent> AddOtherNotCmsPages()
{
    var rootContentReference = _referenceConverter.GetRootLink();
    IList<ContentReference> catalog = _contentRepository.GetDescendents(rootContentReference).ToList();

    var pages = new List<IContent>();
    pages.AddRange(catalog
        .Select(x => _contentRepository.Get<IContent>(x))
        .Where(x => !(x is VariationContent) && !(x is CatalogContent)));

    return pages;
}

// Adding images that belong to commerce pages
public void AddImagesForOtherNotCmsPages(IContent item, XmlDocument doc,
    XmlElement urlNode, SiteDefinition site)
{
    if (item is EntryContentBase entryContentBase)
    {
        var assets = entryContentBase.GetAssetLinks()?.ToList();

        if (assets != null)
        {
            foreach (var asset in assets)
            {
                XmlNode imgNode = doc.CreateElement("image", "image", SitemapManagerConfiguration.XmlNsImage);
                urlNode.AppendChild(imgNode);

                var imageFile = _contentLoader.Get<ImageFile>(asset);

                XmlNode imgLocNode = doc.CreateElement("image", "loc", SitemapManagerConfiguration.XmlNsImage);
                imgNode.AppendChild(imgLocNode);
                imgLocNode.AppendChild(doc.CreateTextNode(SitemapManager.GetItemUrl(asset, site)));

                XmlNode imgCaptionNode = doc.CreateElement("image", "caption", SitemapManagerConfiguration.XmlNsImage);
                imgNode.AppendChild(imgCaptionNode);
                imgCaptionNode.AppendChild(doc.CreateTextNode(imageFile.Caption));

                XmlNode imgTitleNode = doc.CreateElement("image", "title", SitemapManagerConfiguration.XmlNsImage);
                imgNode.AppendChild(imgTitleNode);
                imgTitleNode.AppendChild(doc.CreateTextNode(string.IsNullOrEmpty(imageFile.Title)
                    ? imageFile.Credits
                    : imageFile.Title));
            }
        }
    }
}

// Alternative when priority is not set
public void CreatePriorityPropertyDifferently(XmlDocument doc, XmlElement urlNode, string url)
{
    var priorityNode = doc.CreateElement("priority");
    urlNode.AppendChild(priorityNode);
    priorityNode.AppendChild(doc.CreateTextNode(GetPriority(url)));
}

// Alternative when change frequency is not set
public void CreateChangeFrequencyPropertyDifferently(XmlDocument doc, XmlElement urlNode)
{
    var chgNode = doc.CreateElement("changefreq");
    urlNode.AppendChild(chgNode);
    chgNode.AppendChild(doc.CreateTextNode("weekly"));
}

private static string GetPriority(string url)
{
    var depth = new Uri(url).Segments.Length - 1;

    return Math.Max(1.0 - (depth / 10.0), 0.5).ToString(CultureInfo.InvariantCulture);
}

}

!!!!!!!! FILTER EXAMPLE !!!!!!!!

-- SET FILTER IN A STRUCTUREMAP INITILIZATION MODULE AND REPLACE THE DEFAULT ONE -- if (context.Services.Contains(typeof(ISitemapFilter))) { context.Services.RemoveAll(typeof(ISitemapFilter)); context.Services.AddTransient<ISitemapFilter, CustomSitemapFilter>(); }

-- FILTER CLASS --

public class CustomSitemapFilter : ISitemapFilter { // Include item based on custom implementation public bool IncludeItem(IContent item) { if(item is BlockData){ return true; }

    return false;
}

}