0.5.2

Changes

🧰 Maintenance

• Bump Robots Exclusion Tools to 0.9.1, Sitemap Tools to 0.7.2 by @Turnerj (#147)
• Bump Microsoft.NET.Test.Sdk from 17.2.0 to 17.3.0 by @dependabot (#146)
• Bump Microsoft.NET.Test.Sdk from 17.1.0 to 17.2.0 by @dependabot (#142)
• Bump HtmlAgilityPack from 1.11.42 to 1.11.43 by @dependabot (#143)
• Bump MSTest.TestAdapter from 2.2.8 to 2.2.10 by @dependabot (#141)
• Bump MSTest.TestFramework from 2.2.8 to 2.2.10 by @dependabot (#140)
• Bump Microsoft.NET.Test.Sdk from 17.0.0 to 17.1.0 by @dependabot (#134)

👨🏼‍💻 Contributors

@Turnerj, @dependabot and @dependabot[bot]

0.5.1

Changes

Addresses a binary compatibility issues with mixed dependency versions by bumping SitemapTools to 0.7.1

0.5.0

Changes

🐛 Bug Fixes

• Prevent non-HTTP schemes being crawled by @Turnerj (#133)

🧰 Maintenance

• Bump HtmlAgilityPack from 1.11.30 to 1.11.42
• Bump Microsoft.Extensions.Logging.Abstractions from 3.1.12 to 3.1.22 (for .NET Standard 2.0) / 6.0.0 (for .NET 6.0)
• Bump TurnerSoftware.RobotsExclusionTools from 0.7.0 to 0.8.1
• Bump TurnerSoftware.SitemapTools from 0.6.0 to 0.7.0

0.4.0

Changes

• Increased the amount and the detail of debug logging for crawling

🐛 Bug Fixes

• Fix max pages allowed bug by @Turnerj (#68)

🧰 Maintenance

• Bump HtmlAgilityPack from 1.8.11 to 1.11.30
• Bump Microsoft.Extensions.Logging.Abstractions from 2.2.0 to 3.1.12
• Bump TurnerSoftware.RobotsExclusionTools from 0.3.0 to 0.7.0
• Bump TurnerSoftware.SitemapTools from 0.3.0 to 0.6.0

0.3.0

Improved support for in-page Robots configuration (#7)
Added fragment support for crawl data (#7)
Added support for request timeouts (#7)
Added support for max pages to crawl (#7)
Obey no-follow on anchors (#7)
Various other minor tweaks and fixes (#7)

0.2.0

Replaced crawl contents stream with a string, allowing easier access to contents.

0.1.0

Initial release with core functionality