doc: controller web crawlers article

.playground/server/plugins/robots.ts

@@ -1,4 +1,4 @@
-import { defineNitroPlugin } from 'nitropack/runtime/plugin'
+import { defineNitroPlugin } from '#imports'
 
 export default defineNitroPlugin((nitroApp) => {
   if (import.meta.dev) {

.playground/server/tsconfig.json

@@ -0,0 +1,3 @@
+{
+  "extends": "../.nuxt/tsconfig.server.json"
+}

docs/content/1.getting-started/0.introduction.md

@@ -13,7 +13,10 @@ The core feature of the module is:
 - Telling [crawlers](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) which paths they can and cannot access using a [robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) file.
 - Telling [search engine crawlers](https://developers.google.com/search/docs/crawling-indexing/googlebot) what they can show in search results from your site using a `<meta name="robots" content="index">`{lang="html"} `X-Robots-Tag` HTTP header.
 
-It's important to [learn the difference between](/docs/robots/guides/robots-txt-vs-meta-tag) the two to avoid common SEO pitfalls.
+New to robots or SEO? Check out the [Conquering Web Crawlers](/learn/controlling-crawlers) guide to learn more about why you might
+need these features.
+
+:LearnLabel{label="Conquering Web Crawlers" to="/learn/controlling-crawlers" icon="i-ph-robot-duotone"}
 
 While it's simple to create your own robots.txt file, the module makes sure your non-production environments get disabled from indexing. This is important to avoid duplicate content issues and to avoid search engines serving your development or staging content to users.
 

docs/content/1.getting-started/1.installation.md

@@ -43,7 +43,9 @@ any routes which require authentication should be ignored.
 - [Disabling Site Indexing](/docs/robots/guides/disable-indexing) - If you have non-production environments you should disable indexing for these environments,
 while this works out-of-the-box for most providers, it's good to verify this is working as expected.
 
-Make sure you understand the differences between [robots.txt vs robots meta tag](/docs/robots/guides/robots-txt-vs-meta-tag).
+Make sure you understand the differences between robots.txt vs robots meta tag with the [Conquering Web Crawlers](/learn/conquering-crawlers) guide.
+
+:LearnLabel{label="Conquering Web Crawlers" to="/learn/controlling-crawlers" icon="i-ph-robot-duotone"}
 
 ## Next Steps
 

docs/content/2.guides/1.disable-page-indexing.md

@@ -11,7 +11,9 @@ The best options to choose are either:
 - [Robots.txt](#robotstxt) - Great for blocking robots from accessing specific pages that haven't been indexed yet.
 - [useRobotsRule](#userobotsrule) - Controls the `<meta name="robots" content="...">` meta tag and `X-Robots-Tag` HTTP Header. Useful for dynamic pages where you may not know if it should be indexed at build time and when you need to remove pages from search results. For example, a user profile page that should only be indexed if the user has made their profile public.
 
-If you're still unsure about which option to choose, see the [Robots.txt vs Robots Meta Tag](/docs/robots/guides/robots-txt-vs-meta-tag) guide.
+If you're still unsure about which option to choose, make sure you read the [Conquering Web Crawlers](/learn/conquering-crawlers) guide.
+
+:LearnLabel{label="Conquering Web Crawlers" to="/learn/controlling-crawlers" icon="i-ph-robot-duotone"}
 
 [Route Rules](#route-rules) and [Nuxt Config](#nuxt-config) are also available for more complex scenarios.
 

docs/content/3.nitro-api/0.get-path-robot-config.md

@@ -28,7 +28,6 @@ interface GetPathRobotResult {
 }
 ```
 
-
 ### Arguments
 
 - `e: H3Event`{lang="ts"}: The request event object, used to determine the current path.
@@ -46,7 +45,7 @@ interface GetPathRobotResult {
 ## Example
 
 ```ts twoslash [server/plugins/strip-og-tags-maybe.ts]
-import { getPathRobotConfig, defineNitroPlugin } from '#imports'
+import { defineNitroPlugin, getPathRobotConfig } from '#imports'
 
 export default defineNitroPlugin((nitroApp) => {
   // strip og meta tags if the site is not indexable