robots.txt is not updating in production build

[raufabr]

Hello, I have this robots.txt that I haven't modified from the current version of hydrogen storefront codebase in my robots.txt.

import {type LoaderArgs} from '@shopify/remix-oxygen';

export const loader = ({request}: LoaderArgs) => {
  const url = new URL(request.url);

  return new Response(robotsTxtData({url: url.origin}), {
    status: 200,
    headers: {
      'content-type': 'text/plain',
      // Cache for 24 hours
      'cache-control': `max-age=${60 * 60 * 24}`,
    },
  });
};

function robotsTxtData({url}: {url: string}) {
  const sitemapUrl = url ? `${url}/sitemap.xml` : undefined;

  return `
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /account
${sitemapUrl ? `Sitemap: ${sitemapUrl}` : ''}

# Google adsbot ignores robots.txt unless specifically named!
User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders

User-agent: Pint
<img width="920" alt="Screenshot 2023-09-05 at 19 35 07" src="https://github.com/Shopify/hydrogen/assets/30205551/bd29820a-7058-4fc8-ac06-e0f380408995">
erest
Crawl-delay: 1
`.trim();
}

When I run on localhost and go to localhost:3000/robots.txt I can see the file properly.

However, when I go to the site's Hydrogen storefront URL, I'm getting this.

I'm quite new to SEO and this is an important functionality of our site, so I wanted to check how we could fix this?

[juanpprieto]

It could be due to caching? We cache this page for 24h by default, so you won't see the changes until the next caching cycle. Is this the case?

export const loader = ({request}: LoaderArgs) => {
  const url = new URL(request.url);

  return new Response(robotsTxtData({url: url.origin}), {
    status: 200,
    headers: {
      'content-type': 'text/plain',
      // Cache for 24 hours.  <-------------------------- 
      'cache-control': `max-age=${60 * 60 * 24}`,
    },
  });
};

[raufabr]

Potentially. I've changed it to 1 second but it hasn't updated - do I just have to wait 24 hours to see if it updates?

[raufabr]

I've waited about 36 hours and it hasn't updated still. It works in other environments but not on the production one.

I've also tried to create a new environment and make it the production one but that didn't help. Is there any other way to debug this further?

[raufabr]

The solution was to simply was to switch the Hydrogen site. Oxygen by default prevents crawling of the site when in test mode.

[davidwilliford99]

Here is some documentation: "https://shopify.dev/docs/storefronts/headless/hydrogen/seo#robots-txt"

Shopify states:
"If you deploy to Oxygen, then your robots.txt file is only served in your public production environment, and only if you have a custom domain set.

If you make a non-production deployment accessible with a shareable link or an auth bypass token, then Oxygen overrides the deployment's robots.txt file with a disallow rule for all bots and crawlers. This prevents exposing content prematurely, as well as potential SEO harm from duplicated content."