gatsby-plugin-sitemap always generates sitemap-index even on low page count

[ichik]

Description

According to documentation gatsby-plugin-sitemap should only generate sitemap-index when having more entries than entryLimit option

entryLimit (number = 45000) Number of entries per sitemap file, a sitemap index and multiple sitemaps are created if you have more entries.

With recent updates (4.0) this is not the case anymore. sitemap-index.xml is always generated, which is quite weird for small amount of entries. It is a different problem than the one being solved currently in #31167, but it existence seems related to recent activity.

Steps to reproduce

Branch with reproducible case: https://github.com/ichik/ichik.xyz/tree/sitemap-bug

Expected result

Link to sitemap.xml generated

Actual result

Link to sitemap-index.xml generated

Environment

  System:
    OS: macOS 11.3.1
    CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.16.1 - ~/.nvm/versions/node/v14.16.1/bin/node
    npm: 6.14.13 - ~/.nvm/versions/node/v14.16.1/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 90.0.4430.93
    Firefox: 88.0
    Safari: 14.1
  npmPackages:
    gatsby: ^3.4.1 => 3.4.1 
    gatsby-plugin-humans-txt: ^1.1.4 => 1.1.4 
    gatsby-plugin-image: ^1.4.0 => 1.4.0 
    gatsby-plugin-manifest: ^3.4.0 => 3.4.0 
    gatsby-plugin-mdx: ^2.4.0 => 2.4.0 
    gatsby-plugin-netlify: ^3.4.0 => 3.4.0 
    gatsby-plugin-offline: ^4.4.0 => 4.4.0 
    gatsby-plugin-react-helmet: ^4.4.0 => 4.4.0 
    gatsby-plugin-robots-txt: ^1.5.6 => 1.5.6 
    gatsby-plugin-sharp: ^3.4.1 => 3.4.1 
    gatsby-plugin-sitemap: ^4.0.0 => 4.0.0 
    gatsby-plugin-styled-components: ^4.4.0 => 4.4.0 
    gatsby-plugin-typescript: ^3.4.0 => 3.4.0 
    gatsby-remark-images: ^5.1.0 => 5.1.0 
    gatsby-remark-unwrap-images: ^1.0.2 => 1.0.2 
    gatsby-source-filesystem: ^3.4.0 => 3.4.0 
    gatsby-transformer-sharp: ^3.4.0 => 3.4.0 

[LekoArts]

I don't know if this was done on purpose @moonmeister?

We now use simpleSitemapAndIndex:

gatsby/packages/gatsby-plugin-sitemap/src/gatsby-node.js

Lines 88 to 95 in ea81d3b

	return simpleSitemapAndIndex({
	hostname: siteUrl,
	destinationDir: sitemapPath,
	sourceData: serializedPages,
	limit: entryLimit,
	gzip: false,
	})
	}

According to https://github.com/ekalinin/sitemap.js/blob/4141469ca90e53f9f15b5107a25d16591175984f/README.md#create-sitemap-and-index-files-from-one-large-list:

If you know you are definitely going to have more than 50,000 urls in your sitemap, you can use this slightly more complex interface to create a new sitemap every 45,000 entries and add that file to a sitemap index.

So the behavior changed and thus it's currently working as intended but the docs then might be outdated if entryLimit is only affecting the splitting and not the creation of an index itself.

[moonmeister]

Yeah the index file always exists. Good catch, a PR would be much appreciated to clarify this language.

[ichik]

I think that it's somewhat of an overkill to always generate index in case you might have over 50,000 entries. For a lot of gatsby projects out there previous behavior which resulted in a single sitemap.xml looked way more sensible.

[moonmeister]

Yeah, It may be a little "cleaner", but at the end of the day I couldn't think why it really mattered. The bots don't care.

The reason it changed was because the default behavior of the library we use changed. I wasn't going to spend hours rewriting code to fix it.

If this is important to you, feel free to do the work and submit a PR.

[abheist]

One more bug though: generated index file is containing wrong URLs to individual files:
https://abheist.com/sitemap/sitemap-index.xml:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap>
        <loc>https://abheist.com/sitemap-0.xml</loc>
    </sitemap>
</sitemapindex>

loc link should be: https://abheist.com/sitemap/sitemap-0.xml, missing a sitemap directory.

[moonmeister]

This has been reported and fixed in the next release.

[abheist]

This has been reported and fixed in the next release.

Perfect, thanks!

[SomiDivian]

What's the news?

[prichey]

Related to this, the sitemap-index.xml file also is also mangling my siteUrl. In this example, it's set to https://mysite.com/foo.

sitemap-index.xml:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://mysite.com/sitemap-0.xml</loc>
</sitemap>
</sitemapindex>

The 4. release seems to have lots of issues so I'm going to try downgrading until these are worked out

[moonmeister]

@prichey Please open a separate issue for this and a reproduction or more detailed example of your configuration. and what is happening vs what you expect to happen.

[prichey]

@moonmeister #32080

[justsilencia]

Yeah, It may be a little "cleaner", but at the end of the day I couldn't think why it really mattered. The bots don't care.

The reason it changed was because the default behavior of the library we use changed. I wasn't going to spend hours rewriting code to fix it.

If this is important to you, feel free to do the work and submit a PR.

You mention that "the bots don't care." However, after doing some research, it seems that having the sitemap in a subfolder will only allow google to scan pages in that subfolder. Here's a reference to this:

Stackoverflow answer with link to google docs

Could you please explain if this is incorrect?

[Pandanaax]

I have multi domains and work with Gatsby 2, but since I move for Gatsby 4 my sitemap is like this :

https://inte.****.fr/https://inte.****.it/intimo-premaman
But i need only .it for work for this exemple.

Could you help me please.

[moonmeister]

I have multi domains and work with Gatsby 2, but since I move for Gatsby 4 my sitemap is like this :

https://inte.****.fr/https://inte.****.it/intimo-premaman
But i need only .it for work for this exemple.

Could you help me please.

Hi @Pandanaax !

Sorry to hear you're running into an issue. To help us best begin debugging the underlying cause, it is incredibly helpful if you open a new issue. Please include a minimal reproduction and a copy of any relevant code/config or a link to the repository.

Thanks for using Gatsby! 💜