Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sitemap.xml is not populating a site map #2103

Closed
bradleycamacho opened this issue May 3, 2021 · 13 comments · Fixed by newrelic/gatsby-theme-newrelic#383
Closed

Sitemap.xml is not populating a site map #2103

bradleycamacho opened this issue May 3, 2021 · 13 comments · Fixed by newrelic/gatsby-theme-newrelic#383
Assignees
@rudouglas
Labels
bug a bug or issue eng issues related to site functionality that requires engineering
Milestone
[DevEn] Sprint 3

Comments

@bradleycamacho
Copy link
Contributor

bradleycamacho commented May 3, 2021 •
edited by roadlittledawn
Loading

Description

GTS uses a search tool to allow our support engineers to search multiple content sources at once. The tool was able to index Product Docs using the sitemap here: https://docs.newrelic.com/sitemap.xml/ but that appears to be gone now.

We likely need to update https://www.gatsbyjs.com/plugins/gatsby-plugin-sitemap/ . Complicating things, it seems like there are some issues with Gatsby sitemaps in general: gatsbyjs/gatsby#31095

Expected

It seems like our current config for the sitemap is broken and doesn't generate the sitemap at https://docs.newrelic.com/sitemap/sitemap-index.xml

Eng work

  • add gatsby-plugin-sitemap gatsby-plugin-sitemap output option to theme gatsby-config.js to explicitly set file path.
  • ensure the <link rel="sitemap" /> in <head> has the correct href for the path we set.
  • ensure / update swiftype UI and google search console have correct sitemap path
  • bump all sites to use theme version this is deployed in
@bradleycamacho bradleycamacho added bug a bug or issue eng issues related to site functionality that requires engineering labels May 3, 2021
@jpvajda jpvajda added the P2 label May 3, 2021
@jpvajda jpvajda added this to the [DevEn] Sprint 3 milestone May 3, 2021
@austin-schaefer
Copy link
Contributor

Seems like a potentially significant issue for two reasons:

  • Potentially blocking another team
  • sitemap is a key part of how search engines crawl sites and prioritize content in search, so this may be impacting SEO

@jpvajda jpvajda changed the title Sitemap.xml is not populating Sitemap.xml is not populating a site map May 4, 2021
@jpvajda
Copy link

jpvajda commented May 4, 2021

@austin-schaefer would you say this a P1? There is no work around I can thinking of..

@rudouglas rudouglas self-assigned this May 4, 2021
@rudouglas rudouglas added the sp:2 label May 4, 2021
@rudouglas
Copy link
Contributor

rudouglas commented May 4, 2021 •
edited
Loading

@bradleycamacho Reading through the issue it looks like they changed the default path that they create the sitemap at, I was able to find it here:

https://docs.newrelic.com/sitemap/sitemap-0.xml

The site map index doesn't include the sub-folder that the sitemap files are placed in currently, but there is a PR open to fix that issue:

fix(gatsby-plugin-sitemap): Sitemap path bug #31184

If you are running into issues affected by the above, there seems to be a workaround by setting the output config to /./ referenced here

@roadlittledawn
Copy link
Contributor

Oh weird. I tried that path yesterday but didn't see it. Must've typo'ed it. Nice catch, @rudouglas

This was referenced May 4, 2021
Closed
Merged
@roadlittledawn
Copy link
Contributor

for time being, i updated swiftype UI to use that sitemap path

@bradleycamacho
Copy link
Contributor Author

Oh wow, thanks @rudouglas! I'll share that with our stakeholders.

I will keep this issue open as I believe we still need to fix a few of those remaining issues @rudouglas linked to.

@rudouglas
Copy link
Contributor

Ye @roadlittledawn it seems to throw a 404 when you have a trailing slash 🤷

@rudouglas
Copy link
Contributor

rudouglas commented May 5, 2021 •
edited
Loading

Testing this on Developer site for build times 😛:

It looks like setting output: '/' in the gatsby-config results in the index href being //sitemap-index.xml which does not redirect correctly to the sitemap-0.xml. In addition the actual sitemap is still at

  • localhost:9005/sitemap/sitemap-0.xml

But if we set something specific like output: 'public' it will result in href="/public/sitemap-index.xml", which will redirect correctly to the sitemap-index page, and the sitemap will then be at localhost:9005/sitemap-0.xml which is whats expected i think?

Testing this on the docs site to confirm

@jpvajda
Copy link

jpvajda commented May 5, 2021

@rudouglas @roadlittledawn let me know if we want to ask Gatsby for any insights to this issue. We can drop this in the shared slack room if you need pointers.

@rudouglas
Copy link
Contributor

Ye @jpvajda that might be the best way to go, it works differently on the docs site than it does on the developer site, which may also be down to versioning, but might save time to ask them to weigh in?

@roadlittledawn
Copy link
Contributor

@rudouglas I updated the theme's demo site config to explicitly set output: '/'. i ran yarn workspace demo clean to remove previous build files (i assume that's maybe why you saw the sitemap/sitemap-0.xml as well when you built developer site). it looks like it outputs two files:

  • sitemap-0.xml which has the expected list of pages
  • sitemap-index.xml which looks like an index pointing to available sitemap files. I assume search engines would know to follow that (we can test that when in production to see if google sees it)

It also looked like the <link rel="sitemap" /> in the <head> still pointed to //sitemap-index.xml.

I updated the package to 4.1.0-next.1 and it fixed that double slash. 🤷 Created this PR if you want to take a look.

@jpvajda
Copy link

jpvajda commented May 6, 2021

@roadlittledawn it sounds like we may be good on this issue then? 🥰

@rudouglas
Copy link
Contributor

@roadlittledawn ok ye sitemap-index.xml is usually used when you have multiple sitemaps, so there's one place for search engines to hit, I ran this locally as well in the theme demo and it does get rid of the double slash 😄 not sure why it didn't work in the developer site 😞

@LekoArts LekoArts mentioned this issue May 7, 2021
Merged
@zstix zstix mentioned this issue May 7, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rudouglas rudouglas

Labels
bug a bug or issue eng issues related to site functionality that requires engineering
Projects
None yet
Milestone
[DevEn] Sprint 3
Development

Successfully merging a pull request may close this issue.

Fix sitemap output newrelic/gatsby-theme-newrelic
5 participants
@jpvajda @roadlittledawn @rudouglas @bradleycamacho @austin-schaefer