Choose a new static-site generator

[paulmelnikow]

Gatsby isn't the best fit for us. I think these are the key pain points (though other maintainers may describe them better!)

1. It's laborious to upgrade. The monorepo publishing mechanism issues lots of empty version bumps. We've worked around theis.
2. The project has been slow to implement key features such as Configurable output folder gatsbyjs/gatsby#1878 (which we've worked around) and Support ESM usage with Gatsby gatsbyjs/gatsby#31599
3. The GraphQL functionality feels like bloat: a feature we don't need, which complicates our usage of GraphQL in the server. Probably could be worked around with a different repo organization.
4. Big dependency tree complicates installation in Windows (see Add High res favicons #7974 (comment))

From #7978 (comment):

Straying a bit off topic here, but do you have any thoughts on what a good replacement for Gatsby is? We were using next.js before and decided we didn't like that. I don't really know what the other options are in that space.

I think the ideal criteria are:

• Easy to programmatically generate static pages which contain dynamic React content
• Works well with React and TypeScript
• Well documented

For single-page apps, I use Create React App, though it doesn't generate static pages. I've been meaning to check out Vite, which feels like a fresh look at the problem, though I think it may have the same limitation in the sense that static site generation is not what it does.

As a starting point I would suggest we evaluate these:

• Astro
• Remix
• Docusaurus

There are a few other static site generators for React listed here: https://jamstack.org/generators/

And… I'm curious if other folks might have ideas of options to consider here! (cc @Daniel15)

[Daniel15]

I'm curious if other folks might have ideas of options to consider here!

Docusaurus is pretty great for mostly static sites - I'd recommend trying it out. It does let you embed React components too 😄

For single-page apps, I use Create React App, though it doesn't generate static pages

I don't think it's officially supported but you can use something like react-snap to generate static pages - that's what I do for the home page on https://dnstools.ws/.

[paulmelnikow]

Thanks for replying 😁

Docusaurus is pretty great for mostly static sites

Most of our content is generated at build time. So it's static from the perspective of the consumer, but not from the perspective of the Shields maintainer. Do you think Docusaurus is a good fit for that?

react-snap looks interesting too.

[chris48s]

I've not used it, but I have heard good things about Docusaurus. It is something of a mindset shift from where we are now but there is something tempting about framing and thinking about the frontpage of shields.io as documentation.

[chris48s]

One thing we could do to try out Docusaurus is have a look at it for https://contributing.shields.io/ as a first step. Refs #4139

[chris48s]

OK, so docusaurus is pretty cool and I've produced a proof-of-concept showing how we might use docusaurus for the main shields.io website:

• repo: https://github.com/chris48s/shields-docusaurus-poc/
• live version you can play with: https://chris48s.github.io/shields-docusaurus-poc/
• quick demo vid (I've moved things on a bit since I recorded this, but it'll do): https://user-images.githubusercontent.com/6025893/207102863-53e1b268-ee7a-490c-b01e-4385e5d790e3.mp4

The key concept here is:

In shields, the URL schema is the API. At the moment, we use the data in the badge service files to make this file called service-definitions.yml and then we use that service-definitions.yml to build a custom react frontend which helps users to construct URLs. service-definitions.yml fundamentally contains information like:

• the URL pattern of an endpoint
• query string params we can supply, which ones are required, etc
• example parameter values
• documentation or help to go along with the endpoint

If you squint at this the right way, the contents of service-definitions.yml actually looks a lot like an OpenAPI definition (this can be either JSON or yaml). After all, they're both basically describing a bunch of URL patterns a user might call and how to call them. There are lots of existing tools out there that can consume an OpenAPI definition and spit out a tool that help a user to construct an API call, provide code examples, etc. So.. if we can use the same data to spit out an OpenAPI definition instead of our custom service-definitions.yml then we can leverage one of those existing tools instead of writing and maintaining our own frontend app that already does most of what we want.

..and that's pretty much what I've done.

So as well as docusaurus I have also used https://github.com/cloud-annotations/docusaurus-openapi . This does 99% of what we need out of the box, but I did need to make a few modifications to it:

• APIs that return an image are not common. Docusaurus-openapi assumes your API returns XML, JSON or Text. I added a special-case to render SVG responses as an image.
• There are presets for code examples in a wide variety of languages, but Markdown/rSt/AsciiDoc/HTML were not on the list :) I needed to tweak their code for injecting custom code examples a bit.
• I wanted to remove the next/previous buttons - they don't really make sense for us.

For the moment I've done this by defining a custom theme. Unfortunately, this requires duplicating a large chunk of the docusaurus-theme-openapi code beyond just the bits I wanted to customise. I don't really want to be relying on this as a long-term position. Assuming we want to move forward with this, one of my next jobs is to try and make some upstream contributions which make it easier to apply these customisations without having to hard-fork most of the theme. For the purposes of a proof-of-concept we can evaluate, the approach I've taken will do the job.

Obviously this is not complete, but I think I've tried out a good enough selection of examples to conclude that this is a viable way forward for us. You can get a rough idea of what I have/haven't thought of from the commit history on the repo https://github.com/chris48s/shields-docusaurus-poc/commits/main

I think the big upsides of this approach are:

• Obvious point, but what we have now is basically the worst website ever. Even the out-of-the-box docusaurus look and feel is a huge improvement (in terms of visual appearance, information hierarchy, accessibility, etc) over what we have now.
• I think basically everything I tried out in the proof-of-concept was at least possible if not a massive improvement over our current site.
• Assuming we can contribute my theme changes upstream to docusaurus-theme-openapi this approach is almost entirely declarative. We'd be looking at having close to zero react/frontend/typescript code in the shields repo. Most of the website content could be simple markdown/HTML with all of the 'badge builder' content being defined in the OpenAPI spec and all of the 'badge builder' logic and frontend essentially being outsourced to docusaurus-openapi. The closer we keep it to the defaults (and the defaults are pretty good IMO), the more true this is going to be. Docusaurus allows you to do a lot with markdown/front-matter and HTML.
• Massive reduction in frontend packages we need to manage.
• The combination of the above 2 would make this way easier for us to maintain with our current resources (the reality is that whatever solution we adopt probably needs to be maintained primarily by me).
• Weak/tangential benefit: Generating an OpenAPI spec for the purposes of making our own frontend would allow us to also publish an OpenAPI spec for consumers of the service.
• We've got light/dark mode out of the box
• I think this approach is going to work well enough for the static/dynamic/endpoint badges as well as the standard service badges
• Adding search was really easy and there are a variety of plugins available
• Presenting the badge builder as API documentation should be a pretty natural fit for a project aimed at software developers.

The possible downsides are:

• While it reduces the maintenance/custom dev overhead, the flipside of mostly outsourcing the badge builder rendering to docusaurus-openapi is that we lose some control. There may be some things that could be hard to customise. If the upstream goes in an undesirable direction for us, we may be stuck on an old version or looking for a new solution. In principle, having an OpenAPI spec does mean we could switch to a different tool, but the reality is the other solutions I looked at (e.g: https://github.com/rohit-gohri/redocusaurus/ , https://github.com/PaloAltoNetworks/docusaurus-openapi-docs , other non-docusaurus-based things ) were even less suited to our use-case (which is a bit esoteric and has required some customisations).

• Adopting any package comes with some risk of breaking changes, package becoming unmaintained etc. I think docusaurus itself is a pretty solid bet from this perspective. However, docusaurus-openapi is still unstable (0.x release) and isn't like a super widely used/adopted package (currently on 293 github stars, which isn't the be-all-and-end-all but as a very rough measure of popularity - this is not widely adopted). As such, it feels like this category of risks is more noteworthy than usual. In fact, one of the things I've done (splitting the API spec out into categories) warns WARNING: Loading a directory of OpenAPI definitions is experimental and subject to unannounced breaking changes. Having not attempted to contribute the customisations we'd need upstream yet, I don't know what that process will look like.

• All of that said, our current frontend code is stuck on some old versions of react, gatsby, etc which are proving hard to upgrade. Being [stuck on an old version of docusaurus-openapi with a website that looks good] might be a better bad place to end up in than [stuck on an old version of gatsby with a website that looks terrible] is.

• In OpenAPI, query string params can be optional but there is no such thing as an optional path (positional) parameter, so you can't do

  {
    "name": "branch",
    "in": "path",
    "required": false,  // nope
    "description": "Branch",
    "schema": { "type": "string" }
  }

  This means we need to generate a separate page for every with/without optional param variant. e.g:

  • https://chris48s.github.io/shields-docusaurus-poc/badges/bitbucket-pipelines
  • https://chris48s.github.io/shields-docusaurus-poc/badges/bitbucket-pipelines-with-branch

  This isn't miles off what we do on the existing frontend anyway tbh but we do need to be aware of/more strict about it. I think this mostly doesn't matter as we can basically have only one optional positional param per badge. Can anyone think of an edge case I haven't considered/tested when it comes to optional positional params? My current assessment is this is not a dealbreaker.

• We're going to have to break some existing frontend URLs. I think I'm not too worried about the /category links. We can probably handle /endpoint with a redirect. Is there anything else we need to care about? Don't think this one is a huge deal.

• This approach is going to give us a frontend URL for each badge (which is mostly a good thing - refs Link directly to builder modals #7064 ). We do need to take into account the fact that this pretty much doubles the number of URLs in our schema. I think putting them all under a subdir basically solves this but I know it is something @calebcartwright has expressed some caution over. We could also consider putting the whole frontend under /docs or something and redirecting / to /docs? Again, I don't think this one is a huge issue.

The things that are still unknown/untested are:

• So far I've tested this with a handful of endpoints. The full OpenAPI specs for our whole service catalogue is going to be hundreds of endpoints. How will this affect
  • build time and
  • search performance?
• How are we going to integrate this into shields? At the moment my proof-of-concept is a completely standalone application. I'd like to avoid returning to a process where the badge server and frontend are deployed separately. Can docusaurus be bludgeoned into our existing structure? I don't know yet.
• So far the OpenAPI specs in the proof-of-concept are written by hand. I'm working on the assumption that the code which generates service-definitions.yml can be modified to generate an OpenAPI spec without too much trouble but I've not tried it yet. We might need to make some slight changes to the fields we store in the service definition classes and/or do some special-casing for endpoint/static/dynamic badges. I'm not super worried about this, but it is one of the next assumptions I want to test.
• Will my proposed contributions to docusaurus-openapi be accepted upstream? Not tried yet..

Obviously I've only gone so far with this proof of concept. I've only implemented a subset of features and there's a bunch of content/styling things we can work on further. I think what I'm mainly looking for at this stage is:

• General feedback on the core approach/concept.
• Are there any things I haven't though of, massive holes in my thinking, or additional work I need to do to properly evaluate this approach? What edge cases have I not considered?
• Are any of the risks/downsides I've identified bad enough that we shouldn't move forward with this?

There are things about the proof-of-concept that could be better and extra things we could do, but are there any showstoppers ?

I'd be particularly interested in getting some feedback from @paulmelnikow @calebcartwright @PaulaBarszcz , but anyone involved in this process: please feel free to chime in.

[PaulaBarszcz]

TLDR: I like it. Great job, Chris :)

Link to live Shields POC on Docusaurus by Chris: https://chris48s.github.io/shields-docusaurus-poc/

---

• It has a built-in dark mode - instant +10.

• A search also works nicely and quickly.

 So far, I've tested this with a handful of endpoints. The full OpenAPI specs for our whole service catalogue are going to be hundreds of endpoints. How will this affect build time and search performance?

https://jestjs.io/search?q=mock

Jest docs use Docusaurus. The search for 'mock' (probably the most popular word in Jest :D), had 224 results and was fast.
I think we'd be fine :)
Examples of other pages using Docusaurus: https://docusaurus.io/showcase 

• Great that we show users which queryParams they can use (types / required etc.). It has a Swagger-ish feel :)

•  This means we need to generate a separate page for every with/without optional param variant

If this is a problem, we could limit the number of cases with optional param variants by moving them to query params. That way, e.g. Bitbucket/Github badges would be consistent with GitLab with its glorious repo/subrepo names containing slashes.

•  Will my proposed contributions to docusaurus-openapi be accepted upstream? Not tried it yet.

Before we move forward with a potential Shields migration to Docusaurus, it might be a good idea to open a few issues in docusaurus and/or docusaurus-openapi, with proposals regarding the most important changes needed by us. We'll see if they are fine with the proposed direction.  

• This approach is going to give us a frontend URL for each badge

From my perspective this is a big benefit.

• Some minor comments are below.

1. Cool idea to visually represent the logos of the companies backing Shields on OpenCollective. One question - can we tweak it for all of the logos to look good (not be cropped)? Especially these 2 caught my eye (statushero.com and levups.com).
   

Maybe sth along these lines - https://docsearch.algolia.com/. It is neater and clashes less with the dark mode.

2. This button is illegible:

3. How would the badge look with a very long text as a label/message?

4. Can we add a dropdown to choose parameters from?

5. Will these 2 'Blog' links point to the same place?
   We probably don't want to show the 'Edit this page' button :)
   IMO we don't need a blog; I'm under the impression that people try to scale down on various newsletters etc., to have less cluttered Inboxes.
   Shields already has a Twitter account - if we want to update users on some cool stuff that we're working on, we can let them know via this channel :)
   

6. I'm not sure why the static badge generator didn't work for me here. A deployment was in progress/incorrect link (?)
   

7. Do we want to add Downdetector-like info about Badges availability status somewhere on the page? Sth along the lines of (https://downdetector.co.uk/):
   

8. Those field names could probably use a different styling to differentiate them from the rest of the text:
   

9. In shields.io, we hint to users what can be filled in a field to generate an example badge - I like it. Will we use the same approach here?
   
   

10. Where do we place the info about adding one's GitHub token to increase Badges' pool of tokens?

11. Docusaurus has an out-of-the-box localization. Do we plan on using it?

12. Different styles of badges (+ other possible badges customizations like labelColor and logoWidth). Do we present them near the 'Execute' CTA or mention them in another section on the page?
    

[chris48s]

Thanks for taking the time to give the tyres a good old kick :) Some good points raised there. I'll try to follow up or reply to all that over the next few days but having given it a quick skim, I don't think there is anything there that completely blows the approach out of the water 🤞

[chris48s]

I think I have covered replying to everything here one way or another, but not all in the original order 🙃

---

Jest docs use Docusaurus. The search for 'mock' (probably the most popular word in Jest :D), had 224 results and was fast.
I think we'd be fine :)

Jest isn't quite the right comparison point, but this is a useful prompt to explain this more clearly.

There are multiple options for implemeting search in Docusaurus: https://docusaurus.io/docs/search

Jest are using Algolia, which is a hosted service well suited to dealing with large search indexes.

What I've used on the proof-of-concept is a local search plugin (I've used https://github.com/easyops-cn/docusaurus-search-local - there are others). These ship your entire search index to the user's browser and perform the search client-side. This is a simpler approach (and performant for small sites) but there is going to be an index size where you hit a tipping point and this stops being sensible. This is why I mention search performance/index size and why jest isn't an apples-for-apples comparison. That said, if we outgrow local search it looks like we would meet the criteria to get a free Algolia account. I think it is unlikely to be a showstopper, but I don't really know how big an index shields will generate until we try it with a realistic amount of content.

---

Before we move forward with a potential Shields migration to Docusaurus, it might be a good idea to open a few issues in docusaurus and/or docusaurus-openapi, with proposals regarding the most important changes needed by us. We'll see if they are fine with the proposed direction.

It is only docusaurus-openapi where I'd like to upstream some changes. Nothing to docusaurus core.
I've not opened any issues upstream yet just in case anyone raises anything that means we don't move forward with this approach, but yes. Raising those issues is one of the next jobs for me to look at 👍 agreed.

---

How would the badge look with a very long text as a label/message?

Kinda like this, I guess.. 😄

I think that is OK.

---

Can we add a dropdown to choose parameters from?

Yep. Good question. I didn't have an example showing it originally, but we can do this in OpenAPI using an enum. I've added another example to show what this looks like:

• chris48s/shields-docusaurus-poc@5e9066e
• https://chris48s.github.io/shields-docusaurus-poc/badges/coveralls

(have a look at the VCS Type param)

---

I'm not sure why the static badge generator didn't work for me here..

So this is good feedback confirming that this is not obvious and we'll need to explain better (you'll notice I've brushed over this in the proof of concept by making the docs for this "bla bla bla - explain the static badge here" :D because I knew this would be content that requires a bit of thought to write). This is probably the most difficult badge to explain in this format because (from an OpenAPI perspective) the static badge doesn't take multiple parameters, it takes a single parameter containing either the label, message and color separated by dashes, or the label only and a color separated by dashes. So a valid call would look like either:

or

I think the way to deal with this is probably going to be by including some examples of URLs/badges in the description on this page. I think having a dedicated page we can link to where we can write up some explanation of this and provide examples will still be a big improvement over what we have now, even if this is one of the harder endpoints to document.

---

Do we want to add Downdetector-like info about Badges availability status somewhere on the page?

Yeah we can link to https://stats.uptimerobot.com/PjXogHB5p like we do on https://shields.io/ - its a content design issue though. There are other bits like links out to external sites that we'll want but I've not bothered to reproduce it all completely.

---

Those field names could probably use a different styling to differentiate them from the rest of the text:

Yep. We can embed any HTML in the description, so all that is tweak-able
https://github.com/chris48s/shields-docusaurus-poc/blob/228f1203fda7dd03006be15fa7f68b952b171553/categories/1core.json#L137
Agree that layout needs some work though.

---

In shields.io, we hint to users what can be filled in a field to generate an example badge - I like it. Will we use the same approach here?

Yep. I didn't do it for all of them, but if you look at https://chris48s.github.io/shields-docusaurus-poc/badges/localizely-overall-progress you can see an example of one where I've filled in examples for the params.
https://github.com/chris48s/shields-docusaurus-poc/blob/228f1203fda7dd03006be15fa7f68b952b171553/categories/3other.json#L46-L66

When we come to generate the OpenAPI spec, we already have examples stored on the badge class we can consume.

---

Where do we place the info about adding one's GitHub token to increase Badges' pool of tokens?

Tbc. The proof-of-concept shows we can add arbitrary pages to the site and use HTML in badge descriptions (see https://chris48s.github.io/shields-docusaurus-poc/badges/localizely-overall-progress agsain) so I'm not worried about it at this stage. It will slot in somewhere.

---

Docusaurus has an out-of-the-box localization. Do we plan on using it?

Heh. This aint my first rodeo 🤠 . Having worked on open source projects that accept community translations before, there are several problems you run into:

1. It is basically impossible for me meaningfully review a pull request of the form "translate site into Japanese" (or any other language I don't speak) in the same way I can review a code submission. Is this a high quality translation of the site? Did someone just run it through google translate, call it a day and submit it? Is it mostly our site content but with occasional bits of QAnon propaganda inserted in strategic places? Who knows..
2. A nice person appears and submits a PR tranlsating your site in to Swedish. They mean well. Some time later down the line, you want to change the English content on the site. You now need a Swedish translator. If the person who originally submitted them isn't interested any more for whatever reason, you're stuck. You either have a Swedish version of the site which gradually becomes more out of sync with the English version over time or which progressively has more and more bits of it replaced by English. Eventually you retire the Swedish translations.

I've only ever really seen this work when there is someone on the project core team who speaks the target language and is willing to commit to maintaining the translations over a long period of time, or if you have budget to pay for translations.

In general, as well as looking to improve our website, one of the things I really want to try and achieve here is to also reduce the maintenance overhead of the site. In my experience, publishing a website in multiple langauges is a really good way to massively increase the maintenance burden.

---

Different styles of badges (+ other possible badges customizations like labelColor and logoWidth). Do we present them near the 'Execute' CTA or mention them in another section on the page?

So on the proof-of-concept site, you'll see that logo and logoColor are shown as optional query params on every page. This is achieved by declaring them as global params once
https://github.com/chris48s/shields-docusaurus-poc/blob/228f1203fda7dd03006be15fa7f68b952b171553/categories/2build.json#L17-L38
and then referencing them with JSON pointer in the parameters declaration for each endpoint e.g:
https://github.com/chris48s/shields-docusaurus-poc/blob/228f1203fda7dd03006be15fa7f68b952b171553/categories/2build.json#L63-L64
I think we should document all the global params (style, label, etc) like that. If it end up being useful to link out to a separate page documenting the global params, we can do.

---

[questions about the blog]

Dunno. Tbh, I might drop this. The template site docusaurus spat out had a blog so I decided to roll with it in the proof of concept, but we can change it.. or ditch it completely. This is about content really. It shouldn't influence the tech choice.

---

There is also some feedback about the content I put on https://chris48s.github.io/shields-docusaurus-poc/community

Basically with this page, I just (mostly) re-created what we've got on https://shields.io/community as a starting point and I think most of the comments about this equally apply to https://shields.io/community . The feedback isn't invalid and I don't want to completely dismiss it, but again this is all about content/design. I don't think any of it really influences "should we move forward with docusaurus/docusaurus-openapi" so I'm not going to get too bogged down in it at this stage as that's what I'm really trying to evaluate at this stage.

One thing I will quickly note though is that a lot of this is really feedback about the OpenCollective contributor images. I do fully agree that these are imperfect, but they do also allow us to show a logo/link for everyone that donates (even if some of them look sub-optimal) with zero maintenance overhead to update it over time (we can "set it and forget it"). This effort:functionality ratio is one thing they do have going for them 😄

[PaulaBarszcz]

Thanks for all your answers & explanations, @chris48s.
You’re right in all of the aspects :)

[chris48s]

I've gone a bit quiet on this one lately, but I have been working on some stuff upstream that I hope will help me get back to this again soon.

[calebcartwright]

[questions about the blog]

Dunno. Tbh, I might drop this. The template site docusaurus spat out had a blog so I decided to roll with it in the proof of concept, but we can change it.. or ditch it completely.

Personally I'd like to keep the page even if we drop it from the nav menu (assuming it's not too much trouble). I feel like the one blog posted is worth trying to keep around in perpetuity even if only as an easter egg of sorts

[chris48s]

I think it is unlikely to be a showstopper, but I don't really know how big an index shields will generate until we try it with a realistic amount of content.

Update on this: I'm currently working on a script to chuck out an openapi spec for shields. Its currently a bit half-assed but I've got it to a stage where it can spit out a valid spec for about 500 endpoints with some bits missing. This is less than all of them but a lot closer to the finished product than the ~10 I tested with in the proof of concept so I decided to see how big a search index that would generate. That content generated a search index of just over 3Mb so my guess is we'll be looking at something in the ~4Mb ballpark all told. That is bigger than I'd like, but not completely outrageous. I've seen other docusaurus site using a local search backend with a larger index (up to about 10Mb) so I think I'm fairly happy to roll with local search, at least as a starting point. It is not a showstopper.

[chris48s]

Quick update on this:

• I've had a few PRs merged upstream to docusaurus-openapi supporting all of this. There is now a release (v0.6.4) which contains all of those changes I wanted to make upstream supporting our use-case
• There's a PR open at Export OpenAPI definitions from service examples; affects [dynamic endpoint static] #8966 to generate OpenAPI defs for our entire suite of services, currently waiting on review
• I updated the proof of concept site to use the latest docusaurus-openapi version and also replaced the handful of OpenAPI specs I was using for testing with a copy of the output of running Export OpenAPI definitions from service examples; affects [dynamic endpoint static] #8966 as it currently stands. https://chris48s.github.io/shields-docusaurus-poc/ should now be a pretty realistic indication of what the actual site is going to look like with our full suite of services.
• Doing that yields a search index size of under 5Mb which is in the ballpark I predicted
• The next job is really to work on a PR to rip out all the gatsby stuff and smoosh https://chris48s.github.io/shields-docusaurus-poc/ into this repo in its place, which I am planning to start in the next few days. In theory this is not super-complicated as we've basically got the 2 bits now - I just need to stitch them together. Also experience teaches me there are going to be some fairly substantial intricacies to that and several unrelated things will catch fire during the course of working on it

The wheels continue to turn, slowly.

[chris48s]

We have just launched the new Docusaurus-based frontend https://shields.io/

There is more work to do, but I'm going to track follow up tasks in a series of more focussed issues