Migrate specs to the new content management system

[rufuspollock]

We want to migrate the /specs/ section covering the Frictionless Specs and esp data package are migrated over.

• Specs live in their own repo and should stay there
• Want to have them show up in the main site
• Schemas need to show up somewhere ...

Acceptance

• github.com/frictionlessdata/specs is still home for specs
• specs.fd.io/schemas/data-package.json, specs.fd.io/schemas/registry.csv etc work - https://specs.frictionlessdata.io/schemas/data-package.json
• redirects from fd.io/schemas/data-package.json, fd.io/schemas/registry.csv work - PART OF [v2] Switch over the DNS (and do any Cloudflare level redirects) #432
• fd.io/specs works or redirects - Part of [v2] Switch over the DNS (and do any Cloudflare level redirects) #432
• specs.fd.io is working and looks good - see how it used to look https://web.archive.org/web/20170627080529/http://specs.frictionlessdata.io/ and https://web.archive.org/web/20170704050901/http://specs.frictionlessdata.io/csv-dialect/ - DONE see https://specs.frictionlessdata.io/

Tasks

• Branch in specs for new refactor
• Research how this to used to be when we had old subsite See [epic] Reintroduce readable style for specs datapackage#420 - turns out it was in lektor for ages.
  • Here's commit where we moved to lektor https://github.com/frictionlessdata/specs/tree/9781bc3ee28117cb7c8a72cd1268a2907a746998
  • Here's commit before it https://github.com/frictionlessdata/specs/tree/237fa0b9d5f34e5d127aee48d45a514458ccaa1a
  • Key refactoring was around Commits on Oct 27, 2017 see https://github.com/frictionlessdata/specs/commits/master?after=52863e066e24cc51dec571030c4c142623bcdb02+139**
• Add vuepress build (use ootb theme for now)
• Refactor content from lektor markdown to normal markdown @monikappv
  • Move spec folders up from specs to base directory
  • Sort out frontmatter (very simple - just make one group)
  • Move / remove remaining stuff in specs/ directory
• Add a front page - see old site https://web.archive.org/web/20170704050901/http://specs.frictionlessdata.io/ and current site @monikappv
• Do any custom theming e.g. to support the frontmatter we have @monikappv
  • Use the title field for the main title of specs
  • Use the other frontmatter - see archive.org/web
• Go through all fix relative urls, frontmatter etc -- see "Refactoring each spec to vuepress markdown" below
  • Also remove slug frontmatter - should not have that ...
  • Make sure the frontmatter of all specs is aligned

Sort out the build

• Add access token so gh-pages deploy in gh workflow works https://github.com/JamesIves/github-pages-deploy-action
• Add build specs.fd.io for vuepress to github workflows @monikappv (?)
  • And remove current code (which just triggers a build of the website - and is no longer needed)
• Add build of schemas (npm build) to workflows
• Add tests to workflows
  • Refactor them to work (with new layout)
• Remove parts of grunt.js build that are not needed e.g., generating .md from .lr files https://github.com/frictionlessdata/website/blob/master/Gruntfile.js
• Remove travis (just use github workflows going forward)
  https://github.com/frictionlessdata/specs/blob/master/.travis.yml

Misc

• Set the CNAME
• DNS update - currently specs.frictionlessdata.io redirects to frictionlessdata.io/specs
• Add permanent redirects (301 HTTP status) in fd.io to new locations ... - should be part of [v2] Implement Redirects for v1 URLs #423 turns out you can't do this in Vue ¹ so we have to implement this at cloudflare level
  • /specs/* => specs.fd.io/*
  • /schemas/* => specs.fd.io/schemas/

Later

• Examine schemas build process - can we simplify this ...

Inbox

• (?) Move contributing stuff out of README to contributing

Refactoring each spec to vuepress markdown

• Pay attention to full frictionlessdata.io URLs. Change them to relative URLs.

  [Data Package specification](http://frictionlessdata.io/specs/data-package/)
  
  should become
  
  [Data Package specification](/data-package/)
  

• Document, using Markdown's frontmatter, when a spec is under development. The ones under development are documented in the spreadsheet.
  https://docs.google.com/spreadsheets/d/1qByu4qw56hFUdJIY5IOC7d8TQBud-k31J6720hg1h2w/edit#gid=1449054173
• Double check for broken links.
• Use a spell checker before commiting.

Analysis

Misc

• Why travis.yml and a github workflows?
  • One does test (travis)
  • One does auto build of main website https://github.com/frictionlessdata/specs/blob/master/.github/workflows/website.yml

Do we go with specs.fd.io or fd.io/specs

• fd.io/specs sets up an annoying coupling ... 🐛
  • specs repo has to worry about templating and set up on fd.io
  • fd.io has to submodule in specs at build time
  • 🐛 fd.io does not (currently) rebuild on changes to specs (and would be painful to do)
    • Yes it does: see https://github.com/frictionlessdata/specs/blob/master/.github/workflows/website.yml but sort of Hacky
  • 🐛 specs have weird lektor like layout in markdown and have to have a build script simply to convert md to lr name. For a specs contributor this is weird.
• Painful to modify specs e.g. to have versions (latest vs v1)
• 🐛 we have core schemas living on a perma-url on frictionlessdata.io ... at /schemas/ this is core product functionality on a "presentation" website
• (Separate) We have some complex build structure to build the json schemas ...
• 🐛 building json schemas may prevent the website from deploying

KISS suggests subsite is simpler ...

• SEO is fine
• We can share branding
• Very clean specs site and build
• Easy to revert from if needed

Current flow for deploying specs to the website:

Current build process (via Grunt.js task in website)

• submodule in the repo
• build it
• copy directories around

Details

1. website project runs grunt specs. https://github.com/frictionlessdata/website/blob/master/Gruntfile.js#L87-L100
2. grunt specs clones specs repository and runs npm build and npm test
3. specs npm run build runs build.js with node. https://github.com/frictionlessdata/specs/blob/master/build.js#L108-L110

https://github.com/frictionlessdata/website/blob/master/Gruntfile.js#L87-L100

Footnotes

1. vue router can only do internal redirects. ↩

[rufuspollock]

@monikappv please can you take this on - or, at least the tasks I've specifically marked for you.

I've made a start already on this here in a new branch back-to-own-subsite-2020:

https://github.com/frictionlessdata/specs/tree/back-to-own-subsite-2020

In particular, i've already migrated one spec back to give you an example (see the last commit in that branch).

To start: just clone, switch to that branch, npm install, boot the server and start doing the work. You can then keep pushing to that branch. Once we are complete we can merge back into master - but not before then.

[rufuspollock]

@monikappv can you update with a screenshot of what this now looks like.

[monikappv]

@rufuspollock here is a screenshot of Specs home page and one specs page (CSV dialect)

[nirabpudasaini]

Frontmatter Alignment for Specs:

• Remove slug from all frontmatter
• Change Full URL to relative URL
• Document specs metadata and use them to align frontmatter
• Link to schema from Metatable component
• Mark the specs that are under development in frontmatter
• Check for broken links
• Check for spelling errors using a spellchecker

[rufuspollock]

FIXED. All done that needs to be done. Remaining redirects is in #432 and minor extras can be done later if at all. https://specs.frictionlessdata.io/