Workflow to publish schemas

[ralfhandl]

• GitHub workflow
• bash script called from workflow
• JS script to convert YAML to JSON and patch iteration date
• unit tests for JS script
• Align with outcome of
  • schemas: add WORK-IN-PROGRESS placeholders, don't mention patch numbers, update terminology #4146
• Dependencies between schema files:
  • if meta changes, republish all four files
  • if dialect changes, republish it, schema, and schema_base
  • if schema changes, republish it and schema_base
  • if schema_base changes, republish it
  • new date stamp is maximum date of last commit in the dependency chain

flowchart LR
    schema_base
    schema
    dialect
    meta
    schema --> |default| dialect
    schema_base --> |$ref| schema
    schema_base --> |$ref| dialect
    dialect --> |$ref| meta

Loading

Example PR created by this workflow

• https://github.com/ralfhandl/OpenAPI-Specification/pull/28/files

[karenetheridge]

to confirm, these are both edited in the files?

• dates in the $id keywords
• $comment links in each definition, that currently go to the 3.1.0 spec and should go to 3.1.1

[ralfhandl]

• What about the dates in the $ids?

Those are replaced when converting from YAML to JSON, see example PR https://github.com/ralfhandl/OpenAPI-Specification/pull/26/files

[ralfhandl]

• What about the description links in each definition, that currently go to the 3.1.0 spec and should go to 3.1.1?

These need to be adjusted manually. Changing them in main will then trigger the schema-publish workflow.

[handrews]

Thanks for doing this work! To the extend that I understand current JavaScript, this looks great.

However, I'd like to merge #4146 and update this PR accordingly before publishing any new schemas. Let's make sure that when we resume publication after such a long gap, the first newly-published schemas are what we want.

Unfortunately, this does introduce a complexity, which is that the various 3.1 schema resources (of which there are four) can change independently. Ideally, we would only re-publish and update the $id for each when it changes, although it would not be the end of the world to just publish all four at once with the same updated data whenever any of them change. For the first publication, we'll need to do that anyway, so perhaps supporting independent updating can be something for later to keep the immediate work manageable?

I believe that the search-and-replace could just become a global grep for WORK-IN-PROGRESS once #4146 is merged, rather than worrying about parsing anything. We'd have to know not to put "WORK-IN-PROGRESS" anywhere else in the schemas, but that shouldn't be too hard to document.

It's worth noting that there are more places than id, $id, and $ref where the replacement needs to be done, so a global grep for a unique token would be more robust anyway.

[handrews]

@ralfhandl I realize I am asking for a substantially more complex workflow, so I took a pass at it. I was going to use the "suggest" feature on this PR but... my JavaScript skills are very poor and my bash skills are atrocious.

So here's something that I got working locally that you can use as a proof-of-concept (it doesn't update the markdown but I'm not quite sure what's going on there and I think this should get the point across). Or if you'd prefer, I can try to clean this up and submit it myself. I'm also happy to help with updates to the spec site for the date-using vocabulary and dialect schemas.

In local testing it handled various combinations of only certain schemas being updated correctly, including all updated, only the schema updated (but for both versions) and having the vocab and/or dialect schema changes cause further updates.

thisCommit="$GITHUB_SHA"

# Note that for 3.0, "noDialectSchema" is the only schema
noDialectWIP="schema/WORK-IN-PROGRESS"
vocabWIP="meta/WORK-IN-PROGRESS"
dialectWIP="dialect/WORK-IN-PROGRESS"
strictDialectWIP="schema-base/WORK-IN-PROGRESS"

for schemaDir in schemas/v3* ; do
  version=$(basename "$schemaDir")

  noDialectSchema="$schemaDir/schema.yaml"
  vocabSchema="$schemaDir/meta/base.schema.yaml"
  dialectSchema="$schemaDir/dialect/base.schema.yaml"
  strictDialectSchema="$schemaDir/schema-base.yaml"
  echo $noDialectSchema
  echo $vocabSchema
  echo $dialectSchema
  echo $strictDialectSchema
  echo ""

  if [ -f "$noDialectSchema" ]; then
    noDialectCommit=$(git log -1 --format="%H" -- "$noDialectSchema")
    noDialectCommitDate=$(git log -1 --format="%ad" --date=short -- "$noDialectSchema")
    if [ "$noDialectCommit" = "$thisCommit" ]; then
      updateNoDialect="1"
    fi
  fi

  if [ -f "$vocabSchema" ]; then
    vocabCommit=$(git log -1 --format="%H" -- "$vocabSchema")
    vocabCommitDate=$(git log -1 --format="%ad" --date=short -- "$vocabSchema")
    if [ "$vocabCommit" = "$thisCommit" ]; then
      updateVocab="1"
    fi
  fi

  if [ -f "$dialectSchema" ]; then
    dialectCommit=$(git log -1 --format="%H" -- "$dialectSchema")
    dialectCommitDate=$(git log -1 --format="%ad" --date=short -- "$dialectSchema")
    updateDialect="$updateVocab"
    if [ "$dialectCommit" = "$thisCommit" ]; then
      updateDialect="1"
    fi
  fi

  if [ -f "$strictDialectSchema" ]; then
    strictDialectCommit=$(git log -1 --format="%H" -- "$strictDialectSchema")
    strictDialectCommitDate=$(git log -1 --format="%ad" --date=short -- "$strictDialectSchema")
    updateStrictDialect="$updateDialect"
    if [ -n "$updateNoDialect" ]; then
      updateStrictDialect="1"
    fi
    if [ "$strictDialectCommit" = "$thisCommit" ]; then
      updateStrictDialect="1"
    fi
  fi

  echo $thisCommit
  echo $version $noDialectCommit $noDialectCommitDate $updateNoDialect
  echo $version $vocabCommit $vocabCommitDate $updateVocab
  echo $version $dialectCommit $dialectCommitDate $updateDialect
  echo $version $strictDialectCommit $strictDialectCommitDate $updateStrictDialect
  echo ""

  mkdir -p deploy/oas/$version/schema
  if [ -f "$vocabSchema" ]; then
    mkdir -p deploy/oas/$version/meta
    mkdir -p deploy/oas/$version/dialect
    if [ -n "$updateNoDialect" ]; then
        node scripts/schema-convert.js "$noDialectSchema" $noDialectCommitDate $vocabCommitDate $dialectCommitDate $strictDialectCommitDate > deploy/oas/$version/schema/$noDialectCommitDate
    fi
    if [ -n "$updateVocab" ]; then
        node scripts/schema-convert.js "$vocabSchema" $noDialectCommitDate $vocabCommitDate $dialectCommitDate $strictDialectCommitDate > deploy/oas/$version/meta/$vocabCommitDate
    fi
    if [ -n "$updateDialect" ]; then
        node scripts/schema-convert.js "$dialectSchema" $noDialectCommitDate $vocabCommitDate $dialectCommitDate $strictDialectCommitDate > deploy/oas/$version/dialect/$dialectCommitDate
    fi
    if [ -n "$updateStrictDialect" ]; then
        node scripts/schema-convert.js "$strictDialectSchema" $noDialectCommitDate $vocabCommitDate $dialectCommitDate $strictDialectCommitDate > deploy/oas/$version/schema-base/$strictDialectCommitDate
    fi
  else
    if [ -n "$updateNoDialect" ]; then
        node scripts/schema-convert.js "$noDialectSchema" $noDialectCommitDate > deploy/oas/$version/schema/$noDialectCommitDate
    fi
  fi
done

#!/usr/bin/env node

'use strict';

const fs = require('fs');
const yaml = require('yaml');

function convert(
    filename,
    noDialectDate,
    vocabDate=false,
    dialectDate=false,
    strictDialectDate=false,
) {
    try {
        var s = fs.readFileSync(filename,'utf8');
        s = s.replace(
            /schema\/WORK-IN-PROGRESS/g,
            'schema/' + noDialectDate,
        );
        if (vocabDate) {
            s = s.replace(
                /meta\/WORK-IN-PROGRESS/g,
                'meta/' + vocabDate,
            );
        }
        if (dialectDate) {
            s = s.replace(
                /dialect\/WORK-IN-PROGRESS/g,
                'dialect/' + dialectDate,
            );
        }
        if (strictDialectDate) {
            s = s.replace(
                /schema-base\/WORK-IN-PROGRESS/g,
                'schema-base/' + strictDialectDate,
            );
        }
        const obj = yaml.parse(s, {prettyErrors: true});
        console.log(JSON.stringify(obj,null,2));
    }
    catch (ex) {
        console.warn('  ',ex.message);
        process.exitCode = 1;
    }
}

if (process.argv.length<4) {
    console.warn('Usage: convert-schema.js file.yaml YYYY-MM-DD [YYYY-MM-DD YYYY-MM-DD YYYY-MM-DD]');
}
else {
  if (process.argv.length>4) {
    convert(process.argv[2], process.argv[3], process.argv[4], process.argv[5], process.argv[6]);
  } else {
    convert(process.argv[2], process.argv[3]);
  }
}

[ralfhandl]

@handrews Thanks, will look into this once #4146 is merged.

[ralfhandl]

Replaced with simpler approach:

• Schema Publish workflow #4162