[FEATURE] Support for version 1.4 schema

[madpah]

Creating issue to track adoption of the forthcoming version 1.4 schema specification for CycloneDX.

Preview work can be seen here, but this IS NOT FINAL.

Current estimates are that version 1.4 will be finalised late 2021, early 2022.

[jkowalleck]

lets postpone this feature, until 1.4 is through the RFC phase.
maybe extend the on-hold until 1.4 got finalized

[madpah]

1.4 is now in draft with only minor fixes now expected before it is finalised and released early 2022.

See https://github.com/CycloneDX/specification/tree/v1.4-dev

[madpah]

Work has commenced in branch https://github.com/CycloneDX/cyclonedx-python-lib/tree/feat/schema-version-1.4

[madpah]

Work items:

• Add draft 1.4 schemas to schemas folder
• Add XML schema validation in unit tests only (this prevents lxml from becoming a mainstream dependency)
• Add JSON schema validation in unit tests only (prevent dependency bloat)
• Schema Change (XML): Add support for bom.metadata.tools.tool.externalReferences
• Schema Change (JSON): Add support for bom.metadata.tools.tool.externalReferences
• Schema Change (XML): Make bom.component.version optional
• Schema Change (JSON): Make bom.component.version optional
• Fallback to version == '' for Schema Version < 1.4 when not provided
• Schema Change (XML): Add support for bom.component.releaseNotes
• Schema Change (JSON): Add support for bom.component.releaseNotes
• Schema Change (XML): Vulnerabilities are now part of the core schema in 1.4 - bom.vulnerabilities
• Schema Change (JSON): Vulnerabilities are now part of the core schema in 1.4 - bom.vulnerabilities
• Schema Change: Support for $schema in JSON format SBOM JSON result: add $schema #104
• Schema Change: JSON schema in 1.4 has been hardened - JSF added - BUMPED to [FEAT] add support for JSF Signatures #122
• Assess Parsers for any option to populate new schema elements
• Add FINAL/RELEASED 1.4 schemas to schemas folder

Future work, not specific to version 1.4:

• Complete support for bom.metadata.* - see [FEATURE] More complete support for bom.metadata #6 and Add support for metadata component #118
• Add support for bom.services - see [FEATURE] Add support for generating SaaSBOM (bom.services) #123
• Add support for bom.externalReferences - see [FEATURE] Add support for BOM references (bom.externalReferences) #124
• Add support for bom.dependencies - see [FEATURE]: Add support for dependencies (bom.dependencies) #7
• Add support for bom.compositions
• Add support for bom.properties - see [FEAT] Add support for arbitrary properties at the BOM level (bom.properties) #125

@jkowalleck - appreciate your quick review on the above (first) list of key items to get done for 1.4 spec.

[jkowalleck]

@madpah love the list you accumulated.

some important mentions/additions/summaries/annotations:

• "Vulnerabilities are now part of the core schema in 1.4"
  • cyclonedx-python-lib has an implementation for it already. regarding spec < 1.4
  • so the serializers need to implement the CDX-builtin vuln for JSON & XML for spec > 1.4
    and may fallback to the existing implementation for XML spec < 1.4
    and omit vulns for JSON spec < 1.4 since
    @madpah i think this would a noticable challende that would require more work than the other transitions.
• "JSON schema in 1.4 has been hardened "
  • schema is not relaxed. (per default json schema is relaxed - which allowes to add unspecified/additional properties)
    "additionalProperties": false,` everywhere - nothing but the content of the spec is allowed, - see https://github.com/CycloneDX/specification/blob/v1.4-dev/schema/bom-1.4-SNAPSHOT.schema.json*
  • this means that json extensions are no longer possible with spec 1.4.
    • this should be no issue, since the schema extensions never existed for JSON (https://github.com/CycloneDX/specification/blob/v1.4-dev/schema/ext/vulnerability-1.0.xsd)
  • this means there will be no "strict" JSON schema for v1.4 - as the v1.4 is strict already
  • most important: there must be no property rendered to JSON, that is not specified in the schema. in other words: "for each property in json: property is pecified in schema".
    so it a good time to add schema validator to the library, that others can use

---

@madpah feel free to edit this message, to add more implementation details we need to keep track of

---

PS/Edits:

• pointed out the need of a validator
• removed the note with the empty version here, as it was incorporated into [FEATURE] Support for version 1.4 schema #67 (comment)

[madpah]

Thanks @jkowalleck - agree with your summary - and thanks for pointing out the agreement to set version == '' - had not spotted that!

[jkowalleck]

the thing that worries me the most:
schema 1.4 is not relaxed - is "hardened" - see #67 (comment)
this means there must be no properties in the JSON that are not in the spec - so we should add a validator to the library,
so down-streams/users of the lib can assert that the resulting JSON is valid.

i implemented such schema validators for the php-lib already. some implementation details:

• the library ships all supported schema as snapshot with some modifications:
  • the path to the SPDX-schema was made relative in the schema files,
    so no network traffic is done when validation runs
  • see https://github.com/CycloneDX/cyclonedx-php-library/blob/master/res/README.md
• schema validation may be done against the strict schema for spec < 1.4
• JSON schema validation for spec 1.4 will not go against strict schema, as 1.4 is strict by default

---

you already started the implementation of schema validation, iirc, @madpah .
right?

[madpah]

Understand @jkowalleck - I've already added XML and JSON schema validation for unit tests. But we can easily add that to the main code.

I was avoiding XML schema validation in the main code as that would require lxml, which is a bit bloaty and platform specific. Thoughts?

[jkowalleck]

good point, @madpah .

lets have the validators for internal unit-tests for now and get experienced with it.
there must be multiple xml- and json-schema validator libraries out there - so we need to find one that suites us the most.

we might need to file a decision paper (like this one)
and gather non-functional requirements/constraints,
before we can make a validator public.

if we like the solution we are using for our own tests, we might make it public later.

[jkowalleck]

@madpah when downloading/shipping the spec1.4 json schema, we need to have the JSF schema also.
JSF it is used in spec 1.4 json schema as a relative reference - just like the SPDX schema.

we might not have proper support for JSF implemented, but for schema validation (in unit tests) we need the JSF schema file and the SPDX schema file next to the spec1.4 schema file.

[madpah]

Thanks for the heads up @jkowalleck - haven't got to JSON for 1.4 yet :-)

[madpah]

Pre-release published to PyPi: https://pypi.org/project/cyclonedx-python-lib/1.0.0rc0/

FYI @jkowalleck

[jkowalleck]

will do a review of #108

[madpah]

Released!

Thanks @jkowalleck.