Spectral actions for OpenAPI document

[kulnor]

(this was drafted by @arno-di-loreto)

The question about the use of spectral:oas made me think. We need to talk about the different types of actions we want to do on OpenAPI documents (and if/how to manage them with Spectral rulesets). Based on my understanding, there are 3 types of actions. Does that make sense? Is it correct?

• Format (OpenAPI in that case) validation: is the document "valid"
  • This can be separated in sub-levels of validations/questions
    • Schema validation: Is the document valid against the OpenAPI schema?
      • Can be done with the oas3-schema and oas2 schema rules of oas spectral ruleset
      • A valid schema does not mean a valid document (JSON Schema cannot check consistency between the different objects (example: that if /path/{variable} is defined there's an operation level or path level path parameter defined)
      • Note that for OpenAPI 3.1 there is the possibility of using of JSON schema formats maybe there will be some tweak to do there
    • Document completeness: Is the document complete/are we missing dependencies?
      • For now we are pulling on one document, if it has dependencies we don't have them
      • Detecting that a document has dependencies could be done with Spectral rules or possibly with a parser?
        • For now we can a use a "finder" spectral ruleset to check that and see if that's a real issue for our dataset
      • Note: spectral can unref OpenAPI document leveraging multiple sub-documents, it can use a custom $ref resolver if needed
    • Parser validation: Is the document valid against the OpenAPI specification/valid for parser X?
      • Can possibly be done with Spectral
        • Using a subset of rules of the oas ruleset as describe here but need to be confirmed
        • Creating a custom function leveraging a JS parser
      • Could help us check if we have a complete document or partial one
      • Would it be interesting to check the result of different parsers? (I think it's out of the scope for now, not interesting for an API copilot, but interesting for OAI and parser creators)
        • We can prepare by just adding the info of the first parser we used along with the validation result
• Findings/Finders/Questions
  • We decided to use Spectral to do that
    • These rulesets/rules will probably be highly specific (hence different from usual ones, like the one used in validation category below). They are made to find stuff, not to tell "this is wrong" (though regular rules could be used to do so).
  • This can be done by a single rule (in a ruleset file) or by multiple rules in multiple ruleset files (in that case there will always be a master main file)
  • What matters here is most probably the result of each rule/finding/check
    • Answering questions can be done by a combination of those results
    • By result I mean: rule triggered/not + possibly the targets (inside document) that triggered the rules
• Validation against XXX guidelines created by someone or by us
  • That category probably answers my "why spectral:oas" question here
  • For now, that's pure Spectral stuff
    • But there are other linting formats we could leverage
    • These will be done will ruleset file(s) containing many rules
    • These are "usual" rulesets (opposite to the ones used in findings)
  • What matters is probably not the details of non-respected rules, but the level of validity (high level result)
    • We can start with valid/invalid and then evolve to add a scoring (X% of compliance)
  • Examples (see also your message on real world ruleset)
    • OWASP by Stoplight
    • OWASP by Postman
    • HTTP (we're working on this with Jordan Walsh)
    • Spectral embedded ruleset (recommended, all)
    • Adidas
    • ...
  • These rulesets should be automatically harvested and kept up to date
    • That probably should be managed by a separate "module" (the "high quality/validated spectral rules"?), the KB would just need to pull them from there.

[kulnor]

On the validation topic:

• We are currently using the Python package to validate against the v2.0, v3.0 and v3.1 schemas. The reported errors are OK but sometime a big vague. Do we think a different validation package or tools would produce different results or findings?
• A you mention this will not catch all issues, and this is where Spectral becomes critical to perform second level validation and beyond, and provide more insights around issues.
• Running a report with just the spectral:oas ruleset is a good starting point and example of this

On the findings topic:

• That is definitely the core focus at this point
• The wrong vs right aspect is interesting one. Spectral today is designed to detect what is wrong and fails. It is basically a pessimistic tool :-). It would be nice to have a "pass" reporting level, basically a option to invert the report (only report the rules that do not fail). This would be useful to ask 'what does the JSON document complies with'.

On the guidelines topic:

• This is definitely something we can and want to do in the KB.
• And we also one to come up with new Postman policies to detect common the issues we are finding in the KB to strengthen the platform governance and IDE