Validation of endpoints against a given OpenAPI schema #3645

adamw · 2024-03-28T16:09:28Z

Sometimes it might be desirable to verify if the endpoints match a given OpenAPI schema. To implement this, we would need to parse the schema (see softwaremill/sttp-apispec#2 - although this is already partially done), and then invoke comparison of the two schemas (generated from endpoints & parsed), ignoring fields which do not matter (e.g. descriptions, or examples)

ghik · 2024-04-08T07:44:19Z

What's the "user story" here? Is it something like:

User has a given OpenAPI to implement
Decides to do it manually instead of using codegen
Writes an API in tapir but wants to validate it against the given OpenAPI

?

adamw · 2024-04-08T07:51:35Z

This, or: develops an API, publishes it, and wants to ensure that anything that's changed doesn't break the published contract

ghik · 2024-04-25T07:40:10Z

softwaremill/sttp-apispec#157 introduced SchemaComparator to check compatibility between schemas. This is likely the most complex part of full OpenAPI validation. Schema comparator would be used:

to check compatibility of inputs (parameters, request bodies) - schemas from previous API version (i.e. client's) would be used as "writer" schemas, while schemas from current API version (i.e. server's) would be used as "reader" schemas
to check compatibility of outputs (responses) - schemas from current API version (i.e. server's) would be used as "writer" schemas, while schemas from previous API version (i.e. client's) would be used as "reader" schemas

Possible improvements to SchemaComparator:

Make it understand more Schema patterns and JSON Schema keywords (see SchemaComparator sttp-apispec#157 for more details on comparator's assumptions and limitations)
Make it customizable with pluggable validation logic
The comparator may be too conservative in some cases. For example, it might not understand that some format values are compatible, or that some particular schema patterns are compatible in specific use cases. It may be worth to open SchemaComparator's implementation to extensions which would allow the user to override parts of its logic, in order to fine-tune it to their needs and avoid false negatives.
Possibly rethink the structure of compatibility issues being reported (e.g. by making it completely flat or completely tree-like, deduplicating issues for heavily reused named schemas etc.). Optimal structure largely depends on how exactly we want these results to be presented to the user.

Further work needed to have a full OpenApi comparator:

Make sure OpenAPI can be deserialized (it looks to me like all the decoders are already there, but I haven't tested them)
Implement an OpenAPIComparator similar to SchemaComparator.
This requires comparing the following OpenAPI objects: MediaType, Encoding, RequestBody, Responses, Response, Header, Parameter, Operation, PathItem, and ultimately OpenAPI itself.

ghik · 2024-04-25T13:22:09Z

Proposed structure for compatibility issues reported by full OpenAPI comparator

missing path - server API does not define a path item that client API expects
incompatible path - there are incompatibilities in operations for given path item
- missing operation for given HTTP method
- incompatible operation for given HTTP method
  - a required server operation parameter is missing or optional in client operation
  - client parameter is incompatible with corresponding server parameter
    - client parameter schema is incompatible with server parameter schema
    - client content is incompatible with server content (see below for details)
    - style or explode don't match between client and server
    - allowEmptyValue is true for client but false for server
    - allowReserved is true for client but false for server
  - request body is required in server operation but is missing or optional in client operation
  - client request body is incompatible with server request body
    - client content is incompatible with server content (see below for details)
  - server responses are incompatible with client responses
    - response for some status code or range is defined by the server but not by the client (note: response for a particular code may be defined using a range or default)
    - response for some status code or range defined by the server is not compatible with corresponding definition on the client
      - server response content is incompatible with client response content (see below for details)
      - a response header is required by the client but is missing or optional on the server
      - server response header is incompatible with client response header
        
        server header schema is incompatible with client header schema
        
        server header content is incompatible with client header content (see below for details)
        
        style or explode don't match between server and client
        
        allowEmptyValue is true for server but false for client
        
        allowReserved is true for server but false for client
  - TODO: callbacks, security?

Compatibility issues for content comparison:

media type is defined in writer's content but not defined in reader's content
writer media type is incompatible with reader media type
- writer schema does not match reader schema
- TODO: encoding?

Note: "writer" and "reader" correspond to:

client and server, when comparing input (request) definitions
server and client, when comparing output (response) definitions

adamw · 2024-10-16T10:18:45Z

Current state: we've got the SchemaComparator as describe above.

What's left to do:

ensure that the openapi-circe module in sttp-apispec includes all necessary Decoders, which allow parsing an openapi schema file (write tests + fix if necessary)
introduce an OpenAPIComparator in the sttp-apispec project, as part of the openapi-model module. This should compare OpenAPI instances, as defined above. Non-essential differences (e.g. in examples, doc strings etc.) should be ignored.
in Tapir's openapi-docs module, add an OpenAPIVerifier, which would verify that a given set of endpoints matches the given openapi spec (given as a yaml string). This should have three modes: exact (every endpoint corresponds to exactly one openapi endpoint, and vice versa), at-least (there might be additional endpoints), at-most (there might be more openapi endpoints, than Tapir endpoints)

adamw · 2024-10-16T10:38:36Z

/bounty $750

algora-pbc · 2024-10-16T10:38:40Z

💎 $750 bounty • SoftwareMill

Steps to solve:

Start working: Comment /attempt #3645 with your implementation plan
Submit work: Create a pull request including /claim #3645 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to softwaremill/tapir!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @Saturn225	Oct 16, 2024, 1:17:41 PM	WIP
🔴 @vishwamartur	Nov 16, 2024, 4:02:24 PM	WIP
🟡 @promisingcoder	Dec 12, 2024, 2:44:26 AM	WIP
🟢 @abdelfetah18	Dec 19, 2024, 7:34:57 PM	#4215

Saturn225 · 2024-10-16T13:17:38Z

/attempt #3645

Algora profile	Completed bounties	Tech	Active attempts	Options
@Saturn225	1 bounty from 1 project			Cancel attempt

adamw · 2024-10-21T10:23:40Z

As we are getting first PRs for the bounties, I've published our "How to prepare a good PR" guide. I should have probably done this right away, sorry! :)

kernel-loophole · 2024-11-09T10:59:06Z

@adamw is this issue is still open .would love to work on it .

Saturn225 · 2024-11-09T19:10:08Z

@adamw @kernel-loophole I have worked on it and need a few quick fixes and will draft PR soon

vishwamartur · 2024-11-16T16:02:21Z

/attempt #3645

Options

Cancel my attempt

algora-pbc · 2024-11-16T16:02:27Z

Note

The user @Saturn225 is already attempting to complete issue #3645 and claim the bounty. We recommend checking in on @Saturn225's progress, and potentially collaborating, before starting a new solution.

Related to softwaremill#3645 Implement `OpenAPIVerifier` in the `openapi-docs` module to verify endpoints against a given OpenAPI spec. * Add `OpenAPIVerifier` class in `docs/openapi-docs/src/main/scala/sttp/tapir/docs/openapi/OpenAPIVerifier.scala` - Implement three modes: exact, at-least, and at-most - Use `OpenAPIComparator` to compare `OpenAPI` instances * Add tests for `OpenAPIVerifier` in `docs/openapi-docs/src/test/scalajvm/sttp/tapir/docs/openapi/OpenAPIVerifierTest.scala` - Test all three modes: exact, at-least, and at-most - Use sample endpoints and OpenAPI specs for testing

algora-pbc · 2024-11-23T16:02:31Z

@vishwamartur: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

abdelfetah18 · 2024-11-28T22:22:03Z

Hi @adamw, I've started working on the OpenAPIComparator. Could you take a look at this PR and let me know if I'm heading in the right direction?

softwaremill/sttp-apispec#194

ThijsBroersen · 2024-11-29T10:41:15Z

What about reusing openapi-diff . I have used this tool standalone in CI to check for compatibility with previous openapi specs of projects. But it could easily be included in tests as it is a java module. Could probably save a lot of work and maintenance for this project.
It also has some nice reporting features, very useful for CI release pipelines and notes.

adamw · 2024-11-29T15:45:10Z

@abdelfetah18 yes this looks like the good direction. Would be good to have tests which read .yaml/.json files as well, though.

@ThijsBroersen ah didn't know about that :) We could definitely use it as well - as long as we can provide an appropriate interface on top (comparing endpoints to a specification, ignoring unimportant details (descriptions, examples), and providing exact/at-least/at-most modes

promisingcoder · 2024-12-12T02:44:23Z

/attempt #3645

Options

Cancel my attempt

algora-pbc · 2024-12-12T02:44:31Z

Note

The user @Saturn225 is already attempting to complete issue #3645 and claim the bounty. We recommend checking in on @Saturn225's progress, and potentially collaborating, before starting a new solution.

algora-pbc · 2024-12-19T02:44:33Z

@promisingcoder: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

abdelfetah18 · 2024-12-19T19:34:55Z

/attempt #3645

Algora profile	Completed bounties	Tech	Active attempts	Options
@abdelfetah18	1 softwaremill bounty	Java, C++		Cancel attempt

algora-pbc · 2024-12-19T19:44:59Z

💡 @abdelfetah18 submitted a pull request that claims the bounty. You can visit your bounty board to reward.

algora-pbc · 2024-12-23T13:22:15Z

🎉🎈 @abdelfetah18 has been awarded $750! 🎈🎊

ghik self-assigned this Apr 8, 2024

This was referenced Apr 15, 2024

Reworked Schema to be a complete representation of an OpenAPI 3.1 Schema Object softwaremill/sttp-apispec#152

Merged

sttp-apispec 0.9.0 #3677

Merged

ghik mentioned this issue Apr 24, 2024

SchemaComparator softwaremill/sttp-apispec#157

Merged

ghik removed their assignment Jun 15, 2024

algora-pbc bot added the 💎 Bounty label Oct 16, 2024

vishwamartur mentioned this issue Nov 16, 2024

Add OpenAPI verifier to verify endpoints against OpenAPI spec #4174

Closed

abdelfetah18 mentioned this issue Nov 28, 2024

Implement an OpenAPIComparator softwaremill/sttp-apispec#194

Merged

abdelfetah18 mentioned this issue Dec 19, 2024

Implement OpenAPIVerifier #4215

Merged

algora-pbc bot added the 💰 Rewarded label Dec 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validation of endpoints against a given OpenAPI schema #3645

Validation of endpoints against a given OpenAPI schema #3645

adamw commented Mar 28, 2024

ghik commented Apr 8, 2024

adamw commented Apr 8, 2024

ghik commented Apr 25, 2024

ghik commented Apr 25, 2024

adamw commented Oct 16, 2024

adamw commented Oct 16, 2024

algora-pbc bot commented Oct 16, 2024 •

edited

Loading

Saturn225 commented Oct 16, 2024 •

edited

Loading

adamw commented Oct 21, 2024

kernel-loophole commented Nov 9, 2024

Saturn225 commented Nov 9, 2024

vishwamartur commented Nov 16, 2024 •

edited by algora-pbc bot

Loading

algora-pbc bot commented Nov 16, 2024

algora-pbc bot commented Nov 23, 2024

abdelfetah18 commented Nov 28, 2024 •

edited

Loading

ThijsBroersen commented Nov 29, 2024 •

edited

Loading

adamw commented Nov 29, 2024

promisingcoder commented Dec 12, 2024 •

edited by algora-pbc bot

Loading

algora-pbc bot commented Dec 12, 2024

algora-pbc bot commented Dec 19, 2024

abdelfetah18 commented Dec 19, 2024 •

edited by algora-pbc bot

Loading

algora-pbc bot commented Dec 19, 2024 •

edited

Loading

algora-pbc bot commented Dec 23, 2024

Validation of endpoints against a given OpenAPI schema #3645

Validation of endpoints against a given OpenAPI schema #3645

Comments

adamw commented Mar 28, 2024

ghik commented Apr 8, 2024

adamw commented Apr 8, 2024

ghik commented Apr 25, 2024

ghik commented Apr 25, 2024

Proposed structure for compatibility issues reported by full OpenAPI comparator

adamw commented Oct 16, 2024

adamw commented Oct 16, 2024

algora-pbc bot commented Oct 16, 2024 • edited Loading

💎 $750 bounty • SoftwareMill

Steps to solve:

Saturn225 commented Oct 16, 2024 • edited Loading

adamw commented Oct 21, 2024

kernel-loophole commented Nov 9, 2024

Saturn225 commented Nov 9, 2024

vishwamartur commented Nov 16, 2024 • edited by algora-pbc bot Loading

algora-pbc bot commented Nov 16, 2024

algora-pbc bot commented Nov 23, 2024

abdelfetah18 commented Nov 28, 2024 • edited Loading

ThijsBroersen commented Nov 29, 2024 • edited Loading

adamw commented Nov 29, 2024

promisingcoder commented Dec 12, 2024 • edited by algora-pbc bot Loading

algora-pbc bot commented Dec 12, 2024

algora-pbc bot commented Dec 19, 2024

abdelfetah18 commented Dec 19, 2024 • edited by algora-pbc bot Loading

algora-pbc bot commented Dec 19, 2024 • edited Loading

algora-pbc bot commented Dec 23, 2024

algora-pbc bot commented Oct 16, 2024 •

edited

Loading

Saturn225 commented Oct 16, 2024 •

edited

Loading

vishwamartur commented Nov 16, 2024 •

edited by algora-pbc bot

Loading

abdelfetah18 commented Nov 28, 2024 •

edited

Loading

ThijsBroersen commented Nov 29, 2024 •

edited

Loading

promisingcoder commented Dec 12, 2024 •

edited by algora-pbc bot

Loading

abdelfetah18 commented Dec 19, 2024 •

edited by algora-pbc bot

Loading

algora-pbc bot commented Dec 19, 2024 •

edited

Loading