-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve OpenAPI request/response naming strategy #18321
Closed
Closed
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
45aee60
Improve OpenAPI request/response naming strategy
averche 691e29b
default mappings
averche 1fad3a1
change the mapped names
averche ed954bf
add operationImplied flag
averche a1058a5
Add a test
averche 7a9b75d
changelog
averche 59a692f
requestResponsePrefix -> defaultMountPath
averche 7a39ca9
update -> write
averche 3b90f9a
fix test
averche 91a5134
undo test change
averche 19d5ef0
move the map into openapi_path_mappings.go
averche 2375e5e
Update table with prefix/operation/suffix struct
averche 0d14048
Fix function and test
averche 4d5ccb9
test
averche ac7fe27
mountPathWithPrefix
averche 0be96b8
mountPathWithPrefix
averche a0a65bf
comment
averche 8e34fda
Adjust certain path mappings
averche 77684f5
undo script changes
averche dc6fb71
Merge branch 'main' into ui/openapi-request-response-names
averche dd5767c
spaces
averche 1e6bfe2
don't need path lib
averche 70ec341
dedup
averche 5165291
change KVv1/KVv2 naming structure
averche f1b79a1
Merge branch 'main' into ui/openapi-request-response-names
averche d8d4413
Add back CreateOperationIDs
averche 9ad60e9
comment
averche 66a741b
rename constructOperationID and address PR feedback
averche 9088087
fix tests
averche f1d7eff
add updateOperationOnly flag
averche 27da1e3
fix test
averche 98dae14
remove kv -> secret hack
averche File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
```release-note:improvement | ||
openapi: Improve request/response naming strategy | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I lack the permissions to unresolve the previous conversation, so I need to copy/paste my comment into a new one instead:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair enough, I suppose I can disambiguate the names in some other way. It's just a shame that the libraries will have multiple methods generated which will result in exactly the same behaviour.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The more I think about it, the more I think the library code generation is going to need a way to apply tweaks to the canonical OpenAPI document, to vary the generation process. Let me explain further:
Today, as you have identified, there are various APIs in Vault which are deprecated and replaced by newer APIs. It would be nice if we could exclude the historical cruft from the generated client libraries, whilst it remains important to still acknowledge the APIs exist, to help people understand existing code, and understand the impact of wildcards in policies.
We could (and IMO should) add a simple "deprecated" flag to deprecated APIs, and use that to render them as crossed out in the API explorer UI (something Swagger UI already supports). Then, at the birth of a new autogenerated client library, it would make sense to exclude all these.
But wait! This only works out nice and simply, for the first version of the autogenerated client library. Users expect API stability from their libraries. If we simply exclude all deprecated APIs, then the moment any future API deprecation in Vault would result in the API being removed from the next version of the autogenerated client library, breaking compatibility.
The problem is actually deeper still. There may be a bugfix to how an API is represented in Vault, but that may trigger an API removal in an autogenerated client library if we're not careful. Here is an example:
There is currently this path regexp with the PKI secrets engine:
"keys/generate/(internal|exported|kms)"
and this currently generates three separate APIs. However, the same three values are represented in multiple other PKI secrets engine APIs in this other style:"issuers/generate/intermediate/"+framework.GenericNameRegex("exported")
where the parameter captured by theGenericNameRegex
must have one of the three values,internal
,exported
orkms
. This other pattern only generates one OpenAPI operation. I intend to submit a PR to make the PKI secrets engine internally consistent in how it handles this parameter. When I do, what is currently three operations in the OpenAPI will change to one. If this was after version 1 of an autogenerated client library, there would be a need for the library to maintain deprecated compatibility implementations of the old operations!There are lots more examples to be found in my #18554 PR - various incorrect parameters being fixed or removed, for example.
In summary, then: the OpenAPI straight from Vault is a great starting point for client library generation, but if the library's API is to meet the stability guarantees a developer would hope for, it will be an important requirement to be able to support library specific overrides.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I definitely agree that we will want to support some sort of library-specific overrides. As I mentioned before we have been entertaining an idea of a dedicated repository for generating and hosting
openapi.json
files. The repository will likely contain two flavours: one for library generation and one for documentation purposes. It is likely that the repository will usesdk/openapi.go
directly (and without loading the actual plugins) rather than going though the API endpoint.You bring up very good points regarding deprecation & backwards compatibility issues. Since our conversation last week, I had some time to think about it. In the case of an API endpoint being deprecated and removed from the library-specific API, it should definitely result in a new major version of OpenAPI-for-libraries spec (and consequently a new major version of the libraries). This seems to be a fairly established pattern for OpenAPI-generated libraries. For examples Stripe's go library is currently on major version 74 using OpenAPI version v217. The client code does not need to upgrade right away but if an when they do, they could expect some breaking code changes due to the volatile nature of code generation.
Regarding bug fixes resulting in removal of certain paths, hopefully we can fix all such issues before the 1.0 release. If not, it is still not catastrophic as we can simply bump up the major version for both OpenAPI spec and the generated libraries.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fast-paced new major versions of libraries does work particularly well in the Go world, since with Go modules, a final compiled application can depend upon multiple major versions if it needs to.
In other programming language ecosystems, such as .NET, each compiled application can only depend on one version of each library, so if application A depends on libraries B and C, and B and C need incompatible versions of D, this is a problem. (The "diamond dependency" problem, as if you draw out the dependency graph between A, B, C and D it forms a diamond shape.)
Because of this, what's considered as perfectly reasonable for Go may be seen as less so for .NET. Interestingly, stripe-dotnet has had 30 major versions in the same time stripe-go has gone through 46!
In Vault and Stripe's cases, this might be OK, as both tend to be mostly used for custom business logic for a specific use-case, and unlikely to be required from multiple places deep in a dependency tree, as a common utility library would be.
One thing that might be worth considering, is making sure the library generation system has the capability to apply tweaks on top of the canonical OpenAPI document fetched from the Vault project. As an example, let's imagine that Vault 1.14 hypothetically cleans up the specification of some APIs, in a way which is perfectly compatible at the HTTP layer, but results in incompatible changes to generated code - whilst at the same time adding new APIs that you want to release to users ASAP. You might approach this by keeping a manually curated "overlay" or "patch" in the library generation repo, which temporarily preserves the older forms alongside the newer forms, until such time you need to declare a library major version increment in the future.