tools/importer-rest-api-specs: refactoring the Parser package
#4307
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tombuildsstuff
changed the title
tools/importer-rest-api-specs: refactoring the Parser package
manicminer
added
tool/importer-rest-api-specs
stephybun
previously approved these changes
Jul 31, 2024
|
|
||
| Where possible we use the terminology `API Definition` since these are implementation details - but this more specific terminology will be used where necessary. | ||
|
|
||
| TODO also mention Supplemental Data |
There was a problem hiding this comment.
@manicminer do you have context on this?
manicminer
force-pushed
the
refactor/importer-rest-api-specs-iv
branch
2 times, most recently
from
5585fc2 to
273f87f
Compare
stephybun
reviewed
Oct 1, 2024
tools/importer-rest-api-specs/internal/components/apidefinitions/parser/parsingcontext/build.go
Outdated
Show resolved
Hide resolved
tools/importer-rest-api-specs/internal/components/apidefinitions/parser/parsingcontext/build.go
Outdated
Show resolved
Hide resolved
tools/importer-rest-api-specs/internal/components/apidefinitions/parser/parsingcontext/build.go
Outdated
Show resolved
Hide resolved
| // if this is a parent model, look for implementations. this will currently only work for children defined in | ||
| // the same swagger file. |
There was a problem hiding this comment.
Do we handle the case where the children are defined in a different swagger file somewhere else in build.go?
Context: #3727
There was a problem hiding this comment.
Yes, this should get picked up by https://github.com/hashicorp/pandora/pull/4307/files#diff-d2cb55dda7e9facde77a39139c64afb00da29f5f92150f1e9eec14766b11c4b1 (which I think is your original implementation)
manicminer
force-pushed
the
refactor/importer-rest-api-specs-iv
branch
from
f57a3a2 to
23eccdf
Compare
manicminer
added a commit
that referenced
this pull request
Oct 1, 2024
…ers out into the new structure
…a Data Workaround
…e over to using the SDK types/new path This is a copy and patch up, since I want to leave the existing code path as-is for the moment, to break it apart more effectively
… Service/APIVersion
…rkarounds` package
…age into the new structure
…to the new package This wants refactoring, but moving it over allows cleaning the rest up
…kage There's a bunch of these, and whilst these'll get replaced in time with #4017 but for now this cleans this up
…licates (avoid parsing the same tag twice)
This reverts commit 31b45ca.
… more explicit" This reverts commit 71a166f.
… `go-openapi/jsonreference`, and `go-openapi/spec`
…resolve all remote references
…L-decoded when parsing from the Ref URL fragment
…pect full ancestry
…ame set of resources
…der when parsing operations, as this was never utilised and leads to skipping resources/operations
stephybun
force-pushed
the
refactor/importer-rest-api-specs-iv
branch
from
d9f992a to
1da6920
Compare
This was referenced Oct 10, 2024
zaowang-ms
pushed a commit
to zaowang-ms/pandora
that referenced
this pull request
Oct 22, 2024
…icorp#4307) * `tools/importer-rest-api-specs`: refactoring the `commonschema` Matchers out into the new structure * `tools/importer-rest-api-specs`: refactoring the Constant package * `tools/importer-rest-api-specs`: porting the Network workaround into a Data Workaround * `tools/importer-rest-api-specs`: porting the `dataworkarounds` package over to using the SDK types/new path This is a copy and patch up, since I want to leave the existing code path as-is for the moment, to break it apart more effectively * `tools/importer-rest-api-specs`: scaffolding out the logic to parse a Service/APIVersion * `tools/importer-rest-api-specs`: porting over the "remove unused items" logic * `tools/importer-rest-api-specs`: adding a shim to the updated `dataworkarounds` package * dependencies: updating to `v0.69.0` of `github.com/hashicorp/go-azure-helpers` * `tools/importer-rest-api-specs`: refactoring the `normalization` package into the new structure * `tools/importer-rest-api-specs`: moving all of the `cleanup` items into the new package This wants refactoring, but moving it over allows cleaning the rest up * `tools/importer-rest-api-specs`: adding inner most sdk operation object definition * `tools/importer-rest-api-specs`: moving the `commonids` into a subpackage There's a bunch of these, and whilst these'll get replaced in time with hashicorp#4017 but for now this cleans this up * `tools/data-api-sdk`: removing an empty todo file * `tools/importer-rest-api-specs`: refactoring more logic out to the new package The `parser` package itself needs cleaning, but there's a few things to do first * `tools/importer-rest-api-specs`: moving another `ignore` function * WIP * tools/importer-rest-api-specs: reimplementing hashicorp#4290 atop the rebase The rebase is messed up, but it's a minor patch so should be fine * `tools/importer-rest-api-specs`: adding a placeholder for parsing an APIResource * `tools/importer-rest-api-specs`: refactoring the parser into smaller packages This leaves the old parser logic in place for now, which'll be cleaned up in the next commit * `tools/importer-rest-api-specs`: fixing an issue where the `validate` command didn't pass the API Definitions Directory correctly * `tools/importer-rest-api-specs`: title-casing the ReferenceName - which ensures that when we compare the ObjectDefinition from a previously parsed/currently parsed dataset we have matching values These are normalised post-parsing, so when comparing against the old data set we need to use TitleCase too, so we might as well do this here * `tools/importer-rest-api-specs`: ensuring that TypeSpec examples are filtered out * `tools/importer-rest-api-specs`: trimming `*.json` from the filename to build the api resource name * `tools/importer-rest-api-specs`: removing (most) of the old code path * `tools/data-api-sdk`: making IsCommonType required so this is more explicit * `tools/importer-rest-api-specs`: fixing the validation * `tools/importer-rest-api-specs`: re-enabling most of the tests * `tools/importer-rest-api-specs`: consolidating/porting over the data tests These ensure that the Configs can be parsed and that the Data from them can be loaded * `tools/importer-rest-api-specs`: fixing the test format * tools/importer-rest-api-specs: skipping the local test * `tools/importer-rest-api-specs`: fixing the non-discriminator tests * `tools/importer-rest-api-specs`: updating/reworking the supplementary data test @manicminer is going to work through fixing this one up, this requires parsing the data identifying any parent types and patching any disciminated implementations across * `importer-rest-api-specs`: fix up discriminator handling for datafactory * `importer-rest-api-specs`: shelve SupplementaryData support for now, since it doesn't seem to be required for DataFactory models as they use external Swagger Refs. * `importer-rest-api-specs`: implement `-services` argument * `importer-rest-api-specs`: operate on maps instead of passing them around * `importer-rest-api-specs`: normalize swagger tag names to prevent duplicates (avoid parsing the same tag twice) * `importer-rest-api-specs`: normalize API resources after parsing all of them * `importer-rest-api-specs`: ensure new APIResource is populated * `importer-rest-api-specs`: maps are pointers * `importer-rest-api-specs`: dereference more maps * `importer-rest-api-specs`: remove use of strings.Title, dereference more maps * limit scope of SDK generation test to the SDK being tested * `importer-rest-api-specs`: two loops become one * `importer-rest-api-specs`: workaround inconsistent casing of "VirtualWANs" * Revert "`importer-rest-api-specs`: two loops become one" This reverts commit 31b45ca. * `importer-rest-api-specs`: normalize "attachednetworks" segment of resource IDs * Revert "`tools/data-api-sdk`: making IsCommonType required so this is more explicit" This reverts commit 71a166f. * importer-rest-api-specs: use updated forks for `go-openapi/analysis`, `go-openapi/jsonreference`, and `go-openapi/spec` * importer-rest-api-specs: remove Supplementary Data parser as is not used/needed * importer-rest-api-specs: rework the swagger spec parser and manually resolve all remote references * importer-rest-api-specs: fix format string, ensure model names are URL-decoded when parsing from the Ref URL fragment * importer-rest-api-specs: updated tests for discriminated models to expect full ancestry * importer-rest-api-specs: remove vestiges of supplementary data spike * importer-rest-api-specs: do not set the description, see hashicorp#4434 * importer-rest-api-specs: hack for conflicting constant names in the same set of resources * importer-rest-api-specs: disable check for non-default Resource Provider when parsing operations, as this was never utilised and leads to skipping resources/operations * upper case URL in the go sdk --------- Co-authored-by: Tom Bamford <[email protected]> Co-authored-by: Steph <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the Parser package into
apidefinitions- and removes the old logic/pipelineRight now this mostly works and requires a few fixes around Discriminators/Supplementary Data that @manicminer is going to pick up
Mostly fixes/rounds out #3754