Generating client for Weather.gov API fails to generate compilable code

[zabelc]

In a new swift project in Xcode Version 15.0 beta 4 (15A5195m), using swift-openapi-generator 0.15 I was attempting to generate a client for the Weather.gov API; however, this fails in a couple ways:

1. Part of the spec contains currently unsupported enum features (though I was able work around this)
2. The resulting client contains numerous compilation errors

1. Spec contains unsupported enum features

Generating a client fails with both Json version of the spec: https://api.weather.gov/openapi.json and the YAML version of the spec: https://api.weather.gov/openapi.yaml

Either way, the build fails with the following error:

Error: Disallowed value for a string enum 'Components.Schemas.Zone.statePayload.Case2Payload (#/components/schemas/Zone/state/case2)': ()
LLVM Profile Error: Failed to write file "default.profraw": Operation not permitted

Workaround
I was able to work around this issue after reading the solution to Issue #102, and reading through the currently supported features: https://swiftpackageindex.com/apple/swift-openapi-generator/0.1.5/documentation/swift-openapi-generator/supported-openapi-features

Based on the currently supported features & the diff solution to Issue #102, I simplified the Zone yaml definition as follows

    Zone:
        state:
          $ref: '#/components/schemas/StateTerritoryCode' # added
#          oneOf:
#            -
#              $ref: '#/components/schemas/StateTerritoryCode'
#            -
#              enum:
#                - ''
#              type: string
#              nullable: true

2. The resulting client contains numerous compilation errors

With the modified openapi.yaml file above, I was able to generate a client successfully, but the Client.swift.txt had two types of compilation errors:

1. try await client.send( error: Type of expression is ambiguous without more context
2. Instance method 'setQueryItemAsText(in:name:value:)' requires that 'Components.Schemas.ISO8601Interval' conform to '_StringConvertible

Error 1: try await client.send( error: Type of expression is ambiguous without more context
This occurred for the following operations:

Generated from `#/paths//alerts/active/area/{area}/get(alerts_active_area)`.
Generated from `#/paths//aviation/cwsus/{cwsuId}/cwas/{date}/{sequence}/get(cwa)`.
Generated from `#/paths//aviation/sigmets/{atsu}/{date}/get(sigmetsByATSUByDate)`.
Generated from `#/paths//aviation/sigmets/{atsu}/{date}/{time}/get(sigmet)`.
Generated from `#/paths//stations/{stationId}/observations/{time}/get(station_observation_time)`.
Generated from `#/paths//stations/{stationId}/tafs/{date}/{time}/get(taf)`.
Generated from `#/paths//offices/{officeId}/get(office)`.
Generated from `#/paths//offices/{officeId}/headlines/{headlineId}/get(office_headline)`.
Generated from `#/paths//offices/{officeId}/headlines/get(office_headlines)`.

Error 2: Instance method 'setQueryItemAsText(in:name:value:)' requires that 'Components.Schemas.ISO8601Interval' conform to '_StringConvertible'

Which occurs for the following operations:

Generated from `#/paths//radar/queues/{host}/get(radar_queue)`.
Generated from `#/paths//radar/profilers/{stationId}/get(radar_profiler)`.

Based on some initial analysis, I don't believe any of the generated code is using the Zone datatypes, so I believe that the workaround above is unrelated to these compilation errors.

[czechboy0]

Hi @zabelc, thanks for filing the issue with this great detail!

The Zone problem is tracked by #118, and even though it's an issue in the underlying YAML parser, we might need to work around it in the generator. In the meantime, you found the correct workaround.

The other issues we'll need a bit more time to look into, I'll update this issue as we identify what features we're missing, and where we're tracking them.

[zabelc]

Hi @czechboy0, I'm not sure about the relationship between the YAML & JSON parsers, but I get the same Zone error above with JSON as well, and I was able to work around the error in the same fashion, by changing the original JSON from the original definition (below) to the simplified definition (further down).

Original JSON:

	"state": {
		"oneOf": [
			{
				"$ref": "#/components/schemas/StateTerritoryCode"
			},
			{
				"enum": [
					""
				],
				"type": "string",
				"nullable": true
			}
		]
	},

Simplified JSON:

	"state": {
		"$ref": "#/components/schemas/StateTerritoryCode"
	},

Unfortunately, I get the same 2 compilation errors.

Just to help with troubleshooting, I'm attaching the openapi.yaml file (with the commented out lines), as well as the original and edited openapi.json files with the changes above.
edited-openapi.json.txt
edited-openapi.yaml.txt
original-openapi.json.txt

[czechboy0]

Thanks for all this detail, @zabelc – I took another look. Unfortunately, with build errors, I don't know how many blockers remain. The current ones are:

• Nullable enums with an empty string fail to get generated #118
• Migrate header, path param, query param serialization to URI coder, deprecate _StringConvertible #144 (I just filed this one)

The first one you were able to work around by folding in the enum, but the second one will need to get fixed in the generator. Once it's resolved, we'll be able to move onto the next blocker. Thanks for your help here 🙏

[zabelc]

Thank you for looking into these issues, @czechboy0! I saw Xcode was updated to Beta 5 & this openapi-generator to 0.16, so I took another stab at the weather service openapi definition, I was able to get it to compile...but I had to comment out many operations.

I also discovered a new (to me) issue wherein the generator has a problem with an object type definition that contains two fields with similar names: @type and type. because .Type is part of swift, both type and @type are generated as _type creating a duplicate definition error.

Error 1: on "try await client.send("
I noticed that the Error 1 on "try await client.send(" above has changed "Type of expression is ambiguous without more context" to "Type of expression is ambiguous without a type annotation"

To get around this I simply removed the following operations:
/alerts/active/area/{area}
/stations/{stationId}/observations/{time}
/offices/{officeId}
/offices/{officeId}/headlines/{headlineId}
/offices/{officeId}/headlines

It strikes me that the generator issue may be due to the fact that the various versions of URL's for the operations above are similar, but have parameters embedded in the URL which aren't being handled carefully enough in generation: i.e. the generator should be adding additional type casting annotations to these methods.

This obvious with respect to /offices/, but there are several variations on the /alerts/ and /stations/ paths as well. (It's easy to see a consolidated list of similar paths under the "Specification" tab at the Weather.gov API Documentation Page).

Error 2: Instance method 'setQueryItemAsText(in:name:value:)' requires that 'Components.Schemas.ISO8601Interval' conform to '_StringConvertible'

I believe you covered this in #144. I worked around it by commenting out the following operations:
/radar/queues/{host}
/radar/profilers/{stationId}

NEW: Error 3: Invalid redeclaration of '_type' in Zone with Protocol Conformance Issues
With the above 7 operations commented-out of the API spec, I was able to generate a Client.swift file successfully! ...but it appears that the generator has trouble with the definition of the "Zone" object declaring _type twice. This additionally causes an upstream error that the Zone struct doesn't conform to the Encodable, Equatable, or Hashable protocols.

It appears that because the Zone object definition contains both @type, and type, the generator is creating two fields in the Zone struct, and the Zone.CodingKeys enum:

            /// - Remark: Generated from `#/components/schemas/Zone/@type`.
            public var _type: Components.Schemas.Zone._typePayload?
. . .
            /// - Remark: Generated from `#/components/schemas/Zone/type`.
            public var _type: Components.Schemas.NWSZoneType?

Here is the definition of the zone object:

    Zone:
      type: object
      properties:
        '@context':
          $ref: '#/components/schemas/JsonLdContext'
        geometry:
          $ref: '#/components/schemas/GeometryString'
        '@id':
          type: string
          format: uri
#First "type" definiton 
        '@type':
          enum:
            - 'wx:Zone'
          type: string
        id:
          $ref: '#/components/schemas/NWSZoneID'
# Second "type" definiton 
        type:
          $ref: '#/components/schemas/NWSZoneType'
        name:
          type: string
        effectiveDate:
          type: string
          format: date-time
        expirationDate:
          type: string
          format: date-time
        state:
#        Fix error: Error: Disallowed value for a string enum 'Components.Schemas.Zone.statePayload.Case2Payload (#/components/schemas/Zone/state/case2)': ()
          $ref: '#/components/schemas/StateTerritoryCode'
#          oneOf:
#            -
#              $ref: '#/components/schemas/StateTerritoryCode'
#            -
#              enum:
#                - ''
#              type: string
#              nullable: true
        cwa:
          type: array
          items:
            $ref: '#/components/schemas/NWSForecastOfficeId'
        forecastOffices:
          type: array
          items:
            type: string
            format: uri
        timeZone:
          type: array
          items:
            type: string
            format: iana-time-zone-identifier
        observationStations:
          type: array
          items:
            type: string
            format: uri
        radarStation:
          type: string
          nullable: true
      additionalProperties: false

I can work around and get everything to compile, by commenting out one definition for "_type" either:

#        '@type':
#          enum:
#            - 'wx:Zone'
#          type: string

or:

#        type:
#          $ref: '#/components/schemas/NWSZoneType'

As long as only one of the above definitions are in the spec, the generated Zone struct will be fine.

I've attached my working openapi.yaml file:
openapi-working.yaml.txt

Underlying issue with handling "type" fields in API object definitions
It seems that the issue is how the generator handles the creation of API Spec fields named "type", since Struct.Type is reserved in swift. The default behavior appears to be to add a _ prefix to type fields, but this creates a collision, with the way @ is also translated to _. (I notice that the definition contains both id and @id which generate to Zone.id and Zone._id.)

It appears that additional logic needs to be added to generated field names in this instance. I was unable to find any open issues similar to this problem, so this seems to be a new issue.

[czechboy0]

Well, you're in luck, because SOAR-0001 just recently landed under a feature flag, and it should address the issue with name collisions!

You can try it by adding this to your config file:

featureFlags:
  - proposal0001

Let me know if that at least helps with the collisions! It'll become the default behavior in 0.2.0

[zabelc]

Hi @CzechBoy, I saw there was a 0.18 version update, so I decided to take another look at the WeatherA API. I can confirm that the SOAR-0001 featureFlag took care of the type collisions and the planned approach in that proposal looks good (though I have to admit that I was unaware that "@" is referred to as "commat"). (Actually, this worked in 0.16 last Friday, but I didn't get a chance to write that up before today.)

After re-processing the API I've discovered a couple errors, and a runtime problem that I can't work around:

1. Nullable enums with an empty string fail to get generated #118 didn't succeed in fixing my "Disallowed value for a string enum"
2. Three brand new types of errors relating to a type made up of oneOf two enums
3. I'm still seeing the "Type of Expression is ambiguous without a type annotation" errors
4. Runtime API calls fail because fields were skipped during generation

1. #118 didn't succeed in fixing my "Disallowed value for a string enum"

I hate to say it but, #118 didn't fix the Zone problem above. As a reminder this spec defines a Zone with a state property as follows:

        state:
          oneOf:
            -
              $ref: '#/components/schemas/StateTerritoryCode'
            -
              enum:
                - ''
              type: string
              nullable: true

Unless I change it to a simple $ref, the openapi-generator crashes saying: Command PhaseScriptExecution failed with a nonzero exit code, with an underlying error of:

Error: Disallowed value for a string enum 'Components.Schemas.Zone.statePayload.Case2Payload (#/components/schemas/Zone/state/case2)': ()

You would know better, but the behavior here is actually crashes the generator before it can generate any code (My "Generated Sources" directory is empty)

FWIW, I worked around this issue in the same way as previously.

Three brand new types of errors relating to a type made up of oneOf two enums

I'm seeing a some brand new errors where the compiler is trying to convert instances of various codes to a Date for some reason. Each of the codes is defined with oneOf made up of two enum types

Version 1

Error: Cannot convert value of type 'Components.Parameters.AlertArea?' (aka 'Optional<Array<Components.Schemas.AreaCode>>') to expected argument type '[Date]?'
Arguments to generic parameter 'Element' ('Components.Schemas.AreaCode' and 'Date') are expected to be equal

This occurs in two operations:

Operation 1:
HTTP GET /alerts.
Generated from #/paths//alerts/get(alerts_query).

Client.alerts_query(_ input: Operations.alerts_query.Input) async throws{
        try await client.send(
            input: input,
            forOperation: Operations.alerts_query.id,
            serializer: { input in
		//Other converters ...	
                try converter.setQueryItemAsText(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "area",
                    value: input.query.area <-Error on "area"
                )

Operation 2:
HTTP GET /alerts/active.
Generated from #/paths//alerts/active/get(alerts_active).

Cleint.alerts_active(_ input: Operations.alerts_active.Input) async throws{
        try await client.send(
            input: input,
            forOperation: Operations.alerts_query.id,
            serializer: { input in
		//Other converters ...	
                try converter.setQueryItemAsText(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "area",
                    value: input.query.area <-Error on "area"
                )

Both /alerts & /alerts/active contain similar opanapi.yaml Parameters Definitions:

#...other $refs...
        -
          $ref: '#/components/parameters/AlertArea'
#...other $refs...

AlertArea is defined as:

  parameters:
    AlertArea:
      name: area
      in: query
      description: "State/territory code or marine area code\nThis parameter is incompatible with the following parameters: point, region, region_type, zone\n"
      style: form
      explode: false
      schema:
        type: array
        items:
          $ref: '#/components/schemas/AreaCode'

Version 2

Cannot convert value of type '[Components.Schemas.AreaCode]?' to expected argument type '[Date]?'
Arguments to generic parameter 'Element' ('Components.Schemas.AreaCode' and 'Date') are expected to be equal

This occurs in three different operations:

Operation 1
HTTP GET /stations.
Generated from #/paths//stations/get(obs_stations).

Client.obs_stations(_ input: Operations.obs_stations.Input) async throws
        -> Operations.obs_stations.Output
    {
        try await client.send(
            input: input,
            forOperation: Operations.obs_stations.id,
            serializer: { input in
                //Other converters
                try converter.setQueryItemAsText(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "state",
                    value: input.query.state <-Error on "state"
                )

Operation 2:
HTTP GET /zones.
Generated from #/paths//zones/get(zone_list).

Client.func zone_list(_ input: Operations.zone_list.Input) async throws{
        try await client.send(
            input: input,
            forOperation: Operations.zone_list.id,
            serializer: { input in
                //Other converters
               try converter.setQueryItemAsText(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "area",
                    value: input.query.area <-Error on "Area"
                )

Operation 3:
HTTP GET /zones/{type}.
Generated from #/paths//zones/{type}/get(zone_list_type).

Client.zone_list_type(_ input: Operations.zone_list_type.Input) async throws
        -> Operations.zone_list_type.Output{
        try await client.send(
            input: input,
            forOperation: Operations.zone_list_type.id,
            serializer: { input in
                //Other converters
                try converter.setQueryItemAsText(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "area",
                    value: input.query.area
                )

The YAML defines the state as follows:

      parameters:
       #Other parameters

        -
          name: state #or area
          in: query
          style: form
          explode: false
          schema:
            type: array
            items:
              $ref: '#/components/schemas/AreaCode'

The YAML defines the AreaCode as:

    AreaCode:
      oneOf:
        -
          $ref: '#/components/schemas/StateTerritoryCode'
        -
          $ref: '#/components/schemas/MarineAreaCode'
      description: 'State/territory codes and marine area codes'

Where StateTerritoryCode, and MarineAreaCode are separate enum's of two character strings.

I was able to work around this by replacing the AreaCode with the following definition:

    AreaCode:
      $ref: '#/components/schemas/StateTerritoryCode'
      description: 'State/territory codes and marine area codes'

Version 3
Unfortunately when I made the correction above, I was presented with two new errors of the following form:

Cannot convert value of type '[Components.Schemas.RegionCode]?' to expected argument type '[Date]?'
Arguments to generic parameter 'Element' ('Components.Schemas.RegionCode' and 'Date') are expected to be equal

Like AreaCode is RegionCode denied as:

    RegionCode:
      oneOf:
        -
          $ref: '#/components/schemas/LandRegionCode'
        -
          $ref: '#/components/schemas/MarineRegionCode'

I'm not sure why this wasn't caught in the first compilation, but I worked around it in the same way as the AreaCode

###4. Runtime API calls fail because fields were skipped during generation

Although I can comment-out parts of the API and get compilable code, the generated code breaks at runtime because numerous fields are skipped during generation, which results in DecodingErrors when the service returns those unsupported fields. (In fairness, 0.16 had 176 unsupported feature warnings, and 0.18 now has just 96, so it's definitely improving)

Could be a way to configure the generated code to ignore unexpected/unsupported fields? This could be at build time via a featureFlag, or at runtime via configuration on the Client.

One particularly troubling instance is that most responses of the Weather API feature an @context property that appears in responses as part of weather API's JSON-LD support. The Weather.gov API spec defines the @context as:

    JsonLdContext:
      anyOf:
        -
          type: array
          items: {  }
        -
          type: object

The generator produces the following warning (30 times):

Schema "array ()" is not supported, reason: "not an object-ish schema (object, ref, allOf)"

The @context response is essentially metadata about the response, so not necessary to a client. Also, per JSON-LD, the context is apt not to have a firm data schema or one that would be shared across all types of responses.

In this case, I wonder if array() could be supported by delegating to an underlying JSONSerialization object.

[czechboy0]

Hi @zabelc,

Disallowed value for a string enum

Yes, that only got fixed earlier today, but the fix hasn't been released yet, but you can try with the generator on the main branch: #176

Schema "array ()" is not supported

I got a WIP PR for that up which I hope to land tomorrow: #179

The issue with a query parameter that is an array of a oneOf, where each schema is a string enum, which is tracked by #144, which hasn't been started yet.

Thanks for checking every new version and for your patience, we'll get there 🙂

[czechboy0]

Thanks for your patience, @zabelc - I just confirmed that with Swift OpenAPI Generator 0.1.13, the Weather.gov OpenAPI document produces compiling code.

Note that you'll need to enable all these feature flags, e.g. in your openapi-generator-config.yaml, by adding:

featureFlags:
  - proposal0001
  - multipleContentTypes
  - strictOpenAPIValidation
  - closedEnumsAndOneOfs

They will become the default in the upcoming 0.2.0 as well, at which point you won't need to explicitly specify them.

But you should be unblocked now, please file new issues if you find anything else.