Found several bugs in the OpenAPI generation results

[dhoffer]

We are trying to upgrade a large JAX-RS REST app to use the quarkus-smallrye-openapi library instead of Swagger libraries but we have run into some blocking issues.

List of issues:

1. JDK types should never have new types with the JsonView suffix. JDK types are just the JDK type, no way to have JsonView modified versions.
2. When JsonView types are created they should not append the full JsonView class hierarchy they should just append the view that was specified. E.g Full, Ingest, Abridged, etc. Not Full_Ingest_Abridged.
3. In the Group entity, @JsonIgnoreProperties should only apply to this entity's use case of Role. Currently, it's global because Group was processed before Role.
4. Adding @Schema() in Role entity on the description field does fix the problem were @JsonIgnoreProperties in Group was removing description globally, but now the @JSONVIEW on description field is being ignored and description field is in all JsonViews. Furthermore we have observed that @JSONVIEW is not respected at all if there is a @Schema annotation.

I have created workarounds for 1 & 2 in our OASFilter but it is not possible to fix 3 & 4 in the filter as the schema is completely broken before it reaches the filter.

I have attached a reproducer project.

We are using Quarkus 3.16.2. This is a blocking issue for us so can't use quarkus-smallrye-openapi library until we have fixes or workarounds for all of these issues.

json-view-schema-example.zip

[MikeEdgar]

Thanks for reporting this @dhoffer . PR #2063 should resolve most, if not all, of the items you've listed.

[dhoffer]

Hi @MikeEdgar thanks, I will checkout the changes. What is the best way to pickup these changes, point to Quarkus 999-SNAPSHOT?

Also what about the #4 issue, Furthermore we have observed that @JSONVIEW is not respected at all if there is a @Schema annotation. This is one of the biggest bugs we found but didn't see that listed as fixed.

[MikeEdgar]

@dhoffer I believe that should be handled by this change [1] where previously the presence of a @Schema without hidden = true resulted in the field being present in a schema. Now, in addition to that condition, the field also must not be ignored by a property in the object that references it.

[1] https://github.com/smallrye/smallrye-open-api/pull/2063/files#diff-f03878ebce8de7e5082ce478c94d31ad3d12659258d32feec9c2720f92360742R739-R740

[MikeEdgar]

To answer your other question, this is not yet incorporated in Quarkus, so you would need to build it locally and add the smallrye-open-api-core and smallrye-open-api-jaxrs dependencies to the Quarkus Maven plugin. I'd like to accumulate a few more fixes before releasing a new version, so perhaps later this week or early next week.

[dhoffer]

Hi @MikeEdgar, we have tested your changes and have some feedback. We are almost there but not quite, let me explain.

First I found I had to add these dependencies to pickup your changes:
<dependency> <groupId>io.smallrye</groupId> <artifactId>smallrye-open-api-core</artifactId> <version>4.0.3-SNAPSHOT</version> </dependency> <dependency> <groupId>io.smallrye</groupId> <artifactId>smallrye-open-api-jaxrs</artifactId> <version>4.0.3-SNAPSHOT</version> </dependency> <dependency> <groupId>org.eclipse.microprofile.openapi</groupId> <artifactId>microprofile-openapi-api</artifactId> <version>4.0.2</version> </dependency>

Of my reported issues, #1, 2 & 4 are fixed. However in our testing #3 still fails (it is still being applied globally).

However now that we have thought about this more, we think from an OpenAPI perspective you should ignore the @JsonIgnoreProperties annotation completely. The reason we think you should just ignore this property is that you have no way to know what to call the modified model for the specific use case. Note there could be lots of usages of @JsonIgnoreProperties and you would have to have different models for each combination.

We now don't think OpenAPI should care about this annotation as this is really just a Jackson serialization issue. We welcome your thoughts on this.

[MikeEdgar]

@dhoffer I think @JsonIgnoreProperties is important for the OpenAPI schema generation as it can be used to hide application-private fields that should never be exposed in the public API.

With PR #2063, the model referenced by a field annotated with @JsonIgnoreProperties will be in-lined like roles below. That is, the referenced type is no longer eligible to be set as a reusable schema under /components/schemas and its use is specific to the single location.

smallrye-open-api/extension-jaxrs/src/test/resources/io/smallrye/openapi/runtime/scanner/special.jsonview-with-ignored.json

Lines 5 to 45 in 458fb2f

	"Group_Full" : {
	"type" : "object",
	"properties" : {
	"id" : {
	"type" : "string",
	"format" : "uuid",
	"pattern" : "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"
	},
	"name" : {
	"type" : "string"
	},
	"createdAt" : {
	"$ref" : "#/components/schemas/LocalDateTime"
	},
	"description" : {
	"type" : "string"
	},
	"roleId" : {
	"type" : "string"
	},
	"roles" : {
	"type" : "array",
	"items" : {
	"type" : "object",
	"properties" : {
	"id" : {
	"type" : "string",
	"format" : "uuid",
	"pattern" : "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"
	},
	"name" : {
	"type" : "string"
	},
	"createdAt" : {
	"$ref" : "#/components/schemas/LocalDateTime"
	}
	}
	}
	}
	}
	},

Let me know if this addresses your concern. Did you see different behavior when you tried the changes in 4.0.3-SNAPSHOT?

[dhoffer]

Hi @MikeEdgar but you are also removing the String description from all of the Role models:

Role_Full: type: object properties: name: type: string id: $ref: "#/components/schemas/UUID"

Where is this part?

@Schema() @JsonView(Views.Full.class) private String description;

[MikeEdgar]

It's not in the unit test, but if I do add the RoleResource from your reproducer project, it looks like it renders properly.

@Path("/role")
class RoleResource {
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @JsonView(Views.Full.class)
    @APIResponse(responseCode = "200", content = @Content(schema = @Schema(implementation = Role.class)))
    public Response getRole() {
        return null;
    }

    @POST
    public Response post(@RequestBody @JsonView(Views.Ingest.class) Role role) {
        return null;
    }
}

The resulting schemas are like this (which I see a bug where description should not be present in the Ingest view).

"Role_Full" : {
  "type" : "object",
  "properties" : {
    "id" : {
      "type" : "string",
      "format" : "uuid",
      "pattern" : "[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"
    },
    "name" : {
      "type" : "string"
    },
    "createdAt" : {
      "$ref" : "#/components/schemas/LocalDateTime"
    },
    "description" : {
      "type" : "string",
      "title" : "Title of description"
    }
  }
},
"Role_Ingest" : {
  "type" : "object",
  "properties" : {
    "name" : {
      "type" : "string"
    },
    "description" : {
      "type" : "string",
      "title" : "Title of description"
    }
  }
}

[dhoffer]

@MikeEdgar The problem with that test is it's not applying what the Group processing is doing first. I will attach the full OpenAPI of this sample project.

Also I don't see any problem with Role_Ingest where you mentioned you did see a bug. The problem I see is that Role_Full is missing description because of the @JsonIgnoreProperties("description") in Group on the Roles list.

This is with running the sample app using your SNAPSHOTs and then downloading the openapi it generates.

openapi.yaml.txt

[dhoffer]

@MikeEdgar One more thought on this. I know you said you feel @JsonIgnoreProperties is important and should be supported to allow precise inline OpenAPI models and we are fine with that but I'd like to see this be configurable. One idea would be for smallrye openapi to allow users to specify the list of annotations that smallrye openapi should ignore.

Our current/prior approach that used Swagger libraries skipped @JsonIgnoreProperties for OpenAPI generation and that was fine for us. We would actually prefer it that way just so there is no change with this upgrade to use smallrye openapi, so having it configurable would be ideal.

[MikeEdgar]

@dhoffer I'm not opposed to making things configurable, but I'd like to understand the use case better. I'm struggling to follow why @JsonIgnoreProperties would be used to hide fields from the actual JSON representations, but then those hidden fields would be visible in the OpenAPI schema for the same objects.

[dhoffer]

@MikeEdgar In our case we really aren't trying to hide anything, it was added to break a circular recursion issue when the data was serialized by Jackson. That wouldn't be a problem in the sample reproducer project but it is in our production system. I believe that is the only reason we have to occasionally use @JsonIgnoreProperties.

[MikeEdgar]

Ok, I see. Either way the result is that the JSON is missing a property that is documented by the OpenAPI schema, no? I'm trying to understand if there is a better/different way to make things work for you.

[dhoffer]

Well, lets back up and first cover the remaining bug. Then we can discuss the idea of it being configurable. The current code does not work because @JsonIgnoreProperties is still being applied globally and it should just be applied to the inline model. That is the breaking part.

Then yes being able to configure smallrye openapi to skip @JsonIgnoreProperties would be nice for our use case as we are fine with it always being in the model but no data exists (it doesn't exist in a deep nested level so users don't want the data at that level anyway). I hope that makes sense.

[JakeH10]

@MikeEdgar The issue we see with @JsonIgnoreProperties depends on the order classes are resolved.

In the reproducer, the issue occurs since the scanner resolves group before role. When the annotation scanner resolves group, it finds the embedded role object so it also resolves role within the context of group. Within this context, the description field is ignored via @JsonIgnoreProperties("description"). Later, the annotation scanner finds the Role class but it believes it has already resolved it, so it skips it.

Without @JsonIgnoreProperties on com.example.jsonview.model.Group#roles:

Role:
  type: object
  properties:
    name:
      type: string
    description:
      type: string
    id:
      $ref: "#/components/schemas/UUID"

With @JsonIgnoreProperties on com.example.jsonview.model.Group#roles:

Role:
  type: object
  properties:
    name:
      type: string
    id:
      $ref: "#/components/schemas/UUID"

[MikeEdgar]

Thanks @JakeH10 , I see the issue and I'll look into the fix.

[MikeEdgar]

@dhoffer @JakeH10 , PR #2065 should resolve the issue with @JsonIgnoreProperties modifying Role_Full globally. If you'd like to try it I can leave the PR open a while longer and merge tomorrow.

[dhoffer]

I'm not able to pull down the PR branch for some reason. Are you able to merge so we can test using main branch?

[MikeEdgar]

I'm not able to pull down the PR branch for some reason. Are you able to merge so we can test using main branch?

The branch is in my fork. I can go ahead and merge and we'll address anything new in a separate change.

[JakeH10]

@MikeEdgar Fields are not being hidden when in-line, i.e. when @JsonIgnoreProperties is used. We currently use @Schema and @JsonIgnore and neither hides the fields when the object is in-line. I have updated Role in the reproducer:

public class Role implements Serializable {

    @JsonIgnore
    @Transient
    protected Logger logger = LoggerFactory.getLogger(Role.class);

    @Schema(hidden = true)
    public String getJsonType() {
        return getClass().getSimpleName();
    }

    @Id
    @JsonView(Views.Full.class)
    private UUID id;

    @JsonView(Views.Abridged.class)
    private String name;

    @JsonView(Views.Full.class)
    private String description;

}

The Group schema is created as:

Group:
  type: object
  properties:
    roleId:
      type: string
    roles:
      type: array
      items:
        type: object
        properties:
          logger:
            $ref: '#/components/schemas/Logger'
          name:
            type: string
          id:
            $ref: '#/components/schemas/UUID'
          jsonType:
            type: string
    name:
      type: string
    description:
      type: string
    id:
      $ref: '#/components/schemas/UUID'

The actual Role reference/schema is created as expected:

Role:
  type: object
  properties:
    name:
      type: string
    description:
      type: string
    id:
      $ref: '#/components/schemas/UUID'

[MikeEdgar]

Yeah, I think the logger should have been ignored from that annotation. Just curious, why not make the logger static?

[dhoffer]

In production we have hundreds of entities and its just easier to have this in the base class.

@Schema(hidden = true)
@transient
protected final Logger logger = LoggerFactory.getLogger(getClass());

Also we use @Schema(hidden = true) quite extensively to hide things from the generated OpenAPI schema so its critical that works.

[MikeEdgar]

@dhoffer the third linked PR (now merged to main) should have resolved this. If you test and find more issue on your end, let me know. Thanks

[dhoffer]

@MikeEdgar We have been testing and this looks good. Thank you for all your help. We will let you know if we find anything. How soon could you make the next release with these fixes? We could use that asap.

[MikeEdgar]

I'll cut release 4.0.3 now.