Support for path parameters which can contain slashes

[dimonomid]

I see there are a few issues which ask for the opposite, e.g. this one swagger-api/swagger-js#280 , but what I'm asking for is to add some option to the parameter, which would allow it to contain non-encoded slashes.

My use case: I have an API which allows any arbitrary path to be passed in, for example all of these:

• /api/tags
• /api/tags/foo
• /api/tags/foo/bar/baz

Are valid paths. I tried to describe it as follows:

  /tags{tag_path}:
    get:
      parameters:
        - name: tag_path
          in: path
          required: true
          type: string
          default: "/"

However, https://generator.swagger.io encodes slashes in the path, so it doesn't work. So is there a way to describe my API? It feels to me like a perfectly valid use case, so I'm surprised it's not supported.

If it's indeed not supported, then one possible solution is to add some option for the path parameter, which would allow it to contain non-encoded slashes.

What do you think?

[webron]

The request is not new, and came often around the request to support RFC 6570. At the moment, we've decided to not support it, though we'll probably re-evaluate in the future.

[dimonomid]

I see, thanks for clarifying. However, what were (are) reasons not to support it?

[webron]

As with other features, one of the things we keep in mind is tooling support, and we felt that it would complicate the requirements for OAS 3.0 compliant tools.

[darrelmiller]

@dimonomid Not allowing unescaped slashes in parameter values is for the same reason we don't allow optional path segments. Some tooling needs to be able to deterministically map a URL to an operation. With the current limitations it is easy to match the path segments in the URL to those defined in the operation because the path segments are fixed and the only ambiguity arises when a parameter value matches a constant, and IMO the constant should always win.

However, as soon as we introduce optional path segments then we have the following issue:

Given the URL path:

 /foo/bar/baz

Any one of these paths could be a match (using RFC6570 syntax for illustration only)

      "{/segments}"
      "/foo{/segments}"
      "/foo{/segments}/baz"
      "{/segments}/baz"
      "{/segments}/bar/baz"
      "/foo/bar/baz{/faz}"
      "/foo/bar/baz"

Which then requires us to introduce a priority based matching system, which can be really challenging to debug. Should we define a priority algorithm? Should the definition writer be explicit about priority? Should we use document order?

I think we do need to solve this eventually, as well as using query parameters to distinguish operations. But until we have a very well defined priority algorithm that is easy to understand, I believe we need to keep the constraints in place.

[t1]

Which then requires us to introduce a priority based matching system, which can be really challenging to debug

Isn't this a problem that the REST frameworks already do solve? OpenAPI only has to follow their lead so the documentation matches the implementation, right?

[darrelmiller]

My understanding is that some frameworks don't address it, some do, but many do it in different ways. There is no standardized algorithm for deciding on the right match. Even RFC 6570 punts on this:

Some URI Templates can be used in reverse for the purpose of variable
matching: comparing the template to a fully formed URI in order to
extract the variable parts from that URI and assign them to the named
variables. Variable matching only works well if the template
expressions are delimited by the beginning or end of the URI or by
characters that cannot be part of the expansion, such as reserved
characters surrounding a simple string expression. In general,
regular expression languages are better suited for variable matching.

[t1]

Sure, there's no standard. And it can be a problem. But in practice I rarely encountered this to be a problem. As a developer I think about valid URIs and try to stay away from any ambiguities: They are problematic in the implementation, not in the documentation, are they?

[darrelmiller]

@t1 One of the goals we keep reminding ourselves of when updating this spec is we want it to be as straightforward as possible for people to build tools around it. If we define behaviour but leave ambiguities then we could end up with a ecosystem of tools that are not interoperable. That would not be good.
Yes, you an individual usage of OpenAPI can avoid these ambiguities, but ensuring everyone avoids them in the same way is not nearly as easy.
We do want to achieve this, we just couldn't fit it into the 3.0 timeline. I can assure you that when we designed the current URI template support, we made sure adding the / prefix would be an easy non breaking change. We will simply introduce a new style value.

[wohnemus]

I believe this misses an important point regarding REST APIs, however. It is critical, in fact, that we make our REST APIs discoverable. What does this mean? It means that our Paths are going to be variable and discovered at run-time. Using hypertext (HATEOAS principle) combined with a properly constructed OPTIONS response should make a REST API truly discoverable. Prescribing a specific path definition in the specification actually precludes this sort of dynamic discovery. Having a variable path enables this sort of discovery. It does mean that debugging may be more difficult. It also means that current tools may not be able to understand the specification, but both of those are implementation requirements not design requirements. OpenAPI, I presumed, was a design specification and that as such should be tailored to the needs of REST API design including fully Level 3 compliant (Richardson maturity model) REST APIs. Insisting on a fixed path like this does seem to go against these concepts. APIs we are developing are intended to be RMM level 3 compliant and discoverable. We also use OpenAPI and this creates a conflict for us. We'd hoped that in version 3 this would have been resolved.

[darrelmiller]

@wohnemus OpenAPI has never had the goal of describing a dynamic hypermedia API. The idea of a static API description and hypermedia are contradictory goals. I spent 6 years building a hypermedia based system and had no need for something like OpenAPI.

Most people building what they describe as a REST API are not trying to build a hypermedia API. They are building a HTTP API, which OpenAPI does a good job of describing. OpenAPI v3 introduces the notion of linking, but it is a design time linking concept, not a runtime one.

There are a number of people in the OpenAPI community who would like to see more support for hypermedia style APIs. We may be able to do something to help that in a future release, but as I've said before, the last thing I want is to have OpenAPI do hypermedia badly. And considering OpenAPI is all about communicating to a client an exact description of how a HTTP API works, and hypermedia is all about following your nose, reconciling those two successfully is not going to be easy.

If you have ideas as to how client/server coupling can be kept at a minimum whilst still using OpenAPI, I'd love to hear it. If we can do it successfully, then I will be fully behind it.

[handrews]

@wohnemus if you want a fully dynamic HATEOAS-driven REST API, I encourage you to come over to the JSON Schema project and help us out with JSON Hyper-Schema, which explicitly has those goals as opposed to the static description goals of a service definition like OpenAPI. The two projects serve different purposes (which @darrelmiller nailed perfectly in his hypermedia API vs HTTP API distinction) and I think Hyper-Schema may be more suited to your needs. We are doing a lot of work on JSON Hyper-Schema right now.

I hope that eventually as JSON Hyper-Schema matures, OpenAPI will find ways to adopt it, but Hyper-Schema is definitely not there yet. We're still working out the best practices and features for hypermedia APIs, and it's probably best that we do that separate from any larger project targeting static description such as OpenAPI.

[wohnemus]

@darrelmiller I understand the goals of OpenAPI but it is also true that vendors like WSO2, IBM and many others are adopting OpenAPI as the standard for their tools. As an API developer I need to use such tools for REST APIs, including those that are dynamic. This creates that conflict. @handrews we'd love to join in the discussion there too. We are making use of the HAL-JSON specification for this purpose. The schema definition and design married with OpenAPI specification will help support dynamic APIs.

[handrews]

@wohnemus and anyone else interested in hypermedia description: we're in the final review phase for publishing Draft 06 for the next (probably) 2 weeks or so, but after that I hope to dig into Hyper-Schema a lot for Draft 07. Our main limitation with Hyper-Schema is that we have very few people who are really looking to use it in actual HATEOAS-driven APIs, so getting more actual potential users in the discussion would help us move it forward.

[t1]

@darrelmiller: Maybe the spec should just ban ambiguities, so tools will mark them as errors. If somebody chooses to ignore that, some implementation may still work due to priority management... but that's completely outside of the standard (and utterly confusing).

[webron]

Everyone, just so we're on the same page - the @OAI/tdc has decided this will not be supported in 3.0. This is not to say the requirements and needs are not clear, but we can't afford to reopen it for discussion. It's time for us to make the cut for what gets in and what not.

We encourage further discussions so we can see if/how this gets in future versions.

[bassmanitram]

@handrews You'll be glad to know that Bill ( @wohnemus ) and I (and our teams), are heavily using JSON Schema and JSON Hyperschema in our API definitions and documentation - including in the dynamic description of what the API can do. We'll be coming over with great alacrity :)

@darrelmiller Exactly the approach I was about suggest. AWS API Gateway uses OpenAPI for its API descriptions and simply allows you specify an API to a certain depth and then have a path parameter described as

/seg1/seg2/{restOfPath+}

This basically says ... well it's obvious: "swallow the rest of the path". Yes, I know this isn't standard OpenAPI nor is it standard OpenAPI Extension syntax - but they saw the need and did it. It avoids ambiguity because, after that, you aren't allowed to create ambiguous paths.

IBM's API Connect also uses OpenAPI - but has not chosen to allow this feature.

@darrelmiller Darrel, first thanks for your efforts - a thankless task! I completely get your point that the level of dynamism we're going for doesn't require OpenAPI per sé, but, as the two product citations above imply, in order to get our APIs published and marketed (maybe that's a dirty concept here - sorry :)) we need to put them in a "standard" gateway and the developer portal facilities that those gateways may provide - and that usually means conforming to the best declarative standard out there - OpenAPI. So we end up being constrained by the number of extensions (and liberties) that the implementors add to OpenAPI.

Obviously AWS has taken liberties to get it to work, whereas IBM has constrained itself to the OpenAPI standard in this case, meaning we have to work around it (figure out your longest path, then describe a path parameters for each segment in that path - yuk).

So (and admittedly not having thought this through to the extent I know you'll be able to) would something "restrictive" like AWS's solution be possible as a first pass in, say, version 3.1?

[darrelmiller]

@bassmanitram 6570 URI Templates support what you describe,

/seg1/seg2{/restofPath}

We can easily add this syntax by introducing a new style value. That's not a problem.

I do understand the need. I work on Azure API Management and we support templates like /seg1/seg2/* that does the same thing, and we also use OpenAPI to describe our APIs. So, we have the challenge of OpenAPI not being able to describe some APIs that our customers are defining.

Only using wildcards at the end of a path is one way to limit the ambiguities and it does cover a fairly significant number of use cases. However, it would be nice to find a way to address this in a more general way so that we can solve it for query parameter discrimination too.

Once we get 3.0 out of the way, you can be sure this issue will be high on my list of priorities.

[bassmanitram]

@darrelmiller Re RFC 6570: Exactly. I mentioned AWS's approach simply as an example of a provider who facilitates the functionality in their product. Their syntax is clearly not 6570-compliant, more regex-inspired I suspect.

All understood (Azure API Management is in two weeks time for me - thanks for the heads up :)).

Thanks

[handrews]

@bassmanitram great! Please feel free to introduce yourself on our quiet but still functional mailing list or grab my email address out of the the current drafts on master and say hi :-)

[dimonomid]

So this is not going to be supported soon, and I have to resort to a workaround.

If I have a path /tags{tag_path} and I enter /foo/bar as tag_path, then the actual query request URL will be: /tags%2Ffoo%2Fbar. So, I just added support for that on my backend: the endpoint handler for /tags* urldecodes the path (which is %2Ffoo%2Fbar), and it becomes /foo/bar again.

Yes, a hack, but it works, and it's better than nothing. In my case, tag names can't contain the / character, so there's no conflict. Your mileage may vary, of course.

[rsleggett]

We took a different approach and changed our API to use query parameters instead.

[ePaul]

@wohnemus I described a way of having hypermedia support with an extended variant of OpenAPI in #577. I just need to get to implementing it ...

[bassmanitram]

@ePaul WE ( @wohnemus me and others from our org) could have written you first couple of posts in that issue - it's spot on for what we too are trying to achieve. Now need time to digest it and start commenting on it - first glance looks very promising.

[afocus]

I need the same functionality too

[darrelmiller]

@Joyce-Stack If you are using a DOI as a path parameter then the slash in the DOI should be escaped as it is part of the data element, not part of the resource hierarchy. This is fully supported. It is the case where you people want the parameter to actually define the resource hierarchy that currently isn't supported directly.

[darrelmiller]

@garymazz Is there a problem with the extension solution that I suggested in July? Or is the problem that tooling hasn't implemented support for such an extension?

Also, the moonwalk proposal https://github.com/oai/moonwalk advocates for using for RFC 6570 URL templates, which would support multi segment paths.

I'm not exactly sure what you expect to gain from insulting the efforts of volunteers.

[betonetotbo]

Another good argument to support this is trying to answer this:

• How can I specify a proxy server using OpenAPI?

I have a scenario currently, that I created a microservice backend to handle presigned urls to give secure access to thirty-party integration to APIs that currently require authentication tokens. This are scenarios that the thirty-party software has not possibility to negotiate to obtain tokens.

So, in this case, I need to define an path like this:

• /presigned/{presignedAuthorizationHashKey}/*

The thirty-party will request something like: https://host.com/presigned/AS0DF90ASD9F0ASDDDAFFD/2232/details

[mimkorn]

I'd like to confirm if my understanding is correct that only server-side generation is issue from the practical point of view, not the spec itself.

I'm exploring an API design where a path parameter at the end of the URL can contain slashes, as illustrated below:

NodePath:
  name: path
  in: path
  required: true
  schema:
    type: string
  example: some/path/

We've implemented our server accordingly and observed that generated clients we tried handle these path parameters without issues, ensuring correct server responses. My understanding is that the challenge with slashes mainly pertains to server-side code generation, which we don't rely on.

Could someone confirm if there are other potential pitfalls with this approach, or if the concern is indeed primarily around server-side code generation? Maybe we'd have problems with linters/validators/mock servers/validation proxies, etc..?

The endpoint may look something like this:
@Path("/tables/{table}/platform/{path:.*}")

The OpenAPI specification explicitly states that path parameters must not contain any unescaped "generic syntax" characters such as forward slashes (/), according to RFC3986​1​. But I'd like to know if we may encounter some issues I cannot forsee right now by deliberately ignoring it for this case.

Thank you for your insights!

[oyvindwe]

if the concern is indeed primarily around server-side code generation?

Client side code generation as well, for example API browsers as SwaggerUI. A / in the path parameter will be URL encoded in this case.

If you have control of the toolchain on both the client and server side, there is no problem, but if you want to leverage standard tool support you are in trouble.

[hunjixin]

also the same problem too. need feature like github /blob/{branch}/a/b/c/d.go. any plan for this issue?

[Nikoolayy1]

It seems like /niki/{test+} or /niki/{test++} or /niki.* or /niki/{test=*} or /niki/{test=**} are vendor specific workarounds for wildcard urls for the openapi specification and some vendors support this, some don't and we are still waiting this to be added to the official Openapi spec.

[pkit]

@Nikoolayy1 which vendors though? Lol
So far I haven't found any support anywhere.

[Nikoolayy1]

@pkit F5 BIG-IP AWAF/ASM supports wildcard url in a Openapi/Swagger https://community.f5.com/t5/technical-forum/f5-awaf-asm-support-for-wildcard-url-and-parameter-names-with-a/td-p/322207 in the form of "/{path=**}": . Also here https://stackoverflow.com/questions/42335178/swagger-wildcard-path-parameters Connexion is mentioned.

[handrews]

This has been rolled up into #2653, which in turn will almost certainly be moved over to the Moonwalk (OAS 4) project, which as currently proposed supports full RFC 6570 URI Templates.

[garymazz]

@handrews
Thank you for doing this work moving it over to Moonwalk !!
Is there a initiative for a POC for this feature or is this waiting to the specification update, as I've been told previously ?
How/who is coordinating the implementations between the parser and the generators ?

[handrews]

@garymazz we're not quite to that point with Moonwalk, although we have set an aggressive schedule to release it by the end of 2024. The Moonwalk-specific community calls will be starting at 9AM US-Pacific time on Tuesdays (I think this coming Tuesday will be the first one). We'll be working out details there.

[garymazz]

@handrews
Thank you for the quick reply.
This new work is timely, I'm starting to implement a new, open source version of SNIA CDMI ISO standard. See: https://www.snia.org/cloud/cdmi#:~:text=CDMI%20is%20an%20open%20international,that%20is%20placed%20in%20them.

I would like to use Moonwalk for the server side implementation (C language). I'll try to attend the meetings, I do have another standards meeting conflicting with your 0900/9am PST, I'm trying to resolve. I'll definitely be on upcoming meeting.

[handrews]

@garymazz we're talking about having some alternative time slots, for time zone coverage among other things. If you're not on our Slack you can join using the link in the "Participate" menu on the main OpenAPI web site. The spec and sig-moonwalk channels will be the ones you want in addition to general in order to get updates on this stuff.

[garymazz]

@handrews Thanks, just joined. # spec is a default channel. # sig-moonwalk needs to be manually added.