Invalid URL (jsonschema RefResolutionError)

[ghost]

We probably should install an extra handler for jsonschema, one that prepends file:// and resolves the current directory to all relative file name references. The relevant code that errors out:

    # jsonschema.validators
    def resolve_from_url(self, url):
        url, fragment = urldefrag(url)
        try:
            document = self.store[url]
        except KeyError:
            try:
                document = self.resolve_remote(url)
            except Exception as exc:
                raise RefResolutionError(exc)

        return self.resolve_fragment(document, fragment)

The code in question:

openapi: 3.0.1
servers:
  - url: 'https://example.com/api/v2.0/'
info:
  title: Example API
  description: Our awesome example API
  version: 1.1.0
tags:
  - name: Service
paths:
  $ref: './paths.yaml'

I have also tried file://./paths.yaml, which then results in it looking for /paths.yaml, which is an error of the file resolver jsonschema is using. Either way, we currently cannot resolve any relative file references, which are allowed and correctly working in other tools, including openapi-spec-validator. The latter is a bit odd, since it should be resolving references and validating them.

[ghost]

Tried a quick-fix by providing a spec_url as base, using a file:// url, ending in slash, but that starts going wrong, when you move up directories in your spec. For example:

# spec_uri is set to /home/spec/
/home/
    |- spec/
        |- common/
            |- schemas.yaml
    |- service/
        |- responses.yaml
            - $ref: '../common/schemas.yaml#/PermissionDenied'

The ref will be looked for at /home/common/schemas.yaml instead of /home/spec/common/schemas.yaml. So we will need a resolver that is aware of the file that is making the reference and translates accordingly.

[ghost]

Working on a solution, but I think the bug is with jsonschema. While it stores the referring document, it is never passed to the resolving handler. It's only stored on the object and not used or referenced anywhere else it seems.

[ghost]

The base_uri is required for create_spec if you want to use relative references. This works as advertized, but docs needed and exception message can use a hint. PR forthcoming.

[berislavlopac]

I have ran into another related problem: if a remote has internal references, the validator tries to locate it on the main document.

For example, with the main spec:

openapi: 3.0.2
info:
  title: FooBar
  version: 1.0.0
paths:
  /foo/{id}:
    summary: Path used to retrieve a single Foo.
    get:
      summary: Get a Foo
      description: Gets the details of a single instance of a `Foo`.
      operationId: getFoo
      responses:
        '200':
          $ref: '#/components/responses/FooResponse'
        '404':
          $ref: 'foo-global-schemas.yaml#/components/responses/NotFoundError'
components:
  responses:
    FooResponse:
      description: Successful response - returns a single foo.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Foo'
  schemas:
    Foo:
      type: object
      properties:
        bar:
          type: string

And the remote spec foo-global-schemas.yaml:

components:
  responses:
    NotFoundError:
      description: Resource not found.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/NotFoundError'

And this very basic Python code:

from pathlib import Path

import yaml

from dli.client.openapi import create_spec

path = Path(__file__).parent / 'openapi.yaml'
path = path.resolve()

with open(path) as f:
    spec_dict = yaml.safe_load(f)

spec = create_spec(spec_dict, spec_url=f'file://{path}')

print(spec)

The resulting error will be:

  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/jsonschema/validators.py", line 776, in resolve_fragment
    document = document[part]
KeyError: 'ApiError'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    spec = create_spec(spec_dict, spec_url=f'file://{path}')
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/shortcuts.py", line 22, in create_spec
    return spec_factory.create(spec_dict, spec_url=spec_url)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/specs/factories.py", line 37, in create
    info, list(paths), servers=list(servers), components=components)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/paths/generators.py", line 23, in generate
    yield path_name, Path(path_name, list(operations), parameters)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/operations/generators.py", line 47, in generate
    operation_id=operation_id, tags=list(tags_list)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/operations/models.py", line 14, in __init__
    self.responses = dict(responses)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/responses/generators.py", line 36, in generate
    content=media_types, headers=parameters, links=links)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/responses/models.py", line 16, in __init__
    self.content = content and Content(content) or Content()
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/media_types/generators.py", line 26, in generate
    schema, _ = self.schemas_registry.get_or_create(schema_spec)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_core/schema/schemas/registries.py", line 20, in get_or_create
    schema_deref = self.dereferencer.dereference(schema_spec)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/openapi_spec_validator/validators.py", line 36, in dereference
    with resolver.resolving(ref) as target:
  File "/home/berislav/.pyenv/versions/3.6.8/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/jsonschema/validators.py", line 725, in resolving
    url, resolved = self.resolve(ref)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/jsonschema/validators.py", line 734, in resolve
    return url, self._remote_cache(url)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/jsonschema/validators.py", line 746, in resolve_from_url
    return self.resolve_fragment(document, fragment)
  File "/home/berislav/.virtualenvs/foo-bar/lib/python3.6/site-packages/jsonschema/validators.py", line 779, in resolve_fragment
    "Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'components/schemas/ApiError'

However, when all the internal references in the foo-global-schemas.yaml are prepended with the name of the file itself, all works as expected, printing

<openapi_core.schema.specs.models.Spec object at 0x7f05aac39b70>

After digging through the code a little bit it seems like the issue is related to RefResolver.resolution_scope, which returns the base document; so it might be a jsonschema issue, I'm just putting it here as I tested only with openapi-core.