Supporting $ref to relative path

[hhp21]

I have a Swagger 2.0 spec api/spec.json and inside there I want to have a reference to a local file api/defs.json for my definitions. So in spec.json I'm doing something like:

    "/message": {
      "post": {
        "parameters": [{
          "name": "message",
          "in": "body",
          "schema": {
            "$ref": "defs.json"
          }

where defs.json looks like

{
    "properties": {
      "text": {
        "type": "string"
      }
    },
    "required": ["text"]
}

But I just get #/paths/~1message/post/parameters/0/schema/$ref: Not a valid JSON Reference on init

I've tried several variations like:

"$ref" : "./defs.json"

This starts but fails with #: Reference could not be resolved: ./defs.json when a request is sent.

I have read that $ref supports internal and remote (with an absolute URL), but not been able to find any documentation on relative paths.

Here is a mention that this might be possible already: OAI/OpenAPI-Specification#275 (comment)
Here is a mention that it has just been implemented in swagger-js: swagger-api/swagger-js#417

Are relative paths supported at all?

[whitlockjc]

JSON Reference resolution is handled by json-refs and there is a known issue for supporting this: whitlockjc/json-refs/issues/11. I will keep this open in here until this issue is resolved in json-refs.

[whitlockjc]

This should be fixed but there is currently a bug in json-refs/issues/24 that you need to be careful of. No circular remote/relative references? Things will work like you want.

[glen-84]

@whitlockjc,

It's not working for me. Am I doing something wrong?

http://pastie.org/private/0ozh5wm8li9qrrifl9cbua (swagger.yaml)
http://pastie.org/private/6coezsiq2d9bmawl4yygg (definitions/models/videos/video.yaml)

[whitlockjc]

How are you loading swagger.yaml?

[glen-84]

With Swagger UI.

Also, it seems to request the main file 8 times when I reference the video file, but it doesn't request the video file itself.

[whitlockjc]

swagger-ui does not support relative references.

[whitlockjc]

I'm going to reopen the issue until I have unit tests showing it works but swagger-ui is not related to swagger-tools.

[glen-84]

Thanks @whitlockjc, I guess it's this one: swagger-api/swagger-ui#1456.

[oliverhr]

It would be great to be able to run this example:

https://github.com/swagger-api/swagger-spec/tree/master/examples/v2.0/yaml/petstore-separate
https://github.com/swagger-api/swagger-spec/blob/master/versions/2.0.md#referenceObject

[whitlockjc]

The problem we have right now with supporting this is that swagger-tools does not know where the original document was loaded from so doing relative reference resolution isn't possible without providing that context. swagger-tools uses path-loader and json-refs which both support relative references but they do it based on process.cwd on Node.js and window.location in the browser. Whenever the location of the loaded document is not known and does not conform to the aforementioned defaults, the related json-refs/path-loader APIs have an options.location option to fill in the gap.

That being said, to support this I would need to update all APIs in swagger-tools to support passing this option to json-refs/path-loader. It's not impossible but it would change the APIs a bit. I need to think about this a little more.

[coconitro]

I'm having a little tough time following, is this only an issue only with swagger-ui? I'm unable to get it to work using the swagger-node module loading swagger.yaml (completely in node, no browser in the chain at all)

Relevant snippet

swagger.yaml (snipped)

        default:
          description: Error
          schema:
            $ref: "error.yaml#/ErrorResponse"

error.yaml

ErrorResponse:
  required:
    - errors
  properties:
    errors:
      type: array
      items:
        $ref: "#Error"

Then when running swagger-node's validate I get this:

swagger validate api/swagger/swagger.yaml 

Project Errors
--------------
#/paths/~1events/get/responses/default/schema/$ref: Not a valid JSON Reference
Results: 1 errors, 0 warnings

[whitlockjc]

Your #Error reference is wrong. It should be #/definitions/Error.

[coconitro]

Good catch although it appears to be tangential. I changed errors.yaml as a test to be:

ErrorResponse:
  required:
    - errors
  properties:
    errors:
      type: string

[whitlockjc]

What's the error now? I doubt swagger-node supports relative references just yet since it uses swagger-tools which is known not to support relative references except under certain conditions.

[coconitro]

Sorry I should have been more clear. Its the same error as before.

#/paths/~1events/get/responses/default/schema/$ref: Not a valid JSON Reference

Under what conditions does swagger-tools support relative references? I could maybe force them or look thru swagger-node to figure out how to get them to work.

[whitlockjc]

The paths must be relative to process.cwd which is not ideal of course. So for swagger-node, your relative references would start with ./api/swagger. So if you had a users.yaml in api/swagger, you'd have something like this:

# ...
  $ref: './api/swagger/users.yaml#/definitions/User'
# ...

[glen-84]

Not sure if it's relevant, but what we're doing is using swagger-parser (3.0.0-alpha.5) to dereference our swagger.yaml file (with relative paths), and then we save it as swagger.json and serve that.

[whitlockjc]

swagger-tools dereferences references too but relative references weren't a thing until recently. When we started work on breaking up swagger-tools, we started swagger-core-api and it does work with relative references. In face, swagger-core-api does better validation than swagger-tools at this point. The reason I bring this up is swagger-tools will be deprecated in the future and will be replaced by swagger-core-api and per-server middlewares/plugins that build on top of swagger-core-api.

[diogeneshamilton]

@whitlockjc I use swagger-editor currently, and it seems that I must wait for it to be updated to use swagger-core-api in order to be usable with relative URLs. Am I getting that right?

[whitlockjc]

Yes. We are in the process of migrating swagger-editor to swagger-core-api. Also, supporting relative references for the swagger-editor will likely involve some other work since it's intended right now to be a single-file UI but this could change that.

[dkhanal]

I'm currently in the process of designing a fairly large set of APIs with various widgets/types reused.

I'd like the Swagger Spec itself to be decomposed into one spec file per path referenced through $ref in YAML. Then the referenced spec file itself reference JSON schema definitions all through HTTP-based relative $ref's -- since the hostname would be tier/deployment-specific:

So (in pseudo-code):

myApiDoc.yaml:
    /my/resource/:
        $ref: "swagger-sppecs/myResource.json"

------

Schema definition in swagger-sppecs/myResource.json:
{
    "properties": {
        "propertyA": {
            "$ref":"../schema/propertyA.json"
        }
    }
}

------

Schema definition in schema/propertyA.json:
{
    "properties": {
        "propertyB": {
            "$ref":"../schema/propertyB.json"
        }
    }
}

My questions are:

1. With Swagger-Tools, is it possible to decompose/refactor YAML into specs (it looks like it is possible with the new changes)
2. With Swagger-Tools, is it possible to refactor the referenced JSON schemas themselves to utilize relative ref's in composite widgets and types? Ideally, I would like the schema library to be hosted externally, since the schema definitions could be utilized across technologies and applications.

Any thoughts on the current/upcoming capabilities or recommended patterns around this?

[whitlockjc]

The underlying library swagger-tools uses for JSON Reference resolution (json-refs) supports relative references but to use it requires swagger-tools to know where the loaded Swagger document came from, which it doesn't, and it needs a way to pass this information to json-refs. To support this we need to alter all APIs to take an options object so that we can pass the location where the Swagger document was loaded from to json-refs. There is one situation where this can work now but it is not something work documenting: If your relative references are relative to process.cwd() in Node.js or window.location in the browser.

[dkhanal]

Thank you for the response. I was able to get the Node side to work by making sure the relative refs were off-of process.cwd() like you suggested. However, neither Swagger UI nor Swagger Editor is able to expand the ref'ed schemas. Swagger UI just doesn't work, while Swagger Editor only shows the referenced relative URL for the model.

Is there any way to get Swagger Editor (or UI) to expand the schemas into a more human-friendly view (like it would if the definitions were local and not ref'ed)?

Regarding your comment on window.location, does that imply that the docs would have to served essentially off of process.cwd() for both node and Swagger Editor/UI to work?

[whitlockjc]

swagger-ui is its own thing and they do their own stuff. swagger-editor uses swagger-tools but remember that references then become relative to window.location and I'm sure that the swagger-editor doesn't serve up any documents other than the one. swagger-tools supporting relative references is just part of this as the whole tooling ecosystem needs to support it properly.

I'll work on getting swagger-tools to have an API for specifying the document root so relative references work universally. Then it's up to swagger-editor and others to use the new support.

[dkhanal]

Thanks again for the response. So is your recommendation for now (until all tools in the ecosystem support the doc root) to just go with a single specification document with all references local to the doc?

Having one large YAML with thousands of lines of specification sounds rather monolithic to me -- so as a newcomer to swagger/tools, I would appreciate your insight on what the prevalent industry patterns are for composing a large set of APIs (essentially around reuse and physical refactoring of the specification document).

[whitlockjc]

What server environment are you using?

[dkhanal]

Has anything changed recently with Sway in the way it resolves relative references from within JSON schemas:

SwaggerExpress.create(config, function(err, swaggerExpress) {
  if (err) { throw err; }

  Sway.create({definition: "./api/swagger/swagger.yaml"})
      .then(function (swaggerApi) {
        app.use(new SwaggerUi(swaggerApi.resolved));
      }, function (err) {
        console.error(err.stack);
        process.exit(1);
      });

  swaggerExpress.register(app);

  var port = process.env.PORT || 10010;
  var server = app.listen(port);

  console.log('Server listening at: ' + server.address().address + ':' + server.address().port);
});

I get the following error when I start the project:

Error initializing middleware Error: Swagger document(s) failed validation so the server cannot start
If I remove the relative reference from within JSON, it works. Otherwise it fails. The relative reference is correct and the referenced file is present.

This used to work OK until a couple months ago.

[whitlockjc]

Upgrade, I released a fix Friday.

[dkhanal]

Thanks. I just upgraded Sway. Still get the same error with "sway": "^0.6.0". Anything else that must be upgraded?

[whitlockjc]

I meant to upgrade swagger-tools. Sway hasnt seen a release and shouldn't be causing issues.

[dkhanal]

Upgraded swagger-tools. At "swagger-tools": "^0.9.14" still see the issue.

[whitlockjc]

Try using Sway at master then. I'm not sure what is up because Sway hasn't had any releases to cause any problems and swagger-tools is not doing anything for relative references still. The approach I documented above is probably better but I'm not sure why something just started failing. Run with DEBUG=swagger-tools* to get more information on what the failure is.

[dkhanal]

DEBG=swagger-tools* did not provide anything of much value (first 3 lines below) . The stack looks like this:

swagger-tools:middleware Initializing middleware +0ms
swagger-tools:middleware   Identified Swagger version: 2.0 +0ms
swagger-tools:middleware   Validation: failed +234ms
Error initializing middleware
Error: Swagger document(s) failed validation so the server cannot start
  at ...\node_modules\swagger-tools\index.js:72:13
    at cbWrapper (...\node_modules\swagger-tools\lib\specs.js:1035:5)
    at validateSwagger2_0 (...\node_modules\swagger-tools\lib\specs.js:1030:3)
    at validateSemantically (...\node_modules\swagger-tools\lib\specs.js:1040:5)
    at ...\node_modules\swagger-tools\lib\specs.js:1233:7
    at ...\node_modules\swagger-tools\lib\specs.js:1073:29
    at ...\node_modules\swagger-tools\lib\specs.js:719:12
    at ...\node_modules\swagger-tools\lib\specs.js:695:9

[dkhanal]

Also, why is SwaggeExpress.create() does not seem to follow the callback pattern for errors. Here's a simplified example:

console.log('BEFORE SwaggerExpress.create()');

SwaggerExpress.create(config, function(err, swaggerExpress) {
    console.log('AFTER SwaggerExpress.create()');
    if (err){
        console.log('There was an ERROR.');
    }
    else
    {
        console.log('There was NO ERROR.');
    }
});

And here's the output (it never gets to the callback function). It seems to be throwing, instead.

BEFORE SwaggerExpress.create()
Error initializing middleware
Error: Swagger document(s) failed validation so the server cannot start
at ...\node_modules\swagger-tools\index.js:72:13
at cbWrapper (...\node_modules\swagger-tools\lib\specs.js:1035:5)
at validateSwagger2_0 (...\node_modules\swagger-tools\lib\specs.js:1030:3)
at validateSemantically (...\node_modules\swagger-tools\lib\specs.js:1040:5)
at ...\node_modules\swagger-tools\lib\specs.js:1233:7
at ...\node_modules\swagger-tools\lib\specs.js:1073:29
at ...\node_modules\swagger-tools\lib\specs.js:719:12
at ...\node_modules\swagger-tools\lib\specs.js:695:9

[dkhanal]

Here's another finding that may be helpful.

It looks like the relative references ($ref from one schema file to another) are expected to be relative to the application root.

Previously, I believe the relative references ($ref from one schema file to another) were relative to the schema file containing the $ref declaration.

If I change my $refs to now make them relative to the application root, it seems to work.

Was this a breaking change that was introduced recently?

[dkhanal]

Some more information:

I have a fairly succinct .yaml file where each path contains a $ref to Swagger Specification (in JSON) for the path. Each such specification, then references various other schema (in JSON) files for parameters and responses. Then each schema definition contains one or more $ref's pointing to reusable/common constructs.

This is the working configuration:

$ref's in the swagger.yaml definition are relative to application root.
$ref's in the Swagger Specification JSON (e.g. for parameters and responses) are relative to the specification file containing the $ref.
$ref's in the JSON schema files that point to other JSON schema for reusable constructs are relative to application root. It looks like this is what changed recently. These $ref's, I believe used to be relative to the current schema file (i.e. file containing this $ref).

For example, in the following snippet of a JSON schema, the meta property's $ref must be relative to the application root, rather than relative to the file containing the $ref:

...
...
    },
    "meta": {
      "$ref": "specs/schema/v1/meta.json"
    }
  },
  "required": [
    "data"
  ]
}

Here's the set of dependencies and their versions (higher versions of mw or tools seem to have breaking changes):

"dependencies": {
"express": "^4.12.3",
"swagger-express-mw": "0.0.x",
"swagger-tools": "^0.9.2",
"sway": "^0.1.0"
},

[whitlockjc]

swagger-tools and sway are really old so the chances they have bugs does not surprise me. The more interesting dependency version would be json-refs. Also, you'd be much better off just not using sway and instead using json-refs with the code snippet I pasted above a few days ago.

Upgrading swagger-tools will give you better output for the failure you're seeing. You can also run swagger-tools validate YOUR_FILE and it will let you know the failures as well.

[DannyiCracked]

Project Errors
--------------
#/paths: Additional properties not allowed: $ref
Results: 1 errors, 0 warnings

Are $ref's to other files going to be supported soon?

swagger: "2.0"
info:
  version: "0.0.1"
  title: Test
host: localhost:10010
basePath: /
schemes:
  - http
  - https
consumes:
  - application/json
  - multipart/form-data
  - application/x-www-form-urlencoded
produces:
  - application/json
paths:
  $ref: ./controllers/program.yaml

Inside controllers/program.yaml exists the code you would expect to see under the paths: definition.

The error comes from both swagger-tools cli and swagger-node cli validation.

[whitlockjc]

Will you please read some of the comments before yours? Not only is the answer to your question there but so is the workaround.

On another note, Swagger forbids $ref where you have it so some might say I should be throwing an error there anyways: apigee-127/sway#62

[dexwiz]

Is there a way to set Json-Ref's options.relativeBase from swagger-tools.initializeMiddleware(doc, cb())? I have split by yaml files, but I have to include the full paths for every reference. I would like to be able to set a relative base for the docs instead.

[whitlockjc]

This is the problem with the current swagger-tools API. To allow for you to pass options downstream, every API has to change and that is a breaking change. So I'd have to release 1.0.0 just to support it. That's why I've instead suggested the workarounds above until I can get sway-connect out the door. Thanks for your patience.

[dexwiz]

Thanks for the quick update, good to know!

[dan-kez]

Hello! Is there any update on this? I'd love to be able to split up my swagger.yaml file into multiple directories (e.g. paths, definitions, etc.)

[towertop]

This problem still exists in current version.

I saw json-ref has an additional option 'relativeBase' but spec.js doesn't pass in this option.

As for the amount of invocations in spec.js or other files demanded to modify, I suggest to add a new method in json-ref specified for initial default options globally, including 'relativeBase'. Adding a wrapper above json-ref should also do. This way might ease the code work.

[whitlockjc]

Of course this problem still exists, the issue is still open. The suggested workaround is all you will get for now because rewriting swagger-tools to support this isn't going to happen. I mentioned this above. Just use json-refs to resolve your document and then use the resolved document when providing the Swagger document to the swagger-tools functions.

[canercandan]

Here is another workaround for those who want to use json-refs with the swagger-node express skeletons:

const JsonRefs = require('json-refs');
const YAML = require('js-yaml');
const SwaggerExpress = require('swagger-express-mw');
const app = require('express')();
// export setup Promis for testing
module.exports = new Promise(function (resolve, reject) {

  JsonRefs.resolveRefsAt('./api/swagger/swagger.yaml', {
    // Resolve all remote references
    filter: ['relative', 'remote'],
    loaderOptions: {
      processContent: (res, cb) => cb(undefined, YAML.safeLoad(res.text))
    }
  })
    .then(results => {
      const config = {
        appRoot: __dirname, // required config
        swagger: results.resolved
      };

      SwaggerExpress.create(config, function (err, swaggerExpress) {
        if (err) { throw err; }

        // install middleware
        swaggerExpress.register(app);

        const port = process.env.PORT || 10010;
        app.listen(port, function() {
          if (swaggerExpress.runner.swagger.paths['/hello']) {
            console.log('try this:\ncurl http://127.0.0.1:' + port + '/hello?name=Scott');
          }
          resolve(app);
        });
      });
    })
    .catch(function (err) {
      console.error(err.stack);
      process.exit(1);
    });
  ;
});

[jnazander]

@canercandan thank you very much for the workaround!

@whitlockjc So I guess there will be no fix at all? Did you already close the lid on this one?