[BUG] refs not resolved with a lot of indirection/nesting

[deiferni]

Describe the bug.

When referenced documents have a lot of indirection some of the $refs are not resolved as I'm expecting. I might be missing something or not using the tooling correctly ... but as far as I could tell this might be a bug.

when i bundle the following files a $ref in commonTypes.json is not resolved correctly. The input files look like this:

main.yml:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: "demo"
defaultContentType: application/json
channels:
  'task.v1':
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    task.v1:
      $ref: 'schema/v1/task.json'

schema/v1/task.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "customerOids": {
      "$ref": "commonTypes.json#/definitions/customerOidArrayType",
      "description": "A list of customer Oids"
    }
  }
}

schema/v1/commonTypes.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "definitions": {
    "customerOidType": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128
    },
    "customerOidArrayType": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/customerOidType"
      }
    }
  }
}

I bundle and validate the files as follows in bundle.js:

const { readFileSync, writeFileSync } = require('fs');
const bundle = require('@asyncapi/bundler');
const { DiagnosticSeverity, Parser } = require('@asyncapi/parser');


async function main() {
  // bundle
  const YAMLDocument = readFileSync('main.yml', 'utf-8')
  const bundledDocument = await bundle([YAMLDocument], { referenceIntoComponents: true})
  writeFileSync('bundled.yaml', bundledDocument.yml());

  // validate the document
  const asyncApiDocument = readFileSync('bundled.yaml', 'utf-8')
  const parser = new Parser();
  const parse = await parser.parse(asyncApiDocument);
  const parseErrors = parse.diagnostics.filter((el) => {
    return el.severity == DiagnosticSeverity.Error;
  });
  if (parseErrors.length > 0) {
    const msg = parseErrors.map((err) => {
      return `${err.code}: ${err.message} [${err.path.join(' > ')}]`;
    });
    throw new Error(msg.join('\n'));
  }
}

main().catch((e) => console.error(e));

The output file looks like this bundled.yml:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: demo
defaultContentType: application/json
channels:
  task.v1:
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    task.v1:
      $schema: http://json-schema.org/draft-07/schema#
      type: object
      properties:
        customerOids:
          description: A list of customer Oids
          type: array
          items:
            $ref: '#/definitions/customerOidType'

The issue is in the final line, the $ref #/definitions/customerOidType is not resolved and thus invalid. The validation outputs:

Error: invalid-ref: '#/definitions/customerOidType' does not exist [components > schemas > task.v1 > properties > customerOids > items > $ref]
    at main (./broken/bundle.js:23:11)

I put the failing example in a repo at https://github.com/deiferni/asyncapibundlerbrokenexample for your convenience.

Expected behavior

I would expect the schema to be included in components.schemas and be referenced correctly in the bundled file, something like this:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: demo
defaultContentType: application/json
channels:
  task.v1:
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    customerOidType:
      type: string
      minLength: 1
      maxLength: 128
    task.v1:
      $schema: http://json-schema.org/draft-07/schema#
      type: object
      properties:
        customerOids:
          description: A list of customer Oids
          type: array
          items:
            $ref: '#/components/schemas/customerOidType'

Screenshots

IMO a screenshot is not very useful :)

How to Reproduce

1. check out the broken example from https://github.com/deiferni/asyncapibundlerbrokenexample (or create files above)
2. run
   • either: node bundle.js
   • or:

   npx asyncapi bundle main.yml > bundled.yaml
   npx asyncapi validate bundled.yaml
   

3. You can see the error on your command line and by inspecting bundled.yaml

🥦 Browser

None

👀 Have you checked for similar open issues?

• I checked and didn't find similar issue

#151 might be related, but it explicitly mentions v3. We are using an older version.
#141 might also be related, but I'm not completely sure either.

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue ?

No, someone else can work on it

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[aeworxet]

Up to now, I even transformed all JSONs to YAMLs, removed all suspicious characters, and placed them all in one directory, but they are not dereferenced anyway.

[deiferni]

Hello @aeworxet

Thanks for your quick response. Please let me know if you need any more info/debugging from my side.

[deiferni]

side-note: if anyone else stumbles upon this issue and needs an urgent fix/workaround: https://www.npmjs.com/package/api-ref-bundler (version 0.4.0) seems to work be able to bundle these files correctly

[aeworxet]

I ran Bundler with Parser's validate() on provided files and got the same validation error:

[
  {
    code: 'invalid-ref',
    path: [
      'components',
      'schemas',
      'task.v1',
      'properties',
      'customerOids',
      'items',
      '$ref'
    ],
    message: "'#/definitions/customerOidType' does not exist",
    severity: 0,
    range: { start: [Object], end: [Object] }
  }
]

Changed ./schema/v1/commonTypes.json to

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "definitions": {
    "customerOidType": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128
    },
    "customerOidArrayType": {
      "type": "array",
      "items": {
        "type": "string",
        "minLength": 1,
        "maxLength": 128
      }
    }
  }
}

and got

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: demo
defaultContentType: application/json
channels:
  task.v1:
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    task.v1:
      $schema: http://json-schema.org/draft-07/schema#
      type: object
      properties:
        customerOids:
          description: A list of customer Oids
          type: array
          items:
            type: string
            minLength: 1
            maxLength: 128

(differs from the expected result)

Bundler performs $refs dereferencing by simply forwarding the AsyncAPI Document to @apidevtools/json-schema-ref-parser, so the origin of the bug requires clarification.

@smoya, @KhudaDad414

• Can you please check if the format of the provided ./schema/v1/commonTypes.json is wrong or if the Parser's validation is wrong (I make no assumption that Parser is bug-free?) Which one should be the correct if it should be different?
• Does the behavior of the schemas expected by the user match the behavior that is supposed to be expected according to the AsyncAPI 2.6.0 Specification?

[aeworxet]

@deiferni, please check if the issue still exists in v0.5.0.

[deiferni]

@aeworxet, yes as far as I can see the issue still persists. I did update my example files in https://github.com/deiferni/asyncapibundlerbrokenexample.

After running:

npx asyncapi bundle main.yml -o bundled.yaml
npx asyncapi validate bundled.yaml

I'm still getting the error:

File bundled.yaml and/or referenced documents have governance issues.

bundled.yaml
  1:1       warning  asyncapi-id                  AsyncAPI document should have "id" field.
  1:1       warning  asyncapi-servers             AsyncAPI document should have non-empty "servers" object.
  1:1       warning  asyncapi2-tags               AsyncAPI object should have non-empty "tags" array.
  1:11  information  asyncapi-latest-version      The latest version of AsyncAPi is not used. It is recommended update to the "3.0.0" version.  asyncapi
  2:6       warning  asyncapi-info-contact        Info object should have "contact" object.                                                     info
  2:6       warning  asyncapi-info-description    Info "description" should be present and non-empty string.                                    info
  2:6       warning  asyncapi-info-license        Info object should have "license" object.                                                     info
 14:13      warning  asyncapi2-message-messageId  Message should have a "messageId" field defined.                                              components.messages.task.v1
 32:19        error  invalid-ref                  '#/definitions/customerOidType' does not exist                                                components.schemas.task.v1.properties.customerOids.items.$ref

The file it produces looks like this: https://github.com/deiferni/asyncapibundlerbrokenexample/blob/8d9e933141d57903877cfa3f76b52493e461aff2/bundled.yaml

[aeworxet]

In Bundler v0.5.0:

• The logic of supplying Bundler with the AsyncAPI Documents was changed and now only paths should be provided. The contents of file(s) Bundler will read itself.
• Validation with Parser is now embedded into Bundler, and the dereferenced file is automatically validated first before it is given outside. If the resulted file doesn't pass validation, a list of errors is given out instead of the dereferenced file.

So all bundle.js would now look like

const { writeFileSync } = require('fs');
const bundle = require('@asyncapi/bundler');

async function main() {
  const bundledDocument = await bundle('main.yml')
  writeFileSync('bundled.yaml', bundledDocument.yml());
}

main().catch((e) => console.error(e));

CLI, however, still uses Bundler v0.4.0 and due to the above changes, switching versions is not as easy as changing one symbol in package.json, so I'm examining code dependencies.

[aeworxet]

CLI version 1.12.2 uses Bundler of version 0.5.0, please check the behavior now.

[deiferni]

Hey @aeworxet

as far as i could see the files are now bundled without error with both the CLI and the bundler package. Thanks! (And sorry for the late reply)

Updated files in https://github.com/deiferni/asyncapibundlerbrokenexample/

[aeworxet]

@deiferni
Closing this bug then. Feel free to reopen if you encounter a regression in the future.