Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: parse Avro name field to custom x-parser-schema-id #69

Merged
merged 12 commits into from
Sep 2, 2021
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ channels:
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
payload: # The following is an Avro schema in YAML format (JSON format is also supported)
type: record
name: User
namespace: com.company
doc: User information
fields:
- name: displayName
Expand Down Expand Up @@ -136,6 +138,14 @@ Additional attributes not defined in the [Avro Specification](https://avro.apach

- `example` - Can be used to define the example value from the business domain of given field. Value will be propagated into [examples attribute](https://json-schema.org/draft/2020-12/json-schema-validation.html#rfc.section.9.5) of JSON schema and therefore will be picked for the generated "Example of payload" when using some AsyncAPI documentation generator.

### Support for names and namespaces

If, at the top level of the Avro schema, the 'name' attribute is defined, it will be copied to the corresponding JSON schema's 'x-parser-schema-id' attribute. If the Avro schema also has the 'namespace' attribute defined, then that schema's fully qualified name will be put into that attribute. The fully qualified name is defined by the namespace, followed by a dot, followed by the name.

If there are two schemas that resolve to the same fully qualified name, only the second schema will be returned by the parser, the first will be discarded.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to say this one doesn't let me sleep 😄
what part of the code is responsible for it? why 2nd is returned, then what if 3 schemas resolve to the same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that parser-js keeps a map where the key is x-parser-schema-id. So if there are several schemas that share the same x-parser-schema-id, then only the last one parsed will end up in that map. But if their document has several schemas with the same fully qualified name, I'd hope that they're identical, they repeat them because we force them to put Avro schemas into the message/payload section instead of components/schemas.

Maybe I should just not mention that at all? Or maybe that's an argument for using the title field after all, and adding x-namespace or something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, now it makes sense, you are referring to 👇🏼

  /**
   * @returns {Map<string, Schema>}
   */
  allSchemas() {
    const schemas = new Map();
    const allSchemasCallback = (schema) => {
      if (schema.uid()) {
        schemas.set(schema.uid(), schema);
      }
    };
    traverseAsyncApiDocument(this, allSchemasCallback);
    return schemas;
  }

I think the sentence is ok, but just maybe write
If there are two schemas that resolve to the same fully qualified name, only the last one will be returned by the parser. Make sure names of your schemas are unique ?


If no name attribute is present, the 'x-parser-schema-id' will have a generated unique id with a name like 'anonymous-schema-1' generated by the main parser. 'x-parser-schema-id' is one of the [custom extensions](https://github.com/asyncapi/parser-js/#custom-extensions) supported by the parser.

## Limitations

### Float and double-precision numbers
Expand Down
11 changes: 11 additions & 0 deletions tests/asyncapi-avro-1.9.0-namespace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
asyncapi: 2.0.0
info:
title: My API
version: '1.0.0'
channels:
mychannel:
publish:
message:
schemaFormat: application/vnd.apache.avro;version=1.9.0
payload:
$ref: 'schemas/Person-1.9.0-namespace.avsc'
19 changes: 13 additions & 6 deletions tests/parse.test.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 2 additions & 3 deletions tests/schemas/Person-1.8.2.avsc
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
{
"name": "Person",
"type": "record",
"fields": [
{"name": "name", "type": "string", example: "Donkey"},
{"name": "name", "type": "string", "example": "Donkey"},
{"name": "age", "type": ["null", "int"], "default": null},
{
"name": "favoriteProgrammingLanguage",
Expand All @@ -13,7 +12,7 @@
"type": {
"name": "Address",
"type": "record",
"fields": [{"name": "zipcode", "type": "int", example: "53003"}]
"fields": [{"name": "zipcode", "type": "int", "example": "53003"}]
}
}
]
Expand Down
22 changes: 22 additions & 0 deletions tests/schemas/Person-1.9.0-namespace.avsc
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"name": "Person",
"namespace": "com.company",
"type": "record",
"fields": [
{"name": "name", "type": "string", "example": "Donkey"},
{"name": "age", "type": ["null", "int"], "default": null, "example": "123"},
{
"name": "favoriteProgrammingLanguage",
"type": {"name": "ProgrammingLanguage", "type": "enum", "symbols": ["JS", "Java", "Go", "Rust", "C"], "default": "JS"}
},
{
"name": "address",
"type": {
"name": "Address",
"type": "record",
"fields": [{"name": "zipcode", "type": "int", "example": "53003"}]
}
},
{"name": "someid", "type": "uuid"}
]
}
6 changes: 3 additions & 3 deletions tests/schemas/Person-1.9.0.avsc
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
"name": "Person",
"type": "record",
"fields": [
{"name": "name", "type": "string", example: "Donkey"},
{"name": "age", "type": ["null", "int"], "default": null, example: "123"},
{"name": "name", "type": "string", "example": "Donkey"},
{"name": "age", "type": ["null", "int"], "default": null, "example": "123"},
{
"name": "favoriteProgrammingLanguage",
"type": {"name": "ProgrammingLanguage", "type": "enum", "symbols": ["JS", "Java", "Go", "Rust", "C"], "default": "JS"}
Expand All @@ -13,7 +13,7 @@
"type": {
"name": "Address",
"type": "record",
"fields": [{"name": "zipcode", "type": "int", example: "53003"}]
"fields": [{"name": "zipcode", "type": "int", "example": "53003"}]
}
},
{"name": "someid", "type": "uuid"}
Expand Down
2 changes: 2 additions & 0 deletions tests/to-json-schema.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,15 @@ describe('avroToJsonSchema()', function () {
const result = await avroToJsonSchema({
type: 'record',
doc: 'My test record',
name: 'MyName',
fields: [
{ name: 'key1', type: 'long', doc: 'Key1 docs' },
{ name: 'key2', type: 'string', default: 'value2', doc: 'Key2 docs' },
]
});
expect(result).toEqual({
type: 'object',
'x-parser-schema-id': 'MyName',
description: 'My test record',
properties: {
key1: {
Expand Down
41 changes: 32 additions & 9 deletions to-json-schema.js
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,30 @@ const typeMappings = {
uuid: 'string',
};

const commonAttributesMapping = (avroDefinition, jsonSchema) => {
const commonAttributesMapping = (avroDefinition, jsonSchema, isTopLevel) => {
if (avroDefinition.doc) jsonSchema.description = avroDefinition.doc;
if (avroDefinition.default !== undefined) jsonSchema.default = avroDefinition.default;

const fullyQualifiedName = getFullyQualifiedName(avroDefinition);
if (isTopLevel && fullyQualifiedName) {
jsonSchema['x-parser-schema-id'] = fullyQualifiedName;
}
};

function getFullyQualifiedName(avroDefinition) {
let name;

if (avroDefinition.name) {
if (avroDefinition.namespace) {
name = `${avroDefinition.namespace}.${avroDefinition.name}`;
derberg marked this conversation as resolved.
Show resolved Hide resolved
} else {
name = avroDefinition.name;
}
}

return name;
}

const exampleAttributeMapping = (typeInput, example, jsonSchemaInput) => {
let type = typeInput;
let jsonSchema = jsonSchemaInput;
Expand All @@ -51,15 +70,15 @@ const exampleAttributeMapping = (typeInput, example, jsonSchemaInput) => {
}
};

module.exports.avroToJsonSchema = async function avroToJsonSchema(avroDefinition) {
async function convertAvroToJsonSchema(avroDefinition, isTopLevel) {
const jsonSchema = {};
const isUnion = Array.isArray(avroDefinition);

if (isUnion) {
jsonSchema.oneOf = [];
let nullDef = null;
for (const avroDef of avroDefinition) {
const def = await avroToJsonSchema(avroDef);
const def = await convertAvroToJsonSchema(avroDef, isTopLevel);
// avroDef can be { type: 'int', default: 1 } and this is why avroDef.type has priority here
const defType = avroDef.type || avroDef;
// To prefer non-null values in the examples skip null definition here and push it as the last element after loop
Expand Down Expand Up @@ -94,20 +113,20 @@ module.exports.avroToJsonSchema = async function avroToJsonSchema(avroDefinition
jsonSchema.maxLength = avroDefinition.size;
break;
case 'map':
jsonSchema.additionalProperties = await avroToJsonSchema(avroDefinition.values);
jsonSchema.additionalProperties = await convertAvroToJsonSchema(avroDefinition.values, false);
break;
case 'array':
jsonSchema.items = await avroToJsonSchema(avroDefinition.items);
jsonSchema.items = await convertAvroToJsonSchema(avroDefinition.items, false);
break;
case 'enum':
jsonSchema.enum = avroDefinition.symbols;
break;
case 'record':
const propsMap = new Map();
for (const field of avroDefinition.fields) {
const def = await avroToJsonSchema(field.type);
const def = await convertAvroToJsonSchema(field.type, false);

commonAttributesMapping(field, def);
commonAttributesMapping(field, def, false);
exampleAttributeMapping(field.type, field.example, def);

propsMap.set(field.name, def);
Expand All @@ -116,8 +135,12 @@ module.exports.avroToJsonSchema = async function avroToJsonSchema(avroDefinition
break;
}

commonAttributesMapping(avroDefinition, jsonSchema);
commonAttributesMapping(avroDefinition, jsonSchema, isTopLevel);
exampleAttributeMapping(type, avroDefinition.example, jsonSchema);

return jsonSchema;
};
}

module.exports.avroToJsonSchema = async function avroToJsonSchema(avroDefinition) {
return convertAvroToJsonSchema(avroDefinition, true);
};