Define options for projects by JSON Schemas

[magicmatatjahu]

Today we have more and more projects to maintain and develop in our organization. We have decided that one of our main goals of this year will be to strongly develop the Studio as well as CLI.

One problem to meet this goal is the integration of other tools into those listed. E.g. diff integration into CLI, modelina integration into Studio etc. Currently every integration is the same - we do PR to CLI/Studio/Server-API, then review and we have a new integration. However, I see several (but mainly two) problems here:

• if we do integration in one project, we have to do integration also in another project. Due to the fact that each project (Studio/CLI/Server-API) has different assumptions, the integration itself is usually different, i.e. limitations of a given environment, in Studio we cannot use filesystem etc.
• if we add a new option in a tool (e.g. in diff we recently added the possibility to have output as yaml) we have to add this change also in Studio/CLI/Server-API.

I wonder if we can standardize the integration of tools to others using JSON Schema. The whole idea would be to create a JSON Scheme for each tool (in the source code of project) that would describe the options for that project. E.g. in modelina it would be the language, info how to transform fields (camelCase, snakeCase etc) and a corresponding function that would transform the options described by this JSON Schema to the corresponding options in the project. This is easier to describe with an example. Consider for example modelina:

JSON Schema:

properties:
  language:
    enum:
      - java
      - ts
      - js
      - ...
  options:
    properties:
      namingConvention:
        enum:
          - camelCase
          - snakeCase
          - ...
  presets:
    items:
      oneOf:
        properties:
          preset:
            enum:
              - default
              - jackson
              - ...
        ...

Function:

import { camelCase, snakeCase, ... } from 'naming-conventions';
import { JAVA_JACKSON_PRESET } from '../presets';
import { JavaGenerator } from '../languages';

export function serializeOptions(jsonSchemaData) {
  let options = {};

  let language;
  let namingConvention;

  if (jsonSchemaData.language === 'java') {
    language = JavaGenerator;
  } ...

  if (jsonSchemaData.options.namingConvention === 'camelCase') {
    namingConvention = camelCase;
  } ...

  ...

  return {
    language,
    options: {
      namingConvention,
    }
    ...
  };
}

Above is an example of how it could look like. Why do we need JSON Schema and this function?

• JSON Schema: for parameters/options validation. We will not have to validate those options manually but through JSON Schema - CLI and ServerAPI use case. In the Studio we can use this schema to render a form where the user will enter options for a given tool.
• function: as some tools need references to functions/objects, we can't describe everything with JSON Schema. Additionally, we can split the JSON Schema into several sub-schemas (e.g. by language in the modelina). The task of this function will be just to convert data entered via CLI parameters/forms in Studio/JSON in ServerAPI to proper options representation in tool.

I can see several benefits of such a solution:

• integrates the tool once in Studio/CLI/ServerAPI. To make it easier we can always do it in some generic way, topic to discussion,
• adding a new feature to a given tool involves updating the JSON Schema and function, but in the tool source code, not in the integration side. Then our automatic dependency bump workflow takes care of rest. We don't have to update integrations every time when we add/remove new features in tool.

However, some problems must be noted:

• how should we work with breaking changes on the side of the tools themselves? E.g. we have breaking change on the side of modelina, we have new version 2.0.0 and then should CLI also have breaking change?
• the bigger the project the bigger the JSON Schema and remembering when adding new features/deleting others to update JSON Schema and corresponding function.

What do you think about that idea? Of course, this is just an "suggestion" to help with integration, that we should discuss.

[Souvikns]

how should we work with breaking changes on the side of the tools themselves? E.g. we have breaking change on the side of modelina, we have new version 2.0.0 and then should CLI also have breaking change?

This is something I was thinking about as well. As CLI depends on other tools, when they introduce some breaking changes it will automatically trigger a new release in CLI and it could also break.

I liked the idea, just trying to understand will the spec for diff look something like this.

properties:
  format:
    enum:
      - json
      - yaml
  type:
    oneOf:
      enum:
        - breaking
        - non-breaking
        - unclassified
        - all

Some questions
Will every tool be returning a spec or we would have a separate library that would have the spec.

[magicmatatjahu]

@Souvikns

I liked the idea, just trying to understand will the spec for diff look something like this.

Yes it should look like your example but without oneOf in the type field, only with enum.

Will every tool be returning a spec or we would have a separate library that would have the spec.

Every tool should have JSON Schema (spec) and corresponding function in the codebase due to simple fact: if someone will add new feature or introduce breaking change should also update that spec and function in this same PR, so separate library isn't an option.

EDIT: @Souvikns Also for diff we should define two AsyncAPI spec like:

properties:
  ...
  asyncapis:
    items: ...
    minItems: 2
    maxItems: 2

[aayushmau5]

Just so that I understand this clearly,
We will define all the options that our tool(ex. diff, modelina) will take in a JSON schema, and we will have a corresponding function that will allow certain options from that JSON schema to be passed to our tools. Right?

[magicmatatjahu]

@aayushmau5

We will define all the options that our tool(ex. diff, modelina) will take in a JSON schema, and we will have a corresponding function that will allow certain options from that JSON schema to be passed to our tools. Right?

Yep, but that function should handle some custom logic on passed data (based on JSON Schema). I gave example for modelina where we need change language type to the corresponding language generator (in JSON Schema you cannot pass references to the object), so we also cover such a case.

[Souvikns]

I was playing around with the idea and was thinking if we can create cli plugins in the source code itself. So like in every tool we can have a cli.js file where we export a function and an option to be plugged into CLI.

This is a gist -> https://gist.github.com/Souvikns/65838f8adc23734ed04b82c02b000c06

In place of options, we export CLI parameters like args, flags, descriptions. So the tool itself can update the CLI as they need and when a new build is created the CLI gets automatically updated.

Let me know your thoughts @magicmatatjahu @aayushmau5 maybe we can make this a GSOC idea.

[magicmatatjahu]

@Souvikns Thanks for your response!

I was playing around with the idea and was thinking if we can create cli plugins in the source code itself. So like in every tool we can have a cli.js file where we export a function and an option to be plugged into CLI.

This is a gist -> https://gist.github.com/Souvikns/65838f8adc23734ed04b82c02b000c06

In place of options, we export CLI parameters like args, flags, descriptions. So the tool itself can update the CLI as they need and when a new build is created the CLI gets automatically updated.

I like that idea, but remember that not only CLI should be supported, but also Studio and ServerAPI, that's why I chose JSON Scheme. We can think about the function that performs the logic on the library side, but I would rather go into the JSON Scheme and turn the JSON Scheme into flags on the side of the CLI itself. In the future we can change the framework for CLI again and it will be a big problem and JSON Scheme is more generic :)

Let me know your thoughts @magicmatatjahu @aayushmau5 maybe we can make this a GSOC idea.

It can be propose as idea, however GSOC will end at the September and I don't think if we should wait for it that long. Solving the given problem itself is quite important for me, because I would not like to duplicate code in integrations. Waiting 6-7 months for this will be quite a problem, because we will integrate a significant part of our ecosystem and then start rewriting... Of course if someone would like to have this at GSOC then ok, but still, this is not a feature that can wait a year (in my opinion).

[Souvikns]

Thanks for the clarification @magicmatatjahu 😄

It can be propose as idea, however GSOC will end at the September and I don't think if we should wait for it that long. Solving the given problem itself is quite important for me, because I would not like to duplicate code in integrations. Waiting 6-7 months for this will be quite a problem, because we will integrate a significant part of our ecosystem and then start rewriting... Of course if someone would like to have this at GSOC then ok, but still, this is not a feature that can wait a year (in my opinion).

Yeah, make sense.

I like that idea, but remember that not only CLI should be supported, but also Studio and ServerAPI, that's why I chose JSON Scheme. We can think about the function that performs the logic on the library side, but I would rather go into the JSON Scheme and turn the JSON Scheme into flags on the side of the CLI itself. In the future we can change the framework for CLI again and it will be a big problem and JSON Scheme is more generic :)

I know CLI very well so was just thinking in its terms. I will read up on studio and ServerAPI and try to think how we can have something that would be useful to these and also some new tools that might be written in the future.

[Souvikns]

I have an idea about how we can create one single JSON definition and use it in all other tools be it in CLI and server-api. We need an interface that would let the maintainer of the library control how the input and out format. This way the maintainer can completely change the library usage and still, nothing will break.

For example -

This is the integration function that bundlerwould export

// bundler integration function
const bundle = require('@asyncapi/bundler');

async function bundler(inputs,options, output) {
    const files = inputs.files;
    const base = options.base;
    const format = output.format;

    const document = await bundle(files, {
        base: base
    })

    if(format === 'json') {
        return document.json()
    }
    
    if(format === 'yaml') {
        return document.yml()
    }

    if(format == 'string') {
        return document.string();
    }

    return format.json();
}

module.exports = bundler;

And this is the JSON spec that the bundler would export for this function

type: object
properties:
  name:
    type: 'string'
  inputs:
    type: object
    properties:
      files:
        type: array
        items:
          type: string
  options:
    type: object
    properties:
      base:
        type: string
  output:
    type: object
    properties:
      format:
        enum:
          - json
          - yaml
required: ['name', 'inputs', 'files']

@magicmatatjahu could we use this format to integrate bundler to different tools?

[magicmatatjahu]

@Souvikns It's one of the possible solution but I don't like the output idea. At first I thought it was a description of what the output would look like, but I see that it could be part of the options. The same with input and options. Of course we have libraries where we have a distinction between input and options, but we have to go through all our projects and see if it is actually in any project. In my opinion it should look like this:

properties:
  input:
    type: object
    properties:
      files:
        type: array
        items:
          type: [string, object]
  options:
    type: object
    properties:
      base:
        type: string
      format:
        enum:
          - json
          - yaml

The format itself is in the bundler options -> https://github.com/asyncapi/bundler/blob/master/lib/index.js#L26

[Souvikns]

In my opinion input, option, and output are properties of the function that we are exporting I was not trying to describe bundler I was trying to describe the function that I am exporting which in this case is this -

// bundler integration function
const bundle = require('@asyncapi/bundler');

async function bundler(inputs,options, output) {
    const files = inputs.files;
    const base = options.base;
    const format = output.format;

    const document = await bundle(files, {
        base: base
    })

    if(format === 'json') {
        return document.json()
    }
    
    if(format === 'yaml') {
        return document.yml()
    }

    if(format == 'string') {
        return document.string();
    }

    return format.json();
}

module.exports = bundler;

I was thinking like this because the main reason we are doing this is so that I as a bundler maintainer does not have to care how bundler is implemented because this is the function that is going to run in all other tools, and by the specification I get to control what input and output my function shall get.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[magicmatatjahu]

@Souvikns Sorry for such a delay 😅

To your last comment about inputs and outputs - you can treat output as options and we should treat them in this way. You have case with output in your library but it's not case in other our libraries and usually output we define in options.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️