Improve Builds with CircleCI 2.0 Setup

[philsturgeon]

The recent #333 was a good direct port of CircleCI 1.0 to 2.0, but it has not taken advantage of worklflows just yet, or much of the functionality offered.

A Bigger Thing

It's tough to tell which service is doing which language from looking just at the various CI config files, but there is a list here:

The goal surely would be to get everything into one CI system, and this can easily be done with workflows. A good first step would be to roll Travis into CircleCI as a new workflow.

All of the slowness from these CI builds are down to having all of the "cache" logic for all languages done together, then all of the install steps done together, then all of the test steps done together... serial is slower than parrelllel.

You need to tweak the logic here, to either make a job for each language that does each step, or to split the "dependencies" into parrallel jobs, and "build" into parrallel jobs, etc., and the final test into multiple jobs.

One trivial thing

Multiple docker images can be referenced, so we can pull in the swagger petstore, and set env variables on it like this:

version: 2
jobs:
  build:
    docker:
      - image: openapitools/openapi-generator
      - image: swaggerapi/petstore
          environment:
            SWAGGER_HOST=http://petstore.swagger.io
            SWAGGER_BASE_PATH=/v2

That would cut down on these sort of commands, and the sleeps and trickyness that comes with it.

     - run: docker run -d -e SWAGGER_HOST=http://petstore.swagger.io -e SWAGGER_BASE_PATH=/v2 -p 80:8080 swaggerapi/petstore 

[jmini]

@philsturgeon: this sounds really nice. You seems to know CircleCI very well, are you ready to help?

For example we have some check scripts like:

• bin/utils/detect_merge_conflict.sh
• mvn verify and then bin/utils/ensure-up-to-date

Would it make sense to run those tests as small steps at the beginning? Have you some pointers how this should be defined?

[wing328]

Multiple docker images can be referenced, so we can pull in the swagger petstore, and set env variables on it like this:

@philsturgeon Looks like that's the right way to do it in CircleCI 2.0. Can you please submit a PR with the enhancement?

[philsturgeon]

Thanks for the vote of confidence. I genuinely would love to help you folks out, but my open-source time is completely full with speccy, JSON Schema, OpenAPI TSC calls, openapi.tools, standards.rest, we-call, and faraday-nethttp2.

Whilst CircleCI 2.0 makes workflows easy, it does not seem easy to me (as an outsider, fresh to the project) how to run a single test suite for a single language with the way you have things setup. This change is less about CircleCI and more about refactoring how your CI code, expectations, etc is layed out. I also don't know which language needs which setup steps, so this would be a trial and error nightmare.

If I knew:

1. which dependencies to run for each language
2. which setup to run for each language
3. which command to run to test a specific language

then I would be able to sneak it in amongst all that other stuff, but at that point you'd be half way to writing the workflow yourself. :)

[wing328]

UPDATE: I've submitted #608 to run 2 jobs testing Java8 and Java7 in parallel.

I tried with the below:

version: 2
jobs:
  build:
    docker:
      - image: openapitools/openapi-generator
      - image: swaggerapi/petstore
          environment:
            SWAGGER_HOST=http://petstore.swagger.io
            SWAGGER_BASE_PATH=/v2

but since the executor we used is "machine" (ref: https://circleci.com/docs/2.0/executor-types/), we cannot do the above. (We used "docker" executor before but ran into network issues)

[jmini]

For the moment because we want to run docker (to start the petstore server), it seems that we are forced to use "machine". We might think about changing this also, it would be nice to have access to the workflow features described in this issue.

For example it would be nice to have one workflow step per languages/projects/framework being tested...