[core] Refactor templating management

[jimschubert]

This refactors template management to get logic out of DefaultGenerator
and to provide a cleaner API to template search and read/compile.

Deprecates MockDefaultGenerator, which is not a mock and causes
in-memory retention of file contents. Maintainers should prefer
executing a "dryRun" with new DefaultGenerator(true) or do true
mock/spies if evaluating template intermediaries is truly necessary.
Tests may read written files with lower overhead than the in-process
retention of those bytes.

This attempts to maintain some compatibility with existing templating
adapter interfaces. Any breaking change here would be unintentional but
minimal effort to retarget the new interface.

PR checklist

• Read the contribution guidelines.
• If contributing template-only or documentation-only changes which will change sample output, build the project before.
• Run the shell script(s) under ./bin/ (or Windows batch scripts under.\bin\windows) to update Petstore samples related to your fix. This is important, as CI jobs will verify all generator outputs of your HEAD commit, and these must match the expectations made by your contribution. You only need to run ./bin/{LANG}-petstore.sh, ./bin/openapi3/{LANG}-petstore.sh if updating the code or mustache templates for a language ({LANG}) (e.g. php, ruby, python, etc).
• File the PR against the correct branch: master, 4.3.x, 5.0.x. Default: master.
• Copy the technical committee to review the pull request if your PR is targeting a particular programming language.

[jimschubert]

FYI breaking change, but we should really try sticking to interfaces and move away from abstract base types.

[jimschubert]

this is now handled by template loading functionality, as that is not the responsibility of DefaultCodegen.

[jimschubert]

MockDefaultGenerator duplicates all written bytes. getTemplateBasedFile is used to "inspect" data passed to templates, but this should be tested differently in the future.

[jimschubert]

Note on breaking change without fallback… removal of AbstractGenerator has no fallback abstract class and requires template management construction similar to what is done in this PR.

[jimschubert]

@OpenAPITools/generator-core-team

[wing328]

The following tests are good:

• used -t
• used -t with --library option
• removed template files and got an exception
• removed customized template files and fell back to the default
• wrong template folder (not exist) resulted in an exception

[jimschubert]

@spacether would you like to take a look before merge?

cc @Fjolnir-Dvorak (who's waiting on this work)

[jimschubert]

@wing328 I'd created a GitHub Workflow to cross-compile in linux and Windows, then run a smoke test for the gradle build of the go client which was failing on AppVeyor. This allowed me to finish up more quickly as I was seeing 1.5 hours in queue and 30-45 minutes on AppVeyor. We may want to consider a cross-compile job as part of our PRs for quicker feedback.

Here's the workflow:

name: Build
on:
  push:
    branches:  
      - master
  pull_request:
    branches:
      - master

jobs:
  build:
    name: Build on JDK ${{ matrix.java }} and ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        java: [8, 11]
        os: [ubuntu-latest, windows-latest]
        
    steps:
      - name: Check out code
        uses: actions/checkout@v2

      - name: Set up JDK ${{ matrix.java }}
        uses: actions/setup-java@v1 
        with:
          java-version: ${{ matrix.java }}
          
      - uses: actions/cache@v1
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-maven-
        
      - name: Build with Maven
        shell: bash
        run: mvn -nsu -B --quiet -Djacoco.skip=true -Dorg.slf4j.simpleLogger.defaultLogLevel=error --no-transfer-progress clean install --file pom.xml ${{ matrix.flags }}
        
      - name: Test gradle
        shell: bash
        run: gradle -b modules/openapi-generator-gradle-plugin/samples/local-spec/build.gradle buildGoSdk --stacktrace

[jimschubert]

Note these removals of libraries/<library name> in the Java sample configs and scripts.

While this syntax works if the entirety of the library code is in the named folder, this would cause fallbacks to fail if anyone were to remove something like model.mustache intending to fallback to resources/Java/model.mustache. We should avoid this possible confusion especially when these subdirectories are structured to follow our library lookup logic:

• User customized library path (e.g. custom_template/libraries/feign/model.mustache)
• User customized generator top-level path (e.g. custom_template/model.mustache)
• Embedded library path (e.g. resources/Java/libraries/feign/model.mustache)
• Embedded top-level path (e.g. resources/Java/model.mustache)
• Common embedded path (e.g. resources/_common/model.mustache)
• throw TemplateNotFoundException

[jimschubert]

I'd like to leave --stacktrace here rather than --info. It was taking ~2 hours to get the feedback from AppVeyor after commit and I was unable to repro the issue in Windows locally.

[jimschubert]

Notice regarding the design… template processor no longer requires that we pass that logic through a DefaultGenerator instance, which holds logic for OpenAPI input generation to templated output.

We can now generate without tight coupling to OpenAPI documents. This is work directed toward #841 and somewhat related to #843.

[jimschubert]

I found that .gitignore files sometimes don't load via this.getClass().getClassLoader().getResource(path).

For example, this works: this.getClass().getClassLoader().getResource("_common/.openapi-generator-ignore") while this did not: this.getClass().getClassLoader().getResource("graphql-nodejs-express-server/.gitignore"). This only seems to affect PHP and the GraphQL Express servers as everything else is either gitignore or gitignore.mustache (the formats documented as acceptable on Java documentation).

[jimschubert]

This is required because filenames without extensions would become .hbs or .handlebars in the above loop. A file which isn't intended to be handled like REAMDE.md would become README.hbs or README.handlebars and would result in template not found without this as-is attempt.

[jimschubert]

I had issues executing this test locally with Bash 5.0. Had to update gradle in the example so the bash sourcing would succeed. I've only updated the generated output rather than the embedded files so we're not forcing users to update to a potentially invalid Gradle version as they regenerate the kotlin multiplatform generate.

[jimschubert]

This PR completes my targeted refactor work for the "Template Engine" component defined in #845

[jimschubert]

I'll merge this tomorrow or Friday if there's no additional feedback.

[DenisKnoepfle]

JAR classpath resource path requires '/' as file separator always, must not be OS dependent, e.g. '\' under Windows won't work!
Suggestion: use TemplateManager.getCPResourcePath(libTemplateFile), may extract to similar method like embeddedTemplateExists(String)

[jimschubert]

@DenisKnoepfle are you suggesting this based on code quality or based on an error you've experienced?

This method falls back to embedded templates in the last two conditions, which allows for Windows paths evaluation using exactly what you've mentioned above.

The methods as written here allow us to check for the embedded file in operating systems without windows style paths without incurring the performance overhead of doing that adjustment.

Maybe this method's comment would be better to document it as this to avoid confusion?

     * 1) (template dir)/libraries/(library) [Windows + *nix]
     * 2) (embedded template dir)/libraries/(library) [*nix]
     * 2) (template dir) [Windows + *nix]
     * 3) (embedded template dir) [*nix]
     * 4) (embedded template dir)/libraries/(library) [Windows only]
     * 4) (embedded template dir) [Windows only]

And since we use File.separator in buildLibraryFilePath, the file check works for both Windows and non-Windows separators consistently throughout.

The alternative that you suggest would require transforming via the regular expression in TemplateManager.getCPResourcePath for all operating systems, and we really shouldn't have to pay that price on non-Windows.

[DenisKnoepfle]

Sorry for the late response, I had been on holidays.
@jimschubert I'm suggesting this based on an error I've experienced under Windows using the template path feature (resolve from classpath).

And since we use File.separator in buildLibraryFilePath, the file check works for both Windows and non-Windows separators consistently throughout.

This is what's breaking it. You must not use backslashes ('\') in a java classpath-path, it won't work, not even under Windows. Thus it must always be a forward slash ('/') when forming a classpath-path.
Is it clear now how I meant it? Did you test it under Windows trying to find a resource on a classpath that was not in the root of a jar, e.g. some.jar/some/deep/folder/JavaSpring/pojo.mustache? In my case it always resorted to the embedded dir instead of using my custom mustache template.

[jimschubert]

@DenisKnoepfle thanks for the detailed response. The issue was indeed that I was testing only with a directory structure which mirrors the resources directory in our project. However, I was also testing under Git Bash which I assume is not using the same file separators. I've never done Java development under windows, so I'm not entirely sure how to attach a remote debugger to find out.

If you don't mind, I'd like to tag you in the PR I'll open if you're able to test.

[DenisKnoepfle]

Sorry, I couldn't test it sooner, but I confirmed that it's working properly now for me under Windows (v5.0.0-beta3). Thank you!

[DenisKnoepfle]

Same issue with '\' in resource path under windows, see line 63.