IBM User Build - HowTo - Manage Git context during User Build (and deploiement)

[FALLAI-Denis]

Description of the enhancement requested

Hi,

We use VS Code as source editor.
We work in a Git context, both for editing source codes in ZOE, and in DBB to do the builds.

Our DBB build processes are dependent on the Git context (Git branch) they are in.

When doing a DBB fullBuild or more often a DBB impactBuild, the build.groovy script runs in a workspace containing the cloned Git repository(ies). The build.groovy script uses Git functions to know the context of the Git branch in which it operates.

When doing a DBB userBuild, the build.groovy script does not run in a workspace containing the cloned Git repository: it runs in a workspace created and populated by ZOE "as best it can".
Consequences :

• the build.groovy script don't know in which Git context it operate: feature branch or hotfix branch? (our COBOL compilation options depend on the Git branch, debugging options on feature, performance options on hotfix)
• components may be missing which are necessary for the construction because, except for the includes/copies, ZOE does not know the logical dependencies between the components (cf. IBM User Build - Upload additional files at build run time #297)
• you can't do impactBuild: deliver a COPYBOOK modification and rebuild the programs that use this COPYBOOK, (dependencies information are kept in DBB metadata store, not in Git repository, and ZOE doesn't know DBB metadata store). ZOE could manage the dependencies between COPYBOOK and programs inside the Workspace VS Code, but that would force him to scan all the COBOL sources of the Wokspace to discover the uses of COPYBOOKs, and in addition to do it in cascade because the COPYBOOKs can be referenced by other COPYBOOKs and not directly by COBOL programs, (I don't think it is possible to ask ZOE to do the impact calculation work).

Another problem: after build, you have to deploy the built component(s) (loadmodule, dbrm, listing for debug tools...) for testing.
Here too, the test environment depends on the Git context: we use different test environments (shared between projects) depending on whether we are in development (Git feature branch) or in maintenance (Git hotfix branch).

To solve some of these problems, it would be necessary to be able to pass informations about the current Git context to the build script, at least the name of the Git branch, as parameters.

Either we can do it dynamically: provide a symbolic expression or variable on the name of the Git branch in the parameters of the DBB profile declared in the zapp.yaml file, but it seems that ZOE does not know how to resolve expressions other than those based on the variable ${buildfile) and its variants. In particular ZOE doesn't seem able to resolve VS Code's context variables.

Either you have to do it statically: create as many DBB profiles in the zapp.yaml file with a fixed parameter indicating the type of the Git branch (and not the name of the Git branch which is never the same), but then you need a mechanism that selects the right DBB profile during the "setup" and when running IBM User Build.

What solutions would you suggest to manage these needs?

[phaumer]

@FALLAI-Denis thanks for the feedback. I might misunderstand your environment, but this what I assume and suggest:

• User Build only builds one program at the time. It operates independent of the user using Git or not.
• When running the build operation in the editor it uses the language server to determine the list of copybooks.
• It then checks with the property groups which copybooks are local and which are remote.
• It then uploads the program and the local copybooks to USS for dbb-zappbuild. It ignores remote copybooks and assumes that DBB knows how to find those via its own properties files (something we should optimize in the future to eliminate redundancies).

So for the local copybooks that get uploaded we assume in the Git case that they are either (a) from the same git repo and checked out together with the program or (b) come from another Git repo in a VS Code multi-root workspace and were checked out with the correct branch by the developer as well.

As the programs and copybooks can be different in each branch so have the zapp files to be different and match in their content to this one branch. Each branch could have a different variant of the zapp file. The zapp files are not intended to be orthogonal to the git repo files or know about other branches, but be a part of each branch. This would be the same for Maven pom files, Groovy files, C Makefiles, or Nodejs package.json files in other programming models.

To better support the multi-root/multi-repository use case in which the copybooks are in a separate repo and VS Code workspace we added the multi-root support in 2.2 in which now every workspace can have its own ZAPP file at the top to contain the property-groups for this repository. Z Open Editor will concatenate all the property groups of all the workspaces when evaluating the user build to find all the copybooks in the multi-root workspace.

Here is a sample for such as multi-root configuration with multiple zapp files: https://github.com/IBM/zopeneditor-sample/tree/wazi-main/multiroot

There are other enhancement requests open that ask for more context of the property groups such as zapp files in sub-folders that select different property groups for different programs, which we agree we need to add as well. So setting a context based on the zapp's folder location and perhaps even environment variables or user settings is something we do want to consider.

[FALLAI-Denis]

Hi,

Each branch can have a different variant of the zapp file.

Since zapp files are files among others, they are impacted during Git branch merge operations. I don't see how to have different zapp files per Git branch.
If they are declared in .gitignore, they will not be created during a checkout and it is the developer who will have to create them each time.

multi-root/multi-repository use case where copybooks are in a separate repository and VS Code workspace

This is not our use case. We operate with a simple workspace. The external Git repositories containing the copybooks are dynamically cloned inside the workspace, in a folder declared in .gitignore, by a mechanism that we have put in place.
We have a configuration system that allows you to associate a Git branch of the main Repository (programs) with a list of couples External Git Repository / branch of the external Git Repository (copybooks).

---

Our need is to be able to let DBB know in a userBuild context which Git branch the program sources that have been downloaded come from so that it can adapt the compilation procedure accordingly:

• we do not compile with the same options depending on whether the source program comes from a feature branch or comes from a hotfix branch
• we do not deploy on the same environment depending on whether the source program comes from a feature branch or comes from a hotfix branch

It is therefore necessary to be able to pass as a parameter, to be declared in the DBB profile of the zapp file, the type of the Git branch in use in the Git Repository which is also the VS Code workspace.

[phaumer]

Since zapp files are files among others, they are impacted during Git branch merge operations.

That's a good thing, though. If you make a change to a program or copybook and merge then you also need to check if you need to merge changes into the zapp files as well. That is exactly the strength of Git merging, to require you to go from one consistent state to the next across all your files. If you reject certain changes in the merge it will remember that by looking at the timestamp of your changes and not ask you again.

dynamically cloned inside the workspace, in a folder declared in .gitignore

That sounds like you are working around Git here, which seems a very special solution for your org.

DBB know in a userBuild context

We are going to make some proposals for defining a context in ZAPP. We want this to work similar to Maven Profiles that also become active for such a context condition. So far we are thinking of zapp folder location, environment variables, user settings. We will discuss how those can be utilized for your use cases.

[FALLAI-Denis]

dynamically cloned inside the workspace, in a folder declared in .gitignore
That sounds like you are working around Git here, which seems a very special solution for your org.

We reproduced a mechanism similar to that of Git submodules, but in a proprietary way, because we were advised against using Git submodules.

We are cloning inside mono-root Workspace, so in a subfolder of the main Git Repository, to avoid having conflicts when cloning outside Workspace because several instances of VS Code could be launched in parallel referring to the same imported secondary Git repositories, but not for the same branch nor with the same sparse-checkout filtering.