Refactoring loading build properties

README.md

@@ -13,7 +13,7 @@ zAppBuild is a generic build solution for building z/OS applications using Apach
 For instructions on how to contribute enhancements and bug fixes to zAppBuild, please read the [Contributions Guidelines](CONTRIBUTIONS.md).
 
 ## How zAppBuild works
-The zAppBuild repository is intended to be cloned to a single location on Unix Systems Services (USS) and used to build all of your z/OS applications. This is done by simply copying the supplied `application-conf` folder (located in the [samples folder](samples)) to the application source repository you want to build and then verify/update the contained default configuration property values to ensure they meet the build requirements of your application. See the included [MortgageApplication](samples/MortgageApplication) sample for an example of an application that has been modified to be built by zAppBuild.  
+The zAppBuild repository is intended to be cloned to a single location on Unix Systems Services (USS) and used to build all of your z/OS applications. Global configuration properties are configured in the properties files in the [build-conf](build-conf/) directory. Specifying application-level properties is done by simply copying the supplied `application-conf` folder (located in the [samples folder](samples)) to the application source repository you want to build and then verify/update the contained default configuration property values to ensure they meet the build requirements of your application. See the included [MortgageApplication](samples/MortgageApplication) sample for an example of an application that has been modified to be built by zAppBuild.  
 
 **IMPORTANT** : The [datasets.properties](build-conf/datasets.properties) must be configured for your build machine before executing a build!  See [build-conf/README.md](build-conf/README.md) for more information.
 

build-conf/build.properties

@@ -6,12 +6,16 @@
 
 
 #######################################################################
-# Global build properties used by zappbuild
+# Global build properties used by zAppBuild
 #
 
 #
 # Comma separated list of additional build property files to load
 # Supports both relative path (to zAppBuild/build-conf/) and absolute path
+#
+# These properties files expect to contain centrally managed defaults
+#  such as system datasets, language script specific settings  
+#
 buildPropFiles=datasets.properties,dependencyReport.properties,Assembler.properties,BMS.properties,\
 MFS.properties,PSBgen.properties,DBDgen.properties,ACBgen.properties,Cobol.properties,\
 LinkEdit.properties,PLI.properties,REXX.properties,ZunitConfig.properties,Transfer.properties,\
@@ -20,38 +24,76 @@ CRB.properties
 #
 # Comma separated list of default application configuration property files to load
 # Supports both relative path (to zAppBuild/build-conf/) and absolute path
+# 
+# These properties files expect to contain centrally managed defaults
+#  and also may reference properties files containing configuration
+#  of the language script configurations such as return codes, deploy types   
+# 
+# See also application-conf/application.properties#applicationPropFiles
+# 
+# default: 
+# applicationDefaultPropFiles=defaultzAppBuildConf.properties
+#
+# extended sample to set default language script configurations:
+#
+# applicationDefaultPropFiles=defaultzAppBuildConf.properties,\
+# default-application-conf/searchPaths.properties,\
+# default-application-conf/scriptMappings.properties,\
+# default-application-conf/Cobol.properties,\
+# default-application-conf/BMS.properties,\
+# default-application-conf/PLI.properties,\
+# default-application-conf/Transfer.properties,\
+# default-application-conf/LinkEdit.properties,\
+# default-application-conf/ZunitConfig.properties
+# 
 applicationDefaultPropFiles=defaultzAppBuildConf.properties
 
 #
-# file extension that indicates the build file is really a build list or build list filter
-buildListFileExt=txt
-
+# applicationConfDir
+#
+# required build property that specifies the directory of the
+# application specific build configurations for zAppBuild, also known
+# as application-conf folder
 #
-# Alternate root directory for application-conf locations.  Allows for the deployment of
-# the application-conf directories to an alternate location rather in the application repository.
-# The expectation is that the root directory will have subfolders for all of the applications built
-# by zAppBuild in which the actual application-conf directory is located:
+# a sample of the application-conf can be found in
+#   samples/application-conf
+#
+# zAppBuild expects a file called application.properties in this directory. 
+#
+# The property also allows for the deployment of
+# the application-conf directories to an alternate location rather
+# in the application repository.
+#
+# Default:
+# Location within the application repository and resolving the configuration
+# based on workspace + application and application-conf
+#
+# applicationConfDir=${workspace}/${application}/application-conf
+# 
 #
 # Example: Static location on USS
-# applicationConfRootDir=/u/build/config/applications
+# applicationConfDir=/u/build/config/applications/${application}/application-conf
 # |- /u/build/config/applications
 # |                       |- App1
 # |                       |    |- application-conf
 # |                       |    |        |- application.properties
 # |                       |- App2
 # |                       |    |- application-conf
 #
-# Example: Application cofig files stored in zAppBuild
-# applicationConfRootDir=${zAppBuildDir}/applications
+# Example: Application config files stored in zAppBuild
+# applicationConfDir=${zAppBuildDir}/applications/${application}/application-conf
 # |- /u/build/zAppBuild/applications
 # |                          |- App1
 # |                          |    |- application-conf
 # |                          |    |        |- application.properties
 # |                          |- App2
 # |                          |    |- application-conf
 #
-# Defaults to ${workspace}
-applicationConfRootDir=
+applicationConfDir=${workspace}/${application}/application-conf
+
+#
+# file extension that indicates the build file is really a build list or build list filter
+buildListFileExt=txt
 
 #
 # Determine if a subfolder with a timestamp should be created in the buildOutDir location.

build.groovy

@@ -313,7 +313,7 @@ options:
 
 /*
  * populateBuildProperties - loads all build property files, creates properties for command line
- * arguments and sets calculated propertied for he build process
+ * arguments and sets calculated propertied for the build process
  */
 def populateBuildProperties(def opts) {
 
@@ -333,52 +333,64 @@ def populateBuildProperties(def opts) {
 
 	// load build.properties
 	def buildConf = "${zAppBuildDir}/build-conf"
-	props.load(new File("${buildConf}/build.properties"))
+	if (opts.v) println "** Loading property file ${buildConf}/build.properties"
+	buildUtils.loadBuildProperties("${buildConf}/build.properties")
 
 	// load additional build property files
+	if (opts.v) println "** Loading zAppBuild build properties"
 	if (props.buildPropFiles) {
 		String[] buildPropFiles = props.buildPropFiles.split(',')
 		buildPropFiles.each { propFile ->
 			if (!propFile.startsWith('/'))
 				propFile = "${buildConf}/${propFile}"
-
+			
 			if (opts.v) println "** Loading property file ${propFile}"
-			props.load(new File(propFile))
+			buildUtils.loadBuildProperties(propFile)
 		}
 	}
 
 	// load additional build property files
+	if (opts.v) println "** Loading default application properties"
 	if (props.applicationDefaultPropFiles) {
 		String[] applicationDefaultPropFiles = props.applicationDefaultPropFiles.split(',')
 		applicationDefaultPropFiles.each { propFile ->
 			if (!propFile.startsWith('/'))
 				propFile = "${buildConf}/${propFile}"
 
 			if (opts.v) println "** Loading property file ${propFile}"
-			props.load(new File(propFile))
+			buildUtils.loadBuildProperties(propFile)
 		}
 	}
 
 
 	// load application.properties
-	String appConfRootDir = props.applicationConfRootDir ?: props.workspace
-	if (!appConfRootDir.endsWith('/'))
-		appConfRootDir = "${appConfRootDir}/"
-
-	String appConf = "${appConfRootDir}${props.application}/application-conf"
-	if (opts.v) println "** appConf = ${appConf}"
-	props.load(new File("${appConf}/application.properties"))
-
-	// load additional application property files
-	if (props.applicationPropFiles) {
-		String[] applicationPropFiles = props.applicationPropFiles.split(',')
-		applicationPropFiles.each { propFile ->
-			if (!propFile.startsWith('/'))
-				propFile = "${appConf}/${propFile}"
-
-			if (opts.v) println "** Loading property file ${propFile}"
-			props.load(new File(propFile))
+	String appConf = props.applicationConfDir
+	if (appConf.endsWith('/'))
+		appConf = appConf.substring(0, appConf.length() - 1)
+
+	if (opts.v) println "** Loading application specific properties"
+	if (opts.v) println "** applicationConfDir = ${appConf}"
+
+	applicationProperties = "${appConf}/application.properties"
+	applicationPropertiesFile = new File(applicationProperties)
+
+	if (applicationPropertiesFile.exists()) {
+		if (opts.v) println "** Loading property file ${applicationProperties}"
+		buildUtils.loadBuildProperties(applicationProperties)
+
+		// load additional application property files
+		if (props.applicationPropFiles) {
+			String[] applicationPropFiles = props.applicationPropFiles.split(',')
+			applicationPropFiles.each { propFile ->
+				if (!propFile.startsWith('/'))
+					propFile = "${appConf}/${propFile}"
+
+				if (opts.v) println "** Loading property file ${propFile}"
+				buildUtils.loadBuildProperties(propFile)
+			}
 		}
+	} else {
+		if (opts.v) println "*! Properties file ${applicationProperties} was not found. Build continues."
 	}
 
 	// load property files from argument list
@@ -389,8 +401,8 @@ def populateBuildProperties(def opts) {
 			if (!propFile.startsWith('/'))
 				propFile = "${props.workspace}/${propFile}"
 
-			if (opts.v) println "** Loading property file ${propFile}"
-			props.load(new File(propFile))
+			if (opts.v) println "** Loading property file ${propFile}"	
+			buildUtils.loadBuildProperties(propFile)
 		}
 	}
 

docs/HOWTOS.md

@@ -4,6 +4,7 @@ This page collects information to configure and use specific features of zAppBui
 
 - [How-to and Frequently Asked Questions](#how-to-and-frequently-asked-questions)
   - [Signing load modules and program objects](#signing-load-modules-and-program-objects)
+  - [Reduce necessary **application-conf** configurations within the application repository](#reduce-necessary-application-conf-configurations-within-the-application-repository)
 
 
 ## Signing load modules and program objects
@@ -81,4 +82,53 @@ In the output of amblist, locate the user data section and find the data that th
           DATE         USER DATA
 CSECT:    EPSCMORT
           06/19/2023   MortgageApplication/a21b6ab0
-```
+```
+
+## Reduce necessary **application-conf** configurations within the application repository
+
+**The 3 properties for loading configuration files to configure the build framework**
+
+zAppBuild proposes 3 properties called `buildPropFiles`, `applicationDefaultPropFiles` and `applicationPropFiles`, each referencing a list of properties files that contain important parameters to configure the build process. The referenced properties files are loaded in the below order of precedence: 
+
+1. `buildPropFiles`, managed in [build-conf/build.properties](../build-conf/build.properties), references properties files in the [build-conf](../build-conf/) directory for core zAppBuild settings for the language scripts such as system datasets, naming conventions of build datasets, dataset characteristics and various core properties of the build framework, like the reporting features.
+2. `applicationDefaultPropFiles`, managed in [build-conf/build.properties](../build-conf/build.properties) as well, allows the user to define default application-related (and "language script"-related) properties, that are centrally managed and shared across applications.
+3. `applicationPropFiles` is referencing properties files providing application-level settings. This property is managed in the applications' `application-conf/application.properties` file which is by default located in the applications' [application-conf](../samples/application-conf/) folder. These settings define compiler and link options, deploy types, script mappings and search path configurations for the DBB dependency and impact analysis. A sample is provided in [application-conf](../samples/application-conf/). The location where zAppBuild searches for the `application.properties` file is defined via the `applicationConfDir` setting that is managed in [build-conf/build.properties](../build-conf/build.properties). The default location mandates an `application-conf` folder including the `application.properties` file to be present in the applications' git repository.
+
+The `buildPropFiles` and `applicationDefaultPropFiles` settings define enterprise-level, centrally-controlled properties, that are used by all applications using the build framework. They are shared across all applications.
+
+Historically, a lot of application-level properties are configured and provided via the `applicationPropFiles` within the `application-conf` directory. However, users have reported that zAppBuild is exposing far too many properties to the application team, which also makes it hard to control the update process for new or modified properties. For example, most customers prefer to manage compiler and binder options in the centrally-controlled settings.
+
+This how-to outlines the changes to centrally manage default application settings.
+
+**Centrally defining application-related properties** 
+
+Return codes, deploy types, script mappings or search path configurations for the DBB dependency and impact analysis can be centrally configured especially when application teams follow the similar repository layout and have similar application architectures. It is desireable to avoid having multiple copies of files with the same definitions spread across multiple locations.
+
+Loading common properties files via the `applicationDefaultPropFiles` setting helps to achieve this easily. For instance, the below configuration of `applicationDefaultPropFiles` is loading properties that define the search path configurations, script mappings, and various language settings (such as compiler or linker options) that can be applied to all applications using zAppBuild.
+
+```properties
+..
+# Extended list of applicationDefaultPropFiles to include
+# default application settings for language scripts
+applicationDefaultPropFiles=defaultzAppBuildConf.properties,\
+default-application-conf/searchPaths.properties,\
+default-application-conf/scriptMappings.properties,\
+default-application-conf/Cobol.properties,\
+default-application-conf/BMS.properties,\
+default-application-conf/PLI.properties,\
+default-application-conf/Transfer.properties,\
+default-application-conf/LinkEdit.properties,\
+default-application-conf/ZunitConfig.properties
+..
+```
+This allows to reduce the necessary definitions within the `application-conf` directory of the application repository. This `application-conf` directory now only contains the `application.properties` and `file.properties` files to define the application-specific settings and exceptions, such as [file properties](../docs/FilePropertyManagement.md#dbb-file-properties) for particular build files:
+
+```properties
+# Reduced list of applicationPropFiles
+applicationPropFiles=file.properties
+```
+Potentially, you could go farther and merge the properties defined in `file.properties` into the `application.properties` file.
+
+If an application doesn't need to specify any application-specific settings, there is no need to create an `application.properties` file and zAppBuild will execute the build, even without finding the `application.properties` file expected at the `applicationConfDir` location.
+
+Please note that moving property files to the central build framework implementation disables the capability to perform impactBuilds on a property change for these properties - see `impactBuildOnBuildPropertyChanges` setting at [default](../build-conf/defaultzAppBuildConf.properties).

samples/application-conf/README.md

@@ -1,12 +1,12 @@
 # Application Configuration
 This folder contains application specific configuration properties used by the zAppBuild Groovy build and utility scripts. It is intended to be copied as a high level folder in the application repository or main application repository if the application source files are distributed across multiple repositories. Once copied to the application repository, users should review the default property files and modify any values as needed.
 
-At the beginning of the build, the `application-conf/application.properties` file will automatically be loaded into the [DBB BuildProperties class](https://www.ibm.com/support/knowledgecenter/SS6T76_1.0.4/scriptorg.html#build-properties-class). Use the `applicationPropFiles` property (see table below) to load additional application property files.
+At the beginning of the build, the `application.properties` file will automatically be searched and loaded if it exists into the [DBB BuildProperties class](https://www.ibm.com/docs/en/dbb/2.0?topic=apis-build-properties#build-properties-class). The `application.properties` file is by default searched in the `application-conf` folder of the application, but this can be configured through the `applicationConfDir` property in [build-conf/build.properties](../../build-conf/build.properties). Use the `applicationPropFiles` property (see table below) to load additional application property files.
 
-Properties can be overwritten on a per file basis through DBB Build Properties file properties. The tables below indicate which properties keys can be overwritten. It is recommended to manage these overwrites in file.properties.
+Properties can be overwritten on a per file basis through DBB Build Properties file properties. The tables below indicate which properties keys can be overwritten. It is recommended to manage these overwrites in `file.properties`.
 
 ## Property File Descriptions
-Since all properties will be loaded into a single static instance of BuildProperties, the organization and naming convention of the *property files* are somewhat arbitrary and targeted more for self documentation and understanding.
+Since all properties will be loaded into a single static instance of BuildProperties, the organization and naming convention of the *property files* are somewhat arbitrary and targeted more for self documentation and understanding. Properties related to a language script are prefixed with the name of the language script (i.e `cobol_compileParms`).
 
 ### application.properties
 This property file is loaded automatically at the beginning of the build and contains application specific properties used mainly by `build.groovy` but can also be a place to declare properties used by multiple language scripts. Additional property files are loaded based on the content of the `applicationPropFiles` property.

utilities/BuildUtilities.groovy

@@ -872,6 +872,23 @@ def printLogicalFileAttributes(LogicalFile logicalFile) {
 
 }
 
+/**
+ * method to load build properties into the DBB Build properties.
+ * 
+ * takes the path to the property file, validates if the property file exist
+ *  
+ */
+
+def loadBuildProperties(String propertyFile) {
+	File propFile = new File("$propertyFile")
+	if (propFile.exists()) {
+		props.load(propFile)
+	} else {
+		println "*!* The specified $propertyFile does not exist. Build exits."
+		System.exit(1)
+	}
+}
+
 /**
  * Validates if a buildFile is a zUnit generated test case program
  * 
@@ -884,3 +901,4 @@ def isGeneratedzUnitTestCaseProgram(String buildFile) {
 	}
 	return false
 }
+