Some XSpec for testing CI/CD

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "support/xspec"]
+	path = lib/xspec
+	url = https://github.com/xspec/xspec.git

Makefile

@@ -0,0 +1,54 @@
+include common/make_common.mk
+
+# We'd like to descend to each subdirectory (recursively) that has a makefile
+# Makefile wildcard function does not support that, so we use the shell
+# function with the find utility and examine each Makefile in a child dir
+# relative to this one, excluding this one to use with the FOREACH macro.
+dirs:=$(shell find '.' ! -wholename ./Makefile -name 'Makefile' -printf "%h\n")
+
+# This enables project-level configuration. Each project's own Makefile
+# can arrange its own testing accordingly, while making it easy to expose new tests to
+# the CI/CD runtime, by using any of the targets provided for here.
+
+# Make logic modeled after an example provided by NW: we are still learning
+
+module_path:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
+output_folder:=$(module_path)/test_output
+
+# $folder can be passed in as folder=[folder]
+folder?=.
+
+
+# See Makefile configuration in project files for examples of how to set up calls to 'make'
+# including wiring XSpec tests and wiring XSpec tests to these (top-level) targets 
+
+
+.PHONY: test
+test: ## Run all tests
+	$(call FOREACH_MAKE,$@,$(dirs))
+
+
+.PHONY: pre-checks
+pre-checks: ## Run all "pre checks", enforcing validation contracts across input artifacts
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))
+
+
+.PHONY: smoke-test
+smoke-test: ## Run all "smoke tests", establishing a baseline of sanity across the project
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))
+
+
+.PHONY: spec-test
+spec-test: ## Run all "specification tests", validating an implementation against a spec
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))
+
+
+.PHONY: unit-test
+unit-test: ## Run all unit tests (ci/cd)
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))
+
+
+.PHONY: clean
+clean: ## Remove any generated test or build artifacts
+	rm -fr $(output_folder)/*
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))

README.md

@@ -12,26 +12,49 @@ It is motivated primarily by the need to make available to the community and pub
 
 Most of the code here is not intended to be used standalone, but rather as imports (generally `xsl:import`) from external code. As documented per utility, however, demonstrations may be included to show how to use them, which can be executed independently of any other configuration. Additionally, some functionality has been unit tested (sometimes fairly exhaustively), with unit tests included.
 
-To use these transformations a conformant XSLT 3.0 transformation engine is required such as [Saxon 11](https://saxonica.com/documentation11/documentation.xml) from Saxonica (see SourceForge for the free-to-use HE version), which is available on several platforms and sometimes bundled with commercial software. Outside that practical requirement, this library is free to use and open for contributions.
-
-Applications here for the most part assume inputs to be OSCAL XML, as distinct from OSCAL in other data formats such as JSON and YAML. Please convert your data first into XML before attempting to work further with these tools. XSLT-based data converters for OSCAL (capable of generating OSCAL XML from valid OSCAL JSON) are available in the [main repository](https://github.com/usnistgov/OSCAL/tree/main/xml/convert).
-
-If there is interest in XSLT to support OSCAL JSON, YAML or other notations, please express this requirement to the developers.
+To use these transformations a conformant XSLT 3.0 transformation engine is required such as [Saxon 12](https://saxonica.com/documentation12/documentation.xml) from Saxonica (see [Github](https://github.com/Saxonica/Saxon-HE/) or [Maven](https://central.sonatype.com/artifact/net.sf.saxon/Saxon-HE?smo=true) for the free-to-use HE version), which is available on several platforms and sometimes bundled with commercial software. Outside that practical requirement, this library is free to use and open for contributions.
 
 ###  Project purpose and maturity
 
-The [OSCAL project](https://pages.nist.gov/OSCAL) has published XSLT since 2018. Over this time it has grown in complexity, warranting a reorganization and refactoring of the development efforts that have supported it.
-
-Current refactoring (2022) encapsulates XSLT functionality supporting OSCAL into two repositories, the [OSCAL XSLT](https://github.com/usnistgov/oscal-xslt) repository, and this one. This repository includes logic of general use (i.e., not specific to OSCAL use cases or implementation), and is kept kept separate in order to facilitate reuse of these offerings outside the OSCAL context, potentially (but not necessarily) as a git submodule.
+Some of this code originates with the [OSCAL project](https://pages.nist.gov/OSCAL), which has published XSLT since 2018. Over this time it has grown in complexity and refactored into several supporting repositories, including this one.
 
 Future development of this resource depends largely on uptake and engagement from users who find it valuable. If you have an interest in keeping this code base viable, please make your needs and ideas known to the developers. Demonstrations that take us further into features of XSLT 3 that are demonstrably useful, whether in functionality or architecture (e.g., packaging), are particularly welcome.
 
 See the readme in each project to gauge its scope of application, approach to design, and level of testing. Several of the applications are also accompanied by test suites using the [XSpec XSLT Unit Testing framework](https://github.com/xspec/xspec/).
 
-###  Repository contents
+##  Repository contents
 
 See the subdirectory list for projects and applications currently supported.
 
+Additionally to the project folders are resources at the top level:
+
+
+- `.github` - for Github CI/CD
+- `common` - holds common `bash` scripting - utilities for reuse
+- `support` - contains submodules, dynamic resources and libraries (or scripts for downloading them), also [XSpec support in development](support/xspec-dev/),
+
+## Running and building on applications
+
+As a 4GL, XSLT is well suited for certain types of applications, as it is hoped this site demonstrates. The resources offered here can be used and adapted for many different platforms and runtimes. While an effort has been made to create no dependendencies on tooling that is costly, encumbering in any way, or hard to acquire, at the same time the applications and interfaces offered and described here should not be considered to be normative or necessary, but as examples for evaluation only.
+
+## `make` support
+
+This repository supports command-line access to its tools via `make`, configured in each folder (when supported) by a `Makefile` configuration.
+
+From any directory, `make --help` from a `bash` command line will show scripted logic for that directory, if any is available.
+
+Make tasks with certain target designations can be run under CI/CD - tbd
+
+## CI/CD
+
+Currently CI/CD is set to use an XProc configuration that runs all available XSpec instances.
+
+It can be run locally on any platform with a configured Maven installation, using the `test` target:
+
+> mvn test
+
+Scripts for more granular or targeted application of XSpec under CI/CD are also available in the [XSpec-dev](xspec-dev/) application, requiring `bash` and `make`.
+
 ### Rights and license
 
 See the [LICENSE.md](LICENSE.md) file. As work product of a Bureau of the Department of Commerce (U.S. Government), or of volunteers working in support of our projects, code in this repository is in the public domain unless specifically marked otherwise.

common/make_common.mk

@@ -0,0 +1,59 @@
+# Common makefile utilities for the metaschema-xslt project
+# Note: any makefile including this file will have the following side-effects:
+# 1. The SHELL will be set to bash.
+# 2. A "help" target will be added. If the include statement is at the top the
+#   default target will become "help" unless overridden with .DEFAULT_GOAL.
+# 3. The environment will be populated with the variables seen below
+
+# Gratefully adapted from an original by NW
+
+SHELL:=/usr/bin/env bash
+
+.PHONY: help
+# Run "make" or "make help" to get a list of user targets
+# Adapted from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
+# To include a target in the help text, follow the following format:
+#   <target(s)>: [dependencies] ## Comment
+help: ## Show this help message
+	@grep --no-filename -E '^[a-zA-Z_-]+:.*?##.*$$' $(MAKEFILE_LIST) | awk 'BEGIN { \
+	 FS = ":.*?## "; \
+	 printf "\033[1m%-30s\033[0m %s\n", "TARGET", "DESCRIPTION" \
+	} \
+	{ printf "\033[32m%-30s\033[0m %s\n", $$1, $$2 }'
+
+# The path to the Maven POM to use
+pom_path:=./pom.xml
+
+define EXEC_MAVEN
+	mvn --quiet -f "${pom_path}" exec:java \
+    	-Dexec.mainClass="$1" \
+		-Dexec.args="$2"
+endef
+
+# XML Calabash helper macro
+define EXEC_CALABASH
+	$(call EXEC_MAVEN,com.xmlcalabash.drivers.Main,$1)
+endef
+
+# Saxon helper macro
+define EXEC_SAXON
+	$(call EXEC_MAVEN,net.sf.saxon.Transform,$1)
+endef
+
+define FOREACH
+	for $1 in $2; do {\
+		$3 ;\
+	} done
+endef
+
+# Run a Makefile target for each directory, requiring each directory to have a given target
+define FOREACH_MAKE
+	@echo Running makefile target \'$1\' on all subdirectory makefiles
+	@$(call FOREACH,dir,$2,$(MAKE) -C $$dir $1)
+endef
+
+# Run a Makefile target for each directory, skipping directories whose Makefile does not contain a rule
+define FOREACH_MAKE_OPTIONAL
+	@echo Running makefile target \'$1\' on all subdirectory makefiles that contain the rule
+	@$(call FOREACH,dir,$2,$(MAKE) -C $$dir -n $1 &> /dev/null && $(MAKE) -C $$dir $1 || echo "Makefile target '$1' failed or does not exist in "$$dir". Continuing...")
+endef

directory-manifest/pomx.xml → common/pom.xml

@@ -2,7 +2,7 @@
     <modelVersion>4.0.0</modelVersion>
 
     <groupId>gov.nist.secauto.oscal.tools.core</groupId>
-    <artifactId>ci-cd-deps</artifactId>
+    <artifactId>java-xml-xslt-support</artifactId>
     <version>1.0.0</version>
 
     <properties>
@@ -23,6 +23,10 @@
         redirect. https://github.com/restlet/restlet-framework-java/issues/1390
     -->
     <repositories>
+       <repository>
+          <id>Maven Central</id>
+          <url>https://repo1.maven.org/maven2</url>
+       </repository>
         <repository>
             <id>maven.restlet.org</id>
             <name>maven.restlet.org</name>
@@ -41,6 +45,27 @@
             <artifactId>xmlcalabash</artifactId>
             <version>1.5.3-110</version>
         </dependency>
+
+       <!-- https://mvnrepository.com/artifact/org.nineml/coffeegrinder -->
+       <dependency>
+          <groupId>org.nineml</groupId>
+          <artifactId>coffeegrinder</artifactId>
+          <version>3.2.6</version>
+          <scope>runtime</scope>
+       </dependency>
+       <dependency>
+          <groupId>org.nineml</groupId>
+          <artifactId>coffeefilter</artifactId>
+          <version>3.2.6</version>
+          <scope>runtime</scope>
+       </dependency>
+       <dependency>
+          <groupId>org.nineml</groupId>
+          <artifactId>coffeesacks</artifactId>
+          <version>3.2.6</version>
+          <scope>runtime</scope>
+       </dependency>
+
     </dependencies>
 
     <build>
@@ -61,5 +86,5 @@
                 </executions>
             </plugin>
         </plugins>
-    </buil>
+    </build>
 </project>

common/subcommand_common.bash

@@ -0,0 +1,35 @@
+# Provides common functions for subcommands
+
+set -Eeuo pipefail
+
+# Each subcommand will require Maven to invoke calabash or saxon
+if ! [ -x "$(command -v mvn)" ]; then
+  echo 'Error: Maven (mvn) is not in the PATH, is it installed?' >&2
+  exit 1
+fi
+
+_SUBCOMMAND_COMMON_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+POM_FILE="${_SUBCOMMAND_COMMON_DIR}/pom.xml"
+
+function exec_maven() {
+    mvn --quiet \
+        -f "$POM_FILE" \
+        exec:java \
+        -Dexec.mainClass="$1" \
+        -Dexec.args="$2"
+}
+
+CALABASH_MAIN_CLASS="com.xmlcalabash.drivers.Main"
+
+function invoke_calabash() {
+    exec_maven "$CALABASH_MAIN_CLASS" "$@"
+}
+
+SAXON_MAIN_CLASS="net.sf.saxon.Transform"
+
+function invoke_saxon() {
+    exec_maven "$SAXON_MAIN_CLASS" "$@"
+}
+
+# Clean up unneeded targets
+unset -f _SUBCOMMAND_COMMON_DIR

directory-manifest/html-to-markdown.xsl

@@ -74,13 +74,19 @@
     <xsl:text>`</xsl:text>
   </xsl:template>
 
-  <xsl:template match="em">
-    <xsl:text>*</xsl:text>
-    <xsl:apply-templates/>
-    <xsl:text>*</xsl:text>
-  </xsl:template>
-
-  <xsl:template match="q">
+   <xsl:template match="em | i">
+      <xsl:text>*</xsl:text>
+      <xsl:apply-templates/>
+      <xsl:text>*</xsl:text>
+   </xsl:template>
+
+   <xsl:template match="strong | b">
+      <xsl:text>**</xsl:text>
+      <xsl:apply-templates/>
+      <xsl:text>**</xsl:text>
+   </xsl:template>
+
+   <xsl:template match="q">
     <xsl:text>"</xsl:text>
     <xsl:apply-templates/>
     <xsl:text>"</xsl:text>

directory-manifest/manifest.md

@@ -2,7 +2,7 @@
 
 # Directory Manifest: `directory-manifest`
 
-April 12 2023 5:35 p.m. - 2023-04-12T17:35:46.4135981-04:00 -
+April 3 2024 4:34 p.m. - 2024-04-03T16:34:19.5578974-04:00 -
 
 Listing files suffixed `xml`, `xpl`, `sch`, `xsl`, `xslt`, `xsd` or `xspec`.
 
@@ -20,11 +20,11 @@ Note: Logic is extensible to handle analysis/synopsis of any XML document type
 
 ### directory-manifest.xpl
 
-Document 'c:directory' in namespace http://www.w3.org/ns/xproc-step (13 elements)
+Document 'c:directory' in namespace http://www.w3.org/ns/xproc-step (14 elements)
 
 ### html-to-markdown.xsl
 
-XSLT stylesheet version 3.0 (19 templates)
+XSLT stylesheet version 3.0 (20 templates)
 
 Purpose: Cast HTML into Markdown notation
 
@@ -64,9 +64,13 @@ Reads from (p:with-input) - `directory-listing.xsl`
 
 Reads from (p:with-input) - `html-to-markdown.xsl`
 
-### pomx.xml
+### markdown-from-html.xspec
 
-Failure reading file pomx.xml ::: [err:FODC0002] org.xml.sax.SAXParseException; systemId: file:/C:/Users/wap1/Documents/usnistgov/xslt3-functions/directory-manifest/pomx.xml; lineNumber: 64; columnNumber: 7; The element type "build" must be terminated by the matching end-tag "</build>".(The element type "build" must be terminated by the matching end-tag "</build>".)
+Document 'x:description' in namespace http://www.jenitennison.com/xslt/xspec (29 elements)
+
+### pom.xml
+
+Document 'project', in no namespace (35 elements)
 
 -----
 

directory-manifest/markdown-from-html.xspec

@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<x:description xmlns="http://www.w3.org/1999/xhtml" xmlns:x3f="http://csrc.nist.gov/ns/xslt3-functions"
+   xmlns:x="http://www.jenitennison.com/xslt/xspec" xmlns:xs="http://www.w3.org/2001/XMLSchema"
+   stylesheet="html-to-markdown.xsl" xslt-version="3.0">
+
+   <!--
+      Purpose: A very basic XSpec to demonstrate principles and a pattern for testing
+   HTML to Markdown conversion and mapping.
+
+   Comprehensive coverage even of current functionality is not yet attempted.
+
+   -->
+   <x:scenario label="directory manifest - HTML to Markdown conversion">
+      <x:scenario label="Page structure">
+         <x:context>
+            <html>
+               <head>
+                  <title>A page</title>
+                  <style type="text/css">/* some style */</style>
+               </head>
+               <body>
+                  <h1>Frankenstein</h1>
+                  <h2>Or, the Modern Prometheus</h2>
+               </body>
+            </html>
+         </x:context>
+         <x:expect label="lets its contents through" xml:space="preserve">
+
+# Frankenstein
+
+## Or, the Modern Prometheus</x:expect>
+      </x:scenario>
+
+      <x:scenario label="Basic p">
+         <x:context>
+            <p>Four score and seven years ago --</p>
+         </x:context>
+         <x:expect label="comes through with line feeds" xml:space="preserve">
+
+Four score and seven years ago --</x:expect>
+      </x:scenario>
+
+      <x:scenario label="Basic inlines">
+         <x:context>our fathers brought forth, on this continent, a <em>new nation</em>, conceived in <strong>Liberty</strong> and dedicated</x:context>
+         <x:expect label="are mapped">our fathers brought forth, on this continent, a *new nation*, conceived in **Liberty** and dedicated</x:expect>
+      </x:scenario>
+
+      <x:scenario label="Code scrub">
+         <x:scenario label="A comment">
+            <x:variable name="x3f:a-comment"><!--comment--></x:variable>
+            <x:context select="$x3f:a-comment"/>
+            <x:expect label="is dropped" select="()"/>
+         </x:scenario>
+         <x:scenario label="A Processing instruction">
+            <x:context><?pi type="sample" href="file.css"?></x:context>
+            <x:expect label="is dropped" select="()"/>
+         </x:scenario>
+      </x:scenario>
+   </x:scenario>
+
+</x:description>