Merge pull request #769 from viash-io/develop

Develop

.github/workflows/sbt_test.yml

@@ -16,7 +16,7 @@ jobs:
         java:
           - { ver: '11', run_nextflow: true, run_coverage: false, nxf_ver: "22.04.5" }
           - { ver: '17', run_nextflow: true, run_coverage: true, nxf_ver: latest }
-          - { ver: '20', run_nextflow: true, run_coverage: false, nxf_ver: latest-edge }
+          - { ver: '21', run_nextflow: true, run_coverage: false, nxf_ver: latest-edge }
 
     steps:
     - uses: actions/checkout@v4

.gitignore

@@ -36,4 +36,5 @@ work
 .DS_Store
 
 # ignore automatically generated WorkflowHelper.nf
+src/main/resources/io/viash/runners/nextflow/WorkflowHelper.nf
 src/main/resources/io/viash/platforms/nextflow/WorkflowHelper.nf

CHANGELOG.md

@@ -1,3 +1,155 @@
+# Viash 0.9.0 (2024-09-03): Restructure platforms into runners and engines
+
+This release restructures the introduces changes to the Viash config:
+- The `platforms` field is split into `runners` and `engines`
+- The `.functionality` layer has been removed from the config and all fields have been moved to the top layer
+
+Changes are made to sanitize the built config output and include additional relevant meta data.
+The default `multiple_sep` has been changed from `:` to `;` to avoid conflicts with paths like `s3://foo/bar`.
+
+Implemented a proper way of caching dependency repositories. The cache is stored under `~/.viash/repositories`.
+
+## BREAKING CHANGES
+
+* `runners` and `engines`: The usage of `platforms` is deprecated and instead these are split into `runners` and `engines` (PR #510). 
+  The `platforms` field is still supported but will be removed in a future release.
+  In brief, the `native platform` became a `native engine` and `docker platform` became a `docker engine`.
+  Additionally, the `native platform` and `docker platform` became a `executable runner`, `nextflow platform` became a `nextflow runner`.
+  The fields of `docker platform` is split between `docker engine` and `docker runner`: `port`, `workdir`, `setup_strategy`, and `run_args` (set to `docker_run_args`) are captured by the `runner` as they define how the component is run. The other fields are captured by the `engine` as they define the environment in which the component is run. One exception is `chown` which is rarely set to false and is now always enabled.
+
+* `arguments`: Merge arguments into argument_groups during a json decode prepare step (PR #574). The `--parse_argument_groups` option from `ns list` and `config view` is deprecated as it is now always enabled.
+
+* `arguments`: Change default `multiple_sep` from `:` to `;` to avoid conflicts with paths like `s3://foo/bar` (PR #645).
+  The previous behaviour of using `multiple_sep: ":"` can be achieved by adding a config mod to the `_viash.yaml`:
+  ```yaml
+  config_mods: |
+    .functionality.argument_groups[true].arguments[.multiple == true].multiple_sep := ":"
+  ```
+
+* `functionality`: Remove the `functionality` layer from the config and move all fields to the top layer (PR #649).
+
+* `computational requirements`: Use 1000-base units instead of 1024-base units for memory (PR #686). Additionally, the memory units `kib`, `mib`, `gib`, `tib`, and `pib` are added to support 1024-base definitions.
+
+* `NextflowEngine`: Swap the order of execution of `runIf` and `filter` when calling `.run()`. This means that `runIf` is now executed before `filter` (PR #660).
+
+## NEW FUNCTIONALITY
+
+* `export json_schema`: Add a `--strict` option to output a subset of the schema representing the internal structure of the Viash config (PR #564).
+
+* `config view` and `ns list`: Do not output internal functionality fields (#564). Additionally, add a validation that no internal fields are present when reading a Viash config file.
+
+* `project config`: Add fields in the project config to specify default values for component config fields (PR #612). This allows for a more DRY approach to defining the same values for multiple components.
+
+* `dependencies`: GitHub and ViashHub repositories now get properly cached (PR #699).
+  The cache is stored in the `~/.viash/repositories` directory using sparse-checkout to only fetch the necessary files.
+  During a build, the cache is checked for the repository and if it is found and still up-to-date, the repository is not cloned again and instead the cache is copied to a temporary folder where the files are checked out from the sparse-checkout.
+
+* `ExecutableRunner`: Add a `---docker_image_id` flag to view the Docker image ID of a built executable (PR #741).
+
+* `viash ns query`: Add a query filter that allows selecting a single component by its path in a namespace environment (PR #744).
+
+* `config schema`: Add `label` & `summary` fields for Config, PackageConfig, argument groups, and all argument types (PR #743).
+
+* `NextflowEngine`: Added `runIf` functionality to `runEach` (PR #660).
+
+## MINOR CHANGES
+
+* `testbenches`: Add testbenches for local dependencies (PR #565).
+
+* `testbenches`: Refactor testbenches helper functions to uniformize them (PR #565).
+
+* `logging`: Preserve log order of StdOut and StdErr messages during reading configs in namespaces (PR #571).
+
+* `Java 21 support`: Update Scala to 2.13.12 and update dependencies (PR #602).
+
+* `project config`: Output the project config under the default name `ProjectConfig` instead of `Project` during schema export (PR #631). This is now important as the project config is now part of the component config. Previously this was overridden as the class name was `ViashProject` which was less descriptive.
+
+* `package config`: Renamed `project config` to `package config` (PR #636). Now that we start using the config more, we came to the conclusion that "package" was better suited than "project".
+
+* `ns exec`: Added an extra field `{name}` to replace `{functionality-name}` (PR #649). No immediate removal of the old field is planned, but it is deprecated.
+
+* `BashWrapper`: Added meta-data field `meta_name` as a replacement for `meta_functionality_name` (PR #649). No immediate removal of the old field is planned, but it is deprecated.
+
+* `error message`: Improve the error message when using an invalid field in the config (#PR #662). The error message now includes the field names that are not valid if that happens to be the case or otherwise a more general error message.
+
+* `config mods`: Improve the displayed error message when a config mod could not be applied because of an invalid path (PR #672).
+
+* `docker_engine`: Deprecate `registry`, `organization` and `tag` fields in the `docker_engine` (PR #712). Currently these are hardly ever used and instead the `image` field is used to specify the full image name.
+
+* `docker_engine`: Add `target_package` field to the `docker_engine` (PR #712). This field, together with the `target_organization` is used to specify the full built container image name. The fields use proper fallback for the values set in the component config and package config.
+
+* `organization`: Remove the `organization` field from the component config (PR #712). The value is now directly used by the `docker_engine` as a fallback from the `target_organization` field.
+
+* `ExecutableRunner`: Add parameter `docker_automount_prefix` to allow for a custom prefix for automounted folders (PR #739).
+
+* `ExecutableRunner`: Make Docker runtime arguments configurable via the `---docker_run_args` argument (PR #740).
+
+* `export json_schema`: Add `arguments` field to the `Config` schema (PR #755). Only for the non-strict version, the strict version of the viash config has these values merged into `argument_groups`.
+
+* `scala`: Update Scala to 2.13.14 (PR #764).
+
+* `NextflowEngine`: Also parse `${id}` and `${key}` aside from `$id` and `$key` as identifier placeholders for filenames (PR #756).
+
+## BUG FIXES
+
+* `__merge__`: Handle invalid yaml during merging (PR #570). There was not enough error handling during this operation. Switched to the more advanced `Convert.textToJson` helper method.
+
+* `config`: Anonymize paths in the config when outputting the config (PR #625).
+
+* `schema`: Don't require undocumented fields to set default values and add the `links` and `reference` fields to functionality as they were not meant only to be in the project config (PR #636).
+
+* `export json_schema`: Fix minor inconsistencies and make the strict schema stricter by adapting to what Viash will effectively return (PR #666).
+
+* `deprecation & removal warning`: Improve the displayed warning where a deprecated or removed field could display a double '.' when it field was located at the root level (PR #671).
+
+* `resource path`: Don't finalize the `path` field of a resource until it's written as part of building a component (PR #668).
+
+* `requirements`: Improve the error message when a Python or R requirement uses a single quote in the `.script` field (PR #675).
+
+* `viash test`: Fix Docker id between build and test components not being consistent when using a custom Docker registry (PR #679).
+
+* `MainNSBuildNativeSuite`: Capture the error message when reading the configs so we can capture the expected warning message (PR #688).
+  While almost all tests were already cleanly capturing their expected warning/error messages, this one was still remaining, resulting in warnings being shown in the output.
+
+* `runners & engines`: When applying a filter on empty runners or engines, the fallback default `native engine` and `executable runner` respectively are set before applying the filter (PR #691).
+
+* `dependencies`: Fix resolving of dependencies of dependencies (PR #701). The stricter build config was now lacking the necessary information to resolve dependencies of dependencies.
+  We added it back as `.build_info.dependencies` in a more structured, anonymized way.
+
+* `dependencies`: Fix the `name` field of repositories possibly being outputted in the build config (PR #703).
+
+* `symlinks`: Allow following of symlinks when finding configs (PR #704). This improves symlink functionality for `viash ns ...` and dependency resolving.
+
+* `build_info`: Correctly set the `.build_info.executable` to `main.nf` when building a component with a Nextflow runner (PR #720).
+
+* `vsh organization`: ViashHub repositories now use `vsh` as the default organization (PR #718).
+  Instead of having to specify `repo: vsh/repo_name`, you can now just specify `repo: repo_name`, which is now also the prefered way.
+
+* `testbenches`: Add a testbench to verify dependencies in dependencies from scratch (PR #721).
+  The components are built from scratch and the dependencies are resolved from the local repositories.
+
+* `docker_engine`: Fix a bug in how the namespace separator is handled (PR #722).
+
+* `platforms`: Re-introduce the `--platform` and `--apply_platform` arguments to improve backwards compatibility (PR #725).
+  When the argument is used, a deprecation warning message is printed on stderr.
+  Cannot be used together with `--engine` or `--runner` and/or `--apply_engine` or `--apply_runner`.
+
+* `nextflow_runner`: Fix refactoring error in the `findStates()` helper function (PR #733).
+
+* `viash ns exec`: Fix "relative fields" outputting absolute paths (PR# 737). Additionally, improve path resolution when using the `--src` argument.
+
+* `viash ns`: Fix viash tripping over its toes when it encounters multiple failed configs (PR #761). A dummy config was used as a placeholder, but it always used the name `failed`, so duplicate config names were generated, which we check for nowadays.
+
+* `bashwrapper`: Fix an issue where running `viash test` which builds the test docker container would ignore test failures but subsequential runs would work correctly (PR #754).
+
+* `NextflowEngine`: Fix escaping of odd filename containing special characters (PR #756). Filenames containing a `$` character caused Bash to try to interpret it as a variable.
+
+* `json schema`: Fix repositories types with name incorrectly adding `withname` as type (PR #768).
+
+* `json schema`: Change the '$schema' field to 'http://' instead of 'https://' (PR #768). (Some?) Json validators use this value as a token and not as a URL.
+
+* `viash test`: Fix an issue where the tests would not copy package config settings to determine the docker image name (PR #767).
+
 # Viash 0.8.6 (2024-04-26): Bug fixes and improvements for CI
 
 Fix some issues in some edge cases.
@@ -181,6 +333,8 @@ We added new 'dependencies' functionality to allow for more advanced functionali
 
 * `NextflowPlatform`: Throw error when unexpected keys are passed to the `.run()` method (#512, PR #518).
 
+* `Testbenches`: Add testbenches for the new `dependencies` functionality and other small coverage improvements (PR #524).
+
 * `NextflowPlatform`: Use `moduleDir` instead of `projectDir` to determine the resource directory.
 
 * `NextflowPlatform`: Rename internal VDSL3 variables to be more consistent with regular Viash component variables and avoid naming clashes (PR #553).
@@ -2355,7 +2509,7 @@ If, for some reason, you need to manually specify a mount, you can do this with
 
 For all parameter types (except for `boolean_true` and `boolean_false`), you can specify `multiple: true` in order to turn this argument into an array-based argument. What this does is allow you to pass multiple values for this argument, e.g. `--input file1 --input file2 --input file3:file4:file5`.
 
-The default separator is `:` but this can be overridden by changing the separator by setting it to `multiple_sep: ","` (for example).
+The default separator is `:` but this can be overridden by changing the separator by setting it to `multiple_sep: ";"` (for example).
 
 ### New format
 

Dockerfile

@@ -1,8 +1,8 @@
-FROM openjdk:8
+FROM eclipse-temurin:17
 
 # Install packages
 RUN apt-get update && \
-    apt-get install -y make git gzip
+    apt-get install -y make git gzip gnupg
 
 # Install sbt
 RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list && \
@@ -11,7 +11,7 @@ RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee -a /etc/apt/so
     apt-get install -y sbt
 
 # Install yq
-RUN curl -sSL https://github.com/mikefarah/yq/releases/download/v4.6.1/yq_linux_386 > /usr/bin/yq && \
+RUN curl -sSL https://github.com/mikefarah/yq/releases/download/v4.44.2/yq_linux_386 > /usr/bin/yq && \
     chmod +x /usr/bin/yq
 
 # Run SBT once so that all libraries are downloaded

Makefile.in

@@ -12,7 +12,7 @@ $(obj): $(shell find src/ -name "*.scala")
 	@echo "==================================> $(obj)"
 
 tools: $(obj) $(shell find src/viash)
-	bin/viash ns build -s src/viash -t bin --flatten -c ".functionality.version := '$(VERSION)'"
+	bin/viash ns build -s src/viash -t bin --flatten -c ".version := '$(VERSION)'"
 	@echo "==================================> tools"
 
 .PHONY: with-docker

README.md

@@ -11,9 +11,9 @@ Release](https://img.shields.io/github/v/release/viash-io/viash?style=flat-squar
 CI](https://github.com/viash-io/viash/actions/workflows/sbt_test.yml/badge.svg)](https://github.com/viash-io/viash/actions/workflows/sbt_test.yml)
 [![License](https://img.shields.io/github/license/viash-io/viash?style=flat-square.png)](https://github.com/viash-io/viash/blob/master/LICENSE.md)
 
-Viash helps you turn a script (Bash/R/Python/Scala/JavaScript) into a
-reusable component. By providing some meta-data regarding its
-functionality and the platform on which you want to run the software,
+Viash helps you turn a script (Bash/R/Python/Scala/JavaScript) into a reusable component. 
+By providing some meta-data regarding its
+functionality, a runner and an engine on which you want to run the software,
 Viash can help you:
 
 - Wrap your script in an executable with a CLI and –help functionality,

README.qmd

@@ -13,9 +13,10 @@ format: gfm
 [![Scala CI](https://github.com/viash-io/viash/actions/workflows/sbt_test.yml/badge.svg)](https://github.com/viash-io/viash/actions/workflows/sbt_test.yml)
 [![License](https://img.shields.io/github/license/viash-io/viash?style=flat-square)](https://github.com/viash-io/viash/blob/master/LICENSE.md)
 
-Viash helps you turn a script (Bash/R/Python/Scala/JavaScript) into a reusable component. 
-By providing some meta-data regarding its functionality and
-the platform on which you want to run the software, Viash can help you:
+Viash helps you turn a script (Bash/R/Python/Scala/JavaScript) into a
+reusable component. By providing some meta-data regarding its
+functionality, a runner and an engine on which you want to run the software,
+Viash can help you:
 
 * Wrap your script in an executable with a CLI and --help functionality,
 * Seamlessly execute your component natively on the host platform or in a Docker container

RELEASE_GUIDE.md

@@ -2,7 +2,7 @@
 
 ## Set the version in `build.sbt`
 
-e.g. from `version := "0.7.2 dev"` to `version := "0.7.2"`
+e.g. from `version := "0.7.2-dev"` to `version := "0.7.2"`
 
 ## Finalize the changelog changes
 
@@ -54,7 +54,7 @@ Short summary if you already have an operational envirionment:
 # Post merge
 
 Under the `develop` branch,
-- Update the version in `build.sbt` to the next version with a `dev` suffix.
+- Update the version in `build.sbt` to the next version with a `-dev` suffix.
 - Add a placeholder entry in `CHANGELOG.md` for a future release.
 
 Template:

build.sbt

@@ -1,8 +1,8 @@
 name := "viash"
 
-version := "0.8.6"
+version := "0.9.0"
 
-scalaVersion := "2.13.10"
+scalaVersion := "2.13.14"
 
 libraryDependencies ++= Seq(
   "org.scalactic" %% "scalactic" % "3.2.15" % "test",
@@ -45,7 +45,7 @@ generateWorkflowHelper := {
   import java.nio.file._
 
   val rootDir = (Compile / baseDirectory).value
-  val basePath = (Compile / resourceDirectory).value / "io" / "viash" / "platforms" / "nextflow"
+  val basePath = (Compile / resourceDirectory).value / "io" / "viash" / "runners" / "nextflow"
   val wfHelper = Paths.get(basePath.toString, "WorkflowHelper.nf")
 
   // brute force recursive file listing instead of using a glob that skips files in the root directory
@@ -74,4 +74,4 @@ generateWorkflowHelper := {
 }
 
 assembly := ((assembly) dependsOn generateWorkflowHelper).value
-Test / testOptions := ((Test / testOptions) dependsOn generateWorkflowHelper).value
+Test / testOptions := ((Test / testOptions) dependsOn generateWorkflowHelper).value

codecov.yml

@@ -0,0 +1,12 @@
+coverage:
+  precision: 2
+  round: down
+  status:
+    project:
+      default:
+        target: auto
+        threshold: 1%
+    patch:
+      default:
+        enabled: yes
+        threshold: 5%

docs/reference/config_mods/index.qmd

@@ -3,7 +3,7 @@ title: Dynamic Config Modding
 order: 50
 ---
 
-Viash can modify a [viash config](/reference/config/index.html) at runtime using a custom Domain Specific Language (DSL). This allows making dynamic changes to your components or projects.  
+Viash can modify a [viash config](/reference/config/index.html) at runtime using a custom Domain Specific Language (DSL). This allows making dynamic changes to your components or packages.  
 All Viash subcommands have support for the DSL through the `-c|--config_mod` parameter. The format for these is as follows:
 
 ```bash
@@ -23,27 +23,27 @@ viash COMMAND \
 Change the version of a component:
 
 ```bash
-viash build -c '.functionality.version := "0.3.0"'
+viash build -c '.version := "0.3.0"'
 ```
 
 Change the registry of a docker container:
 
 ```bash
 viash build -c \
-  '.platforms[.type == "docker"].container_registry := "url-to-registry"'
+  '.engines[.type == "docker"].registry := "url-to-registry"'
 ```
 
 Add an author to the list:
 
 ```bash
-viash build -c '.functionality.authors += { name: "Mr. T", role: "sponsor" }'
+viash build -c '.authors += { name: "Mr. T", role: "sponsor" }'
 ```
 
 You can use dynamic config modding to alter the config of multiple components at once:
 
 ```bash
 viash ns build \
-  -c '.functionality.version := "0.3.0"' \
-  -c '.platforms[.type == "docker"].container_registry := "url-to-registry"' \
-  -c '.functionality.authors += { name: "Mr. T", role: "sponsor" }'
+  -c '.version := "0.3.0"' \
+  -c '.engines[.type == "docker"].registry := "url-to-registry"' \
+  -c '.authors += { name: "Mr. T", role: "sponsor" }'
 ```