Skip to content

Commit

Permalink
Merge pull request #112 from Plant-Food-Research-Open/dev
Browse files Browse the repository at this point in the history
Candidate for 0.5.0
  • Loading branch information
GallVp authored Nov 21, 2024
2 parents 24a8f50 + 02e7669 commit ee702d7
Show file tree
Hide file tree
Showing 184 changed files with 7,911 additions and 1,888 deletions.
6 changes: 5 additions & 1 deletion .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ trim_trailing_whitespace = true
indent_size = 4
indent_style = space

[*.{md,yml,yaml,html,css,scss,js}]
[*.{md,yml,yaml,html,css,scss,js,cff}]
indent_size = 2

# These files are edited and tested upstream in nf-core/modules
Expand Down Expand Up @@ -58,3 +58,7 @@ indent_size = unset
# ignore LICENSE
[LICENSE]
indent_size = unset

# ignore bin/genepal_report
[bin/genepal_report*]
indent_size = unset
12 changes: 6 additions & 6 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ If you'd like to write some code for plant-food-research-open/genepal, the stand
1. Check that there isn't already an issue about your idea in the [plant-food-research-open/genepal issues](https://github.com/plant-food-research-open/genepal/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [plant-food-research-open/genepal repository](https://github.com/plant-food-research-open/genepal) to your GitHub account
3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
4. Use `nf-core pipelines schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged

If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
Expand All @@ -37,7 +37,7 @@ There are typically two types of tests that run:
### Lint tests

`nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core pipelines lint <pipeline-directory>` command.

If any failures or warnings are encountered, please follow the listed URL for more documentation.

Expand Down Expand Up @@ -68,7 +68,7 @@ If you wish to contribute a new step, please use the following coding standards:
2. Write the process block (see below).
3. Define the output channel if needed (see below).
4. Add any new parameters to `nextflow.config` with a default (see below).
5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core schema build` tool).
5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core pipelines schema build` tool).
6. Add sanity checks and validation for all relevant parameters.
7. Perform local tests to validate that the new code works as expected.
8. If applicable, add a new test command in `.github/workflow/ci.yml`.
Expand All @@ -79,11 +79,11 @@ If you wish to contribute a new step, please use the following coding standards:

Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.

Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
Once there, use `nf-core pipelines schema build` to add to `nextflow_schema.json`.

### Default processes resource requirements

Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.

The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.

Expand All @@ -96,7 +96,7 @@ Please use the following naming schemes, to make it easy to understand what is g

### Nextflow version bumping

If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core pipelines bump-version --nextflow . [min-nf-version]`

### Images and figures

Expand Down
32 changes: 10 additions & 22 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,46 +9,34 @@ body:
description: A clear and concise description of what the bug is.
validations:
required: true

- type: textarea
id: command_used
attributes:
label: Command used and terminal output
description: Steps to reproduce the behaviour. Please paste the command you used
to launch the pipeline and the output from your terminal.
description: Steps to reproduce the behaviour. Please paste the command you used to launch the pipeline and the output from your terminal.
render: console
placeholder: "$ nextflow run ...
placeholder: |
$ nextflow run ...
Some output where something broke
"
- type: textarea
id: files
attributes:
label: Relevant files
description: "Please drag and drop the relevant files here. Create a `.zip` archive
if the extension is not allowed.
Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
in the directory where you launched the pipeline)_ as well as custom Nextflow
configuration files.
description: |
Please drag and drop the relevant files here. Create a `.zip` archive if the extension is not allowed.
Your verbose log file `.nextflow.log` is often useful _(this is a hidden file in the directory where you launched the pipeline)_ as well as custom Nextflow configuration files.
"
- type: textarea
id: system
attributes:
label: System information
description: "* Nextflow version _(eg. 23.04.0)_
description: |
* Nextflow version _(eg. 23.04.0)_
* Hardware _(eg. HPC, Desktop, Cloud)_
* Executor _(eg. slurm, local, awsbatch)_
* Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
or Apptainer)_
* Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud, or Apptainer)_
* OS _(eg. CentOS Linux, macOS, Linux Mint)_
* Version of plant-food-research-open/genepal _(eg. 1.1, 1.5, 1.8.2)_
"
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/plant-food-r
- [ ] This comment contains a description of changes (with reason).
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/plant-food-research-open/genepal/tree/master/.github/CONTRIBUTING.md)
- [ ] Make sure your code lints (`nf-core lint`).
- [ ] Make sure your code lints (`nf-core pipelines lint`).
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
- [ ] Usage Documentation in `docs/usage.md` is updated.
Expand Down
10 changes: 10 additions & 0 deletions .github/include.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
".":
- ./.github/workflows/**
- ./nf-test.config
tests:
- ./assets/*
- ./bin/*
- ./conf/*
- ./main.nf
- ./nextflow_schema.json
- ./nextflow.config
22 changes: 19 additions & 3 deletions .github/version_checks.sh
Original file line number Diff line number Diff line change
@@ -1,8 +1,24 @@
#!/usr/bin/env bash

config_version=$(sed -n "s/.*version.*= '\(.*\)'.*/\1/p" nextflow.config)
set -euo pipefail

# Check CHANGELOG version
config_version=$(sed -n "/^\s*version\s*=\s*'/s/version//p" nextflow.config | tr -d "=[:space:]'")
cff_version=$(sed -n '/^version: /s/version: //p' CITATION.cff | tr -d '[:space:]')

if [[ $config_version != $cff_version ]]; then
echo 'config_version != cff_version'
exit 1
fi

head -n 10 CHANGELOG.md | grep "## $config_version - " >/dev/null \
# Check CHANGELOG version
head -10 CHANGELOG.md | grep "## v$config_version - " >/dev/null \
|| (echo 'Failed to match CHANGELOG version'; exit 1)

# Check .nf-core.yml version
tail -5 .nf-core.yml | grep "version: $config_version" >/dev/null \
|| (echo 'Failed to match .nf-core.yml version'; exit 1)

# Check multiqc_config.yml version
config_version_plain=$(echo $config_version | sed 's/.*\dev$/dev/; t; s/\+.*//')
head -n 5 assets/multiqc_config.yml | grep "https://github.com/plant-food-research-open/genepal/blob/$config_version_plain/docs/usage.md" >/dev/null \
|| (echo 'Failed to match multiqc_config.yml version'; exit 1)
100 changes: 71 additions & 29 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,58 +1,100 @@
name: nf-core CI
# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
on:
push:
branches:
- dev
pull_request:
release:
types: [published]

env:
NXF_ANSI_LOG: false
NFT_WORKDIR: "nf-test-work"
NFT_DIFF: "pdiff"
NFT_DIFF_ARGS: "--line-numbers --expand-tabs=2"

concurrency:
group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
cancel-in-progress: true

jobs:
nf-test-changes:
name: Check for changes
runs-on: ubuntu-latest
outputs:
nf_test_files: ${{ steps.list.outputs.components }}
steps:
- uses: actions/[email protected]
with:
fetch-depth: 0

- name: List nf-test files
id: list
uses: adamrtalbot/[email protected]
with:
head: ${{ github.sha }}
base: origin/${{ github.base_ref }}
include: .github/include.yaml

- name: print list of nf-test files
run: |
echo ${{ steps.list.outputs.components }}
test:
name: Run pipeline with test data
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'Plant-Food-Research-Open/genepal') }}"
name: ${{ matrix.nf_test_files }} ${{ matrix.profile }} NF-${{ matrix.NXF_VER }}
needs: [nf-test-changes]
if: needs.nf-test-changes.outputs.nf_test_files != '[]'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
NXF_VER:
- "23.04.0"
TEST_PARAMS:
- minimal
- short
- nonmasked
- stub
OPTION_STUB:
- "-stub"
- "24.04.2"

nf_test_files: ["${{ fromJson(needs.nf-test-changes.outputs.nf_test_files) }}"]
profile:
- "docker"

steps:
- name: Check out pipeline code
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
uses: actions/checkout@v4.2.1

- name: Install Nextflow
uses: nf-core/setup-nextflow@v2
with:
version: "${{ matrix.NXF_VER }}"

- uses: actions/[email protected]
with:
python-version: "3.11"
architecture: "x64"

- name: Install pdiff to see diff between nf-test snapshots
run: |
python -m pip install --upgrade pip
pip install pdiff
- uses: nf-core/[email protected]
with:
version: 0.9.0

- name: Disk space cleanup
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
if: matrix.nf_test_files == 'tests/stub/main.nf.test'
uses: jlumbroso/[email protected]

- name: Run nf-test
run: |
nf-test test --verbose ${{ matrix.nf_test_files }} --profile "+${{ matrix.profile }}"
confirm-pass:
runs-on: ubuntu-latest
needs: [test]
if: always()
steps:
- name: All tests ok
if: ${{ !contains(needs.*.result, 'failure') }}
run: exit 0
- name: One or more tests failed
if: ${{ contains(needs.*.result, 'failure') }}
run: exit 1

- name: Run pipeline with test data
- name: debug-print
if: always()
run: |
nextflow run \
${GITHUB_WORKSPACE} \
-profile docker \
-params-file \
./tests/${{ matrix.TEST_PARAMS }}/params.json \
${{ matrix.OPTION_STUB }} \
--outdir ./results \
--max_cpus 2 \
--max_memory '6.GB' \
--max_time '2.h'
echo "toJSON(needs) = ${{ toJSON(needs) }}"
echo "toJSON(needs.*.result) = ${{ toJSON(needs.*.result) }}"
Loading

0 comments on commit ee702d7

Please sign in to comment.