Release version 0.0.1 (#149)

* Fixed configs for develop (#106) * change default-config values for new staging develop env * Bump hosted-git-info from 2.8.8 to 2.8.9 (#108) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump lodash from 4.17.19 to 4.17.21 (#104) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.19...4.17.21) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Get data upload status (#105) * create projects and samples * update projects and samples * delete projects and samples * Add FastMNN and No integration methods (#107) * Worker failing to restart (#111) * Debug logging on helm command that launches worker * Get helm status * Dsiplay helm history on upgrade failure * Do not run helm update concurrently * fix async-lock * Cleanup * Better logging * Run npm audit fix because dependabot is complaining again (#115) * Launch new experiment (#113) * Store new experiment in DynamoDB * Add type to data processing report (#117) Co-authored-by: Martin Fosco <[email protected]> * Add end-to-end authentication and authorization (#110) * Add authentication check for API * fix * Add authorization to the API * add authorization to socketio * fix existing unit tests and add a new type of test * Add tests for authentication * fix-typo * Fix documentation * learn to spell unauthenticated * debug * more-debug * Suppress missing X-Ray context errors for auth middleware lookups * fix-error * Fix canWrite type issues * Throw if experiment does not have rbac_can_write * Fix broken merge * Remove Xray noisy log * Search for noisy Xray log * Try to convince setContextMissingStrategy not to log * Fix non-context Xray traces * Remove debugging logs * Move some commenst from PR to code Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * Add cognito user and admin details to rbac can write (#119) * Add authentication check for API * fix * Add authorization to the API * add authorization to socketio * fix existing unit tests and add a new type of test * Add tests for authentication * fix-typo * Fix documentation * learn to spell unauthenticated * debug * more-debug * Suppress missing X-Ray context errors for auth middleware lookups * fix-error * Fix canWrite type issues * Throw if experiment does not have rbac_can_write * Fix broken merge * Add create experiment endpoint to avoid error message experiment not found when creating new experiment * Add user permissions to saved experiment * Add adminArn * Clear out repetitions before setting rbac_can_write * Remove extra console logs * Test fixes * Add 401 and 403 responses to enabled options * Rename new auth middleware * Move adminArn into config Co-authored-by: Marcell Pek <[email protected]> Co-authored-by: Marcell Pék <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> * load projects and samples (#114) * redoing the ticket * fix * fixes * fixes * comment fixe * fixes * move to inside function * change * Refactor code to work better with async/await * Make sure keys returned to getProjectsFromIds are unqiue. * Convert projectIds into an Array * Fix conversion error between keys * Add array spreading to other reference to projectIds * Add array spreading to other references to projectIds * Make module to filter out orphaned experiments more explicit * Remove based on items Co-authored-by: Marcell Pek <[email protected]> * Recover settings accidentally removed (#121) * api to load experiments (#126) * created gem2s endpoint (#109) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fix manifest grep * fixed some tests * using personal IAC to bypass runner name temporarily * added debug logs * removed old comment * added temporarily pipeline config until the updated one is deployed to iac * added authorization to new endpoints Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Address comments pr 126 (#128) * add route test * add test for getListOfExperiments * Save gem2s pipeline handle (#120) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fixed some tests * Save gem2s pipeline handle Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Get projects test (#123) * test for projects route * write tests for projects * rewrite AWS mock * Handle gem2s response (#127) * created gem2s endpoint * added skeleton parameter to state-machine-definition tests * passing all required params * fixed qc-pipeline skeleton * fixed gem2s pipeline skeleton name * renamed ARN to avoid clashes * converted gem2s pipeline steps into normal ones instead of maps * added last steps for gem2s * removed testing clutter * fixed some tests * Save gem2s pipeline handle * Save response to dynamodb * Add gem2s handle and status report to status * Fix dynamodb updates to experiments and other minor things * Fix tests and minor bugs * Minor fixes Co-authored-by: Pol Alvarez <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Require auth for saving and loading projects (#124) * require auth for saving and loading projects * api to load experiments * add auth to routes * Request GEM2S status from the backend (#118) * add gem2s when requesting backend status * create routes, add tests * change from pipelines to backend-status * remove package-lock.json * Delete project delete experiment [BIOMAGE-948] (#131) * delete experiments * add experimentDelete test * add checks for empty experiment * BIOMAGE-792 Return only experiments and projects that a user has access to after logging in (#122) * Only list projects with access * Check for missing user. No exception on empty list * Fixed query * Accpet empty arrays * Fix length check * Restore change lost on merge * Update tests * Fix transition between gem2s and data processing (#130) * Fix discrepancies in qc and pipeline renaming * Fix various bugs * When no samples, change 500 for 404 * Implement xaviers fix on statusResToSend * Change updateExperiment to work by doing one single request * Refactor changes in previous commit * Fix qc/pipeline discrepancy problem on work requests send * Change names to more readable options * Rename helpers * Minor rafctoring based on agi comment Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Xavier Verges <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * Handle gem2s response refactoring (#129) * Adapt response swagger validation * Add missing change from status to taskName Co-authored-by: Martin Fosco <[email protected]> * Hide some wrong xray missing context logs + typos (#133) * fix nonexistent property access (#137) * [BIOMAGE-1037] - Pass metadata to gem2s (#135) * pass metadata to gem2s * exclude metadata if empty * [BIOMAGE-1036] - Run QC after GEM2S (#134) * run qc after gem2s * add test * await for pipeline runs * [BIOMAGE-1009] Fix error sns messages (#136) * Adapt code in api validation and handler in gem2s to new structure * Fix response handlers * Minor fixes and refactoring * Update test snapshots * Fix indentation * Fix validation swagger files * Fix tests Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> * convert to async/await (#141) * Beter 404 for plots-tables (#140) * Beter 404 for plots-tables * Friendlier logging of expected errors * Broken if omg * Better logging * Log hook execution * Wrong condition, again :-( * Take care of falsy parsedMessage * Sanitize metadata key (#142) * [BIOMAGE-1027] Add safeBatchGetItem that divides batchGetItem in groups of requests of 100 keys (#138) * Add safeBatchGetItem * Add tests * Add explanatory comments * Rewrite safeBatchGetItem to work with limit of 100 for total of items instead of per table * Minor bugfix and add more unit tests * Remove obsolete object Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-989] - Work request throw error if pipeline has not yet succeeded (#139) * new flow for releases (#143) * added deployment for release/hotfixes PRs to master (#145) * using tokens to allow flow to trigger testing * building releases/hotfixes on PR instead of push * Update ci-develop.yaml * Delete files in s3 for deleted samples (#146) * Add deletion of files of deleted project * Add basic delete single sample operation (still needs updates) * Remove samples unnecessary ids field * Fixes and add validation * Fix tests * Fix tests and cleanup code * Reorder * Get sample files instead of hardcoded files * Fix tests Co-authored-by: Martin Fosco <[email protected]> * notifying worker of job done (#151) * notifying worker of job done * Fix louvain cellsets storage in s3 * Add small comment * clean up Co-authored-by: Martin Fosco <[email protected]> * [BIOMAGE-1044] Add a needsRunning flag to gem2s pipeline status (#147) * Add needsRunning flag to gem2s status Adds a paramsHash to the gem2s handle * Minor readibility improvements and cleanup * Refactoring and fix issue with taskparams hash * Refactoring and fix tests * Fix errro during run * Refactor fix Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Pol Alvarez <[email protected]> * Deprecated change for ci.yaml Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anugerah Erlaut <[email protected]> Co-authored-by: Juanlu <[email protected]> Co-authored-by: Xavier Vergés <[email protected]> Co-authored-by: ivababukova <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Marcell Pék <[email protected]> Co-authored-by: Marcell Pek <[email protected]> Co-authored-by: StefanBabukov <[email protected]> Co-authored-by: Alex Pickering <[email protected]> Co-authored-by: Martin Fosco <[email protected]> Co-authored-by: Martin Fosco <[email protected]>
hms-dbmi-cellenics · Jun 15, 2021 · 8091232 · 8091232
1 parent f943c01
commit 8091232
Show file tree

Hide file tree

Showing 101 changed files with 7,855 additions and 930 deletions.
diff --git a/.github/workflows/ci-develop.yaml b/.github/workflows/ci-develop.yaml
@@ -6,10 +6,16 @@ on:
   pull_request:
     branches:
       - develop
+      - master
 jobs:
   test:
     name: Run tests
     runs-on: ubuntu-20.04
+    if: |-
+      !github.event.pull_request == true ||
+      startsWith(github.event.pull_request.head.ref, 'release/') ||
+      startsWith(github.event.pull_request.head.ref, 'hotfix/') ||
+      github.event.pull_request.base.ref == 'develop'
     env:
       CI: "true"
     steps:
@@ -25,6 +31,7 @@ jobs:
       name: Install dependencies
       run: |-
         npm ci
+
     - id: test
       name: Run unit tests
       run: |-
@@ -47,15 +54,18 @@ jobs:
           # This will take a ref like `refs/heads/master`
           # and turn it into `refs-heads-master`
           REF_ID=$(echo $GITHUB_REF | sed 's/\//-/g')
+
           # the final tag is something like:
           # refs-heads-master-a4f8bc313dae
           # this is what we push to ECR
           echo "::set-output name=tag::$REF_ID-$GITHUB_SHA"
+
           # This will take a GitHub repo name like `biomage-ltd/iac`
           # and turns it into `iac`. This will be the name of the
           # ECR repository.
           IMAGE_REPO_NAME=$(echo $GITHUB_REPOSITORY | awk -F '/' '{print $2}')
           echo "::set-output name=repo-name::$IMAGE_REPO_NAME"
+
       - id: set-up-creds
         name: Configure AWS credentials
         uses: aws-actions/configure-aws-credentials@v1
@@ -67,7 +77,7 @@ jobs:
       - id: login-ecr
         name: Login to Amazon ECR
         uses: aws-actions/amazon-ecr-login@v1
-      
+
       - id: create-ecr-registry
         name: Create an ECR repository (if needed)
         # This will fail if the registry already exists, which is fine. If there is some other
@@ -126,25 +136,27 @@ jobs:
         name: Install yq for modifying the deployment spec.
         run: |-
           sudo wget https://github.com/mikefarah/yq/releases/download/3.4.1/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
-      
+
       - id: fill-metadata
         name: Fill out a new HelmRelease resource
         run: |-
           REF_DASHED=$(echo $GITHUB_REF | sed 's/\//-/g')
           echo "::set-output name=ref-dashed::$REF_DASHED"
+
           if [ "${{ matrix.environment }}" = "develop" ]; then
             SANDBOX_ID="default"
             CHART_REF="master"
             DEPLOYMENT_NAME=$(echo $GITHUB_REPOSITORY | awk -F '/' '{print $2}')
             echo "::set-output name=deployment-name::$DEPLOYMENT_NAME"
           fi
+
           if [ "${{ matrix.environment }}" = "staging" ]; then
             SANDBOX_ID="STAGING_SANDBOX_ID"
             CHART_REF="STAGING_CHART_REF"
             DEPLOYMENT_NAME=$(echo $GITHUB_REPOSITORY | awk -F '/' '{print $2}')
             echo "::set-output name=deployment-name::$DEPLOYMENT_NAME"
           fi
-          
+
           yq w -d1 .flux.yaml metadata.name "$DEPLOYMENT_NAME" \
           | yq w -d1 - metadata.namespace "$DEPLOYMENT_NAME-$SANDBOX_ID" \
           | yq w -d0 - metadata.name "$DEPLOYMENT_NAME-$SANDBOX_ID" \
@@ -161,6 +173,7 @@ jobs:
           | yq w -d1 - spec.values.image.tag "$REF_DASHED-$GITHUB_SHA" \
           | yq w -d1 - spec.values.serviceAccount.iamRole "arn:aws:iam::${{ steps.setup-aws.outputs.aws-account-id }}:role/api-role-staging" \
           > $DEPLOYMENT_NAME.yaml
+
           cat $DEPLOYMENT_NAME.yaml
         env:
           IMAGE_NAME: ${{ format('{0}/{1}:{2}', steps.login-ecr.outputs.registry, needs.build-docker.outputs.repo-name, needs.build-docker.outputs.tag) }}
@@ -197,6 +210,7 @@ jobs:
           REF_DASHED: ${{ steps.fill-metadata.outputs.ref-dashed }}
         run: |-
           mv $DEPLOYMENT_NAME.yaml $REF_DASHED.yaml
+
       - name: Push staging deployment template to iac
         if:
           matrix.environment == 'staging'
@@ -220,3 +234,4 @@ jobs:
           repo: iac
           enforce_admins: true
           retries: 8
+
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -68,7 +68,7 @@ jobs:
       - id: login-ecr
         name: Login to Amazon ECR
         uses: aws-actions/amazon-ecr-login@v1
-      
+
       - id: create-ecr-registry
         name: Create an ECR repository (if needed)
         # This will fail if the registry already exists, which is fine. If there is some other
@@ -127,7 +127,7 @@ jobs:
         name: Install yq for modifying the deployment spec.
         run: |-
           sudo wget https://github.com/mikefarah/yq/releases/download/3.4.1/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
-      
+
       - id: fill-metadata
         name: Fill out a new HelmRelease resource
         run: |-
@@ -145,7 +145,7 @@ jobs:
             SANDBOX_ID="STAGING_SANDBOX_ID"
             CHART_REF="STAGING_CHART_REF"
           fi
-          
+
           yq w -d1 .flux.yaml metadata.name "$DEPLOYMENT_NAME" \
           | yq w -d1 - metadata.namespace "$DEPLOYMENT_NAME-$SANDBOX_ID" \
           | yq w -d0 - metadata.name "$DEPLOYMENT_NAME-$SANDBOX_ID" \

diff --git a/.github/workflows/draft-new-release.yaml b/.github/workflows/draft-new-release.yaml
@@ -0,0 +1,50 @@
+name: Create a new release
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'The version you want to release.'
+        required: true
+
+jobs:
+  create-release:
+    name: ${{ format('Create release {0}', github.event.inputs.repo) }}
+    runs-on: ubuntu-20.04
+    steps:
+
+      - id: checkout
+        name: Check out source code
+        uses: actions/checkout@v2
+        with:
+          token: ${{ secrets.BIOMAGE_ACTIONS_TOKEN }}
+          persist-credentials: false # otherwise, the token used is the BIOMAGE_ACTIONS_TOKEN, instead of your personal access token
+          fetch-depth: 0
+
+      - id: create-release
+        name: Create release branch
+        run: git checkout -b release/${{ github.event.inputs.version }}
+
+      - id: initialize-git-info
+        name: Initialize mandatory git info
+        run: git config user.name "Github Actions"
+
+      - name: Push changes
+        uses: ad-m/github-push-action@master
+        with:
+          github_token: ${{ secrets.BIOMAGE_ACTIONS_TOKEN }}
+          branch: ${{ format('release/{0}', github.event.inputs.version ) }}
+
+      - id: create-pr
+        name: Create pull request
+        uses: thomaseizinger/[email protected]
+        env:
+          GITHUB_TOKEN: ${{ secrets.BIOMAGE_ACTIONS_TOKEN }}
+        with:
+          head: release/${{ github.event.inputs.version }}
+          base: master
+          title: Release version ${{ github.event.inputs.version }}
+          body: |
+            This PR was created in response to a manual trigger of the release workflow here: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}.
+
+            Merging this PR will create a GitHub release, update master & develop branches, and upload any assets that are created as part of the release build.
diff --git a/.github/workflows/prevent-pr-to-master.yaml b/.github/workflows/prevent-pr-to-master.yaml
@@ -1,4 +1,4 @@
-name: Prevent pull requests modifying Flux CI paths
+name: Accept only release and hotfix pull requests to master
 
 on:
   pull_request:
@@ -8,10 +8,11 @@ on:
 jobs:
   run:
     runs-on: ubuntu-20.04
+    if: (!startsWith(github.event.pull_request.head.ref, 'release/') && !startsWith(github.event.pull_request.head.ref, 'hotfix/')) # allow PRs to master for releases & hotfixes
     steps:
     - uses: superbrothers/close-pull-request@v2
       with:
         comment: |-
-            Pull requests to master are not allowed, merge into develop.
+            Pull requests to master are not allowed unless they are releases or hotfixes, merge into develop.
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/publish-new-release.yaml b/.github/workflows/publish-new-release.yaml
@@ -0,0 +1,55 @@
+name: "Publish new release"
+
+on:
+  pull_request:
+    branches:
+      - master
+    types:
+      - closed
+
+jobs:
+  release:
+    name: Publish new release
+    runs-on: ubuntu-20.04
+    # only trigger on merged pull requests named release/* or hotfix/*
+    if: |
+        github.event.pull_request.merged == true &&
+        (startsWith(github.event.pull_request.head.ref, 'release/') || startsWith(github.event.pull_request.head.ref, 'hotfix/'))
+    steps:
+      - name: Extract version from branch name (for release branches)
+        if: startsWith(github.event.pull_request.head.ref, 'release/')
+        run: |
+          BRANCH_NAME="${{ github.event.pull_request.head.ref }}"
+          VERSION=${BRANCH_NAME#release/}
+
+          echo "RELEASE_VERSION=$VERSION" >> $GITHUB_ENV
+
+      - name: Extract version from branch name (for hotfix branches)
+        if: startsWith(github.event.pull_request.head.ref, 'hotfix/')
+        run: |
+          BRANCH_NAME="${{ github.event.pull_request.head.ref }}"
+          VERSION=${BRANCH_NAME#hotfix/}
+
+          echo "RELEASE_VERSION=$VERSION" >> $GITHUB_ENV
+
+      - name: Create Release
+        uses: thomaseizinger/[email protected]
+        env:
+          GITHUB_TOKEN: ${{ secrets.BIOMAGE_ACTIONS_TOKEN }}
+        with:
+          target_commitish: ${{ github.event.pull_request.merge_commit_sha }}
+          tag_name: ${{ env.RELEASE_VERSION }}
+          name: ${{ env.RELEASE_VERSION }}
+          draft: false
+          prerelease: false
+
+      - name: Merge master into dev branch
+        uses: thomaseizinger/[email protected]
+        env:
+          GITHUB_TOKEN: ${{ secrets.BIOMAGE_ACTIONS_TOKEN }}
+        with:
+          head: master
+          base: develop
+          title: Merge master into dev branch
+          body: |
+            This PR merges the master branch back into develop.
diff --git a/.gitignore b/.gitignore
@@ -5,5 +5,6 @@ npm-debug.log
 .DS_Store
 *.log
 coverage/
-.vscode
+.vscode/*
+!.vscode/settings.json
 chart-instance
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,24 @@
+{
+    "editor.codeActionsOnSave": {
+        "source.fixAll.eslint": true
+    },
+    "[javascript]": {
+        "editor.tabSize": 2,
+        "editor.insertSpaces": true,
+        "editor.detectIndentation": false,
+        "editor.formatOnPaste": true,
+        "editor.formatOnSave": true
+    },
+    "markdownlint.config": {
+        "default": true,
+        "MD003": { "style": "setext_with_atx" },
+        "MD034": false,
+        "MD046": false
+    },
+    "[yaml]" : {
+        "editor.formatOnType" : true,
+    },
+    "yaml.format.enable": true,
+    "yaml.format.singleQuote": true,
+    "yaml.validate": true
+}
diff --git a/Makefile b/Makefile
@@ -13,6 +13,8 @@ check: ## Checks code for linting/construct errors
 	@npm run lint
 	@npm run detect-secrets
 	@echo "    [✓]\n"
+test: ## Runs UI tests
+	@npm test
 build: ## Empty target for uniform make interfaces because API does not neet to be build
 run: ## Runs the UI 
 	@npm start

diff --git a/api.code-workspace b/api.code-workspace
@@ -4,22 +4,5 @@
 			"path": "."
 		}
 	],
-	"settings": {
-		"editor.codeActionsOnSave": {
-			"source.fixAll.eslint": true
-		},
-		"[javascript]": {
-			"editor.tabSize": 2,
-			"editor.insertSpaces": true,
-			"editor.detectIndentation": false,
-			"editor.formatOnPaste": true,
-			"editor.formatOnSave": true
-		},
-		"markdownlint.config": {
-			"default": true,
-			"MD003": { "style": "setext_with_atx" },
-			"MD034": false,
-			"MD046": false
-		}
-	}
+	"settings": {}
 }