Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Biomage changes 2 #427

Merged
merged 212 commits into from
Feb 27, 2023
Merged
Changes from 1 commit
Commits
Show all changes
212 commits
Select commit Hold shift + click to select a range
e60fe4b
Move to one line
cosa65 Dec 22, 2022
6b7e100
Implement subsetSeurat setupSubsetSamples hook
cosa65 Dec 22, 2022
22203e0
handle pipeline timeout error
aerlaut Dec 23, 2022
d95a94a
update snapshot
aerlaut Dec 23, 2022
85dde7d
add catch in step
aerlaut Dec 23, 2022
4871f0c
add catch
aerlaut Dec 23, 2022
e74d700
insert auth JWT into error pipeline
aerlaut Dec 23, 2022
c967144
insert auth in gem2s
aerlaut Dec 23, 2022
eb0665a
move gem2s authJWT from taskParams
aerlaut Dec 23, 2022
9382143
add authJWT to gem2s context
aerlaut Dec 23, 2022
191a2da
update snapshots
aerlaut Dec 23, 2022
8408256
correct arguments
aerlaut Dec 23, 2022
21b8d6d
correctlly get props
aerlaut Dec 23, 2022
2d81777
Update comment
cosa65 Dec 26, 2022
0a03297
Start returning can_rerun_gem2s so the UI can use it
cosa65 Dec 26, 2022
1811914
Add check on whether the experiment can have gem2s rerun or not
cosa65 Dec 26, 2022
731f2af
Start creating the experiment execution entries
cosa65 Dec 26, 2022
67ee2c9
Fix experiment executuion sql update to point to nthe new experiment …
cosa65 Dec 26, 2022
6e43bd5
More explicit check
cosa65 Dec 26, 2022
2a92646
Move canRerunGem2s check to the controller
cosa65 Dec 26, 2022
c09c190
FIx, pass sampleIdMap to copyTo so no new sample ids are created
cosa65 Dec 26, 2022
9462b9f
Merge branch 'master' into 2293-handle-subset-updates
cosa65 Dec 26, 2022
2b21d72
TMP - Disable tsts to be able to stage
cosa65 Dec 26, 2022
b0db7ea
Merge branch 'master' into 2293-handle-subset-updates
cosa65 Dec 26, 2022
c9b7031
Merge branch 'master' into handle-step-functions-timeout
aerlaut Jan 2, 2023
ce449a7
add mark as failed
aerlaut Jan 2, 2023
c9ca5ee
enable test in staging
aerlaut Jan 2, 2023
358e010
mark error as failed
aerlaut Jan 2, 2023
acca57a
prevent double reporting
aerlaut Jan 2, 2023
90baf5e
add test for pipeline error route
aerlaut Jan 2, 2023
9ae1b81
Revert "add test for pipeline error route"
aerlaut Jan 2, 2023
e2b788e
delete pipeline error endpoint
aerlaut Jan 2, 2023
1c08398
move pipeline error as a helper
aerlaut Jan 2, 2023
6279ff5
better error message
aerlaut Jan 2, 2023
baf3f07
add controller tests
aerlaut Jan 2, 2023
069297b
add test for error handler helper
aerlaut Jan 2, 2023
0b94492
restore default timeout
aerlaut Jan 2, 2023
9ac074f
handle catch for qc map step
aerlaut Jan 3, 2023
6071617
add catch steps for data integration
aerlaut Jan 3, 2023
b503fb8
Merge branch 'master' into handle-step-functions-timeout
aerlaut Jan 3, 2023
9165609
remove timeout error handler from local
aerlaut Jan 4, 2023
5a49c4a
Merge branch handle-step-functions-timeout of https://github.com/biom…
aerlaut Jan 4, 2023
55e6112
Merge branch 'master' into handle-step-functions-timeout
aerlaut Jan 4, 2023
c97fe47
remove create failed step
aerlaut Jan 4, 2023
28a7096
Merge branch 'handle-step-functions-timeout' of https://github.com/bi…
aerlaut Jan 4, 2023
17a1408
remove unrequired error handler
aerlaut Jan 4, 2023
902d391
remove pipelineErrorHandler
aerlaut Jan 4, 2023
3d10154
batch catch general error
aerlaut Jan 4, 2023
6ff55ad
fix catch steps
aerlaut Jan 5, 2023
6d14133
update tests
aerlaut Jan 5, 2023
5a9b336
Merge branch 'master' into 2293-handle-subset-updates
StefanBabukov Jan 5, 2023
7094961
add catch for uploadToAWS step
aerlaut Jan 6, 2023
80319bc
use states.format
aerlaut Jan 6, 2023
578f329
make error message use input
aerlaut Jan 6, 2023
c8e1a54
fix path to error
aerlaut Jan 6, 2023
ac3165c
restore catch steps in local pipeline
aerlaut Jan 6, 2023
9b0677e
fix invalid path
aerlaut Jan 6, 2023
16e72c1
try simple message
aerlaut Jan 6, 2023
20dd5e9
do not use input
aerlaut Jan 6, 2023
7cda376
format message correctly
aerlaut Jan 10, 2023
a3b6400
update snapshot
aerlaut Jan 10, 2023
96484bb
remove console.log
aerlaut Jan 10, 2023
74d039d
add comment
aerlaut Jan 10, 2023
63ac13f
Bump knex from 1.0.7 to 2.4.0
dependabot[bot] Jan 11, 2023
a4c0591
Merge pull request #85 from biomage-org/handle-step-functions-timeout
aerlaut Jan 11, 2023
2a18c33
implement forgotten revert
aerlaut Jan 11, 2023
e4f3692
Merge pull request #89 from biomage-org/revert-before-merging
aerlaut Jan 11, 2023
a1a241a
added constraint
StefanBabukov Jan 11, 2023
db09624
Merge branch 'master' into 2293-handle-subset-updates
cosa65 Jan 11, 2023
bbc4991
Merge branch 'master' into unique-sample-names
StefanBabukov Jan 11, 2023
94098e8
Add edpoint for getting rerun status
Jan 11, 2023
0d39362
Compare old and new hash
Jan 11, 2023
1998025
Save paramsHash on pipeline rerun
Jan 11, 2023
1d5f190
Update migration for subset and begin fetching the parent_experiment_id
cosa65 Jan 11, 2023
1f2b719
Insert the experiment parent row when a subset experiment is being cr…
cosa65 Jan 11, 2023
d62fd66
Update openapi validation
cosa65 Jan 11, 2023
d460908
Update snapshot and add mock for ExperimentParent
cosa65 Jan 11, 2023
20f2678
Update tests of Experiment model
cosa65 Jan 11, 2023
c192645
Update gem2sController tests
cosa65 Jan 11, 2023
50f12a1
Update test
cosa65 Jan 11, 2023
405578d
Update subsetController runSubset test
cosa65 Jan 11, 2023
8bac3ff
Enable tests again
cosa65 Jan 11, 2023
c89f188
Add can rerun gem2s again
cosa65 Jan 12, 2023
478d879
Remove unused parameters
Jan 12, 2023
e86d004
Rename migration, change it to also remove can_rerun_gem2s
cosa65 Jan 12, 2023
bf0e086
Remove can_rerun_gem2s references
cosa65 Jan 12, 2023
4291784
Fix
cosa65 Jan 12, 2023
e6aecaf
Fix parentExpeirmentId checking on attempt to run gem2s
cosa65 Jan 12, 2023
9e451f9
Update runGem2s test
cosa65 Jan 12, 2023
7a2fabd
Add test for MethodNotAllowedError on runGem2s
cosa65 Jan 12, 2023
c8ad8a9
Add test for subsetSeurat hook
cosa65 Jan 12, 2023
9f85f84
Change line order
cosa65 Jan 13, 2023
49bc830
Fix tests
cosa65 Jan 13, 2023
69691cf
Merge pull request #90 from biomage-org/unique-sample-names
StefanBabukov Jan 13, 2023
ec3ec1d
add deployment name to config
aerlaut Jan 13, 2023
29c0dad
fix linting
aerlaut Jan 13, 2023
94ac75a
add deployment to Slack message
aerlaut Jan 13, 2023
c10a838
fix tests
aerlaut Jan 13, 2023
113a554
enable error for staging
aerlaut Jan 13, 2023
259045c
change type from plain text to markdown
aerlaut Jan 13, 2023
387ede9
better message format
aerlaut Jan 14, 2023
cd2d029
remove deployment name
aerlaut Jan 16, 2023
629808c
Merge branch 'master' into 2293-handle-subset-updates
cosa65 Jan 16, 2023
113145e
Merge pull request #92 from biomage-org/show-pipeline-error-origin
aerlaut Jan 17, 2023
851bf8f
remove deployment name
aerlaut Jan 17, 2023
fb2486a
Merge pull request #93 from biomage-org/revert-changes-before-merge
aerlaut Jan 17, 2023
b039ae4
Fix merge conflcits
cosa65 Jan 17, 2023
23ff170
Switch usage of paramsHash, use shouldRerun instead
cosa65 Jan 17, 2023
2a8b2b2
Add migration to add last_gem2s_params to the experiment_execution table
cosa65 Jan 17, 2023
cd1a44b
Merge branch 'master' into 2293-handle-subset-updates
cosa65 Jan 18, 2023
1465923
Add last_gem2s_params to an experiment fields reference
cosa65 Jan 18, 2023
4530bea
Uncomment auth middleware
cosa65 Jan 18, 2023
cf2ce71
Move shouldRerun in backendStatus
Jan 18, 2023
284c5b4
Update file structure
Jan 18, 2023
4cdf791
Remove /rerunStatus endpoint
Jan 18, 2023
163fc20
WIP - tests
Jan 18, 2023
a245f36
Comment testing
Jan 18, 2023
e739b23
Merge pull request #87 from biomage-org/2293-handle-subset-updates
cosa65 Jan 18, 2023
01b57db
Fix merge conflicts
cosa65 Jan 19, 2023
99b5696
Add metadata of ExperimentExecution to camelcase exceptions
cosa65 Jan 19, 2023
e6790a1
Merge branch 'master' into dependabot/npm_and_yarn/knex-2.4.0
ivababukova Jan 23, 2023
2bec2f1
first working version
kafkasl Jan 23, 2023
25b0df1
Merge branch 'master' of github.com:biomage-org/api into upload-metadata
kafkasl Jan 23, 2023
cea15ad
cleanup and tests
kafkasl Jan 23, 2023
ba068cb
removed unused call
kafkasl Jan 23, 2023
90252ee
added handling of wrong sample name
kafkasl Jan 23, 2023
1bff136
Merge pull request #88 from biomage-org/dependabot/npm_and_yarn/knex-…
ivababukova Jan 24, 2023
bb3302f
Bump cookiejar from 2.1.3 to 2.1.4
dependabot[bot] Jan 24, 2023
26ae8b8
Bump got, node-jq and nodemon
dependabot[bot] Jan 24, 2023
673de3d
Testing
Jan 24, 2023
afe5c8e
Merge branch 'master' into backend-hash
Jan 24, 2023
1208545
Uncomment codecov in ci
Jan 24, 2023
bdc2c4d
PR review improvements
kafkasl Jan 24, 2023
b1dc708
Remove unused params hash references
Jan 25, 2023
66e2fc1
accepting any kind of spacing in the parseTSV function
kafkasl Jan 24, 2023
2c3db8b
added clone experiment to userid
StefanBabukov Jan 25, 2023
43c0ea6
Fix: make getSamples return always all the samples, even if they don'…
cosa65 Jan 25, 2023
8eb3046
Renaming and removing of newlines
Jan 26, 2023
ee6659c
WIP - fixing comments
Jan 26, 2023
1362d02
Remove mock of replaceNullsWithObject, we can just check the knex cal…
cosa65 Jan 26, 2023
eaa8015
improved error messags
kafkasl Jan 26, 2023
ad3081b
Fetch samples once. Remove gem2s files
Jan 26, 2023
1102d25
Remove unused parameter
Jan 26, 2023
41ce2f9
tests fix
StefanBabukov Jan 26, 2023
fc197a3
Return new samples order to the ui doesnt need to think
cosa65 Jan 26, 2023
c4f4be5
Update test
cosa65 Jan 26, 2023
116ca38
Merge branch 'master' of github.com:biomage-org/api into upload-metadata
kafkasl Jan 27, 2023
ba02389
Resolve comments
Jan 27, 2023
566abbf
Merge pull request #95 from biomage-org/upload-metadata
kafkasl Jan 27, 2023
be5a63a
Merge branch 'master' into backend-hash
StefanBabukov Jan 27, 2023
0a69c89
Merge branch 'master' into 2310-fix-to-many-samples
cosa65 Jan 27, 2023
9dda1fb
Merge pull request #99 from biomage-org/2310-fix-to-many-samples
cosa65 Jan 27, 2023
1bad7f0
Fix circular dependancy
Jan 29, 2023
0fdc0c3
Fix circular dependancy, remove shouldRerun param from experiment bod…
Jan 29, 2023
28ebae7
Merge branch 'master' into backend-hash
Jan 29, 2023
0c4d855
Handle case for subsets
Jan 29, 2023
9e568d3
fix tests
Jan 29, 2023
251d11f
tell batch ignore ssl-cert
aerlaut Jan 31, 2023
979a463
Minor improvements
Jan 31, 2023
0ceac36
Update test
cosa65 Jan 31, 2023
affe8e5
Add test for isSubset return false
cosa65 Jan 31, 2023
bfb1eb8
add test for isSubset when experiment has a parent
Jan 31, 2023
08c43b4
Refactor getting gem2s comparison parameters
Feb 1, 2023
ba6280f
Merge branch 'master' into clone-experiment-to-user
StefanBabukov Feb 1, 2023
a02078b
Fix comments
Feb 1, 2023
97badd9
Add console logs temp
cosa65 Feb 1, 2023
ef8ee17
Add expId log tmp
cosa65 Feb 1, 2023
5ace8aa
Fix parameter not being passed correctly
cosa65 Feb 1, 2023
5ed3740
Merge pull request #91 from biomage-org/backend-hash
cosa65 Feb 1, 2023
a012a88
Merge branch 'master' into clone-experiment-to-user
StefanBabukov Feb 2, 2023
16be5b5
Merge branch 'master' into batch-ignore-ssl-certificate
aerlaut Feb 2, 2023
46e2e2c
Return null if no samples
cosa65 Feb 2, 2023
41d79b2
Remove console logs
cosa65 Feb 2, 2023
5bcb241
Merge pull request #106 from biomage-org/fix-backend-status-fetch-on-…
cosa65 Feb 2, 2023
9168e1d
Merge branch 'master' into dependabot/npm_and_yarn/cookiejar-2.1.4
ivababukova Feb 3, 2023
a4d2c97
Merge branch 'master' into dependabot/npm_and_yarn/got-and-node-jq-an…
ivababukova Feb 3, 2023
2a95fef
Merge pull request #96 from biomage-org/dependabot/npm_and_yarn/cooki…
ivababukova Feb 3, 2023
2f8f1b9
Merge branch 'master' into dependabot/npm_and_yarn/got-and-node-jq-an…
ivababukova Feb 3, 2023
d20fe44
Merge pull request #97 from biomage-org/dependabot/npm_and_yarn/got-a…
ivababukova Feb 3, 2023
5c1444c
Bump http-cache-semantics from 4.1.0 to 4.1.1
dependabot[bot] Feb 3, 2023
a601a2b
Merge branch 'master' into clone-experiment-to-user
StefanBabukov Feb 3, 2023
7988734
Merge branch 'master' into batch-ignore-ssl-certificate
cosa65 Feb 3, 2023
8afede6
change to using Boolean
aerlaut Feb 3, 2023
da1cbda
Merge pull request #105 from biomage-org/batch-ignore-ssl-certificate
aerlaut Feb 3, 2023
957dddf
Merge branch 'master' into clone-experiment-to-user
StefanBabukov Feb 3, 2023
0d7130d
Add work request body of ScTypeAnnotate
cosa65 Feb 6, 2023
dae2111
Add migration for fields
Feb 7, 2023
1c06d35
Add query for example experiments
Feb 7, 2023
da79cde
Add validation
Feb 7, 2023
a7107cf
Simplify migration. Add test.
Feb 7, 2023
a6da3ff
Remove console log
Feb 7, 2023
d97f4cf
Remove unnecessary comments
Feb 7, 2023
6dba17b
Remove unnecassary newlines
Feb 7, 2023
2eac40e
Merge remote-tracking branch 'biomage/master' into biomage-changes-2
alexvpickering Feb 7, 2023
aaf329d
fix tests
alexvpickering Feb 7, 2023
37f8aa7
Merge pull request #109 from biomage-org/2358-add-sctype-ui-comp
cosa65 Feb 8, 2023
b8499a9
Merge branch 'master' into clone-experiment-to-user
cosa65 Feb 8, 2023
c0fe9b7
test added
StefanBabukov Feb 8, 2023
0550483
merge confs and tests added
StefanBabukov Feb 8, 2023
4eef5ef
update seurat stuff
alexvpickering Feb 8, 2023
9511935
fix seurat
alexvpickering Feb 8, 2023
5e54b0b
fix shouldPipelineRerun
alexvpickering Feb 8, 2023
5ec06ef
Merge pull request #100 from biomage-org/clone-experiment-to-user
StefanBabukov Feb 9, 2023
3b91abe
fix tests
alexvpickering Feb 9, 2023
ee38012
Merge branch 'master' into example-experiment-fields
kafkasl Feb 13, 2023
154a533
Merge pull request #111 from biomage-org/example-experiment-fields
kafkasl Feb 13, 2023
f718ee1
remove unused mocks of S3 delete
ivababukova Feb 20, 2023
d8b7b5b
Merge pull request #115 from biomage-org/not-used-mock
ivababukova Feb 20, 2023
3bc1ab8
Merge branch 'master' into dependabot/npm_and_yarn/http-cache-semanti…
ivababukova Feb 21, 2023
97cccaa
Merge pull request #108 from biomage-org/dependabot/npm_and_yarn/http…
ivababukova Feb 21, 2023
bcab067
Merge remote-tracking branch 'biomage/master' into biomage-changes-2
alexvpickering Feb 21, 2023
0dd279b
remove tags
alexvpickering Feb 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
prevent double reporting
  • Loading branch information
aerlaut committed Jan 2, 2023

Unverified

The committer email address is not verified.
commit acca57a06d0d71b543dbefa2b634ba8cc1c9c344
4 changes: 2 additions & 2 deletions src/api.v2/constants.js
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ const ASSIGN_POD_TO_PIPELINE = 'assignPodToPipeline';
const PIPELINE_ERROR = 'pipelineError';

// Pipeline step name
const HANDLE_ERROR_STEP = 'HandleErrorStep';
const HANDLE_TIMEOUT_ERROR_STEP = 'HandleTimeoutErrorStep';
const END_OF_PIPELINE = 'EndOfPipeline';

// Pipeline states as defined in
@@ -45,7 +45,7 @@ module.exports = {
OLD_QC_NAME_TO_BE_REMOVED,
SUBSET_PROCESS_NAME,
PIPELINE_ERROR,
HANDLE_ERROR_STEP,
HANDLE_TIMEOUT_ERROR_STEP,
END_OF_PIPELINE,
RUNNING,
FAILED,
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@ const constructPipelineStep = (context, step) => {
return createNewStep(context, step, args);
}
case 'create-handle-error-step': {
return createHandleErrorStep(context, step);
return createHandleErrorStep(context, step, args);
}
case 'mark-as-failed': {
return createFailedStep(context, step);
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
const config = require('../../../../../config');
const { PIPELINE_ERROR } = require('../../../../constants');
const { getActivityId } = require('../utils');

const buildErrorMessage = (
@@ -22,8 +21,9 @@ const buildErrorMessage = (
},
});

const createHandleErrorStep = (context, step) => {
const createHandleErrorStep = (context, step, args) => {
console.log('*** context', context);
console.log('*** args', args);
const {
environment,
accountId,
@@ -34,14 +34,18 @@ const createHandleErrorStep = (context, step) => {
authJWT,
} = context;

const { errorType } = args;

const activityId = getActivityId(activityArn);

const errorMessage = buildErrorMessage(sandboxId,
const errorMessage = buildErrorMessage(
sandboxId,
experimentId,
PIPELINE_ERROR,
errorType,
processName,
activityId,
authJWT);
authJWT,
);

console.log('*** errorMessage', errorMessage);

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const config = require('../../../../../config');
const { HANDLE_ERROR_STEP } = require('../../../../constants');
const { HANDLE_TIMEOUT_ERROR_STEP } = require('../../../../constants');

const createNewJobIfNotExist = (context, step) => {
const { accountId, activityArn, processName } = context;
@@ -19,9 +19,9 @@ const createNewJobIfNotExist = (context, step) => {
},
Catch: [
{
ErrorEquals: ['States.ALL'],
ErrorEquals: ['States.Timeout'],
ResultPath: '$.error-info',
Next: HANDLE_ERROR_STEP,
Next: HANDLE_TIMEOUT_ERROR_STEP,
},
],
};
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ const { QC_PROCESS_NAME, GEM2S_PROCESS_NAME, SUBSET_PROCESS_NAME } = require('..
const getGeneralParams = require('./paramsGetters/getGeneralParams');
const getQCParams = require('./paramsGetters/getQCParams');
const getSubsetParams = require('./paramsGetters/getSubsetParams');
const { HANDLE_ERROR_STEP } = require('../../../../constants');
const { HANDLE_TIMEOUT_ERROR_STEP } = require('../../../../constants');

const buildParams = (context, stepArgs) => {
let stepParams;
@@ -38,9 +38,9 @@ const createNewStep = (context, step, stepArgs) => {
...!step.End && { Next: step.Next },
Catch: [
{
ErrorEquals: ['States.ALL'],
ErrorEquals: ['States.Timeout'],
ResultPath: '$.error-info',
Next: HANDLE_ERROR_STEP,
Next: HANDLE_TIMEOUT_ERROR_STEP,
},
],
};
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const config = require('../../../../../config');
const { HANDLE_ERROR_STEP } = require('../../../../constants');
const { HANDLE_TIMEOUT_ERROR_STEP } = require('../../../../constants');

const submitBatchJob = (context, step) => {
const {
@@ -68,9 +68,9 @@ const submitBatchJob = (context, step) => {
},
Catch: [
{
ErrorEquals: ['States.ALL'],
ErrorEquals: ['States.Timeout'],
ResultPath: '$.error-info',
Next: HANDLE_ERROR_STEP,
Next: HANDLE_TIMEOUT_ERROR_STEP,
},
],
};
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
const { buildQCPipelineSteps, qcPipelineSteps } = require('./qcPipelineSkeleton');
const { gem2SPipelineSteps } = require('./gem2sPipelineSkeleton');
const subsetPipelineSteps = require('./subsetPipelineSteps');
const { END_OF_PIPELINE, HANDLE_ERROR_STEP } = require('../../../../constants');
const { END_OF_PIPELINE, HANDLE_TIMEOUT_ERROR_STEP } = require('../../../../constants');


const createLocalPipeline = (nextStep) => ({
@@ -78,8 +78,11 @@ const buildInitialSteps = (clusterEnv, nextStep, runInBatch) => {
};

const buildErrorHandlingSteps = () => ({
[HANDLE_ERROR_STEP]: {
[HANDLE_TIMEOUT_ERROR_STEP]: {
XStepType: 'create-handle-error-step',
XConstructorArgs: {
errorType: 'timeout',
},
Next: 'MarkAsFailed',
},
MarkAsFailed: {
2 changes: 1 addition & 1 deletion src/utils/send-failed-slack-message.js
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ const sendFailedSlackMessage = async (message, user, process, stateMachineArn) =
type: 'plain_text',
text: `
The process for ${process} has failed for experiment ${experimentId}
Step: ${message.input.taskName}
Step: ${message.input.taskName ? message.input.taskName : 'timeout'}
State machine arn: ${stateMachineArn}
`,
},
Original file line number Diff line number Diff line change
@@ -178,7 +178,7 @@ exports[`test for pipeline services Create Subset pipeline works: createStateMac
Array [
Array [
Object {
"definition": "{\\"Comment\\":\\"Subset Pipeline for clusterEnv 'test'\\",\\"StartAt\\":\\"RequestPod\\",\\"States\\":{\\"RequestPod\\":{\\"ResultPath\\":null,\\"Next\\":\\"WaitForPod\\",\\"Comment\\":\\"Send a message through SNS so that the API assigns a pod to the pipeline\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:::sns:publish\\",\\"Parameters\\":{\\"TopicArn\\":\\"arn:aws:sns:eu-west-1:000000000000:work-results-test-default-v2\\",\\"Message\\":\\"{\\\\\\"taskName\\\\\\":\\\\\\"assignPodToPipeline\\\\\\",\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"apiUrl\\\\\\":\\\\\\"test-public-api-url\\\\\\",\\\\\\"input\\\\\\":{\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"sandboxId\\\\\\":\\\\\\"default\\\\\\",\\\\\\"activityId\\\\\\":\\\\\\"pipeline-test-mock-uuid\\\\\\",\\\\\\"processName\\\\\\":\\\\\\"subset\\\\\\"}}\\",\\"MessageAttributes\\":{\\"type\\":{\\"DataType\\":\\"String\\",\\"StringValue\\":\\"PipelineResponse\\"}}}},\\"WaitForPod\\":{\\"ResultPath\\":null,\\"Next\\":\\"SubsetSeurat\\",\\"Type\\":\\"Map\\",\\"ItemsPath\\":\\"$.retries\\",\\"MaxConcurrency\\":1,\\"Retry\\":[{\\"ErrorEquals\\":[\\"NoPodAssigned\\"],\\"IntervalSeconds\\":1,\\"MaxAttempts\\":13,\\"BackoffRate\\":1.5}],\\"Iterator\\":{\\"StartAt\\":\\"GetAssignedPod\\",\\"States\\":{\\"GetAssignedPod\\":{\\"Next\\":\\"IsPodAssigned\\",\\"Type\\":\\"Task\\",\\"Comment\\":\\"Retrieves first unassigned and running pipeline pod.\\",\\"Resource\\":\\"arn:aws:states:::eks:call\\",\\"Parameters\\":{\\"ClusterName\\":\\"biomage-test\\",\\"CertificateAuthority\\":\\"AAAAAAAAAAA\\",\\"Endpoint\\":\\"https://test-endpoint.me/fgh\\",\\"Method\\":\\"GET\\",\\"Path\\":\\"/api/v1/namespaces/pipeline-test-namespace/pods\\",\\"QueryParameters\\":{\\"labelSelector\\":[\\"type=pipeline,activityId=pipeline-test-mock-uuid\\"]}}},\\"IsPodAssigned\\":{\\"Type\\":\\"Choice\\",\\"Comment\\":\\"Redirects to an error state if the pipeline pod is not assigned yet.\\",\\"Choices\\":[{\\"Variable\\":\\"$.ResponseBody.items[0]\\",\\"IsPresent\\":false,\\"Next\\":\\"NoPodAssigned\\"}],\\"Default\\":\\"ReadyToRun\\"},\\"NoPodAssigned\\":{\\"Type\\":\\"Fail\\",\\"Cause\\":\\"No available and running pipeline pods.\\",\\"Error\\":\\"NoPodAssigned\\"},\\"ReadyToRun\\":{\\"Type\\":\\"Pass\\",\\"End\\":true}}}},\\"SubsetSeurat\\":{\\"Next\\":\\"PrepareExperiment\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"subsetSeurat\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"parentExperimentId\\":\\"fromExperimentId\\",\\"subsetExperimentId\\":\\"toExperimentId\\",\\"cellSetKeys\\":[\\"louvain-1\\",\\"louvain-2\\"]},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.ALL\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleErrorStep\\"}]},\\"PrepareExperiment\\":{\\"Next\\":\\"UploadToAWS\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"prepareExperiment\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"experimentName\\":\\"toExperimentName\\",\\"authJWT\\":\\"mockAuthJWT\\"},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.ALL\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleErrorStep\\"}]},\\"UploadToAWS\\":{\\"Next\\":\\"EndOfSubset\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"uploadToAWS\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"experimentName\\":\\"toExperimentName\\",\\"authJWT\\":\\"mockAuthJWT\\"},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.ALL\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleErrorStep\\"}]},\\"EndOfSubset\\":{\\"Type\\":\\"Pass\\",\\"End\\":true},\\"HandleErrorStep\\":{\\"Next\\":\\"MarkAsFailed\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:::sns:publish\\",\\"Parameters\\":{\\"TopicArn\\":\\"arn:aws:sns:eu-west-1:000000000000:work-results-test-default-v2\\",\\"Message\\":\\"{\\\\\\"taskName\\\\\\":\\\\\\"pipelineError\\\\\\",\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"apiUrl\\\\\\":\\\\\\"test-public-api-url\\\\\\",\\\\\\"input\\\\\\":{\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"sandboxId\\\\\\":\\\\\\"default\\\\\\",\\\\\\"activityId\\\\\\":\\\\\\"pipeline-test-mock-uuid\\\\\\",\\\\\\"processName\\\\\\":\\\\\\"subset\\\\\\"}}\\",\\"MessageAttributes\\":{\\"type\\":{\\"DataType\\":\\"String\\",\\"StringValue\\":\\"PipelineError\\"}}}},\\"MarkAsFailed\\":{\\"Type\\":\\"Fail\\"},\\"EndOfPipeline\\":{\\"Type\\":\\"Pass\\",\\"End\\":true}}}",
"definition": "{\\"Comment\\":\\"Subset Pipeline for clusterEnv 'test'\\",\\"StartAt\\":\\"RequestPod\\",\\"States\\":{\\"RequestPod\\":{\\"ResultPath\\":null,\\"Next\\":\\"WaitForPod\\",\\"Comment\\":\\"Send a message through SNS so that the API assigns a pod to the pipeline\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:::sns:publish\\",\\"Parameters\\":{\\"TopicArn\\":\\"arn:aws:sns:eu-west-1:000000000000:work-results-test-default-v2\\",\\"Message\\":\\"{\\\\\\"taskName\\\\\\":\\\\\\"assignPodToPipeline\\\\\\",\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"apiUrl\\\\\\":\\\\\\"test-public-api-url\\\\\\",\\\\\\"input\\\\\\":{\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"sandboxId\\\\\\":\\\\\\"default\\\\\\",\\\\\\"activityId\\\\\\":\\\\\\"pipeline-test-mock-uuid\\\\\\",\\\\\\"processName\\\\\\":\\\\\\"subset\\\\\\"}}\\",\\"MessageAttributes\\":{\\"type\\":{\\"DataType\\":\\"String\\",\\"StringValue\\":\\"PipelineResponse\\"}}}},\\"WaitForPod\\":{\\"ResultPath\\":null,\\"Next\\":\\"SubsetSeurat\\",\\"Type\\":\\"Map\\",\\"ItemsPath\\":\\"$.retries\\",\\"MaxConcurrency\\":1,\\"Retry\\":[{\\"ErrorEquals\\":[\\"NoPodAssigned\\"],\\"IntervalSeconds\\":1,\\"MaxAttempts\\":13,\\"BackoffRate\\":1.5}],\\"Iterator\\":{\\"StartAt\\":\\"GetAssignedPod\\",\\"States\\":{\\"GetAssignedPod\\":{\\"Next\\":\\"IsPodAssigned\\",\\"Type\\":\\"Task\\",\\"Comment\\":\\"Retrieves first unassigned and running pipeline pod.\\",\\"Resource\\":\\"arn:aws:states:::eks:call\\",\\"Parameters\\":{\\"ClusterName\\":\\"biomage-test\\",\\"CertificateAuthority\\":\\"AAAAAAAAAAA\\",\\"Endpoint\\":\\"https://test-endpoint.me/fgh\\",\\"Method\\":\\"GET\\",\\"Path\\":\\"/api/v1/namespaces/pipeline-test-namespace/pods\\",\\"QueryParameters\\":{\\"labelSelector\\":[\\"type=pipeline,activityId=pipeline-test-mock-uuid\\"]}}},\\"IsPodAssigned\\":{\\"Type\\":\\"Choice\\",\\"Comment\\":\\"Redirects to an error state if the pipeline pod is not assigned yet.\\",\\"Choices\\":[{\\"Variable\\":\\"$.ResponseBody.items[0]\\",\\"IsPresent\\":false,\\"Next\\":\\"NoPodAssigned\\"}],\\"Default\\":\\"ReadyToRun\\"},\\"NoPodAssigned\\":{\\"Type\\":\\"Fail\\",\\"Cause\\":\\"No available and running pipeline pods.\\",\\"Error\\":\\"NoPodAssigned\\"},\\"ReadyToRun\\":{\\"Type\\":\\"Pass\\",\\"End\\":true}}}},\\"SubsetSeurat\\":{\\"Next\\":\\"PrepareExperiment\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"subsetSeurat\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"parentExperimentId\\":\\"fromExperimentId\\",\\"subsetExperimentId\\":\\"toExperimentId\\",\\"cellSetKeys\\":[\\"louvain-1\\",\\"louvain-2\\"]},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.Timeout\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleTimeoutErrorStep\\"}]},\\"PrepareExperiment\\":{\\"Next\\":\\"UploadToAWS\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"prepareExperiment\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"experimentName\\":\\"toExperimentName\\",\\"authJWT\\":\\"mockAuthJWT\\"},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.Timeout\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleTimeoutErrorStep\\"}]},\\"UploadToAWS\\":{\\"Next\\":\\"EndOfSubset\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:eu-west-1:000000000000:activity:pipeline-test-mock-uuid\\",\\"ResultPath\\":null,\\"TimeoutSeconds\\":10800,\\"HeartbeatSeconds\\":10,\\"Parameters\\":{\\"experimentId\\":\\"toExperimentId\\",\\"taskName\\":\\"uploadToAWS\\",\\"processName\\":\\"subset\\",\\"server\\":\\"remoter-server-toExperimentId.pipeline-test-namespace.svc.cluster.local\\",\\"experimentName\\":\\"toExperimentName\\",\\"authJWT\\":\\"mockAuthJWT\\"},\\"Catch\\":[{\\"ErrorEquals\\":[\\"States.Timeout\\"],\\"ResultPath\\":\\"$.error-info\\",\\"Next\\":\\"HandleTimeoutErrorStep\\"}]},\\"EndOfSubset\\":{\\"Type\\":\\"Pass\\",\\"End\\":true},\\"HandleTimeoutErrorStep\\":{\\"Next\\":\\"MarkAsFailed\\",\\"Type\\":\\"Task\\",\\"Resource\\":\\"arn:aws:states:::sns:publish\\",\\"Parameters\\":{\\"TopicArn\\":\\"arn:aws:sns:eu-west-1:000000000000:work-results-test-default-v2\\",\\"Message\\":\\"{\\\\\\"taskName\\\\\\":\\\\\\"timeout\\\\\\",\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"apiUrl\\\\\\":\\\\\\"test-public-api-url\\\\\\",\\\\\\"input\\\\\\":{\\\\\\"experimentId\\\\\\":\\\\\\"toExperimentId\\\\\\",\\\\\\"sandboxId\\\\\\":\\\\\\"default\\\\\\",\\\\\\"activityId\\\\\\":\\\\\\"pipeline-test-mock-uuid\\\\\\",\\\\\\"processName\\\\\\":\\\\\\"subset\\\\\\"}}\\",\\"MessageAttributes\\":{\\"type\\":{\\"DataType\\":\\"String\\",\\"StringValue\\":\\"PipelineError\\"}}}},\\"MarkAsFailed\\":{\\"Type\\":\\"Fail\\"},\\"EndOfPipeline\\":{\\"Type\\":\\"Pass\\",\\"End\\":true}}}",
"loggingConfiguration": Object {
"level": "OFF",
},
@@ -251,9 +251,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -305,9 +305,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -354,9 +354,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -402,9 +402,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -460,9 +460,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -503,10 +503,10 @@ Array [
"End": true,
"Type": "Pass",
},
"HandleErrorStep": Object {
"HandleTimeoutErrorStep": Object {
"Next": "MarkAsFailed",
"Parameters": Object {
"Message": "{\\"taskName\\":\\"pipelineError\\",\\"experimentId\\":\\"testExperimentId\\",\\"apiUrl\\":\\"test-public-api-url\\",\\"input\\":{\\"experimentId\\":\\"testExperimentId\\",\\"sandboxId\\":\\"default\\",\\"activityId\\":\\"pipeline-test-mock-uuid\\",\\"processName\\":\\"qc\\"}}",
"Message": "{\\"taskName\\":\\"timeout\\",\\"experimentId\\":\\"testExperimentId\\",\\"apiUrl\\":\\"test-public-api-url\\",\\"input\\":{\\"experimentId\\":\\"testExperimentId\\",\\"sandboxId\\":\\"default\\",\\"activityId\\":\\"pipeline-test-mock-uuid\\",\\"processName\\":\\"qc\\"}}",
"MessageAttributes": Object {
"type": Object {
"DataType": "String",
@@ -530,9 +530,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
@@ -594,9 +594,9 @@ Array [
"Catch": Array [
Object {
"ErrorEquals": Array [
"States.ALL",
"States.Timeout",
],
"Next": "HandleErrorStep",
"Next": "HandleTimeoutErrorStep",
"ResultPath": "$.error-info",
},
],
Loading