[WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows #7392

salonishah11 · 2024-03-27T21:16:19Z

This PR adds support for fetching Github token for user from ECM to support private workflows.

Testing
I have tested these changes in Workflows and Cromwell Runner apps in dev by manually editing the config as well as updating the deployment with Cromwell image containing the changes. Screenshot of a successful run of private workflow.

Closes

cjllanwarne · 2024-03-28T14:18:52Z

services/src/main/scala/cromwell/services/auth/ecm/EcmService.scala

+  private val getGithubAccessTokenApiPath = "api/oauth/v1/github/access-token"
+
+  /*
+     ECM doesn't have a standard error response format. Some of the responses contains HTML tags in it. This helper


I think technically ECM always returns json format, but it's possible for things in the network layer between you and ECM (like the apache proxies, etc) to return html pages

That's true. I have updated the comment. I also looked at the error handling in ECM and it seems it should return a standard JSON response in format of ErrorReport in all other status codes. Hence I have updated the helper method to

only return custom error message in case its 401

try and parse response body as JSON and extract the value associated with message key if found

return the response body in case parsing failed or key message was not found

services/src/main/scala/cromwell/services/auth/ecm/EcmService.scala

cjllanwarne · 2024-03-28T14:30:23Z

services/src/main/scala/cromwell/services/auth/impl/GithubAuthVendingActor.scala

+
+  lazy val enabled: Boolean = serviceConfig.getBoolean("enabled")
+
+  private lazy val ecmConfigOpt: Option[EcmConfig] = EcmConfig.apply(serviceConfig)


FYI this should work but apply is a pseudo-reserved word for scala because it gets auto-called when you run var = EcmConfig(serviceConfig). So the convention would be for an apply to return an instance, not an option of an instance.

In this case, I would just rename to something like fromConfig

cjllanwarne · 2024-03-28T14:32:14Z

services/src/main/scala/cromwell/services/auth/impl/GithubAuthVendingActor.scala

+
+  lazy val enabled: Boolean = serviceConfig.getBoolean("enabled")
+
+  private lazy val ecmConfigOpt: Option[EcmConfig] = EcmConfig.apply(serviceConfig)


Because it's in reference.conf there will always be some kind of ecm config in the config (you can't remove values by adding a config on top, only redefined them).

I think what you really want is for the baseUrl to be optional, and for that to be what we're switching on.

That's a good point that you can't remove values and only redefine them in additional configs. Then in this case it means that ECM base url will always be present so I think we might not need for either base url or the EcmConfig to be Optional. What do you think about simply removing the Optional and assuming ecmConfig will always be defined?

Based on Janet's comment, I have updated the reference.conf file and commented out ecm.base-url. Hence base url is now optional and so I have updated EcmConfig to have Optional baseUrl instead of EcmConfig object being Optional. Since L28 now returns EcmConfig object in all cases I have kept the apply method name as is. See ea97924

services/src/main/scala/cromwell/services/auth/impl/GithubAuthVendingActor.scala

cjllanwarne · 2024-03-28T14:37:15Z

services/src/main/scala/cromwell/services/auth/ecm/EcmService.scala

+    def responseEntityToFutureStr(responseEntity: ResponseEntity): Future[String] =
+      responseEntity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String)


I know akka is picky and can throw exceptions if we don't read the bytes in time (within 1 second?). How comfortable are we that this future will evaluate in time?

I copied the pattern we have in Cromwell (see WorkflowCallbackActor.scala and TesAsyncBackendJobExecutionActor.scala) and didn't realize that 1 second was the default timeout. Is there an implicit timeout being imported in these 2 references that I also need to update here? 🤔

It's possible that we're hitting this error in cases where we aren't choosing to read the bytes at all, rather than cases where we take too long to read them. We're planning to do WX-1525 next week, which should confirm that one way or the other.

In this case, should the whole response be loaded into memory as Strict entity using .toStrict(<timeout>) method?

I would just make sure that responseEntity.dataBytes.runFold is actually getting executed in all code paths. It looks like you're calling it on both success and failure codes, so I think you're probably okay!

cjllanwarne · 2024-03-28T14:38:04Z

services/src/test/scala/cromwell/services/auth/ecm/EcmServiceSpec.scala

+  forAll(testCases) { (testName, statusCode, responseBodyAsStr, expectedErrorMsg) =>
+    it should testName in {
+      assert(ecmService.extractErrorMessage(statusCode, responseBodyAsStr) == expectedErrorMsg)
+    }


Woohoo for table based tests!

services/src/test/scala/cromwell/services/auth/impl/GithubAuthVendingActorSpec.scala

cjllanwarne

Looking good, only a few minor comments left

services/src/main/scala/cromwell/services/auth/impl/GithubAuthVendingActor.scala

cjllanwarne · 2024-03-29T18:39:52Z

services/src/test/scala/cromwell/services/auth/impl/GithubAuthVendingActorSpec.scala

-        )
-      )
+  private val testCases = Table(
+    ("test name", "service config", "use TestEcmService class", "terra token", "response"),


Can we add a case for "valid config / no token for user in ECM"?

Do you think this is covered by the last test case return failure message if ECM service returns non-successful response? It seems that ECM would return a non-successful response if there was no Github token associated with the user and the last test case does simulate getGithubAccessToken() returning a Failed future which is what would happen in "no token for user in ECM" case too.

services/src/test/scala/cromwell/services/auth/impl/GithubAuthVendingActorSpec.scala

jgainerdewar · 2024-03-29T16:29:44Z

core/src/main/resources/reference.conf

-      #   - don't include the 'Bearer' before the token
-      #   - this config value should be removed when support for fetching tokens from ECM has been added to Cromwell
-      access-token = "dummy-token"
+      ecm.base-url = "https://externalcreds.dsde-dev.broadinstitute.org"


I would discourage setting the default to a real Terra service here - might be confusing for non-Terra users.

I see. In this case should there be no default? I can comment this out and add comment describing what service it is looking for?

Something like this?

config { enabled = false auth.azure = false # Set this to the service that Cromwell should retrieve Github access token associated with user's token. # ecm.base-url = "https://example.org" }

Yeah, that looks good!

jgainerdewar · 2024-03-29T19:04:14Z

services/src/main/scala/cromwell/services/auth/ecm/EcmService.scala

+    def responseEntityToFutureStr(responseEntity: ResponseEntity): Future[String] =
+      responseEntity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String)


It's possible that we're hitting this error in cases where we aren't choosing to read the bytes at all, rather than cases where we take too long to read them. We're planning to do WX-1525 next week, which should confirm that one way or the other.

jgainerdewar · 2024-04-01T15:55:54Z

core/src/main/resources/reference.conf

-      #   - this config value should be removed when support for fetching tokens from ECM has been added to Cromwell
-      access-token = "dummy-token"
+      # Set this to the service that Cromwell should retrieve Github access token associated with user's token.
+      # ecm.base-url = "https://externalcreds.dsde-dev.broadinstitute.org/"


My previous comment disappeared because of another change to this file (thanks Github) so just calling out that we talked about removing this URL in a now-outdated thread.

Oops, I had the change locally but forgot to push it. ecm.base-url should now have value https://example.org

cjllanwarne

I do just have some quick, minor questions about the config layout and "enabled" value

cjllanwarne · 2024-04-02T13:35:29Z

core/src/main/resources/reference.conf

@@ -578,7 +578,8 @@ services {
    config {
      enabled = false
      auth.azure = false
-      ecm.base-url = "https://externalcreds.dsde-dev.broadinstitute.org"
+      # Set this to the service that Cromwell should retrieve Github access token associated with user's token.
+      # ecm.base-url = "https://example.org"


Really minor: Can we just leave this blank?

cjllanwarne · 2024-04-02T13:36:12Z

services/src/main/scala/cromwell/services/auth/ecm/EcmConfig.scala

@@ -3,8 +3,8 @@ package cromwell.services.auth.ecm
 import com.typesafe.config.Config
 import net.ceedubs.ficus.Ficus._

-final case class EcmConfig(baseUrl: String)
+final case class EcmConfig(baseUrl: Option[String])


TOL is there a reason the enabled and auth.azure fields are not here?

Discussed via Slack but posting here for reference: Since these values are more related to enabling GithubAuthVending service they are separate from the ecm sub-section in the config.

cjllanwarne · 2024-04-02T13:36:26Z

services/src/main/scala/cromwell/services/auth/ecm/EcmConfig.scala


 object EcmConfig {
-  def apply(config: Config): Option[EcmConfig] = config.as[Option[String]]("ecm.base-url").map(EcmConfig(_))
+  def apply(config: Config): EcmConfig = EcmConfig(config.as[Option[String]]("ecm.base-url"))


What if enabled is false?

Discussed via Slack but posting here for reference: Since it is separate from ecm config it is handled separately here

cjllanwarne · 2024-04-02T13:37:41Z

services/src/test/scala/cromwell/services/auth/ecm/EcmConfigSpec.scala

-    actualEcmConfig shouldBe defined
-    actualEcmConfig.get.baseUrl shouldBe "https://mock-ecm-url.org"
+    actualEcmConfig.baseUrl shouldBe defined
+    actualEcmConfig.baseUrl.get shouldBe "https://mock-ecm-url.org"
  }

  it should "return None when ECM base url is absent" in {


Should we also have a case for the other config values (to continue the theme... like enabled!)

Yes they should be covered here

salonishah11 · 2024-04-02T17:00:15Z

Not sure why codecov/patch is reporting a low number. I have added unit tests for the modified code as possible but it doesn't seem to be recognizing it 🤔

@kevinRenaersBio

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <[email protected]> * PR feedback Co-authored-by: Brian Reilly <[email protected]> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <[email protected]> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <[email protected]> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <[email protected]> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <[email protected]> * Apply PR feedback in second place Co-authored-by: Adam Nichols <[email protected]> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <[email protected]> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <[email protected]> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) * Update cromwell version from 85 to 86 * WX-905 (broadinstitute#7012) Co-authored-by: Thomas Wiseman <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-719 modernize `cromwell-publish` image (broadinstitute#7013) * WX-930 Add `CODEOWNERS` file (broadinstitute#7015) * Update FiveMinuteIntro: Java 11 requirement (broadinstitute#6830) Co-authored-by: Adam Nichols <[email protected]> * WX-952 Fix Cromwell version update in Helm chart (broadinstitute#7014) * Reference Disk Manifest Builder App Test (broadinstitute#7017) * [WM-1696] Update Womtool (broadinstitute#7019) Co-authored-by: Tom Wiseman <[email protected]> * WX-958 write_map() should write its last entry with a newline (broadinstitute#7022) * Centaur slurm (broadinstitute#7083) * Remove Deploy Key (broadinstitute#7084) * WX-950 Set user agent in Java client (broadinstitute#7087) * WX-950 Upgrade Azure libs to probably fix extraneous log (broadinstitute#7088) * WX-984 Revert "WX-950 Upgrade Azure libs to probably fix extraneous log" (broadinstitute#7090) * Add cron run integrations each day of the week (broadinstitute#7089) * 85 release (#28) * Update cromwell version from 83 to 84 * BW-1255 Implement POST /runs endpoint (broadinstitute#6779) * Adding route * Fixing HTTP method error * All formFields made optional * A compliling state * Saving * Saving * All three endpoints functioning as expected; updated RESTAPI.md * Updated response for submission from 200 to 201 to pass tests * Test submission response * Moved updated submission response to askSubmit * test * updating RESTAPI.md * saving * Adding utility file for submitRequest * cleanup * Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814) * BW-1305 Swagger Update (broadinstitute#6818) * Properly documenting metadataArchiveStatus in WorkflowQueryResult model * Update docs * BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817) BT-710 Add configs for BlobPathBuilderFactory * BW-1305 Make "name" optional in workflow query response (broadinstitute#6821) * BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816) Modify blobPathBuilder to fallback to creating a filesystem if one is not found * Logging updates: (broadinstitute#6813) * [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824) * first pass on BlobTokenGenerator with E2E test * update BlobPathBuilder constructor args in test * account -> container level client * [BT-687] specify correct types (broadinstitute#6829) * specify correct types * fix test with new type * remove type declarations in function call * remove unnecessary sas-token config * BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833) * Add tests, getting frid of WesRunRoutes.scala * wesWorkflowId fix, ec implicits errors gone * Refactoring path for GET /runs * Indentation fix * Commit to rollback * Revert "Indentation fix" This reverts commit 63fc484. * PR trigger * Optimize imports * Missed import * BW-1354 - Porting CBAS preliminary step (broadinstitute#6837) * Getting rid of shared utility file; Adding/Updating WES version of submit. * Edit spec file * Adding Wes-like error * BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826) * Update cromwell version from 84 to 85 * BW-1393 Release doc updates (broadinstitute#6839) * BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <[email protected]> * PR feedback Co-authored-by: Brian Reilly <[email protected]> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <[email protected]> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <[email protected]> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <[email protected]> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <[email protected]> * Apply PR feedback in second place Co-authored-by: Adam Nichols <[email protected]> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <[email protected]> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <[email protected]> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Katrina P <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Saloni Shah <[email protected]> Co-authored-by: kshakir <[email protected]> Co-authored-by: mspector <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Brian Reilly <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Trevyn Langsford <[email protected]> Co-authored-by: Miguel Covarrubias <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Blair Murri <[email protected]> * Remove Yaml (broadinstitute#7098) * Azure blob read/write integration test (broadinstitute#7024) Co-authored-by: Tom Wiseman <[email protected]> * Goodbye Travis (broadinstitute#7100) * Single workflow runner (broadinstitute#7106) * WX-975 Revert special TES path munging (broadinstitute#7103) * Centaur blob filesystem (broadinstitute#7104) * WX-1010 Update Github Actions set output syntax (broadinstitute#7111) * WX-1001 Upgrade JSON Small and Fast Parser (broadinstitute#7116) * WX-1015 Override glob behavior for TES to use absolute path (broadinstitute#7120) * WX-990 Make TES request backoff behavior configurable (broadinstitute#7122) * WX-1103 Revert CI to old TES polling behavior (broadinstitute#7126) * Develop aws (#29) * stuck on globbing * efs works, no callcaching * update readme * extended EFS support * fix for globbing in nested scatters * updated config for globbing, to prevent issues with empty folders * WX-1103 Accelerate TES CI (broadinstitute#7130) * Fix Horicromtal Deadlock Test (broadinstitute#7131) * WM-1963: Validate PRs begin with Jira tags (broadinstitute#7127) * WX-1106 Add logging for failed docker manifest pulls (broadinstitute#7135) Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * [WX-1108] Disable drs tests (broadinstitute#7145) * ID-377 Setup pact for consumer contract testing. (broadinstitute#7123) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1092 Support `size` engine function for public HTTP files (broadinstitute#7128) * WX-1105 Fix interpretation of full http blob paths (broadinstitute#7138) Co-authored-by: Adam Nichols <[email protected]> * Revert "ID-377 Setup pact for consumer contract testing." (broadinstitute#7146) * WX-966 Add Cascades, remove Directory from Biscayne (broadinstitute#7105) * WX-966 Post-merge branch cleanup + test fix (broadinstitute#7149) * WX-1122 Enable Azure ApplicationInsights (broadinstitute#7143) * [WX-1108] Add tests back (broadinstitute#7148) * [WX-1136] Docker Build Test (broadinstitute#7151) * [WX-1136] Self Hosted (broadinstitute#7152) * WX-1145 Fix minor regression introduced in WDL 1.1 foundation (broadinstitute#7153) * ID-377 Setup pact for consumer contract testing. Retry (broadinstitute#7147) Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Ivan <[email protected]> * WX-1133 Shorten TES localized blob paths (broadinstitute#7150) * WX-1154 Fix Slack Messaging (broadinstitute#7159) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32) * WX-1160 Fix bad MariaDB version assumptions (broadinstitute#7160) * WX-1126 Upgrade to modern Python, 3.8 not available in package repo anymore (broadinstitute#7164) * WX-1122 Use legacy AppInsights to get better control over logging (broadinstitute#7157) * WX-1137 Replace 4-byte unicode chars in PAPI event descriptions (broadinstitute#7166) * WX-1101 Update library versions to support batch (broadinstitute#7155) Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1195 Update Azure identity libs (broadinstitute#7173) * WX-1114 Initial inclusion of Azure NIO 'fork' (broadinstitute#7168) * WX-1110[risk=low] Added endpoint to fetch failed tasks by root workflow id (broadinstitute#7165) * WX-1210 Added JIRA ID for Cromwhelm auto commit message (broadinstitute#7180) * WX-1179 DRS image build updates, remove self-hosted runners (broadinstitute#7179) * WX-1210-action-fix Use PR title to find JIRA ID for cromwhelm commit message (broadinstitute#7184) * WX-1110-query-fix Corrected Query to pull in attributes outside of executionStatus and backendStatus (broadinstitute#7185) * WX-1179 GCP Batch new base (broadinstitute#7177) Co-authored-by: dspeck <[email protected]> Co-authored-by: jarroyo28 <[email protected]> Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Tristan Garwood <[email protected]> * WM-2100: Update terra-helmfile on publish (broadinstitute#7187) * WM-2100: Correct commit and PR description of terra-helmfile auto update (broadinstitute#7188) * WX-1230 Ignore `invalidate_bad_caches_use_good_local.test` (broadinstitute#7197) * WX-1078 ACR support (broadinstitute#7192) * WX-1179 Enable GCP Batch Integration Tests (broadinstitute#7199) Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-1179 GCP Batch Docs Update (broadinstitute#7196) Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * ID-734 Increase Timeout for DRSHub Communication (broadinstitute#7198) * [WX-1156] internal_path_prefix for TES 4.4 (broadinstitute#7190) * [WM-2184] Remove brackets from Jira ID (broadinstitute#7206) * WX-1153 Azure blob read md5 from metadata for large files (broadinstitute#7204) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1156 Fix internal_path_prefix (broadinstitute#7208) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1256 Temporarily turn off engine hashing for blob files (broadinstitute#7209) Co-authored-by: Adam Nichols <[email protected]> * WX-1173 Reopen filesystem for blob storage outside workspace (broadinstitute#7178) Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-1174 Adjust NIO Copy functionality (broadinstitute#7207) Co-authored-by: Adam Nichols <[email protected]> * [WX-1168] TES Log Paths (broadinstitute#7210) * Develop aws (#34) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * WX-1264 Don't expire an unexpirable filesystem (broadinstitute#7216) * [WX-495] DRS Parallel Downloads (broadinstitute#7214) * WX-1225 Print TES error messages to job logger (broadinstitute#7220) * WX-1217 Workflow completion callback (broadinstitute#7213) Co-authored-by: Chris Llanwarne <[email protected]> * [WM-2199] Changeset for modifying ownership of tables for WORKFLOWS app (broadinstitute#7218) Co-authored-by: Chris Llanwarne <[email protected]> * [WX-1234] Update CHANGELOG.md (broadinstitute#7227) Merging past CI since this is doc only. * WM-2252: Configurable metadata write role (broadinstitute#7225) Co-authored-by: dvoet <[email protected]> * Update cromwell version from 86 to 87 * WX-1282 Update failedJobs Query to use `lo_get` instead of INNER JOIN against pg_largeobject (broadinstitute#7228) * [WX-1234] Update Release Process with Docker Instructions (broadinstitute#7231) Co-authored-by: Adam Nichols <[email protected]> * WM-2294: Allow role-setter action to run on change (broadinstitute#7233) * WM-2296: Callback should supply fully qualified output names (broadinstitute#7234) * [WX-499] DRS Parallel Downloads Follow-up (broadinstitute#7229) * WX-1318 gcp batch: Add GPU driver install (broadinstitute#7235) Co-authored-by: Adam Nichols <[email protected]> * WX-1232 Include useful workflow ids in TES tags (broadinstitute#7221) * Options to publish status only (#36) * add options to publish status only * updated readme.md --------- Co-authored-by: quekx <[email protected]> * WX-1307 Azure E2E test (broadinstitute#7239) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Fix aws unit tests (#39) * checkpoint * fix ecr and batch tests * fix AwsBatchJobSpec.scala --------- Co-authored-by: quekx <[email protected]> * return bucket directly instead of listing and checking it (#38) Co-authored-by: quekx <[email protected]> * WX-1340 GCP Batch: Mount with extra colon issue and multiple zones support (broadinstitute#7240) Co-authored-by: Adam Nichols <[email protected]> * WX-1339 Make `throwExceptionOnExecuteError` false for PAPI aborts (broadinstitute#7245) * WX-1338 Fix cron invocation of E2E test (broadinstitute#7244) * WX-1341 Better logging when a runner stops picking up new workflows (broadinstitute#7246) * Add evaluteOnExit for aws batch retry (#40) Co-authored-by: quekx <[email protected]> * [WX-1260] Acquire sas token from task runner (broadinstitute#7241) Co-authored-by: Adam Nichols <[email protected]> * [WX-1183] Ice Lake (broadinstitute#7252) * WX-1333 Improve logging visibility for load management (broadinstitute#7253) * [WX-1391] Fix Bash Bug (broadinstitute#7326) * [WX-1393] Add Content Length to Curl request (broadinstitute#7328) * [WX-1346] Scalafmt (broadinstitute#7257) * WX-1351 Speed up `Centaur Horicromtal PapiV2 Beta` (broadinstitute#7329) * WX-1351 CI CWL cleanup (broadinstitute#7327) * WX-1351 Remove slow/misbehaving localization test (broadinstitute#7330) * WX-1351 Split out restart tests (broadinstitute#7333) * [WX-1345] Automatic Token Acquisition for TES Config (broadinstitute#7256) * WX-1387 Remove unused case classes (broadinstitute#7334) * [WX-1394] Upgrade Logback Core to address CVE-2023-6378 (broadinstitute#7332) * WX-1351 Another round of CI cleanup (broadinstitute#7336) * [WX-1395] Scalafmt GitHub Action (broadinstitute#7337) * WX-1351 SBT compile speedup (broadinstitute#7339) * WX-1407 Fix contract tests (broadinstitute#7340) Co-authored-by: Ivan <[email protected]> * WX-1409 Java 17 (broadinstitute#7342) * Revert "WX-1409 Java 17 (broadinstitute#7342)" (broadinstitute#7343) * WX-1409 Restore Java 17 (broadinstitute#7342) (broadinstitute#7344) * Revert "WX-1409 Java 17" (broadinstitute#7346) * [WX-1395] Scala formatter Github Action (broadinstitute#7341) Co-authored-by: ScalaFmt Fixer <[email protected]> Co-authored-by: Broad Bot <[email protected]> * Improved tagging support (#37) * efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation * support for tagging instances and volumes used in jobs, some support for spaces in file paths * corrected workflow id in tagging * redirect exit code 137 to retry-with-more-memory routine --------- Co-authored-by: Henrique Ribeiro <[email protected]> * add gpu count (#41) * add gpu count * fix typo --------- Co-authored-by: quekx <[email protected]> Co-authored-by: Henrique Ribeiro <[email protected]> * WX-1385 Reject blob URLs with external SAS tokens as unparsable (broadinstitute#7347) * WX-1411 Require preinstalled `jq` and `curl` for just-in-time SAS feature (broadinstitute#7350) * WX-1396 Cromwell client allow additional properties (broadinstitute#7352) * WM-2428: Include full error context when failing to abort TES jobs (broadinstitute#7354) * WX-1420 Fix GCP Batch label regex restriction (broadinstitute#7355) Co-authored-by: Beibei Chen <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1385 Remove SAS tokens from TES input log printouts (broadinstitute#7358) * [WX-1184] PostgreSQL Docker Image for Local Cromwell (broadinstitute#7172) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1444 Use MySQL LTS in DBMS tests (broadinstitute#7360) * WX-1449 Add `latest` Docker tag (broadinstitute#7362) * WX-767 Upgrade Cloud SDK to 461.0.0 (broadinstitute#7361) * WX-1443 Adopt `gcloud storage` for localization only (broadinstitute#7359) * WX-964 suffix() (broadinstitute#7363) * WX-1417 New database role strategy (broadinstitute#7366) * [WM-2291] Callback API contract tests between Cromwell and CBAS (broadinstitute#7251) Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-1445 Update docker image regex to handle python:3 correctly (broadinstitute#7367) * WX-1252 Runtime attributes cleanup – CWL runtime attributes (broadinstitute#7370) * WX-1485 Upgrade postgres lib (broadinstitute#7371) * WM-2454: Private GitHub support on describe api (broadinstitute#7365) * WX-1252 Runtime attributes cleanup – CWL runtime environment (broadinstitute#7369) * WX-1462 POSIX-flavored sub() (broadinstitute#7374) * [WX-963] Unzip Engine Function (broadinstitute#7368) * WX-1489 Hopefully reduce CI flakiness by modernizing deadlock test (broadinstitute#7376) * [WX-965] quote() and squote() engine functions. (broadinstitute#7375) Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * [WX-1317] Remove Akka 'server' header from all HTTP responses (broadinstitute#7379) * WM-2461] Add support for running private workflows on Azure (broadinstitute#7373) * WX-1252 Per-backend runtime attributes (broadinstitute#7380) * ID-347 Remove Martha References (broadinstitute#7384) * WX-1461 Remove `womtool upgrade` command (broadinstitute#7382) * WX-1488 Supply cloud platform when making DRS requests (broadinstitute#7381) * WX-757 Fix workflow stuck in aborting after WDL type error (broadinstitute#7385) * [WX-1506] Add jobLogger output for subworkflow running (with parent and subworkflow ID) (broadinstitute#7388) Co-authored-by: jlester-msft <[email protected]> * WX-757 Fix `stdout`, `stderr` in workflow body causing crashes (broadinstitute#7386) * WX-1519 Stop leaking DRS Localizer images on every CI run (broadinstitute#7390) * [WX-1460] WDL 1.1 Struct Literal Parsing (broadinstitute#7391) Co-authored-by: Janet Gainer-Dewar <[email protected]> * [WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows (broadinstitute#7392) * WX-1542 Fix workflow cancellation (broadinstitute#7398) * WX-1550 Don't pointlessly continue running integration tests (broadinstitute#7400) * WX-1557 Fix default values from Configuring.md and Scaling.md (broadinstitute#7393) Co-authored-by: Adam Nichols <[email protected]> * WX-1557 Fix actor-factory in Batch101.md (broadinstitute#7377) Co-authored-by: Adam Nichols <[email protected]> * WX-1557 Add more tests to the GCP Batch backend (broadinstitute#7394) Co-authored-by: Adam Nichols <[email protected]> * [WX-1531] Struct Literal Type Checking (broadinstitute#7402) * [WX-1468] Implement `returnCodes` runtime attribute (broadinstitute#7389) Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> * [WM-2555] Cromwell -> ECM contract test (broadinstitute#7405) * [WX-1568] Bump Akka max-response-reason-length limit (broadinstitute#7406) * WX-1530 Strip pesky URL bits when creating local paths for HTTP inputs (broadinstitute#7404) * WX-1538 Add more tests to the GCP Batch backend (broadinstitute#7410) * [WX-1605] Fix Codecov Report Uploads (broadinstitute#7418) * [WX-1410] Sanitize 4 byte UTF-8 characters before inserting into METADATA_ENTRY (broadinstitute#7414) * WX-1566 Special Docker build for debugging (broadinstitute#7417) Run `sbt -Dproject.isDebug=true server/docker` * [WX-1448] Add verbose logging and timeout for getm (broadinstitute#7416) Co-authored-by: Adam Nichols <[email protected]> * WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (broadinstitute#7419) * WX-1566 Fix RTD so this ticket's docs get published (broadinstitute#7420) * fix womFileType coercion (#47) Co-authored-by: quekx <[email protected]> * PR #44: Multi-aserisk globbing by @kevinRenaersBio. Reapplied selection of changes on aws_develop branch * PR #45 reapplied to aws_develop: support for sharedMemory. skipped EFS fixes for now while waiting for feedback * upstream PR broadinstitute#7488 : fix compilation error due to swagger2markup * Fix issue with caching on EFS by replacing touch with existOrThrow routine * make sure that md5 sibling file of EFS data is newer than main file to be valid * revise sharedmemory attribute to take reguluar GB/MB specification * updated AWS README file --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Saloni Shah <[email protected]> Co-authored-by: Brian Reilly <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Katrina P <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Justin Variath Thomas <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Trevyn Langsford <[email protected]> Co-authored-by: Miguel Covarrubias <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Thomas Wiseman <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Stephen Fleming <[email protected]> Co-authored-by: Henrique Ribeiro <[email protected]> Co-authored-by: mspector <[email protected]> Co-authored-by: Tristan Garwood <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: dspeck1 <[email protected]> Co-authored-by: jarroyo28 <[email protected]> Co-authored-by: Jacob Jennings <[email protected]> Co-authored-by: Dillon Scott <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: dvoet <[email protected]> Co-authored-by: xquek <[email protected]> Co-authored-by: quekx <[email protected]> Co-authored-by: ScalaFmt Fixer <[email protected]> Co-authored-by: Broad Bot <[email protected]> Co-authored-by: Beibei Chen <[email protected]> Co-authored-by: Maria Yazykova <[email protected]> Co-authored-by: rsaperst <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: jlester-msft <[email protected]> Co-authored-by: Alexis Hernandez <[email protected]> Co-authored-by: dkj <[email protected]> Co-authored-by: Ryan Saperstein <[email protected]> Co-authored-by: Alexis Hernandez <[email protected]>

salonishah11 added 11 commits March 14, 2024 17:58

wip

a6c1092

extract error message

9467e26

comment

fb0d6bf

attempting to fix deduplicate error

9a26105

test new WSM client and updated jakarta-annotation version

2b105ac

add jersey-hk2 back with updated version

c5495f9

scalafmt

60ca5ed

revert WSM client upgrade

29a412a

remove pinned versions from previous PR for WSM client

1372959

call ECM api via HTTP request

dcbc763

cleanup, comments and tests

a9861d7

salonishah11 requested a review from a team as a code owner March 27, 2024 21:16

Merge branch 'develop' into sps_ecm_http_call

93f98ea

cjllanwarne reviewed Mar 28, 2024

View reviewed changes

salonishah11 added 3 commits March 28, 2024 15:23

PR feedback

3d3935e

fix tests; use table based testing

4701188

Merge branch 'develop' into sps_ecm_http_call

ab3a534

cjllanwarne approved these changes Mar 29, 2024

View reviewed changes

jgainerdewar reviewed Mar 29, 2024

View reviewed changes

salonishah11 added 2 commits March 29, 2024 17:56

PR feedback

ea97924

scala formatter

dbd6efd

jgainerdewar reviewed Apr 1, 2024

View reviewed changes

remove reference to ECM url

2d7dd21

salonishah11 requested review from jgainerdewar and cjllanwarne April 1, 2024 17:44

jgainerdewar approved these changes Apr 1, 2024

View reviewed changes

cjllanwarne reviewed Apr 2, 2024

View reviewed changes

cjllanwarne approved these changes Apr 2, 2024

View reviewed changes

PR feedback

05b1e15

salonishah11 merged commit d7def8d into develop Apr 2, 2024
33 of 34 checks passed

salonishah11 deleted the sps_ecm_http_call branch April 2, 2024 17:00

aednichols mentioned this pull request May 13, 2024

CROM-4572 Allow describing wdl with zipped imports. #7133

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows #7392

[WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows #7392

salonishah11 commented Mar 27, 2024 •

edited

Loading

cjllanwarne Mar 28, 2024

salonishah11 Mar 28, 2024

cjllanwarne Mar 28, 2024

cjllanwarne Mar 28, 2024

salonishah11 Mar 28, 2024

salonishah11 Mar 29, 2024

cjllanwarne Mar 28, 2024

salonishah11 Mar 28, 2024

jgainerdewar Mar 29, 2024

salonishah11 Mar 29, 2024

jgainerdewar Apr 1, 2024

cjllanwarne Mar 28, 2024

cjllanwarne left a comment

cjllanwarne Mar 29, 2024

salonishah11 Mar 29, 2024

jgainerdewar Mar 29, 2024

salonishah11 Mar 29, 2024 •

edited

Loading

salonishah11 Mar 29, 2024

jgainerdewar Apr 1, 2024

jgainerdewar Mar 29, 2024

jgainerdewar Apr 1, 2024

salonishah11 Apr 1, 2024

cjllanwarne left a comment

cjllanwarne Apr 2, 2024

cjllanwarne Apr 2, 2024

salonishah11 Apr 2, 2024 •

edited

Loading

cjllanwarne Apr 2, 2024

salonishah11 Apr 2, 2024 •

edited

Loading

cjllanwarne Apr 2, 2024

salonishah11 Apr 2, 2024

salonishah11 commented Apr 2, 2024


		lazy val enabled: Boolean = serviceConfig.getBoolean("enabled")

		private lazy val ecmConfigOpt: Option[EcmConfig] = EcmConfig.apply(serviceConfig)

		def responseEntityToFutureStr(responseEntity: ResponseEntity): Future[String] =
		responseEntity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String)

[WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows #7392

[WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows #7392

Conversation

salonishah11 commented Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjllanwarne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjllanwarne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salonishah11 commented Apr 2, 2024

salonishah11 commented Mar 27, 2024 •

edited

Loading

salonishah11 Mar 29, 2024 •

edited

Loading

salonishah11 Apr 2, 2024 •

edited

Loading

salonishah11 Apr 2, 2024 •

edited

Loading