Optional value was not set and no 'default' attribute was provided

[DavyCats]

It looks like cromwell doesn't properly handle optional variables in tasks. If they are undefined this error occurs: "Optional value was not set and no 'default' attribute was provided". Setting a default (either in the input block or like so: ~{default="" optionalValue}) will cause the workflow to run, but this seems to me to go entirely against the reason for having an optional datatype to begin with.

It also looks like setting an optional with default to null in the input JSON is ignored, which is in conflict with the SPECs here.

I'm getting this error when using WDL version 1.0 (draft-3). The error occurs in both cromwell version 33 and 34.

Full stacktrace:

2018-07-24 09:54:27,543 cromwell-system-akka.dispatchers.backend-dispatcher-53 ERROR - DispatchedConfigAsyncJobExecutionActor [UUID(514f031f)AlignStar.star:NA:1]: Error attempting to Execute
java.lang.Exception: Failed command instantiation
        at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:536)
        at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:471)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:265)
        at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:264)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
        at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
        at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:599)
        at scala.util.Try$.apply(Try.scala:209)
        at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:599)
        at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:599)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:912)
        at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:904)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:208)
        at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
        at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
        at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
        at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
        at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
        at akka.actor.Actor.aroundReceive(Actor.scala:514)
        at akka.actor.Actor.aroundReceive$(Actor.scala:512)
        at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:208)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: common.exception.AggregatedMessageException: Error(s):
Optional value was not set and no 'default' attribute was provided
Optional value was not set and no 'default' attribute was provided
        at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:60)
        at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:56)
        at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:534)
        ... 39 common frames omitted

[aednichols]

@DavyCats thank you for the report.

Would you mind posting the WDL that you're using to induce the error? (Or if it contains sensitive information, a minimal repro case.)

[DavyCats]

@aednichols It happens in two workflows that I have tried, both can be found here: https://github.com/biowdl/aligning/tree/BIOWDL-61 (align-bwamem.wdl and align-star.wdl).

[cjllanwarne]

Thanks @DavyCats I think that this is really two issues in one:

• How to interpolate a String? into a command section.
  • I tried to find something in the SPEC to back up how Cromwell is currently working, but I couldn't find a definitive statement on this either way!...
  • Since "optionals require a default" is how Cromwell has always worked, I'm reluctant to change it unilaterally without thinking through what the right answer should be.
    • If you can point me to a definitive SPEC statement on this, I'll change Cromwell to match the spec as a bug-fix
    • Otherwise I'd suggest making a clarification PR against the SPEC to make your case and let the discussion play out in the community.
• Cromwell failing to override set optional values with nulls
  • This does indeed look like a Cromwell bug - thanks for reporting it!
  • I think this matches what @orodeh reported here: Override an optional with null #3819

[DavyCats]

@cjllanwarne I cannot find an explicit statement regarding this in the SPECs either, but there are some sections which suggest that no default is required for optional values. Most importantly, here the statement is made that a placeholder for an optional input can be instantiated in two ways: as a value, or nothing (no defaults are defined in the example).

For draft-2 workflows this was handled as such, if an optional input was not assigned a value, its placeholder was (effectively) replaced by an empty string.

[cjllanwarne]

@DavyCats thanks for checking that - I think you're right, and we should definitely fix this in our 1.0 support. Thanks!