Localisation of input 'File' that is a folder

[EvanTheB]

I have a task with an input 'File' (WDL concept) that is actually a folder. I did not know what would happen, as I could not find explanation of the concept in either wdl or cromwell docs.

For the 'File's that are actually files, cromwell reports Localisation via hard-link has failed ... invalid cross device link as expected (different disks), and then seems to successfully soft-link (as expected). For the folder, the same hard link error appears, but then there is no soft link error, and the folder is (recursively) copied. The folder hard link would also fail due to folders not being hard-linkable. But the soft-link should probably succeed.

I am running an SGE backend, with no modification of the 'localisation' settings https://cromwell.readthedocs.io/en/latest/backends/HPC/#shared-filesystem

What is the expected behaviour? Should I avoid using folders for WDL 'File's? Any advice would be appreciated.

Thanks

[EvanTheB]

openwdl/wdl#173

[Adrien-Evo]

I'm having the same kind of issues with cromwell 32. Hard link are failing but symbolic link are working for the first task.
Then the task fail due to, again hard link not permitted , then due to soft link not permitted due to too many level of symbolic links.

[geoffjentry]

@EvanTheB Yeah, File in WDL is not supposed to reference a directory (as you noted in your followup, that's a whole separate topic). It manages to work-ish by happenstance on shared filesystem backends, as you discovered - we came across that a little while back.

I'd call it one of those things where if it Works For You then knock yourself out but keep in mind that it's not supported behavior and if your WDL relies on it they'll not be portable in cloud environments.

[EvanTheB]

OK thanks for the clarification @geoffjentry , I am happy, I guess it is up to you if this should be in the docs or be a warning or have different behaviour.

[geoffjentry]

@EvanTheB It's a good point that we should slip a warning in the docs somewhere as this isn't the first time it has come up (or, alternatively, push the OpenWDL group to ratify a directory type!)

@ruchim @cjllanwarne Any thoughts on if we should put this somewhere and if so where? I could see it being part of the WDL docs maybe? Now that I say that I think the Cromwell docs are more appropriate but I have no idea which section would be most appropriate