Add a way to preserve or set file permissions on generated content

[mlorbetske]

Currently, all content is created with the system default permissions since the original permissions aren't inspected when being read from the template source.

We could either:

• Find a way to inspect the source permissions & expose that through the IFile abstraction & apply them (P/Invoke or a managed wrapper from somewhere) - this would be a breaking change

• Add a post-action to set permissions on generated files in the case that the system being run on supports chmod

Thoughts?
@prafullbhosale @seancpeters @mhutch @sayedihashimi

[rmunn]

Long-term solution (the right way to go)

NuGet packages are .zip files, and the .zip file format includes 4 bytes (32 bits) for "external file attributes". In theory, if the .zip file was created from a Unix-based system with the standard Unix zip tools (from the Info-Zip project), it will include the file's original Unix permissions in the upper 16 bits of that 32-bit field. This is not well-documented anywhere, but the code for the standard Unix zip utilities is freely available, and the way Unix attributes are stored in the .zip file is visible in that code. Links to info sources:

• Documentation of the "external file attributes" field in the ZIP file format
• Git clone of the Info-Zip source
• Source code documenting how the external file attributes field is used
• Where those attributes are actually written to the zip file

See also https://stackoverflow.com/a/6297838/2314532 for more discussion of the zip file format's "external file attribute" field, including its use in the standard Python library's zipfile module.

Now, in practice, since NuGet was originally developed on (and for) Windows AFAICT, I rather doubt that .zip files created by NuGet are going to preserve the Unix file attributes in the same way that the Info-Zip tools do. But the best solution, ultimately, would be to make sure that that happens. This would probably also involve including an ExternalFileAttributes field (an int) in https://msdn.microsoft.com/en-us/library/system.io.compression.ziparchiveentry(v=vs.110).aspx, and ensuring that other Zip-handling tools in the .Net ecosystem learn about this field, and use it when they're running on a Unix-like system.

Short-term solution (the right-now fix)

All of the above doesn't do a thing to solve the immediate problem, because it will take years for the right fix (make .Net aware of this field in Zip files, and use it properly) to make its way through all the bits of code that need it. So the right now fix, I think, is something like the post-action suggestion made by @mlorbetske. HOWEVER, in the interest of keeping things as declarative as possible*, I suggest a specific post-action that goes along the lines of: "This is the list of filenames that should have their execute bit set if deploying on Unix." Then the template author doesn't have to write the chmod calls himself, he just has to make sure that any files that are supposed to be executable (e.g., build.sh scripts) are included in the list (wherever it may end up being) that controls that specific post-unpacking step.

* Declarative style always gives you more flexibility than imperative style, because with a declarative style, external tools can inspect the package and know what it will produce. Whereas if you have imperative steps in your packaging system, external tools cannot know what those imperative steps are going to do and whether that's a security hole.

[rmunn]

https://github.com/dotnet/corefx/issues/10223 and https://github.com/dotnet/corefx/issues/15516 (and https://github.com/dotnet/corefx/issues/14853 as well) are relevant to the ZIP file permissions issue.

[rmunn]

#1042 is a good temporary fix for this, though I'd prefer for this issue to be left open until the long-term solution is correctly implemented. But I've gone ahead and documented how to use #1042 at https://github.com/dotnet/templating/wiki/Post-Action-Registry#change-file-permissions.

[andycmaj]

not sure the chmod post action is actually working...
looks like it isn't locating the chmod command or maybe doesn't have the cwd set correctly?

error

 /Workspace/ims/ApplicationBlocks    master  dotnet new solution                                                                                                                                                                                    ✔  10:57:52
The template "our standard solution foundation" was created successfully.

Processing post-creation actions...
The files /Workspace/ims/ApplicationBlocks/chmod and /Workspace/ims/ApplicationBlocks/+x do not exist.
Unable to apply permissions +x to "build.sh".
Post action failed.
Description: Make scripts executable
Manual instructions: Run 'chmod +x *.sh'

 /Workspace/ims/ApplicationBlocks    master ?  ls                                                                                                                                                                                                   ✔  10:57:58
Analyzers.ruleset               NuGet.config                    build.cake                      global.json                     stylecop.json ApplicationBlocks.sln README.md                       build.sh                        omnisharp.json                  tools.packages.config

my template:

{
  "$schema": "http://json.schemastore.org/template",
  "author": "Andy Cunningham",
  "classifications": ["dotnet", "cake"],
  "identity": "NexxusNew.Solution",
  "name": "our standard solution foundation",
  "shortName": "solution",
  "sourceName": "NexxusNew",
  "preferNameDirectory": true,
  "postActions": [
    {
      "condition": "(OS != \"Windows_NT\")",
      "description": "Make scripts executable",
      "manualInstructions": [{ "text": "Run 'chmod +x *.sh'" }],
      "actionId": "cb9a6cf3-4f5c-4860-b9d2-03a574959774",
      "args": {
        "+x": "build.sh"
      },
      "continueOnError": true
    }
  ]
}

[rmunn]

@andycmaj I've seen this work on Linux and gotten reports of it working on OS X. The post-action code simply calls "chmod" via System.Diagnostics.Process.Start, passing UseShellExecute = true so that the chmod command will be searched for on the PATH. In your setup, the PATH is apparently not being searched, because it complains about not finding a /Workspace/ims/ApplicationBlocks/chmod file.

And the complaint about not finding /Workspace/ims/ApplicationBlocks/+x is even more interesting: whatever implementation of System.Diagnostics.Process.Start you have there, it's completely failing to parse a standard command line like "chmod +x filename" and trying to run that as three separate commands: chmod, +x, and filename. Strange. What's your setup? Are you running in a chroot, and is the chmod command accessible in that chroot?

[TheAngryByrd]

@rmunn I'm encountering that bug with dotnet 2.1.300 so now MiniScaffold is broke again

The files /private/var/folders/14/mp4bnvkn3fq5mqcgqscm5c040000gn/T/7f3575aecf3e420794111d7bd0dd9e66/fsharp-data-sample/chmod, /private/var/folders/14/mp4bnvkn3fq5mqcgqscm5c040000gn/T/7f3575aecf3e420794111d7bd0dd9e66/fsharp-data-sample/+x, and /private/var/folders/14/mp4bnvkn3fq5mqcgqscm5c040000gn/T/7f3575aecf3e420794111d7bd0dd9e66/fsharp-data-sample/*.sh do not exist.
Unable to apply permissions +x to "*.sh".
Post action failed.
Description: Make scripts executable
Manual instructions: Run 'chmod +x *.sh'

😿 😿 😿 😿 😿

[rmunn]

@TheAngryByrd -

"The files /long/path/chmod, /long/path/+x, and /long/path/*.sh don't exist"? Oh dear. It should be looking for chmod in the PATH, not in the local directory. Sounds like someone broke the post-action chmod script in 2.1.300 and didn't notice because they were on Windows. This will probably require a bug report to a .Net Core repo somewhere...

[rmunn]

Can confirm that it fails on Linux as well:

Processing post-creation actions...
xdg-open: unexpected argument '+x'
Try 'xdg-open --help' for more information.
Unable to apply permissions +x to "*.sh".
Post action failed.
Description: Make scripts executable
Manual instructions: Run 'chmod +x *.sh'

xdg-open is the Linux equivalent of the "Start whatever program can deal with this file" command (e.g., the start command in Windows. Apparently someone changed the logic of System.Diagnostics.Process.Start to ignore the UseShellExecute parameter at some point, so that a command that was intended to be run by the Linux shell is instead getting passed as parameters to xdg-open, e.g. it looks like Linux is seeing xdg-open chmod +x *.sh as the command being attempted.

Sigh... and this is in the released version, too, which means waiting a LONG time for the bug fix...

[rmunn]

It looks like dotnet/corefx@75f34a5 is what broke the chmod post-action. That's when the UseShellExecute=true logic was changed from "use the shell" to "use xdg-open", which I don't understand the rationale for. xdg-open is not the shell, and anything written with UseShellExecute in mind will break (as we've seen) if it's not executed by sh. Why was that change made?

[rmunn]

Looks like https://github.com/dotnet/corefx/issues/24704 is relevant here. They changed the meaning of UseShellExecute in order to be cross-platform compatible, which broke a lot of Linux-based stuff (because Linux devs think that UseShellExecute means "use /bin/sh to execute this", and that's no longer what it means).

[rmunn]

#1555 should fix the post-build chmod action, though it will probably remain broken in dotnet 2.1.300 (which was released before the bug was fixed) so template authors will have to issue special instructions to people with that version.

I wonder if there's any way, inside the template.json file, to detect the current dotnet version and print something to the output iff the version number is 2.1.300? Eh, the manualInstructions section of the chmod post-action is probably good enough for that purpose.

[rmunn]

BTW, looks like the fix did not make it into dotnet 2.1.301 (tested on Linux Mint 18, which should be equivalent to Ubuntu 16.04).

[mlorbetske]

Unfortunately the earliest this change could show up in CLI would be 2.1.400, but we're likely too late to make it in there.

[rmunn]

Can confirm that this change did NOT make it into the 2.1.401 preview:

$ dotnet --version
2.1.401-preview-009202
$ dotnet new -i MiniScaffold
  Restoring packages for /home/rmunn/.templateengine/dotnetcli/v2.1.401-preview-009202/scratch/restore.csproj...
  Installing MiniScaffold 0.9.5.
  Generating MSBuild file /home/rmunn/.templateengine/dotnetcli/v2.1.401-preview-009202/scratch/obj/restore.csproj.nuget.g.props.
  Generating MSBuild file /home/rmunn/.templateengine/dotnetcli/v2.1.401-preview-009202/scratch/obj/restore.csproj.nuget.g.targets.
  Restore completed in 253.46 ms for /home/rmunn/.templateengine/dotnetcli/v2.1.401-preview-009202/scratch/restore.csproj.
$ dotnet new mini-scaffold -lang F#
The template "MiniScaffold" was created successfully.

Processing post-creation actions...
xdg-open: unexpected argument '+x'
Try 'xdg-open --help' for more information.
Unable to apply permissions +x to "*.sh".
Post action failed.
Description: Make scripts executable
Manual instructions: Run 'chmod +x *.sh'
$

😞

EDIT: I'm currently unable to find all the necessary packages to test the SDK 2.2 preview: https://github.com/dotnet/core-sdk/blob/master/README.md#installers-and-binaries has 2.2.100-preview1-009298-1, but https://github.com/dotnet/core-setup#daily-builds doesn't yet have any 2.2 builds listed. My attempts to guess the URL for those builds let me to find https://dotnetcli.blob.core.windows.net/dotnet/Runtime/release/2.2/dotnet-runtime-deps-latest-x64.deb, but that's 2.2.0~preview-26725-02. And trying https://dotnetcli.blob.core.windows.net/dotnet/Runtime/release/2.2.100/dotnet-runtime-deps-latest-x64.deb and https://dotnetcli.blob.core.windows.net/dotnet/Runtime/release/2.2.1xx/dotnet-runtime-deps-latest-x64.deb failed. I'll update this issue with the results of testing against the .NET Core 2.2 SDK as soon as I can find the right packages to install, but I can't do so yet.

[mlorbetske]

That’s correct unfortunately, 2.1.4xx is not a “product construction” build, so it didn’t pick up any newer bits than 2.1.3xx. 2.2.xxx is a product construction build & will have the changes. I spoke with one of the PMs today & we’re going to try to see if there’s another 2.1.xxx release we can manually get this in to.

[donJoseLuis]

This issue was last touched some years ago. We are working on a new delivery road map. Please reopen if this is something we want & we'll properly assess its' priority compared to other work aimed at improving the overall templating UX.

[fdw]

I think this issue is still unsolved in .NET 8, right?