Skip to content

Commit

Permalink
Address additional feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
swaroop-sridhar committed Feb 14, 2020
1 parent 11912ba commit a1d1f65
Showing 1 changed file with 39 additions and 32 deletions.
71 changes: 39 additions & 32 deletions accepted/single-file/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Here's the overall experience for publishing a HelloWorld single-file app. The n
* Published files: `HelloWorld.exe`, `HelloWorld.pdb`
* Single-file publish Windows: `dotnet publish -r win-x64 /p:PublishSingleFile=true`
* Published files: `HelloWorld.exe`, `HelloWorld.pdb`, `coreclr.dll`, `clrjit.dll`, `clrcompression.dll`, `mscordaccore.dll`
* Single-file publish Windows with Extraction: `dotnet publish -r win-x64 /p:PublishSingleFile=true /p:IncludeContentInSingleFile=true`
* Single-file publish Windows with Extraction: `dotnet publish -r win-x64 /p:PublishSingleFile=true /p:IncludeNativeLibrariesInSingleFile=true`
* Published files: `HelloWorld.exe`, `HelloWorld.pdb`

## Build System Interface
Expand Down Expand Up @@ -72,20 +72,23 @@ All other files, including platform-specific native binaries and symbol files, a

### Optional Settings

| Property | Behavior when set to `true` |
| ---------------------------- | ------------------------------------------------------------ |
| `IncludeContentInSingleFile` | All published files (except symbol files) are embedded within the single-file bundle. The files that cannot be processed directly from the bundle are extracted out to disk at startup. |
| `IncludeSymbolsInSingleFile` | The symbol files (IL `.pdb` file, and the native `.ni.pdb` / `app.guid.map` files generated by ready-to-run compiler) are embedded within the symbol file, and extracted out to disk on startup. |
The following settings can be used to package additional files into the single-file app. However, when using these options, the files that cannot be processed directly from the bundle will be extracted out to disk during startup.

When including content in single files, certain files can be explicitly excluded from being embedded in the single-file by setting following `ExcludeFromSingleFile` meta-data element. For example, to place some files in the publish directory but not bundle them in the single-file:
| Property | Behavior when set to `true` |
| ------------------------------------ | ------------------------------------------------------------ |
| `IncludeNativeLibrariesInSingleFile` | Bundle published native binaries into the single-file app. |
| `IncludeAllContentInSingleFile` | Bundle all published files (except symbol files) into single-file app. This option is proposed to replicate the .net core 3 version of single-file apps. |
| `IncludeSymbolsInSingleFile` | Bundle symbol files (IL `.pdb` file, and the native `.ni.pdb` / `app.guid.map` files generated by ready-to-run compiler) into the single file app. |

Certain files can be explicitly excluded from being embedded in the single-file by setting following `ExcludeFromSingleFile` meta-data element. For example, to place some files in the publish directory but not bundle them in the single-file:

```xml
<PropertyGroup>
<PublishSingleFile>true</PublishSingleFile>
<IncludeContentInSingleFile>true</IncludeContentInSingleFile>
</PropertyGroup>
<ItemGroup>
<Content Update="*.xml">
<Content Update="*-exclude.dll">
<CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
<ExcludeFromSingleFile>true</ExcludeFromSingleFile>
</Content>
Expand Down Expand Up @@ -124,26 +127,26 @@ On Startup, the [host components](https://github.com/dotnet/core-setup/blob/mast

* Implements a bundle-probe function, which will be used by the runtime to probe contents of the bundle when resolving assemblies.

* Bundle probe has the following signature:
```C++
/// <summary>
/// <param name="path"> Relative-path to the file being probed. </param>
/// <param name="size"> Out-param: size of the file, if found. </param>
/// <param name="path"> Out-param: offset within the bundle, if found</param>
/// <returns> true if the requested file is found in the bundle,
/// false otherwise. </returns>
/// </summary>
bool bundle_probe(const char *path, int64_t *size, int64_t *offset);
```

* The `bundle_probe` function pointer is passed to the runtime (encoded as a string) through a property named `BUNDLE_PROBE`.

```C++
/// <summary>
/// <param name="path"> Relative-path to the file being probed. </param>
/// <param name="size"> Out-param: size of the file, if found. </param>
/// <param name="path"> Out-param: offset within the bundle, if found</param>
/// <returns> true if the requested file is found in the bundle,
/// false otherwise. </returns>
/// </summary>
bool bundle_probe(const char *path, int64_t *size, int64_t *offset);
```
* When probing for assemblies, the the [assembly resolution logic](https://github.com/dotnet/runtime/blob/4f9ae42d861fcb4be2fcd5d3d55d5f227d30e723/docs/design/features/assembly-conflict-resolution.md#probe-ordering) will treat bundled assemblies similar to assemblies in the app directory. The probe ordering will be in the order: servicing location, shared store, framework directory(s) from higher to lower, the *single-file bundle*, app directory, additional locations specified in runtimeconfig.dev.json file.

* The `bundle_probe` function pointer is passed to the runtime (encoded as a string) through a property named `BUNDLE_PROBE`.
* The assemblies on disk are listed in `TRUSTED_PLATFORM_ASSEMBLIES` using absolute paths. The assemblies to be loaded from the single-file bundle will be listed in `TRUSTED_PLATFORM_ASSEMBLIES` using relative paths (as a distinguishing convention).

* The contents of the bundle are not included in `TRUSTED_PLATFORM_ASSEMBLIES`. Any additional assemblies on disk (ex: due to `additional-deps` /`AdditionalProbingPaths`) are included in `TRUSTED_PLATFORM_ASSEMBLIES`.

* The runtime probes the satellite assemblies in the bundle before looking in `PLATFORM_RESOURCE_ROOTS`. There is no change anticipated to `PLATFORM_RESOURCE_ROOTS` list.

* The extraction directory (if any) will be added to `NATIVE_DLL_SEARCH_DIRECTORIES` as the first destination to probe for native binaries.
* Similarly, the paths to directories containing satellite assemblies within the bundle are listed as relative paths in `PLATFORM_RESOURCE_ROOTS`.

* The extraction directory (if any) will be added to `NATIVE_DLL_SEARCH_DIRECTORIES` as the first destination to probe for native binaries.

### Host Builds

Expand Down Expand Up @@ -179,7 +182,7 @@ Therefore, we propose that targeting `win7-*` should not be supported when publi

### Dependency Resolution

In order to resolve assemblies, runtime will first query the bundle probe for it followed by exiting search in `TRUSTED_PLATFORM_ASSEMBLIES` followed by any existing additional probing.
The Host includes bundled assemblies as relative paths in `TRUSTED_PLATFORM_ASSEMBLIES`. When such a path is encountered, the runtime probes the bundle using the `bundle_probe` callback to obtain the location of the assembly within the single-file bundle. It then proceeds to memory-map and load the assembly as described [below](#PEImage-loader).

### PEImage loader

Expand All @@ -189,20 +192,18 @@ In order to resolve assemblies, runtime will first query the bundle probe for it
* ReadyToRun Assemblies
* On Linux, ReadyToRun assemblies are loaded directly from bundle. The various sections in the PE file are mapped at appropriate addresses, and offsets are fixed up.
* On MAC, ReadyToRun assemblies are loaded similar to Linux. However, Mojave hardened runtime doesn't allow executable mappings of a file. Therefore, the contents of an assembly are read from the bundle into pre-allocated executable memory.
* On Windows, due to certain limitations in memory mapping routines described below, ReadyToRun assemblies are loaded by memory mapping the file and copying sections to appropriate offsets.
* On Windows, due to [certain limitations](#Windows-Limitations) in memory mapping routines described below, ReadyToRun assemblies are loaded by memory mapping the file and copying sections to appropriate offsets.

* ReadyToRun [Composite](https://github.com/dotnet/runtime/blob/master/docs/design/features/readytorun-composite-format-design.md) Assemblies are expected to be loaded similar to ReadyToRun assemblies.

#### Windows Limitations
The Windows mapping routines have the following limitations:
* [CreateFileMapping](https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga) has no option to create a mapping for a part of the file (no offset argument).
This means that:
* We cannot use the (SEC_IMAGE) attribute to perform automatic section-wise loading (circumventing alignment requirements) of bundled assemblies directly.
* Instead we need to map each section independently.
Therefore, we cannot use the (SEC_IMAGE) attribute to perform automatic section-wise loading (circumventing alignment requirements) of bundled assemblies directly. Instead we need to map each section independently.
* [MapViewOfFile]( https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffileex) can only map parts of a file aligned at [memory allocation granularity]( https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/ns-sysinfoapi-system_info), which is 64KB.
This means that each section within the assemblies should be aligned at 64KB – which is not guaranteed by the crossgen compiler.
This means that each section within the assemblies should be aligned at 64KB – which is not guaranteed by the crossgen compiler. Therefore, without substantial changes to the runtime and the ready-to-run file format, we cannot load ready-to-run files using direct file mappings.

We therefore map the assembly and perform an in-memory copy of the sections to appropriate offsets. In the long term, the solution to the mapping problem would involve considerations such as:
We therefore map ReadyToRun assemblies as-is, and subsequently perform an in-memory copy of the sections to appropriate offsets. In the long term, the solution to the mapping problem would involve considerations such as:
* Compile all assemblies in a version bubble into one PE assembly, with a few aligned sections.
* Embed the big composite assembly into the host with proper alignment, so that the single-exe bundle can be loaded without copies at run time.

Expand All @@ -213,19 +214,25 @@ We therefore map the assembly and perform an in-memory copy of the sections to a
There are a few options to consider for the `Assembly.Location` property of a bundled assembly:

* A fixed literal (ex: `null`) indicating that no actual location is available.
* Throw an `AssemblyLoadedFromBundle` exception
* The empty string, similar to assemblies loaded from [byte-array](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assembly.location?view=netcore-3.1#property-value),
* The simple name of the assembly (with no path).
* The path of the assembly as if it were not to be packaged into the single-file.
* A special UNC notation such as `<bundle-path>/:/asm.dll` to denote files that come from the bundle.
* A configurable selection of the above, etc.

Proposed solution is for `Assembly.Location` to return `null` for bundled assemblies, which is the default behavior for assemblies loaded from memory.
Proposed solution is for `Assembly.Location` to return the empty-string for bundled assemblies, which is the default behavior for assemblies loaded from memory.

Most of the app development can be agnostic to whether the app is published as single-file or not. However, the parts of the app that deal with physical locations of files need to be aware of the single-file packaging.

#### `AppContext.BaseDirectory`

`AppContext.BaseDirectory` will be the directory where the AppHost (the single-file bundle itself) resides.

This is a change from [.net core 3 semantics](design_3_0.md#API-Impact) for single-file apps, where `AppContext.BaseDirectory` returned the directory where the bundle-contents were extracted. In .net 5, extraction is not performed by default. Therefore, the details about extraction directory are not exposed through the `AppContext.BaseDirectory` API.

However, if files not handled by the runtime are bundled into the single-file (due to `IncludeAllContentInSingleFile` property which emulates .net core 3 bundling behavior) the app will need a (new) API to locate the extraction directory.

## Handling Content

The app may want to access certain embedded content for reading, rather than loading via host/runtime. For example: bundled payload/data files. In this case, the recommended strategy is to embed the content files within appropriate managed assemblies as resources, and access them through [resource handling APIs](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assembly.getmanifestresourceinfo?view=netcore-3.1).
Expand Down

0 comments on commit a1d1f65

Please sign in to comment.