diff --git a/accepted/single-file/bundler.md b/accepted/single-file/bundler.md index 07350e952..48f4187bc 100644 --- a/accepted/single-file/bundler.md +++ b/accepted/single-file/bundler.md @@ -1,6 +1,6 @@ # The Single-file Bundler -### Requirements +## Requirements Ideally, the bundler should be: @@ -9,46 +9,45 @@ Ideally, the bundler should be: * Deterministic (generate the exact same single-file on multiple runs) * Amenable to post-processing (ex: signing tools) -### Bundle Transformation +## Bundle Layout -Given a .net core app published with respect to a specific runtime, the bundler transforms the `AppHost` binary to a single-file bundle by: +| Bundle Layout (ver 2.0) | +| ------------------------------------------------------------ | +| **AppHost**
`(Bundle-Marker)` | +| **Embedded Files**
`app.dll`
`app.deps.json`
`app.runtimeconfig.json`
`dependency.managed.dll`
`...`
| +| **Bundle Header**
`2.0` (Version #)
`#Number of Embedded Files`
`Bundle-ID`
`Needs Extraction?`
`deps.json file location (offset, size)`, if any.
`runtimeconfig.json file location (size,offset)`, if any. | +| **Bundle Manifest**
For each bundled file:
`Location (Offset, Size)`
`Type: IL, ReadyToRun, deps.json, runtimeconfig.json`, or `other` (extract) | -* Appending the managed app and its dependencies as a binary-blob at the end of the `AppHost` executable. -* Writing meta-data to identify the binary as a single-file bundle, and a manifest of the embedded files. +### Bundle Marker -#### Bundle Layout +Every `AppHost` has a static variable that identifies the location of the single-file header (if any). By default, this value is zero, which indicates that the `AppHost` is not a bundle. The bundler tool rewrites this value with the location of the bundle header. -The bundling tool will append the following contents to the `AppHost` binary: +Using a special marker for recognizing bundle-header (instead of simply writing the header at the end of the file) enables compatibility with other post-processing tools (such as `signtool`) which require their own content to be at the end of the file. -* The actual files to be published into the single file (including the managed app) -* A bundle header containing: - * The bundler tool version - * A bundle identifier: which is a *path-compatible* cryptographically strong name. - * This identifier is used to distinguish bundles for different versions of the same app. - * This identifier is used as part of the bundle extraction mechanism as described in [this document](extract.md). - * Currently, a new bundle identifier is generated for each bundle transformation. - In future, the bundle identifiers should be generated by hashing the contents of the bundle -- so that bundle transformation is deterministic. - * Offset of the bundle manifest +### Bundle Identifier -* A bundle manifest that describes: - * The location of embeded files (offset and size) - * The type of the embedded files: MSIL assemblies, ready-to-run assemblies, configuration files (ex: `app.deps.json` `app.runtimeconfig.json`), or others. +The bundle identifier is a *path-compatible* cryptographically strong name. -Every `AppHost` has a static variable that identifies the location of the single-file header (if any). -By default, this value is zero, which indicates that the `AppHost` is not a bundle. -The bundler tool rewrites this value with the location of the bundle header. -This ensures that the bundle-header is not position constrained (ex: at the end of the file), so that tools like `signtool` can post-process the bundle file. +* This identifier is used to distinguish bundles for different versions of the same app. +* This identifier is used as part of the bundle extraction mechanism as described in [this document](extract.md). +* A new bundle identifier is generated for each bundle transformation. -### Implementation +## Bundle Transformation -The bundler should ideally be located close to the core-host, since their implementation is closely related. Therefore, the bundler will be implemented in the `core-setup` repo. +Given a .net core app published with respect to a specific runtime, the bundler transforms the `AppHost` binary to a single-file bundle through the following actions: + +* Append the managed app and its dependencies as a binary-blob at the end of the `AppHost` executable. +* Write meta-data headers and manifest that help locate the contents of the bundle. +* Set the bundle-indicator in the `AppHost` to the offset of the bundle-header. + +## Implementation The bundling tool is implemented as a library in the [Microsoft.NET.HostModel](https://www.nuget.org/packages/Microsoft.NET.HostModel/) package. This library is used by the SDK in order to publish a .net core app as a single-file. -### Further work +The bundler should generate the correct format of single-file bundles based on the target framework. -#### Codesign on Mac +## Limitations -Codesign tool on mac-os performs strict consistency checks, and cannot tolerate the additional files appended at the end of the `AppHost` executable. -[Further work](https://github.com/dotnet/core-setup/issues/7065) is necessary to update the binary headers to make the bundle file compatible for signing. +* Currently, a new bundle identifier is generated for each bundle transformation. In future, the bundle identifiers should be generated by hashing the contents of the bundle -- so that bundle transformation is deterministic. +* `Codesign` tool on mac-os performs strict consistency checks, and cannot tolerate the additional files appended at the end of the `AppHost` executable. [Further work](https://github.com/dotnet/core-setup/issues/7065) is necessary to update the binary headers to make the bundle file compatible for signing. diff --git a/accepted/single-file/design.md b/accepted/single-file/design.md index ff415d6f2..88156ddb6 100644 --- a/accepted/single-file/design.md +++ b/accepted/single-file/design.md @@ -1,62 +1,28 @@ # Single-file Publish -Design for publishing apps as a single-file in .Net Core 3.0 +This document describes the design single-file apps in .net 5.0. +The design of single-file apps in .net core 3.0 can be found [here](3.0/bundler.md) ## Introduction The goal of this effort is enable .Net-Core apps to be published and distributed as a single executable. -There are several strategies to implement this feature -- ranging from bundling the published files into zip file (ex: [Warp](https://github.com/dgiagio/warp)), to native compiling and linking all the binaries together (ex: [CoreRT](https://github.com/dotnet/corert)). These options, along with their cost/benefit analysis is explored in this [staging document](staging.md). +There are several strategies to implement this feature -- ranging from bundling the published files into zip file to native compiling and linking all the binaries together. These options, along with their cost/benefit analysis is discussed in the [staging document](staging.md) and [related work](related.md). #### Goals -In .Net Core 3.0, we plan to implement a solution that +In .Net Core 5.0, we plan to implement a solution that: -* Is widely compatible: Apps containing MSIL assemblies, ready-to-run assemblies, native binaries, configuration files, etc. can be packaged into one executable. -* Can run framework dependent pure managed apps directly from bundle: - * Executes IL assemblies, and processes configuration files directly from the bundled executable. - * Extracts ready-to-run and native binaries to disk before loading them. -* Usable with debuggers and tools: The single-file should be debuggable using the generated symbol file. It should also be usable with profilers and tools similar to a non-bundled app. - -This feature-set is described as Stage 2 in the [staging document](staging.md), and can be improvised in further releases. +* Is widely compatible: Apps containing MSIL assemblies, ready-to-run assemblies, composite assemblies, native binaries, configuration files, etc. can be packaged into one executable. +* Can run managed components of the app directly from bundle, without need for extraction to disk. +* Usable with debuggers and tools. #### Non Goals -* Optimizing for development: The single-file publishing is typically not a part of the development cycle. It is typically used in release builds as a packaging step. Therefore, the single-file feature will be designed with focus on consumption rather than production. +* Optimizing for development: The single-file publishing is typically not a part of the development cycle, but is rather a packaging step as part of a release. Therefore, the single-file feature will be designed with focus on consumption rather than production. * Merging IL: Tools like [ILMerge](https://github.com/dotnet/ILMerge) combines the IL from many assemblies into one, but lose assembly identity in the process. This is not a goal for single-file feature. -#### Existing tools - -Single-file packaging for .Net Core apps is currently supported by third-party tools such as [Warp](https://github.com/dgiagio/warp) and [Costura](https://github.com/Fody/Costura). We now consider the pros and cons of implementing this feature within .Net Core. - -##### Advantages - -* A standardized experience available for all apps -* Integration with dotnet CLI -* A more transparent experience: for example, external tools may utilize public APIs such `AssemblyLoadContext.Resolving` or `AssemblyLoadContext.ResolvingUnmanagedDll` events to load files from a bundle. However, this may conflict with the app's own use of these APIs, which requires a cooperative resolution. Such a situation is avoided by providing inbuilt support for single-file apps. - -##### Limitations - -* Inbox implementation of the feature adds complexity for customers with respect to the list of deployment options to choose from. -* Independent tools can evolve faster on their own schedule. -* Independent tools can provide richer set of features catering to match a specific set of customers. - -We believe that the advantages outweigh the disadvantages in with respect to implementing this feature inbox. - -## Design - -There are two main aspects to publishing apps as a self-extracting single file: - -- The Bundler: A tool that embeds the managed app, its dependencies, and the runtime into a single host executable. -- The host: The "single-file" which facilitates the extraction and/or loading of embedded components. - -### The Bundler - -#### Bundling Tool - -The bundler is a tool that embeds the managed app and its dependencies into the native `AppHost` executable. The functional details of the bundler are explained in [this document](bundler.md). - -#### Build System Interface +## Build System Interface Publishing to a single file can be triggered by adding the following property to an application's project file: @@ -66,70 +32,185 @@ Publishing to a single file can be triggered by adding the following property to ``` -* The `PublishSingleFile` property applies to both framework dependent and self-contained publish operations. -* The `PublishSingleFile` property applies to platform-specific builds with respect to a given runtime-identifier. The output of the build is a native binary for the specified platform. - When `PublishSingleFile` is set to `true`, it is an error to leave `RuntimeIdentifier` undefined, or to set `UseAppHost` to `false`. -* Setting the `PublishSingleFile`property causes the managed app, managed dependencies, platform-specific native dependencies, configurations, etc. (basically the contents of the publish directory when `dotnet publish` is run without setting the property) to be embedded within the native `apphost`. +When the `PublishSingleFile` property is set to true, -By default, the symbol files are not embedded within the single-file, but remain as separate files in the publish directory. -This includes both the IL `.pdb` file, and the native `.ni.pdb` / `app.guid.map` files generated by ready-to-run compiler. -Setting the following property causes the symbol files to be included in the single-file. +* `RuntimeIdentifier` must be defined. Single-file builds generate a native binary for the specific platform and architecture. +* `UseAppHost` cannot be set to `false`. +* If `TargetFramework` is + * `netcoreapp5.0`, single-file publish works as described in this document. + * `netcoreapp3.0` or `netcoreapp3.1` single-file publish works as described [here](design_3_0.md). + * An earlier framework, causes a compilation error. -```xml - - true - -``` +Setting the `PublishSingleFile` property causes the managed app, runtime configuration files (`app.deps.json`, `app.runtimeconfig.json`), and managed binary dependencies to be embedded within the native `apphost`. All managed binaries (IL and ready-to-run files) that would be written to the publish directory and any sub-directories are bundled with the apphost. -Certain files can be explicitly excluded from being embedded in the single-file by setting following meta-data: +All other files, including platform-specific native binaries and symbol files, are left alongside the app by default. However, the set of files left unbundled alongside the app is expected to be small, such as: data-files (ex: `appsettings.json`) and custom native binary dependencies of the application. Further details regarding the files left next to the app is discussed in the [Host build](#Host-Builds) section. -```xml -true -``` +### Optional Settings -For example, to place some files in the publish directory but not bundle them in the single-file: +| Property | Behavior when set to `true` | +| ------------------------------- | ------------------------------------------------------------ | +| `IncludeContentInSingleFile` | All published files (except symbol files) are embedded within the single-file bundle. | +| `IncludeSymbolsInSingleFile` | The symbol files (IL `.pdb` file, and the native `.ni.pdb` / `app.guid.map` files generated by ready-to-run compiler) are embedded within the symbol file. | +| `ExtractContentsFromSingleFile` | The files that cannot be processed directly from the bundle are extracted out to disk at startup. | + +When including all content in single files, certain files can be explicitly excluded from being embedded in the single-file by setting following `ExcludeFromSingleFile` meta-data element. For example, to place some files in the publish directory but not bundle them in the single-file: ```xml + + true + true + true + - - PreserveNewest + + PreserveNewest true - - + + ``` -#### Interaction with other tools +### Alternatives -Once the single-file-publish tooling is added to the publish pipeline, other static binary transformation tools may need to adapt its presence. For example: +The behavior of `PublishSingleFile` property described above, is significantly different from `.net core 3.x` SDK. As an alternative, we could leave `PublishSingleFile` semantics unchanged (bundle all content to an actual single file), and have a different property `PublishFewFiles` to only bundle content that can be directly processed from the single-file. -* The MSBuild logic in `dotnet SDK` should be crafted such that [IlLinker](https://github.com/dotnet/core/blob/master/samples/linker-instructions.md), [crossgen](https://github.com/dotnet/coreclr/blob/master/Documentation/building/crossgen.md), and the single-file bundler run in that order in the build/publish sequence. -* External tools like [Fody](https://github.com/Fody/Fody) that use `AfterBuild`/`AfterPublish` targets may need to adapt to expect the significantly different output generated by publishing to a single file. The goal in this case is to provide sufficient documentation and guidance. +### Interaction with External Tools -### The Host +Once the single-file-publish tooling is added to the publish pipeline, other static binary transformation tools may need to adapt its presence. For example, tools like [Fody](https://github.com/Fody/Fody) that use `AfterBuild`/`AfterPublish` targets may need to adapt to expect the significantly different output generated by publishing to a single file. The goal in this case is to provide sufficient documentation and guidance. -On Startup, the AppHost checks if it has embedded files. If so, it +## The Bundler -* Memory maps the entire bundle file. -* Extracts the necessary files (ex: native binaries) to disk (if any) as explained in this [document](extract.md). -* Sets up data-structures so other components can access files embedded directly, as explained in this [document](direct.md) +The bundler is a tool that embeds the managed app and its dependencies into the native `AppHost` executable. The functional details of the bundler are explained in [this document](bundler.md). -#### Dependency Resolution +## The Host + +### Startup + +On Startup, the [host components](https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/host-components.md) perform the following functions: + +* **AppHost**: The AppHost identifies itself as a single-file bundle (by checking the [bundle marker](bundler.md#bundle-marker)) before invoking [HostFxr](https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/host-components.md#host-fxr). + +* **HostFxr**: If invoked from a single-file app, HostFxr process the `runtimeconfig.json` and `deps.json` files directly from the bundle. The location of these `json` files are identified directly from the bundle-header for simple access. + +* **HostPolicy**: Much of the bundle-processing is performed within HostPolicy. + + * If the app needs extraction, extracts out the appropriate files as explained in [this document](extractor.md). + + * Reads the `deps.json` file directly from the bundle and resolves dependencies. + + * Processes the Bundle manifest to maintain internal data-structures to locate bundled assemblies when probed by the runtime. + + * Implements a bundle-probe function, which will be used by the runtime to probe contents of the bundle when resolving assemblies. + + * Bundle probe has the following signature: + + ```C++ + /// + /// Relative-path to the file being probed. + /// Out-param: size of the file, if found. + /// Out-param: offset within the bundle, if found + /// true if the requested file is found in the bundle, + /// false otherwise. + /// + bool bundle_probe(const char *path, int64_t *size, int64_t *offset); + ``` + + * The `bundle_probe` function pointer is passed to the runtime (encoded as a string) through a property named `BUNDLE_PROBE`. + + * The contents of the bundle are not included in `TRUSTED_PLATFORM_ASSEMBLIES`. Any additional assemblies on disk (ex: due to `additional-deps` /`AdditionalProbingPaths`) are included in `TRUSTED_PLATFORM_ASSEMBLIES`. + +### Host Builds + +The .net 5 AppHost will be built in a few different configurations, and consumed by the SDK in appropriate scenarios as noted below. + +| Code Name | Build | Scenario | +| ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| AppHost | Current AppHost build | All non-single-file apps
Framework-dependent single-file apps | +| StaticHost | AppHost with HostFxr and HostPolicy statically linked. | Self-contained single-file apps on Windows | +| SuperHost | StaticHost with CoreCLR runtime components statically linked | Self-contained single-file apps on Unix systems | + +Ideally, we should use the SuperHost for self-contained single-file apps on Windows too. However, due to certain limitations in debugging experience and the ability to collect Watson dumps, etc., the CoreCLR libraries are left on disk beside the app. + +The files typically published for self-contained apps on Windows are: + +* The `App.Exe`-- StaticHost along with assemblies and configuration files bundled as a single-file app. +* `coreclr.dll`(runtime), `clrjit.dll` (JIT compiler), and `clrcompression.dll` (native counterpart of BCL for certain compression algorithms). + * It may be possible to link these DLLs together as CoreCLR.dll, but this work is not prioritized, in deference to building the full super-host once debugging framework supports it. +* mscordaccore.dll (to enable Watson dumps) + +Certain additional binaries may be optionally included with the app to enable debugging scenarios. +The work to not require these binaries as separate files on disk alongside the app is ongoing. + +* For F5 debugging on VS and VS-Core: `mscordbi.dll`, `libmscordbi.so` +* For Linux mini-dumps: `createdump`, `libmscordaccore.so` +* For ETW / LTTng: `clretwrc.dll`, `libcoreclrtraceptprovider.so` +* For exception stack trace source information, error string resources on Windows: `Microsoft.DiaSymReader.Native.amd64.dll`, `mscorrc.debug.dll`, `mscorrc.dll` + +When targeting `win7` platform, several additional DLLs are necessary (`api-*.dll`) to handle API compatibility. These files must be alongside the AppHost for the app to even start execution. +Therefore, we propose that targeting `win7-*` should not be supported when publishing apps as a single-file. + +## The Runtime + +### Dependency Resolution + +In order to resolve assemblies, the embedded resources are probed first, followed by other probing paths encoded in `TRUSTED_PLATFORM_ASSEMBLIES`. + +### PEImage loader + +* IL assemblies are loaded directly from the bundle. + * The portion of the single-file bundle containing the required assembly is memory mapped, and the contents are appropriately interpreted by the runtime. + +* ReadyToRun Assemblies + * On Linux, ReadyToRun assemblies are loaded directly from bundle. The various sections in the PE file are mapped at appropriate addresses, and offsets are fixed up. + * On MAC, ReadyToRun assemblies are loaded similar to Linux. However, Mojave hardened runtime doesn't allow executable mappings of a file. Therefore, the contents of an assembly are read from the bundle into pre-allocated executable memory. + * On Windows, due to certain limitations in memory mapping routines described below, ReadyToRun assemblies are loaded by memory mapping the file and copying sections to appropriate offsets. + +* ReadyToRun [Composite](https://github.com/dotnet/runtime/blob/master/docs/design/features/readytorun-composite-format-design.md) Assemblies are expected to be loaded similar to ReadyToRun assemblies. + +#### Windows Limitations +The Windows mapping routines have the following limitations: +* [CreateFileMapping](https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga) has no option to create a mapping for a part of the file (no offset argument). + This means that: + * We cannot use the (SEC_IMAGE) attribute to perform automatic section-wise loading (circumventing alignment requirements) of bundled assemblies directly. + * Instead we need to map each section independently. +* [MapViewOfFile]( https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffileex) can only map parts of a file aligned at [memory allocation granularity]( https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/ns-sysinfoapi-system_info), which is 64KB. + This means that: + * Each section within the assemblies should be aligned at 64KB – which is not guaranteed by the crossgen compiler. + * In order to memory-map one embedded assembly at a time, the assemblies in the bundle must be aligned at 64KB boundaries. + * A prototype bundler with this behavior for testing purposes is available in this branch: https://github.com/swaroop-sridhar/core-setup/tree/single-exe + +We therefore map the assembly and perform an in-memory copy of the sections to appropriate offsets. In the long term, the solution to the mapping problem would involve considerations such as: +* Compile all assemblies in a version bubble into one PE assembly, with a few aligned sections. +* Embed the big composite assembly into the host with proper alignment, so that the single-exe bundle can be loaded without copies at run time. + +### API Semantics -An app may choose to only embed some files (ex: due to licensing restrictions) and expect to pickup other dependencies from application-launch directory, nuget packages, etc. In order to resolve assemblies and native libraries, the embedded resources are probed first, followed by other probing paths. +#### `Assembly.Location` -### New API +There are a few options to consider for the `Assembly.Location` property of a bundled assembly: -In this section, we propose adding a few APIs to facilitate common operations on bundled-apps. +* A fixed literal (ex: `null`) indicating that no actual location is available. +* The simple name of the assembly (with no path). +* The path of the assembly as if it were not to be packaged into the single-file. +* A special UNC notation such as `/:/asm.dll` to denote files that come from the bundle. +* A configurable selection of the above, etc. -The binaries that are published in the project are expected to be handled transparently by the host. However, explicit access to the embedded files is useful in situations such as: +Proposed solution is: `://asm.dll`, where the `:` denotes the location within the bundle. The `:` is not a valid character in file-names on Windows. The `:` is already used as a path-separator on Unix systems. -- Reading additional files packaged into the app (ex: data files). -- Open an assembly for reflection/inspection -- Load plugins built as single-file class-libs using existing Loader APIs. +Most of the app development can be agnostic to whether the app is published as single-file or not. However, the parts of the app that deal with physical locations of files need to be aware of the single-file packaging. + +#### `AppContext.BaseDirectory` -Therefore, we propose adding an API similar to [GetManifestResourceStream](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assembly.getmanifestresourcestream?view=netframework-4.7.2#System_Reflection_Assembly_GetManifestResourceStream_System_String_) to obtain a stream corresponding to an embedded file. +`AppContext.BaseDirectory` will be the directory where the AppHost (the single-file bundle itself) resides. -This is only a draft of the proposed APIs. The actual shape of the APIs will be decided via API review process. +## Handling Content + +The app may want to access certain embedded content for reading, rather than loading via host/runtime. For example: bundled payload/data files. In this case, the recommended strategy is to embed the content files within appropriate managed assemblies as PE resources, and access them through [resource handling APIs](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assembly.getmanifestresourceinfo?view=netcore-3.1). + +However, explicit access to the embedded files is useful in situations such as: +- Reading data-files from the app's native code +- Open an assembly for reflection/inspection + +Therefore, we propose adding an API similar to [GetManifestResourceStream](https://docs.microsoft.com/en-us/dotnet/api/system.reflection.assembly.getmanifestresourcestream?view=netcore-3.1) to obtain a stream corresponding to an embedded file. This is only a draft of the proposed APIs. The actual shape of the APIs will be decided via API review process. ```cs namespace System.Runtime.Loader @@ -137,67 +218,32 @@ namespace System.Runtime.Loader public partial class Bundle { // Check whether an app is running from a single-file bundle - public static bool IsBundle(Assembly assembly); + public static bool IsBundle(); // Get the location where contents of the bundle are extracted public static string GetContentRoot(Assembly assembly); - // Open a file embedded in the bundle built for the specified assembly - public static System.IO.Stream GetFileStream(Assembly assembly, string name); + // Open a file embedded in the bundle + public static System.IO.Stream GetFileStream(string path); } } ``` -We can also provide an abstraction that abstracts away the physical location of a file (bundle or disk). For example, add a variant of `GetFileStream` API that looks for a file in the bundle, and if not found, falls back to disk-lookup. However such abstractions are also easy to build outside of the .Net Framework. - -### Existing API - -We need to determine the semantics of current APIs such as `Assembly.Location` that return the information about an assembly's location on disk. - -#### `Assembly.Location` - -There are a few options to consider for the `Assembly.Location` property of a bundled assembly: - -* A fixed literal (ex: `null`) indicating that no actual location is available. -* The simple name of the assembly (with no path). -* A special UNC notation such as `/:/asm.dll` to denote files that come from the bundle. -* The path of the assembly as if it were not to be packaged into the single-file. -* In the case of files spilled to disk (ex: ready-to-run assemblies) -- the actual location of the extracted file. -* A configurable selection of the above, etc. - -We propose keeping the default behavior of `Assembly.Location`. That is, - -* If the assembly is loaded directly from the bundle, return empty-string -* If the assembly is extracted to disk, return the location of the extracted file. - -Most of the app development can be agnostic to whether the app is published as single-file or not. However, the parts of the app that deal with physical locations of files need to be aware of the single-file packaging. - -#### `AppContext.BaseDirectory` - -A few options to consider for the `AppContext.BaseDirectory` are: - -* The directory where embedded files are extracted out: This option enables easy access to content files spilled to disk. However, if no files need to be extracted to disk for an application bundle, there's no extraction directory. -* The extraction-directory if files are extracted, the AppHost directory otherwise. The limitation to this approach is that the value of `AppContext.BaseDirectory` changes with the way applications are packaged/executed. -* The directory where `AppHost` binary resides (always). - -We propose that `AppContext.BaseDirectory` should always be set to the directory where the `AppHost` bundle resides. This scheme doesn't provide an obvious mechanism to access the contents of the extraction directory -- by design. The recommended method for accessing content files from the bundle are: - -- Do not bundle application data files into the single-exe; instead them next to the bundle. This way, the application binary is a single-file, but not the whole application. -- Embed data files as managed resources into application binary, and access them via resource management [APIs](). - ## Testing -* Unit Tests to achieve good coverage on apps using managed, ready-to-run, native code -* Tests for the newly added API -* Tests to verify that framework-dependent and self-contained apps can be published as single file. -* Tests for ensure that every app model template supported by .NET Core (console, wpf, winforms, web, etc.) can be published as a single file. -* A subset of CoreCLR Tests -* Tests to ensure that MSIL files with embedded PDBs are handled correctly -* End-to-End testing on real world apps +* Unit Tests: + * Apps using managed, ready-to-run, native code + * Framework-dependent and self-contained apps + * Apps with content explicitly annotated for inclusion/exclusion in the single-file bundle + * MSIL files with embedded PDBs + * PDBs included/excluded from bundle +* Tests for ensure that every app model template supported by .NET Core can be published as a single file. +* Tests to ensure cross-platform publishing of single-file bundles. +* Manual end-to-end testing on real world apps #### Measurements -Measure publish size and run-time (first run, subsequent runs) for HelloWorld (framework-dependent and self-contained), Roslyn and MusicStore. +Measure publish size and startup time for a few real-world apps. #### Telemetry @@ -205,86 +251,35 @@ Collect telemetry for single-file published apps with respect to parameters such * Framework-dependent vs self-contained apps. * Whether the apps are Pure managed apps, ready-to run compiled apps, or have native dependencies. -* Embedding of additional/data files, and use of file access API. +* Embedding of additional/data files. ## Further Work -#### Bundler Optimizations - -Since all the files of an app published as a single-file live together, we can perform the following optimizations - -- R2R compile the app and all of its dependent assemblies in a single version-bubble - - Single-file apps compiled cross-platform may have this optimization disabled until the ready-to-run compiler (`crossgen`) supports cross-compilation. - -- Investigate whether collectively signing the files in an assembly saves space for certificates. - -#### Compression - -Currently the bundler does not compress the contents embedded at the end of the host binary. -Compressing the bundled files and meta-data can significantly reduce the size of the single-file output (by about 30%-50% as determined by prototyping). - -#### Single-file Plugins - -The above design should be extended to seamlessly support single-file publish for plugins. - -- The bundler tool will mostly work as-is, regardless of whether we publish an application or class-lib. The binary blob with dependencies can be appended to both native and managed binaries. -- For host/runtime support, the options are: - - Implement plugins using existing infrastructure. For example: Take control of assembly/native binary loads via existing `AssemblyLoadContext` callbacks and events. Extract the files embedded within the single-file plugin using the `GetFileStream()` API and load them on demand. - - Have new API to load a single-file plugin, for example: `AssemblyLoadContext.LoadWithEmbeddedDependencies()`. - -#### VS Integration - -Developers should be able to use the feature easily from Visual Studio. The feature will start with text based support -- by explicitly setting the `PublishSingleFile` property in the project file. In future, we may provide a single-file publish-profile, or other UI triggers. +* **Compression**: Currently the bundler does not compress the contents embedded at the end of the host binary. Compressing the bundled files and meta-data can significantly reduce the size of the single-file output (by about 30%-50% as determined by prototyping). +* **Single-file Plugins** Extended the above design to seamlessly support single-file publish for plugins. ## User Experience -To summarize, here's the overall experience for creating a HelloWorld single-file app - -* Create a new HelloWorld app: `HelloWorld$ dotnet new console` - -#### Framework Dependent HelloWorld - -* Normal publish: `dotnet publish` - - * Publish directory contains the host `HelloWorld.exe` , the app `HelloWorld.dll`, configuration files `HelloWorld.deps.json`, `HelloWorld.runtimeconfig.json`, and the PDB-file `HelloWorld.pdb`. - -* Single-file publish: `dotnet publish -r win10-x64 --self-contained=false /p:PublishSingleFile=true` - - * Publish directory contains: `HelloWorld.exe` `HelloWorld.pdb` - - * `HelloWorld.dll`, `HelloWorld.deps.json`, and `HelloWorld.runtimeconfig.json` are embedded within `HelloWorld.exe`. - -* Run: `HelloWorld.exe` - - * The app runs completely from the single-file, without the need for intermediate extraction to file. - -#### Self-Contained HelloWorld - -- Normal publish: `dotnet publish -r win10-x64` - - * Publish directory contains 221 files including the host, the app, configuration files, the PDB-file and the runtime. - -- Single-file publish: `dotnet publish -r win10-x64 /p:PublishSingleFile=true` - - - Publish directory contains: `HelloWorld.exe` `HelloWorld.pdb` - - The remaining 219 files are embedded within the host `HelloWorld.exe`. - -- Run: `HelloWorld.exe` +To summarize, here's the overall experience for creating a HelloWorld single-file app: - * The bundled app and configuration files are processed directly from the bundle. - * Remaining 216 files will be extracted to disk at startup. - * If reuse of extracted files is enabled, subsequent runs of the app may skip the extraction step. +* Create a new HelloWorld app: `HelloWorld$ dotnet new console` +* Framework Dependent Publish -Most applications are expected to work without any changes. However, apps with a strong expectation about absolute location of dependent files may need to be made aware of bundling and extraction aspects of single-file publishing. No difference is expected with respect to debugging and analysis of apps. + * Normal publish: `dotnet publish` + * Published files: `HelloWorld.exe`, `HelloWorld.dll`, `HelloWorld.deps.json`, `HelloWorld.runtimeconfig.json`, `HelloWorld.pdb` -## Related Work + * Single-file publish: `dotnet publish -r win10-x64 --self-contained=false /p:PublishSingleFile=true` + * Published files: `HelloWorld.exe`, `HelloWorld.pdb` -* [Mono/MkBundle](https://github.com/mono/mono/blob/master/mcs/tools/mkbundle) -* [Fody.Costura](https://github.com/Fody/Costura) -* [Warp](https://github.com/dgiagio/warp) -* [dotnet-warp](https://github.com/Hubert-Rybak/dotnet-warp) -* [BoxedApp](https://docs.boxedapp.com/index.html) -* [LibZ](https://github.com/MiloszKrajewski/LibZ) +* Self-Contained Publish + * Normal publish: `dotnet publish -r win10-x64` + * Published files: `HelloWorld.exe`, `HelloWorld.pdb`, and 224 more files + * Single-file publish Linux: `dotnet publish -r linux-x64 /p:PublishSingleFile=true` + * Published files: `HelloWorld.exe`, `HelloWorld.pdb` + * Single-file publish Windows: `dotnet publish -r win10-x64 /p:PublishSingleFile=true` + * Published files: `HelloWorld.exe`, `HelloWorld.pdb`, `coreclr.dll`, `clrjit.dll`, `clrcompression.dll`, `mscordaccore.dll` + * Single-file publish Windows with Extraction: `dotnet publish -r win10-x64 /p:PublishSingleFile=true /p:IncludeContentInSingleFile=true /p:ExtractContentsFromSingleFile=true` + * Published files: `HelloWorld.exe`, `HelloWorld.pdb` + diff --git a/accepted/single-file/design_3_0.md b/accepted/single-file/design_3_0.md new file mode 100644 index 000000000..e72b1ce58 --- /dev/null +++ b/accepted/single-file/design_3_0.md @@ -0,0 +1,136 @@ +# Single-file Publish + +The design of single-file apps in .net core 3.0. + +## Introduction + +In .net core 3 implements Stage 1 (described in the [staging document](staging.md)) support for single-file apps. + +## Build System Interface + +Publishing to a single file can be triggered by adding the following property to an application's project file: + +```xml + + true + +``` + +* The `PublishSingleFile` property applies to both framework dependent and self-contained publish operations. +* The `PublishSingleFile` property applies to platform-specific builds with respect to a given runtime-identifier. The output of the build is a native binary for the specified platform. + When `PublishSingleFile` is set to `true`, it is an error to leave `RuntimeIdentifier` undefined, or to set `UseAppHost` to `false`. +* Setting the `PublishSingleFile`property causes the managed app, managed dependencies, platform-specific native dependencies, configurations, etc. (basically the contents of the publish directory when `dotnet publish` is run without setting the property) to be embedded within the native `apphost`. + +By default, the symbol files are not embedded within the single-file, but remain as separate files in the publish directory. This includes both the IL `.pdb` file, and the native `.ni.pdb` / `app.guid.map` files generated by ready-to-run compiler. Setting the following property causes the symbol files to be included in the single-file. + +```xml + + true + +``` + +Certain files can be explicitly excluded from being embedded in the single-file by setting following meta-data: + +```xml +true +``` + +For example, to place some files in the publish directory but not bundle them in the single-file: + +```xml + + + PreserveNewest + true + + +``` + +## The Bundler + +The bundler is a tool that embeds the managed app and its dependencies into the native `AppHost` executable, as described [here](bundler.md). The bundle layout in .net core 3.x is described below. + +| Bundle Layout (ver 1.0) | +| ------------------------------------------------------------ | +| **AppHost**
`(Bundle-Marker)` | +| **Embedded Files**
`app.dll`
`app.deps.json`
`app.runtimeconfig.json`
`dependency.dll`
`...`
| +| **Bundle Header**
`1.0` (Version #)
`#Number of Embedded Files`
`Bundle-ID` | +| **Bundle Manifest**
For each bundled file:
`Location (Offset, Size)`
`Type :IL, ReadyToRun, other` (This information is not used in 3.0) | + +## The Host + +On Startup, the AppHost checks if it has embedded files. If so, it + +* Memory maps the entire bundle file. +* Checks if the contents are already extracted to the extraction location (described below) + * If all files are intact in the extraction location, the execution continues reusing the extracted files. + * If not, extract out the missing components +* If an extraction is not available, the host extracts the embedded files to disk as explained in this [document](extract.md). +* Invokes the runtime (through other host components). + +### Extraction Location + +For a single-file app, the extraction directory is `//` + +* `` is + * `DOTNET_BUNDLE_EXTRACT_BASE_DIR` environment variable, if set. + * If not, defaults to: + * Windows: `%TEMP%\.net` + * Unix: + * `${TMPDIR}/.net/${UID}` if `${TMPDIR}` is set; otherwise, + * `/var/tmp/.net/${UID}` if `/var/tmp` exists, and is writable; otherwise, + * `/tmp/.net/${UID}` if `/tmp` exists, and is writable; otherwise fail. + +* `` is the name of the single-exe binary + +* `` is a unique Bundle-identifier. + +## API Impact + +Most of the app development can be agnostic to whether the app is published as single-file or not. However, the parts of the app that deal with physical locations of files need to be aware of the single-file packaging. + +* `Assembly.Location`: Returns the actual location within the extraction-location. +* `AppContext.BaseDirectory`: Returns the extraction directory, the location where `app.dll` resides. + +## User Experience + +To summarize, here's the overall experience for creating a HelloWorld single-file app + +* Create a new HelloWorld app: `HelloWorld$ dotnet new console` + +#### Framework Dependent HelloWorld + +* Normal publish: `dotnet publish` + + * Publish directory contains the host `HelloWorld.exe` , the app `HelloWorld.dll`, configuration files `HelloWorld.deps.json`, `HelloWorld.runtimeconfig.json`, and the PDB-file `HelloWorld.pdb`. + +* Single-file publish: `dotnet publish -r win10-x64 --self-contained=false /p:PublishSingleFile=true` + + * Publish directory contains: `HelloWorld.exe` `HelloWorld.pdb` + + * `HelloWorld.dll`, `HelloWorld.deps.json`, and `HelloWorld.runtimeconfig.json` are embedded within `HelloWorld.exe`. + +* Run: `HelloWorld.exe` + + * The app runs completely from the single-file, without the need for intermediate extraction to file. + +#### Self-Contained HelloWorld + +- Normal publish: `dotnet publish -r win10-x64` + + * Publish directory contains 221 files including the host, the app, configuration files, the PDB-file and the runtime. + +- Single-file publish: `dotnet publish -r win10-x64 /p:PublishSingleFile=true` + + - Publish directory contains: `HelloWorld.exe` `HelloWorld.pdb` + - The remaining 219 files are embedded within the host `HelloWorld.exe`. + +- Run: `HelloWorld.exe` + + * On first run, the 219 embedded files will be extracted to disk at startup. + * Subsequent runs of the app will reuse the extraction, without incurring startup overhead. + +#### Debugging + +No difference is expected with respect to debugging and analysis of apps. + diff --git a/accepted/single-file/direct.md b/accepted/single-file/direct.md deleted file mode 100644 index 1a8da2541..000000000 --- a/accepted/single-file/direct.md +++ /dev/null @@ -1,78 +0,0 @@ -# Accessing Bundled Content - -This document outlines the process by which the host and runtime components access files embedded within a single-file bundle directly (that is, without extracting out to temporary files on disk). - -### AppHost - -The AppHost bundle-handling code implements the following service, which will be registered for use by other components. - -```c++ -// If the requested file is available in the bundle, the function provides -// - A pointer to a buffer with the contents of the file in memory -// - The size of the file. -// The function returns true if the file is found, false otherwise. -static bool read_bundled_file(const char* name, - const void** buffer, - size_t* size); -``` - -### HostFxr - -The `Apphost` provides the a bundle-reader for `HostFxr` via the updated API: - -```C++ -SHARED_API int hostfxr_main_startupinfo(const int argc, - const pal::char_t* argv[], - const pal::char_t* host_path, - const pal::char_t* dotnet_root, - const pal::char_t* app_path, - bool (*bundle_reader)(const char* name, - const void** buffer, - size_t* size)); -``` - -The `HostFxr` uses this reader to read `app.runtimeconfig.json` and any other files it may need from the bundle. If the `bundle_reader` is invalid or fails to find the requested file, it falls back to look for an actual file on disk. - -### HostPolicy - -The `HostFxr` registers the `bundle_reader` with `HostPolicy` via the updated API: - -```C++ -struct corehost_initialize_request_t -{ - size_t version; - strarr_t config_keys; - strarr_t config_values; - bool (*bundle_reader)(const char* name, const void** buffer, size_t* size); -}; - -SHARED_API int __cdecl corehost_initialize( - const corehost_initialize_request_t *init_request, - int32_t options, - /*out*/ corehost_context_contract *context_contract); -``` - -Host policy uses this reader to attempt to process the `app.deps.json` file and check for the existence of other component files within the bundle. - -### Runtime Components - -The `HostPolicy` registers the `bundle-reader` with the runtime via a new API: - -```C++ -// Register a bundle-reader with CoreCLR -// -// Parameters: -// reader - The bundle-reader to use for this application -// Returns: -// S_OK - -CORECLR_HOSTING_API(coreclr_register_bundle_reader, - bool (*bundle_reader)(const char* name, - const void** buffer, - size_t* size)); -``` - -`HostPolicy` maintains a list of bundled assemblies internally, and communicates the assemblies on disk (unbundled or extracted) to the runtime in `TRUSTED_PLATFORM_ASSEMBLIES`. - -In order to resolve an assembly, the TPA-Binder first checks if it is available in the bundle (by invoking the `bundle_reader` callback). If the assembly is available in the bundle, the runtime loads the binary from memory via PELoader. If the assembly is not available in the bundle, the runtime continues to resolve it using the TPA-list. - diff --git a/accepted/single-file/extract.md b/accepted/single-file/extract.md index ef346892d..e361b8cb1 100644 --- a/accepted/single-file/extract.md +++ b/accepted/single-file/extract.md @@ -1,69 +1,59 @@ # Extracting Bundled Files to Disk -When running the single-file, the host will need to extract some bundled files resources out to the disk before loading them. This is particularly true for native DLLs because of operating system API restrictions -- for example, Windows doesn't support a `LoadLibrary()` API that reads in from a stream. +When running the single-file, the [host components](https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/host-components.md) may need to extract some bundled files out to the disk. This is particularly true for bundled native DLLs because of operating system API restrictions -- for example, Windows doesn't support a `LoadLibrary()` API that reads in from a memory stream. -We now explore the options with respect to the location and life-time of the extracted files. +## Extraction Requirements -## Extraction Options - -### Extraction Time - -* **Startup**: The host spills the necessary files at startup, and communicates the location to the runtime. -* **Lazy:** The runtime provides callbacks to extract a bundled file to disk when it is actually necessary to be loaded. - -Extraction at startup is easier to implement, as the runtime needs no knowledge of the extraction. - -Lazy extraction saves startup cost, particularly if many ready-to-run images are bundled. The lazy extraction feature is only useful when files are extracted on every run. - -### Extraction Lifetime - -#### Temporary Extraction - -Extract out the necessary files to a random temporary location, and attempt to clean them out on exit. - -##### Pros - -* Simpler implementation: fewer complexities with respect to app-updates and concurrency. -* User need not worry about explicit cleanup. -* Suits customers who prefer click-and-run model for their apps, rather than app-install model. - -##### Cons - -* Higher startup cost, because of extraction on every run -- especially for self-contained apps. -* Cleanup may fail if the application crashes. - -#### Persistent Extraction - -Extract files to a specific location on first run, and re-use the extraction on subsequent runs. - -##### Pros - -- Saves startup cost, particularly for self-contained apps, or apps containing several native binary dependencies. -- No need for implementing lazy-extraction. - -##### Cons - -- Complex implementation. -- Unsuitable for customer scenarios that cannot tolerate persistent disk state after the app terminates. -- Cleanup is manual. +- *Fault-tolerance*: Recover from failure after partial extraction on first run. +- *Concurrency*: Handle concurrent launches on first start. +- *Upgrade*: Each version of the app extracts to a unique location, supporting side-by-side use of multiple versions. +- *Manual Cleanup*: Users can identify and delete extracted files when the app is no longer needed. +- *Amortization*: It is desirable that the cost of extraction is amortized over several runs. For example, by extracting on first-run, and reusing the same extracted files on subsequent runs. -### Current Tools +## Extraction Options -* **Extraction Time**: Mono extracts native libraries at startup. Costura extracts and explicitly loads the native-libraries at startup. -* **Lifetime**: Mono and Costura extract native dependencies to temporary files on every run. +* ***Extraction Time*** + * **Startup**: The host spills the necessary files at startup, and communicates the location to the runtime. + * Simple implementation + * High startup cost + * Current tools: Mono extracts native libraries at startup. Costura extracts and explicitly loads the native-libraries at startup. + * **Lazy:** The runtime provides callbacks to extract a bundled file to disk when it is actually necessary to be loaded. + * Lazy extraction is risky because the time when a particular file is needed is not always known to the runtime (ex: dependencies of native DLLs loaded by the runtime). + * Low startup cost +* ***Lifetime*** + * **Extract Always**: Extract out the necessary files to a random temporary location on each run + * Simple implementation, since the app need not keep track of previous extractions + * Higher startup cost, because of extraction on every run. + * Current tools: Mono and Costura extract native dependencies to temporary files on every run. + + * **Reuse extraction**: Extract files to a specific location on first run, and re-use the extraction on subsequent runs. + * Amortizes the startup cost over several runs. + * Complex implementation: During startup, apps need to look for existing extractions, verify if they are usable, and extract missing pieces if necessary. + * Unsuitable for customer scenarios that cannot tolerate persistent disk state after the app terminates. +* ***Location*** + * **Temporary Directory** + * The extracted files are cleaned up automatically by the OS + * If extracted files are reused, the app should tolerate missing files + * **Persistent directory** + * Robust solution for reusing extracted files + * Solution feels like an app-install, rather than a click-and-run executable. ## Proposed Solution -For .Net Core 3.0, we propose to implement the persistent-extraction scheme. Based on customer feedback, we can implement other options discussed above. +### Extraction Mechanism -### Extraction Requirements +During startup, the host performs the following steps: -- *Reuse*: Extract on first-run, reuse on subsequent runs. -- *Fault-tolerance*: Recover from failure after partial extraction on first run. -- *Concurrency*: Handle concurrent launches on first start. -- *Upgrade*: Each version of the app extracts to a unique location, supporting side-by-side use of multiple versions. -- *Cleanup*: Users can identify and delete extracted files when the app is no longer needed. -- *Access control*: Have an option for elevated runs to extract files to admin-only-writable locations. +* Check if the [extraction directory](Extraction Location) exists for this app: + * If not, perform a new extraction: + * Extract appropriate files to a temporary location `//` . This ensures that concurrent launches of the app do not interfere with each other. + * After successful extraction of all necessary files to `//`, the commit the extraction to the actual location `//`. + - If `//` already exists by now, another process may have completed the extraction, so remove the directory `//`. Verify and reuse the extraction at `//`. + - If not, `//` is renamed to `//` + - The rename should be performed with appropriate retries to accommodate file-locking by antivirus software. + * If it does, verify if all expected files are intact in the extraction location. + * If all files are intact, the continue execution reusing the extracted files. + * If not, extract out the missing components: For each missing file, extract it within `//`, and rename it to the corresponding location within `//` , as described above. ### Extraction Location @@ -73,62 +63,48 @@ The desirable characteristics of the extraction directory are: - Should be semi-permanent: Not a directory that is lost frequently (ex: on machine reboot), but considered recyclable storage, since any subsequent run can re-extract. - Be short, to reduce the risk of running over path length limit. -For a single-file app, the extraction directory is `//` +For a single-file app, the extraction directory is -* `` is +```bash +// +``` +where: + +* `` is * `DOTNET_BUNDLE_EXTRACT_BASE_DIR` environment variable, if set. - * If not, on Windows, defaults to `%TEMP%\.net` - * On Unix-like systems, where multiple users may use a single system, an approach that removes the possibility of name collisions and other users creating files to prevent an application to start (by a malicious user creating a predictable directory name) is used instead: - * For .NET 3, the extraction directory follows the following order: - * `${TMPDIR}/.net/${UID}` if `${TMPDIR}` is set; otherwise, - * `/var/tmp/.net/${UID}` if that exists; otherwise, - * `/tmp/.net/${UID}` if that exists; fails otherwise. - * For .NET 5, the proposal is to follow this scheme: - * The directory to extract the bundle is created with `mkdtemp()`, using the `$TMPDIR` environment variable (if set), `/var/tmp/` (if exists, because it survives reboots), falling back to `/tmp` (does not survive reboots, it's often a ramdisk) in the template (e.g. `/var/tmp/dotnet--XXXXXX`); - * A symbolic link to the directory created by `mkdtemp()` is created in a predictable location: - * If `${XDG_CACHE_HOME}` is set, the symlink is created under `${XDG_CACHE_HOME}/.cache/dotnet//` (See [the XDG spec for information](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)); otherwise, - * Symlink is created under `~/.cache/dotnet//` - * On startup: - * If the symbolic link exists and isn't stale (points to a directory owned by the user, with correct permissions (`0700`), etc.), that's what it is used; - * If the symbolic link does not exist (or exists and is stale), it is removed, a new directory is created with `mkdtemp()`, and the link is re-created. + * If not, defaults to: + * Windows: `%TEMP%\.net` + * Unix: + * `${TMPDIR}/.net/${UID}` if `${TMPDIR}` is set; otherwise, + * `/var/tmp/.net/${UID}` if `/var/tmp` exists, and is writable; otherwise, + * `/tmp/.net/${UID}` if `/tmp` exists, and is writable; otherwise fail. * `` is the name of the single-exe binary +* `` is the unique [Bundle-identifier](bundler.md#bundle identifier). -* `` is a unique Bundle-identifier. - - Each single-file bundle contains a path-compatible cryptographically strong identifier embedded within it. This identifier is used as part of the extraction path in order to ensure that files from multiple versions of an app do not interfere with each other. - -#### Cleanup +#### Unix: An Alternate Proposal -The cleanup of extracted files in the install-location will be manual in this version. +On Unix-like systems, where multiple users may use a single system, an approach that removes the possibility of name collisions and other users creating files to prevent an application to start (by a malicious user creating a predictable directory name) is used instead: -We can consider providing helper scripts to cleanup the extracted files for an executable. However, future versions are expected to spill fewer artifacts to disk, making the cleanup commands a lower priority feature. +* The directory to extract the bundle is created with `mkdtemp()`, using the `$TMPDIR` environment variable (if set), `/var/tmp/` (if exists, because it survives reboots), falling back to `/tmp` (does not survive reboots, it's often a ramdisk) in the template (e.g. `/var/tmp/dotnet--XXXXXX`); + * To facilitate reuse of extraction, the extraction directory must be predictable. To achieve this, a symbolic link to the directory created by `mkdtemp()` is created in a predictable location: + * If `${XDG_CACHE_HOME}` is set, the symlink is created under `${XDG_CACHE_HOME}/.cache/dotnet//` (See [the XDG spec for information](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)); otherwise, + * Otherwise, the symlink is created under `~/.cache/dotnet//` + * In order to check for reuse on startup: + * If the symbolic link exists and isn't stale (points to a directory owned by the user, with correct permissions (`0700`), etc.), that's what it is used; + * If the symbolic link does not exist (or exists and is stale), it is removed, a new directory is created with `mkdtemp()`, and the link is re-created. -### Extraction Mechanism - -During startup, the Apphost: - -- Checks if pre-extracted files are available for the app, and if so, reuses them. - - In this version, the extracted files are assumed to be correct by construction. That is, if `//` exists, the files there are reused. - - If the contents of the extraction directory are corrupted post-extraction, they will need to be explicitly cleaned. - - This situation is no different from having a managed app with one of the dependent DLLs corrupted. -- Extracts appropriate files to a temporary location `//` . This ensures that concurrent launches of the app do not interfere with each other. -- After successful extraction of all necessary files to `//`, the app host commits the extraction to the actual location `//` . - - If `//` already exists by now, another process may have completed the extraction, so the host removes the directory `//`, and continues execution with files in `//` . - - If not, `//` is renamed to `//` -- The `//` is now considered app-root, and execution continues. +#### Cleanup -### Extraction Configurations +The cleanup of extracted files in the install-location will be manual, or through OS cleanup routines that remove files within the temporary directory. -The following settings can be used to configure the extraction mechanism: +## Extraction Implementation -* Extract all embedded dependencies in each run (instead of loading certain files directly from the bundle) - * Build time: Set the property `ExtractAllFiles= true` - * Run-time: Set the environment variable `DOTNET_BUNDLE_EXTRACT_ALL_FILES=true` -* Set the base directory within which the files embedded in a single-file app are extracted, as explained in sections above. - * Run-time: Set the `DOTNET_BUNDLE_EXTRACT_BASE_DIR` environment variable to the full path. - * There is no build-time configuration, since this is a path-setting on the machine where the app is run. +In .net core 3.0, the AppHost itself performed the extraction of bundled components. This was necessary because, the [HostFxr](https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/host-components.md#host-fxr) and [HostPolicy](https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/host-components.md#host-policy) DLLs themselves may be bundled (in self-contained builds), and need to be extracted out before extraction. In .net 5.0, this problem is solved by alleviating the need to extract host components, as explained in the [Host Builds](design.md#host-builds) section. -The environment variables mentioned above are expected to be useful in debugging scenarios. +In .net 5.0, the bundle-extraction functionality will be implemented in the HostPolicy component. The advantages of this approach are: +* The AppHost is designed to be a minimal wrapper that finds and invokes the framework. It needs to be compatible with several .net versions, and is not easily serviceable. Therefore, moving the extraction out of the AppHost better suits its architecture. +* For Framework dependent apps, moves functionality into the framework, thus reducing AppHost size. +* For Framework dependent apps, makes extraction code easily serviceable. \ No newline at end of file diff --git a/accepted/single-file/related.md b/accepted/single-file/related.md new file mode 100644 index 000000000..a4c39ab1d --- /dev/null +++ b/accepted/single-file/related.md @@ -0,0 +1,27 @@ +# Related Work + +Single-file packaging for .Net Core apps is currently supported by third-party tools such as [Warp](https://github.com/dgiagio/warp) and [Costura](https://github.com/Fody/Costura). We now consider the pros and cons of implementing this feature within .Net Core. + +#### Advantages + +* A standardized experience available for all apps +* Integration with dotnet CLI +* A more transparent experience: for example, external tools may utilize public APIs such `AssemblyLoadContext.Resolving` or `AssemblyLoadContext.ResolvingUnmanagedDll` events to load files from a bundle. However, this may conflict with the app's own use of these APIs, which requires a cooperative resolution. Such a situation is avoided by providing inbuilt support for single-file apps. + +#### Limitations + +* Inbox implementation of the feature adds complexity for customers with respect to the list of deployment options to choose from. +* Independent tools can evolve faster on their own schedule. +* Independent tools can provide richer set of features catering to match a specific set of customers. + +We believe that the advantages outweigh the disadvantages in with respect to implementing this feature inbox. + +## References + +* [Mono/MkBundle](https://github.com/mono/mono/blob/master/mcs/tools/mkbundle) +* [Fody.Costura](https://github.com/Fody/Costura) +* [Warp](https://github.com/dgiagio/warp) +* [dotnet-warp](https://github.com/Hubert-Rybak/dotnet-warp) +* [BoxedApp](https://docs.boxedapp.com/index.html) +* [LibZ](https://github.com/MiloszKrajewski/LibZ) +