diff --git a/docs/internals.md b/docs/advanced/internals.md similarity index 69% rename from docs/internals.md rename to docs/advanced/internals.md index 2f62b5724..c9c23ff25 100644 --- a/docs/internals.md +++ b/docs/advanced/internals.md @@ -1,4 +1,4 @@ -# igir Internals +# Internal Operations Information about the inner workings of `igir`. @@ -8,20 +8,20 @@ Information about the inner workings of `igir`. 1. Scan each DAT input path for every file and parse them, if provided (`--dat `) 2. Scan each ROM input path for every file (`--input `) - - Detect headers in those files, if applicable (see [header docs](roms/headers.md)) -3. Scan each patch input path for every file (`--patch `) (see [patching docs](roms/patching.md)) + - Detect headers in those files, if applicable (see [header docs](../roms/headers.md)) +3. Scan each patch input path for every file (`--patch `) (see [patching docs](../roms/patching.md)) 4. Then for each DAT: - - Parent/clone information is inferred if the DAT has none (see [DATs docs](./dats/processing.md#parentclone-inference)) - - Parent/clone ROMs sets are merged or split (`--merge-roms `) (see [arcade docs](./usage/arcade.md)) - - ROMs in the DAT are filtered to only those desired (`--filter-*` options) (see [filtering & preference docs](roms/filtering-preferences.md)) + - Parent/clone information is inferred if the DAT has none (see [DATs docs](../dats/processing.md#parentclone-inference)) + - Parent/clone ROMs sets are merged or split (`--merge-roms `) (see [arcade docs](../usage/arcade.md)) + - ROMs in the DAT are filtered to only those desired (`--filter-*` options) (see [filtering & preference docs](../roms/filtering-preferences.md)) - Input files are matched to ROMs in the DAT - Patch files are matched to ROMs found - - ROM preferences are applied (`--single`, see [filtering & preference docs](roms/filtering-preferences.md#preferences-for-1g1r)) + - ROM preferences are applied (`--single`, see [filtering & preference docs](../roms/filtering-preferences.md#preferences-for-1g1r)) - ROMs are combined (`--zip-dat-name`) - ROMs are written to the output directory, if specified (`copy`, `move`, `symlink`) - Written ROMs are tested for accuracy, if specified (`test`) - - A "dir2dat" DAT is created, if specified (`dir2dat`) (see [dir2dat docs](./dats/dir2dat.md)) - - A "fixdat" is created, if specified (`fixdat`) (see [fixdats docs](./dats/fixdats.md)) + - A "dir2dat" DAT is created, if specified (`dir2dat`) (see [dir2dat docs](../dats/dir2dat.md)) + - A "fixdat" is created, if specified (`fixdat`) (see [fixdats docs](../dats/fixdats.md)) 5. "Moved" input ROMs are deleted (`move`) -6. Unknown files are recycled from the output directory, if specified (`clean`, see [cleaning docs](output/cleaning.md)) -7. An output report is written to the output directory, if specified (`report`, see [reporting docs](output/reporting.md)) +6. Unknown files are recycled from the output directory, if specified (`clean`, see [cleaning docs](../output/cleaning.md)) +7. An output report is written to the output directory, if specified (`report`, see [reporting docs](../output/reporting.md)) diff --git a/docs/advanced/logging.md b/docs/advanced/logging.md new file mode 100644 index 000000000..4b554e79c --- /dev/null +++ b/docs/advanced/logging.md @@ -0,0 +1,55 @@ +# Logging + +By default, `igir` will print the following log levels: + +- `ERROR`: an unexpected error has prevented an intended [command](../commands.md) +- `WARN`: a preventable error has prevented an intended [command](../commands.md) + +There are additional levels of verbosity that can be enabled with the `-v` flag: + +- **`INFO` (`-v`): print information about actions taken.** + + This includes: + + - Files being copied, zipped, and symlinked + - [dir2dat](../dats/dir2dat.md) files being created + - [Fixdat](../dats/fixdats.md) files being created + - Input files deleted after being moved + - Output files being [cleaned](../output/cleaning.md) + - [Report](../output/reporting.md) files being created + + Usage: + + ```shell + igir [commands..] [options] -v + ``` + + This level is helpful to turn on if you want to know every action that is resulting in a file being created, modified, or deleted. + +- **`DEBUG` (`-vv`): print information about actions taken and skipped.** + + This includes: + + - Everything from the `INFO` level above + - Files skipped from being copied, zipped, or symlinked because the output file exists and an `--overwrite` option wasn't provided + - [Fixdat](../dats/fixdats.md) files skipped from being created because all games were found + + Usage: + + ```shell + igir [commands..] [options] -v + ``` + + This level is helpful to turn on if you want debug why an action didn't take place. + +- **`TRACE` (`-vvv`): print information about actions taken, skipped, and additional information that can be helpful to debug issues.** + + Usage: + + ```shell + igir [commands..] [options] -v + ``` + + !!! note + + Trace logs are required when submitting [bug reports](https://github.com/emmercm/igir/issues/new/choose) as they include information that can help diagnose your unique situation! diff --git a/docs/advanced/troubleshooting.md b/docs/advanced/troubleshooting.md new file mode 100644 index 000000000..9e0ae0fc5 --- /dev/null +++ b/docs/advanced/troubleshooting.md @@ -0,0 +1,49 @@ +# Troubleshooting + +See the sibling page about [logging](./logging.md) for more information to help troubleshoot issues. + +## Fatal crashes + +### Heap space out-of-memory + +These types of crashes will print something similar to this to the console: + +```text +<--- Last few GCs ---> + +[5789:0x3f6eca0] 79353 ms: Scavenge 2020.1 (2075.5) -> 2016.5 (2076.5) MB, 9.57 / 0.00 ms (average mu = 0.566, current mu = 0.422) allocation failure; +[5789:0x3f6eca0] 79403 ms: Scavenge 2021.2 (2076.5) -> 2018.3 (2079.5) MB, 11.10 / 0.00 ms (average mu = 0.566, current mu = 0.422) allocation failure; +[5789:0x3f6eca0] 80721 ms: Scavenge 2024.0 (2079.5) -> 2020.6 (2097.0) MB, 1280.87 / 0.00 ms (average mu = 0.566, current mu = 0.422) allocation failure; + + +<--- JS stacktrace ---> + +FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory +----- Native stack trace ----- + + 1: 0xbd3a28 node::Abort() [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 2: 0xaa1af8 [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 3: 0xdfa1a0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 4: 0xdfa574 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 5: 0x10197b7 [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 6: 0x1031839 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 7: 0x100b547 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 8: 0x100c184 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] + 9: 0xfec37e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] +10: 0x143b35c v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/nix/store/d6lkbndr98lcn8spbqxfq52f2ldvqhks-nodejs-20.11.0/bin/node] +11: 0x7fe14fed9ef6 +``` + +The issue is that `igir` ran out of memory likely due to low system limits, large DAT packs, or large ROM collections. + +You likely need to process your ROM collection in batches, just be careful when using the [`igir clean` command](../commands.md). If you don't need every DAT from a pack, you can try reducing the number of DATs being processed with the [`--dat-*-regex ` and `--dat-*-regex-exclude ` options](../dats/processing.md#dat-filtering) like this: + +```shell +igir [commands..] --dat "*.dat" --dat-name-regex "/nintendo/i" +``` + +You can also try increasing V8's "old memory" limit above [Node.js' default (version dependent)](https://medium.com/geekculture/node-js-default-memory-settings-3c0fe8a9ba1) to ~75% of your system's available RAM in megabytes like this: + +```shell +npx --node-options='--max-old-space-size=6144' igir@latest [commands..] [options] +``` diff --git a/mkdocs.yml b/mkdocs.yml index 51eae91ff..81e8c0859 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -96,7 +96,9 @@ nav: - output/reporting.md - output/cleaning.md - Advanced: - - internals.md + - advanced/logging.md + - advanced/troubleshooting.md + - advanced/internals.md - Misc: - rom-dumping.md - Terms and Conditions: @@ -130,6 +132,7 @@ plugins: 'examples.md': 'usage/collection-sorting.md' 'input/archives.md': 'input/reading-archives.md' 'input/dats.md': 'dats/introduction.md' + 'internals.md': 'advanced/internals.md' 'output/arcade.md': 'usage/arcade.md' 'reporting.md': 'output/reporting.md' 'rom-filtering.md': 'roms/filtering-preferences.md'