Error "Failed to read RomFS directory cache!" occurring on very recent Nintendo published games

[RQWorldblender]

This program has served me very well for many years, but it seems that it may come to an end for some newer games, unless this issue can be fixed soon.
For most Nintendo published games released within the past 5 months this year, I've ran into errors being "Failed to read RomFS directory cache!" for the following titles (specifically with the NCA files holding the actual game data, not those containing stuff like icons and IP notices):

• Luigi's Mansion 2 HD [010048701995E000] (2024/06/27)
• The Legend of Zelda Echoes of Wisdom [01008CF01BAAC000] (2024/09/26)
• Super Mario Party Jamboree [0100965017338000] (2024/10/17)

The most recent game I was able to extract game data from:

• Emio - The Smiling Man Famicom Detective Club [0100A9501759E000] (2024/08/29)

Games I haven't extracted data from yet:

• Nintendo World Championships NES Edition [0100AD10185F8000] (2024/07/18)

I suspect that this issue is related to #109 (compressed RomFS), since the ExeFS can be opened successfully. There will most likely be more Nintendo published games that utilize a compressed RomFS, and I fear that this issue will only get worse if this is not fixed at all. There may also be 3rd party games with this same issue, but my issue only focuses on Nintendo published games.

Games announced but not yet released with release dates:

• Mario & Luigi: Brothership (2024/11/07)
• Donkey Kong Country Returns HD (2025/01/16)
• Xenoblade Chronicles X: Definitive Edition (2025/03/20)

On a side note, I should also mention that the latest firmware version is now 19.0.1 as of this post, and version 19.0.0 introduced another master key revision, 0x12. The info section doesn't seem to be able to report which key goes to which firmware version past 10.0.0 (but it's something I can fix). The games that which have the error I'm running into have master key revisions no earlier than 0x10.

My workaround for the time being is to use nxdumptool (https://github.com/DarkMatterCore/nxdumptool), specifically its rewrite that supports USB devices to dump any of these problematic games, but it works only as long as my Nintendo Switch is still working. If it breaks or I no longer have it for any reason, I will be unable to dump these and likely future Nintendo published games.

I'm capable of helping out if necessary; I just don't want to lose access to dumping newer Nintendo published games even if I no longer have a Nintendo Switch.

[SciresM]

I began working on https://github.com/Atmosphere-NX/hac2l to replace this tool, but got busy, and then distracted, and kind of forgot about it.

I suppose I really, really should finish hac2l. But that said, hac2l should already support extracting games with compressed romfs, to my knowledge. Does it? If not, what error is returned?

[RQWorldblender]

I began working on https://github.com/Atmosphere-NX/hac2l to replace this tool, but got busy, and then distracted, and kind of forgot about it.

I suppose I really, really should finish hac2l. But that said, hac2l should already support extracting games with compressed romfs, to my knowledge. Does it? If not, what error is returned?

Thank you for letting me know about this program, I just learned about this the moment you mentioned it. Since I haven't heard of this program before, I'll have to give it a try sometime to see if it will solve my issue in question. What makes you say that hac2l isn't complete, I wonder?

Can it also extract from sparse NCA files? Two Nintendo published (or exclusive) games are already using this, but no 2024 titles are using them.

• Bayonetta 3 [01004A4010FEA000]
• Splatoon 3 [0100C2500FC20000]

Back to hactool: I actually looked through the code before when I first ran into this issue last month, and this code section in ./romfs.c may tell me what's going on (lines 83 to 87):

        fseeko64(ctx->file, ctx->romfs_offset + ctx->header.dir_meta_table_offset, SEEK_SET);
        if (fread(ctx->directories, 1, ctx->header.dir_meta_table_size, ctx->file) != ctx->header.dir_meta_table_size) {
            fprintf(stderr, "Failed to read RomFS directory cache!\n");
            exit(EXIT_FAILURE);
        }

I don't get any detailed error messages besides what I already mentioned, and using strace didn't help me that much. Going by this code section, it looks like a compressed RomFS can cause this to happen.

[SciresM]

Yes, hac2l has full support for both compressed and sparse NCA files. It's the non-NCA files that it doesn't really have proper support for yet, as well as e.g. partially extracting what it can from NCAs which are corrupted in some way.

[Whovian9369]

If you want a PC command line application that's actively maintained (with no disrespect towards SciresM) and works with Sparse or Compressed NCAs, I'd suggest hactoolnet instead. It should do what you want without having to mess with hactool code or the not super actively maintained hac2l.

[SciresM]

None taken. I should really finish it though....

[RQWorldblender]

If you want a PC command line application that's actively maintained (with no disrespect towards SciresM) and works with Sparse or Compressed NCAs, I'd suggest hactoolnet instead. It should do what you want without having to mess with hactool code or the not super actively maintained hac2l.

I've heard of hactoolnet before, but I just wasn't sure whether it could run on Linux distros. Now that I see that there's Linux packages in the release section, as well as .NET already being supported on Linux distros, I feel more confident in giving hactoolnet a try (I can compile it myself if all else fails). I think if it's primarily command line, it should run without much issue, as long I can reproduce extracting files without corruption in the same behavior that hactool does.

[RQWorldblender]

None taken. I should really finish it though....

Perhaps you can tell me what's unfinished about hac2l? I might be able to help since I have some coding experience, but anything requiring more specialized knowledge means I may take longer to make contributions.