Suggestion: Optional mod provided metadata

[MattMills]

Is your feature request related to a problem? Please describe.

Stellaris (and other games I assume) needs a better mod metadata format to improve mod compatibility and reduce "support cost" for known issues that can relatively easily be solved. Ideally it'd be up to the game developer to implement such a format, but since they aren't I figured I'd come discuss it here and hopefully could demonstrate the usefulness and "reverse engineer" better functionality into the games.

Describe the solution you'd like

It would be ideal if someone defined an optional metadata file format that mods could include to define:

1. Other mods that should be loaded before this mod
2. Other mods that should be loaded after this mod
3. Other mods that should not be loaded with this mod (known conflicts)
4. Other mods that resolve conflicts between this mod and a second mod (patch mods for known conflicts)
5. Other mods this mod is dependent on (required)
6. Other mods this mod suggests
7. More complex version compatibility language (IE minimum 3.0.1, maximum 3.5.3, or alternatively just being able to specify multiple entries for compatible versions rather than a single entry, personally I'd also like >= minimum with no maximum).
8. A mod unique identifier that can authoritatively "resolve" multiple different platform IDs into a single mod.
9. A mod version identifier (not sure if this is a good idea TBH, might be better to use a hash of all files or a hash of hashes).

I think for first pass completely ignoring mod versioning might be the best strategy...

Describe alternatives you've considered
RimPy implements this for rimworld as a "community rule list"... which might also be a useful functionality for mods that chose not to / refuse to provide their own metadata...

Additional context
Similar implementations:

https://wiki.factorio.com/Tutorial:Mod_structure#Dependency

https://rimworldwiki.com/wiki/About.xml

https://github.com/rimpy-custom/communityRules/releases

[bcssov]

Recently had a discussion on Irony discord in regards to something like that.

TLDR; What is good such a feature if community will not use it? It's going to be tedious to advocate such a feature and modders won't certainly tell people please use Irony. Just look at CWTools, lots of users very few contributors.

[lyratcm]

While that is true for CWTools there are a couple of reasons I don't think it applies to this.
firstly CWTools mostly works, other than more edge case stuff like parmatise scripts, the tool is going to get stuff right so there's not much of a need to work out how to improve it, unlike automated mod ordering which is non-existence outside of z-a and pray

secondly improving CWTools is more complicated than adding more metadata is, adding stuff to tools required a basic level of coding knowledge and interesting learning how the back end works. whereas the metadata can be made far more simple.
if the metadata is similar to rimpy in complexity the user would only need to learn a mods id, name and the name of the other mods that it is being ordered around reducing barrier to entry

lastly, updating isn't as needed, once a mod has its relations added unless it is updated for more compat the part of the file don't need to be touched, unlike CWTools that any file can be needed to be updated when pdx updates the script so even if there is a low amount of updaters its less of an issue. also, the vast majority of users use the main mods e.g. gigas, uiod or NSC so only a couple of mods relations would be needed to be super focused on and amount them most take steps to avoid overwrites so reducing the amount of work needed further

[TTFTCUTS]

Heck, I've tried to take a look at contributing to cwtools myself, and despite a pretty decent level of programming experience I can't make heads nor tails of F#. I'm at the point where I am seriously considering trying to write my own tool...

As for the metadata, I think it'd probably see a higher level of adoption, since really it's not a whole lot of work to add this kind of data to a mod and many authors are willing to add larger compat stuff. I bet even more so if it means fewer bug reports due to users ordering things incorrectly. I'm certainly willing.

(Hi, I'm currently the lead on Gigastructural Engineering)

[bcssov]

I'm not talking about CWTools source code (main repo) I'm talking about the stellaris config. Without it CWTools will not really be what it is. I'm using that as a baseline comparison.

In the words of the CWTools author: Stellaris cwtools config is mostly maintained by a pdx employee. HOI4 is maintained by some really large modding groups. Other games lack many of the resources that these 2 games have, Victoria 3 seems to be kicking up (at least Dayshine kickstarted it).

[MattMills]

To add from my perspective, pretty much all the mods I am familiar with advocate for using Irony over the Paradox launcher due to the multitude of issues the Paradox launcher seems to cause.

I can't speak to the CWTools issues, largely because I don't really do much in the paradox scripting language (I generally try to avoid it) and don't know it well enough to contribute to any syntax tools.

I think the simplest approach would just be to add a file that mods can include to provide metadata, but in the Rimpy approach, where there is a "community" database is where I personally am most interested in the possibilities, largely because I think there are significant possibilities for automated analysis and automated testing.

For example, the capability to batch modifier updates was added in Stellaris 3.0.1, so any mod that uses that is incompatible with any older version. One of the things I've been interested in doing is building a "performance baseline" testing mod, something that will allow an automated test to run on each new version of the game and baseline the performance of a series of scenarios that have been or are currently problematic (IE: measuring the frame rate with a large number of fleets in a single system); basically, having a setup that can have test cases added to it and be performance measured repeatable over time against each version of the game to track improvements or regressions. I think one of the rather easy things to add to this would be taking popular mods (or mod combinations that have been identified as problematic somehow) and doing either a performance baseline or a functionality baseline test (IE, does combining mod x and y cause the game to crash at startup).

Ultimately I think you are correct, in that this is something that the games itself need to require, but I don't think that will happen without an existing implementation that is showing usefulness.

[bcssov]

OK. So full disclosure about what Irony was supposed to have when I started the project. It's been mentioned briefly before but never got from the ground (lack of help) so I pretty much shelved it. It's also the main reason why I'm still hesitant to pick up anything related to it.

I've never played the games you're referencing but I did play KSP. And my inspiration(s) come from there. Irony was intended to have a similar db such as this but based upon CKAN (comprehensive kerbal archive network). Given how Irony came in late to the scene (3-4 years) it was difficult to advocate for such a feature. CKAN already has metadata which could have been repurposed for Stellaris ie. requirements, min\max version, recommends, conflicts with and so on. Some metadata of course is missing but you get the idea: https://github.com/KSP-CKAN/CKAN-meta/blob/master/APPLE/APPLE-1-1.1.9.ckan
This repository is the central db from which you pull mod metadata so transparent and easy to use (to me at least). Of course Irony would not be able to download mods (maybe one day once I figure out legal stuff).

Once this was solved then there would have been phase 2, which would have been based on ModuleManager: https://github.com/sarbian/ModuleManager/wiki/Module-Manager-Syntax
Ever since I wrote my first scripts (via python template automation for some repetitive stuff, you see I wanted to make initially a transpiler -- go figure) the whole thing pretty much reminded me (just reminded) of KSP configuration files. And the constant problems you had with multiple mods editing the same stock files which resulted with conflicts (vanilla support has improved greatly). For KSP this was solved by module manager. The crude example for stellaris which I used as an example:

[country_types:swarm]:BEFORE[mod-id1]:AFTER[mod-id2]
{
 ai = {
   enabled = no
 }
}

The syntax states: Patch country type and (swarm) before mod-id1 and after-mod-id2 and add the following line(s). Pretty sure you all understood what it means but just in case.

This whole logic should have (hope) reduced number of compatibility patches. This was the reason why I wrote 2nd iteration of Irony's parser (a general dumb parser) due to many reasons. One of them due to CWTools edge cases (not understanding how to parse HSV - I think), failing to parse certain name scripts. And more importantly to allow the development of this feature since most likely the paths of Irony and CWTools would diverge therefore a dedicated parser would be needed.

The idea was abandoned due to arguments:

• That would mean I'd have to tell people to use Irony
• Not many people want to use third party tools
• Lack of interest
  ...

And we're here at the present.

[Siveen]

Recently had a discussion on Irony discord in regards to something like that.

TLDR; What is good such a feature if community will not use it? It's going to be tedious to advocate such a feature and modders won't certainly tell people please use Irony. Just look at CWTools, lots of users very few contributors.

I think if we can convince major mod dev (leads) to adopt this format, the community will follow suit over time. (TTFTCUTS could provide some insight in how feasible it would be to adopt it in Gigaengineering/how much resistance there would be?)
If we could convince Orrie (UI Overhaul Dynamic), CaptainX3 (NSC2, though unlikely I think?), Gatekeeper (PD mods) and Guilliman (Planet Modifiers and Features amongst others) as well and have them push for mass adoption of IMM, I think we'd have a very strong case to make. Might be a long push however.
There's also a discussion to be had about the rules being applied. Are they community provided? -> you need a Community to provide them first and foremost. Are they mod author provided? ->What if a mod author provides complete bollocks? Do you step in and tell them they're in the wrong? Then you'd have to either manually add a rule that ignores the mod authors defined rules or leave it be and have it cause issues; more upkeep on maintaining rules as well.
What about a mix of both? Ask mod creators to submit the rules to the community repo to be reviewed by the community repo maintainers? Which still sort of leaves you with the community to provide them issue.
I have no easy answer to any of these. It might not be a bad idea to ask Paladin (RimPy Community Rules Maintainer atm AFAIK) how that project got started.

Of course Irony would not be able to download mods (maybe one day once I figure out legal stuff)

CKAN does not download mods. CKAN uses SteamCMD to download mods (and also helps the user to set SteamCMD up). SteamCMD is covered by the Steam Workshop Agreement AFAIK, so there should be no legal issues involved. (I am not a lawyer, this is not legal advice.) Have you talked to HerbaruSan about this by any chance? He can probably give you more insight on the issue.

[bcssov]

CKAN does not download mods

Wrong. It does:

• Github
• CurseForce
• Spacedock
  Probably more.

Also, I haven't spoken to anyone about anything.

[MattMills]

I think the scripting-patch-language that you mention would be going much further than necessary, and would likely have issues with adoption as you mentioned; but I don't think a simple file that points out recommended load order and incompatibilities between mods would.

As for downloading mods, personally I don't think that's very important, the current functionality of Open in Steam or Open in browser works fine.

[Siveen]

CKAN does not download mods

Wrong. It does:

• Github
• CurseForce
• Spacedock
  Probably more.

Also, I haven't spoken to anyone about anything.

My mistake. RimPy does that in any case. Are you considering reaching out? I don't see a reason why they would refuse a short exchange on information.

[bcssov]

Are you considering reaching out?

Not really. This feature is not on the roadmap and all the theoretical knowledge is pretty much acquired. Lastly you just don't drop unannounced to people you've never met and start bothering them with random questions (CKAN etc).

[lyratcm]

so i feel like theres 2 ides at play in this post #416 (comment)
part 1 which is a auto mod order which I can see lot of promise and avoids some of the issues,
That would mean I'd have to tell people to use Irony
once the mod set has been ordered irony isn't required anymore so can be shared with others without forcing them to use it.

Not many people want to use third party tools
this is still an issue either way but having to press 1 button to order it rather than learning mod conflict merging is a far lower barrier

Lack of interest
while I again agree the sort of person who going to express an opinion about irony need a feature is those who have used it for a while and probably know ordering/merging by experience.
I am a mod on the stellaris official discord and the most common questions I see about mods is asking how to order stuff? or why their game isn't working with mods?, both of which can be fixed with an auto mod ordering (lots of newer ppl issues with mods breaking is from order/a lack of patches)

last note it could be possible to have the mod tools in some webpage where someone uploads their list and it orders it for them, removing most of issues 1/2. but that's a lot of extra work for that

[corsairmarks]

As a modder, I'm interested in providing this metadata. Preferably it could be some form of text file that I include in the root mod directory. I vote for something json-formatted since that is easy to parse in many programming languages, and won't require additional tools to be used by modders to enter the data into an external website (etc).

Stellaris files have two priority systems. I often choose to use per-item overrides rather than full-file overrides, and mod load order means nothing to partial overrides. I know Irony provides some levels of game-object conflict detection, but some sort of spec for including that info in metadata is attractive to me. As an example, a modder had to reorder their mods before Irony detected multiple, conflicting overrides for a vanilla building. My mod in question was Building: Aquaponics Farm but I unfortunately don't know the other conflicting mods.

[bcssov]

Full disclosure, I ran a poll to see the community interest here are the results after a month or 2 of it running:

Democratic majority vote wins. Users and modders.

Now let's move this in discussion. About specs, how would it be acceptable.

For a start the idea is to simply have in your mod folders a piece of info storing a "remote id" lookup. The whole database would be hosted on GitHub and community maintained.

@MattMills @TTFTCUTS @corsairmarks @lyratcm