36 occurrences where different JSON objects share same type and id

[int-ua]

Describe the bug

While preparing to make tools/json_tools/util.py work with copy-from I've discovered not only that there are >500 instances where same id is used on different JSON objects (which is not ideal but seems to be expected) but also that there are 36 cases where they share both type and id, which is questionable.

Lists by status

Included in the PR:

type	id
AMMO	oxygen
event_statistic	num_broken_bone
harvest	demihuman
json_flag	ELECTRIC_IMMUNE
ammo_effect	LIGHTNING
trait_group	Exp_Eng_Electrical_Group
MONSTER	mon_wasp_small
effect_type	magnesium
MIGRATION	l_enforcer_45
ammunition_type	oxygen
TOOL	tool_anfo_charge
TOOL	tool_anfo_charge_act
GENERIC	cable_instrument
item_group	light_reading
item_group	my_precious
item_group	mussto_windinst
item_group	mussto_stringinst
item_group	ic_merch_vending
item_group	ic_merch
mission_definition	MISSION_ISHERWOOD_JESSE_2
talk_topic	TALK_JESSE_CARLOS
talk_topic	CWH_EVACUEE_5_IDEAS1
talk_topic	TALK_LEAVE_NOW
talk_topic	TALK_TEST_SPEAKER_EFFECT_COMPOUND_SENTINEL_CONDITIONAL

@mlangsdorf is fixingfixed these:

type	id
vehicle_part	seat
vehicle_part	seat_leather
vehicle_part	reclining_seat
vehicle_part	reclining_seat_leather

Intentional:

type	id
talk_topic	TALK_FRIEND_CONVERSATION

Steps To Reproduce

diff --git a/tools/json_tools/util.py b/tools/json_tools/util.py
index de33d82..61364b5 100755
--- a/tools/json_tools/util.py
+++ b/tools/json_tools/util.py
@@ -3,7 +3,7 @@
 """

 import argparse
-from collections import Counter, OrderedDict
+from collections import Counter, OrderedDict, defaultdict
 from fnmatch import fnmatch
 import json
 import re
@@ -42,6 +42,15 @@ def import_data(json_dir=JSON_DIR, json_fmatch=JSON_FNMATCH):
                             errors.append("Problem parsing data from file %s, reason: expected a list." % json_file)
                     else:
                         data += candidates
+    all_ids = defaultdict(set)
+    for obj in data:
+        obj_id = obj.get('id')
+        obj_type = obj.get('type')
+        if obj_id and not isinstance(obj_id, list):
+            if obj_id not in all_ids[obj_type]:
+                all_ids[obj_type].add(obj_id)
+            else:
+                print(obj_type, obj_id)
     return (data, errors)

Expected behavior

At least type and id together provide enough info to identify a JSON object globally. But then, there is abstract now too. :(

Versions and configuration

My fork, about 40 commits behind right now.

Additional context

#42958
#42959
https://discordapp.com/channels/598523535169945603/598535827169083403/744539312674308138

[jbytheway]

Did you include mod data in this?

[int-ua]

No, should I?

[jbytheway]

I think that some mods deliberately reuse ids to overwrite the base game version of an item (not totally sure if this is a thing or not) so if you had included mods then that might explain some of this, but given that you haven't all these are probably genuine bugs. Thanks for finding them. Someone with permissions should make this issue a blocker for the release, otherwise it's liable to get lost.

[int-ua]

Can someone help with reviewing talk_topic duplicates and deciding what should be done?

[mlangsdorf]

The TALK_FRIEND_CONVERSATION stuff is legal and intentional.

The TALK_LEAVE_NOW repeats look like a bug.

I'll fix the duplicate seat entries.

[int-ua]

I've updated OP and split the table by status.

[int-ua]

On AMMO oxygen: original was added here and then a duplicate was added here. I'm not sure if I should remove the original, please advise. @KorGgenT

[int-ua]

Removed the old one in the PR.