feat: OneHotEncoder.inverse_transform now maintains the column order from the original table

[Marsmaennchen221]

Closes #109.

Summary of Changes

OneHotEncoder.inverse_transform now maintains the column order from the original table (#109)
Fixed bug with OneHotEncoder.inverse_transform to not work if not all columns were fitted
New feature columns in OneHotEncoder will now be inserted where the combined columns were in the original table

[codecov]

Codecov Report

Merging #195 (b1856b1) into main (bea976a) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #195      +/-   ##
==========================================
+ Coverage   93.97%   94.01%   +0.04%     
==========================================
  Files          42       42              
  Lines        1476     1486      +10     
==========================================
+ Hits         1387     1397      +10     
  Misses         89       89              

Impacted Files	Coverage Δ
...ds/data/tabular/transformation/_one_hot_encoder.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[lars-reimann]

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
✅ PYTHON	black	2	0	0	0.63s
✅ PYTHON	mypy	2		0	1.65s
✅ PYTHON	ruff	2	0	0	0.04s
✅ REPOSITORY	git_diff	yes		no	0.02s

See detailed report in MegaLinter reports
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by

[lars-reimann]

Solving this issue is a little more complicated than that: Say you call fit with this table and run the encoder only on column col1:

col1	col2
"a"	0
"b"	1

You store the schema of this table in self._original_schema. Now you call transform with the following table:

col2	col1
0	"a"
1	"b"

If you now call inverse_transform on the result, you will get

col1	col2
"a"	0
"b"	1

because the relative order of columns is now determined by the table the transformer was fitted with. This is not the table before transform, though.

A more thorough solution would be that transform already maintains the order of columns by replacing a column with its one-hot-encoded version instead of appending all new columns at the end. For example, calling transform on

col2	col1
0	"a"
1	"b"

should result in

col2	col1_a	col1_b
0	1	0
1	0	1

and calling transform on

col1	col2
"a"	0
"b"	1

should result in

col1_a	col1_b	col2
1	0	0
0	1	1

Then the inverse_transform can simply maintain the column order.

[lars-reimann]

I've added another test that should work but doesn't.

Since the column name and the value could contain underscores, it's not possible to figure out which original column a column in the transformed table corresponds to. For example, a_b_c could correspond to the column called a_b (and value c) or a column called a (and value b_c).

Because of this, we'll need to store the mapping from original columns to transformed columns during fit (dict[str, list[str]], with original column names as keys and names of corresponding transformed columns as values). This will also be useful for #190 later. During transform and inverse_transform you can then use this mapping to figure out how the columns need to be sorted instead of relying on the startswith check.

[lars-reimann]

I've created another issue (#201) about conflicting column names when using a OneHotEncoder. That can be done in a separate PR.

[lars-reimann]

🎉 This PR is included in version 0.11.0 🎉

The release is available on:

• v0.11.0
• GitHub release

Your semantic-release bot 📦🚀