[Lens] Inspector and CSV download does not filter partial columns from formula

[dej611]

Describe the bug:

Partial columns (hidden, generated by the formula's feature) are exported and downloaded in the CSV file generated.

Steps to reproduce:

1. Create a Lens XY chart with a formula like unique_values( fieldA ) / count()
2. Download the CSV
3. 3 columns for unique_values( fieldA ) will be present as Part-of X , together with the final formula column (all with the same colum name)

Expected behavior:

Part-of but in general hidden columns should be removed from the export.
Eventually, it can be considered to let partial results be downloaded as an extra feature, but only on user's request and with a better column description.

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

@ghudgins Discussed with the app services team how to solve this and we have a good plan, but there is one product question related to this - do we want to offer a "mode" for Inspect view and/or CSV export to include these intermediary columns? Or should we just always filter them? I'm leaning towards filtering them at all times to keep things simple - what do you think?

Technically, Lens should handle this like Visualize charts - the vis expression function should use its vis-specific knowledge to only register the actually needed columns in the inspector adapter.

[ghudgins]

I'm leaning towards filtering them at all times to keep things simple - what do you think?

agreed - I wouldn't expect the intermediate columns in the output as a user. keep it simple seems like the right call here...can always reconsider in the "more complexity" direction a lot easier than the other way around

[flash1293]

Sounds good - cc @dej611 we know how to proceed in this case. This is how all of the vis functions should do it (including the Lens ones):

kibana/src/plugins/chart_expressions/expression_heatmap/common/expression_functions/heatmap_function.ts

Line 198 in dd9d846

const logTable = prepareLogTable(data, argsTable);

[flash1293]

@alexwizp I think #126786 is also fixing this one, right?

[zakwanhaj]

I have the same problem. It's very annoying. I hope you can include it in the short term plan and that it will be fixed soon.

[flash1293]

@zakwanhaj Actually I opened a PR yesterday that will fix this issue: #128297

It will be released with the next upcoming version 8.2