fix: Datatime formatter in small dataset and improve performace

[cyantangerine]

Description

When trainning ctgan, we used the DataLoader to load data as chunk. While, due to the historical reasons, DatetimeFormatter using a simple list to format columns. When we using DataLoader, the formatter format the data by chunk, which lost it's index. So, when we concat the next formatted chunk column to chunk table, the table result (index beginning by 1*chunk_size) will be NaN becase it can not match the zero-based index of chunk column.

Motivation and Context

I changed the method of formatting, instead of using the 'apply' function of DataFrame/Series. It fixed the problem and improved the performance by using 'for' cycling.

How has this been tested?

A whole test for DatetimeFormatter has been given.

Types of changes

• Maintenance (no change in code, maintain the project's CI, docs, etc.)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[cyantangerine]

I suggest that we should have a check for all code to replace 'for' to pandas.apply morely. I saw a lot of implement like this in code. It can improve the performance a lot.

[jalr4ever]

I suggest that we should have a check for all code to replace 'for' to pandas.apply morely. I saw a lot of implement like this in code. It can improve the performance a lot.

Thank you very much! Regarding this issue, I think it would be best to have a use case that can immediately demonstrate the system's performance problems, so we can then discuss whether to make this optimization. If there is interest in this performance issue, we can create an ISSUE for separate analysis and tracking.

[Wh1isper]

Looks reasonable for me, @jalr4ever what do you think?

[Wh1isper]

And yes, for is usually bad in performance. I've open an issue for it #245. If anyone has interested in it, feel free to draft a PR!

[cyantangerine]

@Wh1isper
Something to attention: It's a bug.

Due to the original method, when the dataset length is bigger than DataLoader chunksize, the result of processed data is FULL of NaN except the first chunk.

[cyantangerine]

default chunk_size = 10000
so for 15000 length dataset, it has 5000 NaN