[Bug] 无法将datetime类型数据转换成日期格式

[weilai201]

Search before asking

• I had searched in the issues and found no similar issues.

Version

1.3.1

What's Wrong?

doris-spark-connector无法将doris库中datetime类型数据在spark中转换为java Date或timestamp类型。目前统一转换为String类型，对于使用非常不方便！

What You Expected?

doris中的datetime类型转换为java中的Date或Timestamp类型

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[gnehil]

Thanks for your suggestion, there are some incompatibilities between spark2 and spark3, we are working on fixing this issue.

[dingyufei615]

Would it be possible to add a parameter that allows users to choose whether to convert datetime to timestamp? In Spark, using string-type dates can sometimes cause confusion and require additional conversion steps for users.

[gnehil]

Would it be possible to add a parameter that allows users to choose whether to convert datetime to timestamp? In Spark, using string-type dates can sometimes cause confusion and require additional conversion steps for users.

For the current code structure, all functions are
in one module. Since different spark versions map the date and timestamp
types in catalyst to different java classes, it is difficult to be
compatible with different versions of the API.
But the good news is that an important refactoring is underway and will be
released in the next version.
So after this, different data deserialization can be performed for
different spark versions, and this problem will not be difficult to solve.
This optimization can be planned to be released in the version after next,
and you are also welcome to contribute this optimization to the community.

[dingyufei615]

Would it be possible to add a parameter that allows users to choose whether to convert datetime to timestamp? In Spark, using string-type dates can sometimes cause confusion and require additional conversion steps for users.

For the current code structure, all functions are in one module. Since different spark versions map the date and timestamp types in catalyst to different java classes, it is difficult to be compatible with different versions of the API. But the good news is that an important refactoring is underway and will be released in the next version. So after this, different data deserialization can be performed for different spark versions, and this problem will not be difficult to solve. This optimization can be planned to be released in the version after next, and you are also welcome to contribute this optimization to the community.

Nice! Has the refactored code been merged already? I would be happy to contribute code for this optimization.

[dingyufei615]

Search before asking

• I had searched in the issues and found no similar issues.

Version

1.3.1

What's Wrong?

doris-spark-connector无法将doris库中datetime类型数据在spark中转换为java Date或timestamp类型。目前统一转换为String类型，对于使用非常不方便！

What You Expected?

doris中的datetime类型转换为java中的Date或Timestamp类型

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

我也遇到同样的问题了，目前我临时的解决办法是调整了部分代码，rowBatch处理的时候把datetimev2转为timestamp，
https://github.com/dingyufei615/doris-spark-connector/tree/support_datetimev2_to_timastamp
新增了一个参数doris.read.datetimev2.as.timestamp.enabled ，设置为true 即可，我当前使用的是Spark3.3 Scala2.12，目前测试使用没遇到问题。社区后面也说了会重构这部分，如果不着急的话可以等后续的版本。

[weilai201]

Search before asking

• I had searched in the issues and found no similar issues.

Version

1.3.1

What's Wrong?

doris-spark-connector无法将doris库中datetime类型数据在spark中转换为java Date或timestamp类型。目前统一转换为String类型，对于使用非常不方便！

What You Expected?

doris中的datetime类型转换为java中的Date或Timestamp类型

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

我也遇到同样的问题了，目前我临时的解决办法是调整了部分代码，rowBatch处理的时候把datetimev2转为timestamp， https://github.com/dingyufei615/doris-spark-connector/tree/support_datetimev2_to_timastamp 新增了一个参数doris.read.datetimev2.as.timestamp.enabled ，设置为true 即可，我当前使用的是Spark3.3 Scala2.12，目前测试使用没遇到问题。社区后面也说了会重构这部分，如果不着急的话可以等后续的版本。

👍！我目前也是通过修改源码做了支持！期望你的代码能够合并进去