[Bug] [dolphinscheduler-api] HDFS failed to upload large file #10340

github-keep · 2022-06-02T07:16:04Z

Search before asking

I had searched in the issues and found no similar issues.

What happened

在配置HDFS之后使用资源中心上传过大的文件（90MB）失败，小文件可以。猜想：不能上传超过HDFS数据块大小（64MB）的文件，查看日志错误信息：没有合适的分解器。

What you expected to happen

在之前版本（2.0.5）中，该功能正常可以使用，升级版本3.0.0-alpha，3.0.0-beta-1均不能正常使用。

How to reproduce

集群部署完成后，配置HDFS，使用的Hadoop 集群 NameNode 配置了 HA ，完成之后上传一个128MB（超过HDFS数据块）以上的文件

Anything else

问题每次都出现，相关的日志再安装目录下面的api-server/logs下的最新文件中

Version

3.0.0-beta-1

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

github-actions · 2022-06-02T07:16:28Z

Search before asking

I had searched in the issues and found no similar issues.

What happened

After configuring HDFS, uploading too large files (90MB) using the resource center fails, but small files are ok. Guess: cannot upload files larger than the HDFS data block size (64MB), check the log error message: no suitable decomposer.

What you expected to happen

In the previous version (2.0.5), this function can be used normally, but the upgraded version 3.0.0-alpha and 3.0.0-beta-1 cannot be used normally.

How to reproduce

After the cluster is deployed, configure HDFS, use the Hadoop cluster NameNode configured with HA, and upload a file over 128MB (more than HDFS data blocks) after completion

Anything else

The problem occurs every time, and the relevant logs are in the latest files under api-server/logs under the installation directory.

Version

3.0.0-beta-1

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

github-actions · 2022-06-02T07:16:45Z

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

EricGao888 · 2022-06-09T02:13:20Z

I encountered this error, too. May I ask is there any update or potential solutions to this issue?

fhygh · 2022-06-09T12:01:12Z

i upload 200+M file is ok, 3.0.0 beta-1 version

GodWon · 2022-06-20T01:24:08Z

我上传 200+M 文件没问题，3.0.0 beta-1 版本

我300M也没报错传900M的报错了

SunHeng98 · 2022-06-24T06:40:46Z

I encountered this error, too. May I ask is there any update or potential solutions to this issue?

你好，请问解决了吗？我也遇到这个问题了，好像是上传时间超过15秒就会自动取消。

yaowj2 · 2022-07-07T06:43:14Z

从debug springboot报的实际异常是org.eclipse.jetty.io.EofException: Early EOF。原因在前端用axios 提交时设置了超时时间为15s，导致在上传过程中前端超时提前中断了，见下面在ui service.ts的代码片段部分，修改加大timeout时间后上传大文件通过
const baseRequestConfig: AxiosRequestConfig = { baseURL: import.meta.env.MODE === 'development' ? '/dolphinscheduler' : import.meta.env.VITE_APP_PROD_WEB_URL + '/dolphinscheduler', timeout: 15000,

CriysHot · 2022-07-12T07:17:07Z

从debug springboot报的实际异常是org.eclipse.jetty.io.EofException: Early EOF。原因在前端用axios 提交时设置了超时时间为15s，导致在上传过程中前端超时提前中断了，见下面在ui service.ts的代码片段部分，修改加大timeout时间后上传大文件通过
const baseRequestConfig: AxiosRequestConfig = { baseURL: import.meta.env.MODE === 'development' ? '/dolphinscheduler' : import.meta.env.VITE_APP_PROD_WEB_URL + '/dolphinscheduler', timeout: 15000,
Excuse me,i wanna know how to change the conf conveniently？should I compile the code？

EricGao888 · 2022-07-12T08:23:07Z

related: #10509

yangjf2019 · 2022-07-12T08:53:59Z

I'm having the same problem, what specific file configuration needs to be changed, please? The service.ts file was not found. @CriysHot @yaowj2

The file url is https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-ui/src/service/service.ts.

yangjf2019 · 2022-07-12T10:04:59Z

Solved, the following 4 files need to be modified.

/api-server/ui/assets/service.766f4632.js
/api-server/ui/assets/service.766f4632.js.gz

/ui/assets/service.766f4632.js
/ui/assets/service.766f4632.js.gz

and find the following configuration baseURL:"/dolphinscheduler",timeout:15e3,then modify 15e3 to 15e5 or greater.

EricGao888 · 2022-07-12T10:07:08Z

Solved, the following 4 files need to be modified.
/api-server/ui/assets/service.766f4632.js
/api-server/ui/assets/service.766f4632.js.gz

/ui/assets/service.766f4632.js
/ui/assets/service.766f4632.js.gz
and find the following configuration baseURL:"/dolphinscheduler",timeout:15e3,then modify 15e3 to 15e5 or greater.

@yangjf2019 Great job! Would you like to submit a PR to fix this?

yangjf2019 · 2022-07-12T10:10:26Z

Thanks, I can try.

yangjf2019 · 2022-07-12T10:24:36Z

Hi, @EricGao888 is it possible to increase the value of this parameter so that it becomes 30 minutes?

yangjf2019 · 2022-07-12T10:25:42Z

In general, and frankly, it's not recommended to use dolphinscheduler to upload files that are too big!

EricGao888 · 2022-07-12T10:56:42Z

Hi, @EricGao888 is it possible to increase the value of this parameter so that it becomes 30 minutes?

May I ask whether it is possible to make it configurable for users?

yangjf2019 · 2022-07-12T10:58:43Z

Yes, I think it should be done too, please let me take another look, thanks.

zhongjiajie · 2022-07-28T09:25:29Z

Thanks, I can try.

Do you still want to submit pull request to fix this? @yangjf2019

xpf-demo · 2022-08-15T10:25:18Z

我改了那四个js文件，，为啥还是不行啊，有大佬有相同的情况吗

github-keep · 2022-08-17T08:50:47Z

我改了那四个js文件，，为啥还是不行啊，有大佬有相同的情况吗

The larger the file, the longer the upload time. It is recommended to set the time value larger

xpf-demo · 2022-08-18T07:35:45Z

我改了那四个js文件，，为啥还是不行啊，有大佬有同样的情况吗

文件越大，上传时间越长。建议将时间值设置得大一些

谢谢，我的问题已经解决了。我后面修改了js文件之后，清空了下浏览器的缓存之后，就能正常上传了，后台也没有报错了

EricGao888 · 2022-08-23T02:40:01Z

Yes, I think it should be done too, please let me take another look, thanks.

Hello @yangjf2019, may I ask whether you are still working on this issue? We have received feedback from many users that they get blocked by this issue. IMHO we could have a hot-fix for it at the first step, simply increasing the threshold. Then we could move it further and make it configurable for users. Thanks.

EricGao888 · 2022-08-29T16:24:19Z

Yes, I think it should be done too, please let me take another look, thanks.

Hello @yangjf2019, may I ask whether you are still working on this issue? We have received feedback from many users that they get blocked by this issue. IMHO we could have a hot-fix for it at the first step, simply increasing the threshold. Then we could move it further and make it configurable for users. Thanks.

Hot fix #11694

TCGOGOGO · 2023-01-31T07:18:10Z

How about make this configurable? modify code seems tricky

TheWindIsRising · 2023-02-18T03:12:15Z

我也遇到了这个错误。请问这个问题是否有任何更新或潜在的解决方案？

你好，请问解决了吗？我也遇到这个问题了，好像是上传时间超过15秒就会自动取消。

我的版本是3.1.2
存储位置是MinIO
部署方式是K8s使用heml部署的，请问您是怎么解决的

TheWindIsRising · 2023-02-18T03:12:57Z

我也遇到了这个错误。请问这个问题是否有任何更新或潜在的解决方案？

你好，请问解决了吗？我也遇到这个问题了，好像是上传时间超过15秒就会自动取消。

我的版本是3.1.2 存储位置是MinIO 部署方式是K8s使用heml部署的，请问您是怎么解决的

我是上传一个超过200多M的时候发现的问题

TheWindIsRising · 2023-02-18T03:35:52Z

我的版本是3.1.2请问这个问题前端是否解决？

TheWindIsRising · 2023-02-18T03:36:41Z

解决了，需要修改以下4个文件。
/api-server/ui/assets/service.766f4632.js
/api-server/ui/assets/service.766f4632.js.gz

/ui/assets/service.766f4632.js
/ui/assets/service.766f4632.js.gz
并找到以下配置，然后将 15e3 修改为 15e5 或更高版本。baseURL:"/dolphinscheduler",timeout:15e3
@yangjf2019干得好！您想提交 PR 来解决此问题吗？

可否留个联系方式

TheWindIsRising · 2023-02-18T03:59:14Z

/api-server/ui/assets/service.766f4632.js

请问这个文件在哪/api-server/ui/assets/service.766f4632.js
由于我是k8s部署的ds，/ui/assets/service.766f4632.js这个文件我在api的pod里面的ui找到了，但是上面那个不知道在哪

Jacky-YC · 2023-04-22T00:15:35Z

my version 3.1.5

From the results, more than 15 seconds of requests will be canceled

dalaopo · 2023-07-26T06:47:36Z

/api-server/ui/assets/service.766f4632.js

请问这个文件在哪/api-server/ui/assets/service.766f4632.js 由于我是k8s部署的ds，/ui/assets/service.766f4632.js这个文件我在api的pod里面的ui找到了，但上面那个不知道在哪

docker没有 ui/assets/下文件，忽略

github-keep added bug Something isn't working Waiting for reply Waiting for reply labels Jun 2, 2022

SbloodyS assigned github-keep Jun 2, 2022

SbloodyS added backend and removed Waiting for reply Waiting for reply labels Jun 2, 2022

songjianet added this to the 3.0.0-beta-2 milestone Jun 2, 2022

davidzollo added this to DolphinScheduler Roadmap Jun 6, 2022

davidzollo moved this to In Progress in DolphinScheduler Roadmap Jun 6, 2022

devosend modified the milestones: 3.0.0-beta-2, 3.0.0-beta-3 Jul 6, 2022

EricGao888 mentioned this issue Jul 12, 2022

[Bug] [Module Name] error when uploading large files #10509

Closed

3 tasks

yangjf2019 mentioned this issue Jul 25, 2022

[Bug] [dolphinscheduler-api] Upload file failed because the file over 50M #11132

Closed

3 tasks

zhongjiajie assigned songjianet and yangjf2019 and unassigned songjianet Jul 28, 2022

zhongjiajie added the priority:high label Jul 28, 2022

zhongjiajie modified the milestones: 3.0.0, 3.0.1 Aug 5, 2022

EricGao888 added the UI ui and front end related label Aug 23, 2022

EricGao888 mentioned this issue Aug 29, 2022

[Improvement][UI] Hot fix for timeout when uploading large files to resource center #11694

Closed

zhuangchong modified the milestones: 3.0.1, 3.0.2 Sep 18, 2022

SbloodyS mentioned this issue Feb 18, 2023

dolphinscheduler A problem occurred when I tried the last 200M file in the Content Center #13580

Closed

3 tasks

zhongjiajie mentioned this issue May 6, 2023

[Bug] [resource center] The resource center module fails to upload jar files. #14027

Closed

3 tasks

devosend mentioned this issue Jun 2, 2023

[Fix-10340][UI] Fix resource center upload file timeout bug #14268

Merged

zhongjiajie closed this as completed in #14268 Jun 2, 2023

github-project-automation bot moved this from In Progress to Done in DolphinScheduler Roadmap Jun 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] [dolphinscheduler-api] HDFS failed to upload large file #10340

[Bug] [dolphinscheduler-api] HDFS failed to upload large file #10340

github-keep commented Jun 2, 2022

github-actions bot commented Jun 2, 2022

github-actions bot commented Jun 2, 2022

EricGao888 commented Jun 9, 2022 •

edited

Loading

fhygh commented Jun 9, 2022 •

edited

Loading

GodWon commented Jun 20, 2022

SunHeng98 commented Jun 24, 2022

yaowj2 commented Jul 7, 2022 •

edited

Loading

CriysHot commented Jul 12, 2022

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022 •

edited

Loading

yangjf2019 commented Jul 12, 2022 •

edited

Loading

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

zhongjiajie commented Jul 28, 2022 •

edited

Loading

xpf-demo commented Aug 15, 2022

github-keep commented Aug 17, 2022

xpf-demo commented Aug 18, 2022

EricGao888 commented Aug 23, 2022

EricGao888 commented Aug 29, 2022

TCGOGOGO commented Jan 31, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

Jacky-YC commented Apr 22, 2023 •

edited

Loading

dalaopo commented Jul 26, 2023

[Bug] [dolphinscheduler-api] HDFS failed to upload large file #10340

[Bug] [dolphinscheduler-api] HDFS failed to upload large file #10340

Comments

github-keep commented Jun 2, 2022

Search before asking

What happened

What you expected to happen

How to reproduce

Anything else

Version

Are you willing to submit PR?

Code of Conduct

github-actions bot commented Jun 2, 2022

Search before asking

What happened

What you expected to happen

How to reproduce

Anything else

Version

Are you willing to submit PR?

Code of Conduct

github-actions bot commented Jun 2, 2022

EricGao888 commented Jun 9, 2022 • edited Loading

fhygh commented Jun 9, 2022 • edited Loading

GodWon commented Jun 20, 2022

SunHeng98 commented Jun 24, 2022

yaowj2 commented Jul 7, 2022 • edited Loading

CriysHot commented Jul 12, 2022

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022 • edited Loading

yangjf2019 commented Jul 12, 2022 • edited Loading

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

EricGao888 commented Jul 12, 2022

yangjf2019 commented Jul 12, 2022

zhongjiajie commented Jul 28, 2022 • edited Loading

xpf-demo commented Aug 15, 2022

github-keep commented Aug 17, 2022

xpf-demo commented Aug 18, 2022

EricGao888 commented Aug 23, 2022

EricGao888 commented Aug 29, 2022

TCGOGOGO commented Jan 31, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

TheWindIsRising commented Feb 18, 2023

Jacky-YC commented Apr 22, 2023 • edited Loading

dalaopo commented Jul 26, 2023

EricGao888 commented Jun 9, 2022 •

edited

Loading

fhygh commented Jun 9, 2022 •

edited

Loading

yaowj2 commented Jul 7, 2022 •

edited

Loading

yangjf2019 commented Jul 12, 2022 •

edited

Loading

yangjf2019 commented Jul 12, 2022 •

edited

Loading

zhongjiajie commented Jul 28, 2022 •

edited

Loading

Jacky-YC commented Apr 22, 2023 •

edited

Loading