Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature-10495][Resource Center] Resource Center Refactor #12076

Merged
merged 56 commits into from
Oct 26, 2022

Conversation

EricPyZhou
Copy link
Contributor

@EricPyZhou EricPyZhou commented Sep 21, 2022

Purpose of the pull request

close #10495

Brief change log

  • In DB, I created a table named t_ds_relation_resources_task containing an auto-incremented id, fullName and type, which can be found in dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql .
  • id params were removed both in frontend resource api (dolphinscheduler-ui/src/service/modules/resources/index.ts) and backend resourceController.java.
  • To test the resource center, we need to execute table creation and field update SQL, which can be found in the dolphinscheduler-dao/src/main/resources/sql/upgrade/3.1.0_schema/mysql/dolphinscheduler_ddl.sql.

Most of the changes are in dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java, dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ResourcesController.java, dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java and dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java. The original Ids of resources files were all replaced with their unique path (fullName). In the parameter, notice fileName is different from fullName.
fullName is in the format of "bucketName/TenantCode/resources(udfs)/fileName"
fileName is just the last part of the fullName.
resTenantCode is the tenant code of the resource. It is used for the permission check where the current login user (ordinary user) can not read resource files if they are not the correct tenant. the scenario is possible because tenantCode is part of the URI.

In the frontEnd. To access resources with the right permission. all URL path has a query pattern of "fullName" and "tenantCode"
E.g. prefix=dolphinscheduler-test/tenant1/ds/resources/testDDD1/&tenantCode=tenant1/ds
In this example. fullName and tenantCode are the fullName and resTenantCode mentioned above.

Future plan:

  • Work on PyDolphinScheduler, particularly modify implementation about resource centre.
  • Add sanitization on user inputs, like fullname

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

…r resource update in taskDefinition related tables
jieguangzhou
jieguangzhou previously approved these changes Oct 26, 2022
Copy link
Member

@jieguangzhou jieguangzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, until the above comments are resolved

Copy link
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @EricPyZhou

@sonarcloud
Copy link

sonarcloud bot commented Oct 26, 2022

@SbloodyS SbloodyS merged commit 489e7fe into apache:dev Oct 26, 2022
zhongjiajie added a commit to zhongjiajie/dolphinscheduler that referenced this pull request Nov 29, 2022
in apache#12076 we refactor our resource center, remove the resource table,
it is a good refactor but it failed python api upload, this patch try
to fix python api upload function

ref: apache#12076
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Nov 29, 2022
* Fix resource upload function related to apache/dolphinscheduler#12076
* And some documentation about how to use resource uploading

ref: apache/dolphinscheduler#13042
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Nov 29, 2022
* Fix resource upload function related to apache/dolphinscheduler#12076
* And some documentation about how to use resource uploading

ref: apache/dolphinscheduler#13042

[run-it]
zhongjiajie added a commit that referenced this pull request Dec 2, 2022
in #12076 we refactor our resource center, remove the resource table,
it is a good refactor but it failed python api upload, this patch try
to fix python api upload function

ref: #12076
shangeyao pushed a commit to shangeyao/dolphinscheduler that referenced this pull request Dec 3, 2022
in apache#12076 we refactor our resource center, remove the resource table,
it is a good refactor but it failed python api upload, this patch try
to fix python api upload function

ref: apache#12076
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Dec 4, 2022
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Dec 4, 2022
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Dec 5, 2022
zhongjiajie added a commit to zhongjiajie/dolphinscheduler-sdk-python that referenced this pull request Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend feature new feature UI ui and front end related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement] [API] Resource center API Optimization
7 participants