-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature-10495][Resource Center] Resource Center Refactor #12076
Conversation
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
Outdated
Show resolved
Hide resolved
…r resource update in taskDefinition related tables
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, until the above comments are resolved
...cheduler-service/src/main/java/org/apache/dolphinscheduler/service/storage/impl/S3Utils.java
Outdated
Show resolved
Hide resolved
…uler/service/storage/impl/S3Utils.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @EricPyZhou
SonarCloud Quality Gate failed. 0 Bugs |
in apache#12076 we refactor our resource center, remove the resource table, it is a good refactor but it failed python api upload, this patch try to fix python api upload function ref: apache#12076
* Fix resource upload function related to apache/dolphinscheduler#12076 * And some documentation about how to use resource uploading ref: apache/dolphinscheduler#13042
* Fix resource upload function related to apache/dolphinscheduler#12076 * And some documentation about how to use resource uploading ref: apache/dolphinscheduler#13042 [run-it]
in apache#12076 we refactor our resource center, remove the resource table, it is a good refactor but it failed python api upload, this patch try to fix python api upload function ref: apache#12076
* Fix resource upload function related to apache/dolphinscheduler#12076 ref: apache/dolphinscheduler#13042 [run-it]
* Fix resource upload function related to apache/dolphinscheduler#12076 ref: apache/dolphinscheduler#13042 [run-it]
* Fix resource upload function related to apache/dolphinscheduler#12076 ref: apache/dolphinscheduler#13042 [run-it]
* Fix resource upload function related to apache/dolphinscheduler#12076 ref: apache/dolphinscheduler#13042 [run-it]
Purpose of the pull request
close #10495
Brief change log
t_ds_relation_resources_task
containing an auto-incremented id, fullName and type, which can be found indolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql
.id
params were removed both in frontend resource api (dolphinscheduler-ui/src/service/modules/resources/index.ts) and backend resourceController.java.dolphinscheduler-dao/src/main/resources/sql/upgrade/3.1.0_schema/mysql/dolphinscheduler_ddl.sql
.Most of the changes are in
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/ResourcesService.java
,dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ResourcesController.java
,dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java
anddolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/S3Utils.java
. The original Ids of resources files were all replaced with their unique path (fullName). In the parameter, notice fileName is different from fullName.fullName is in the format of "bucketName/TenantCode/resources(udfs)/fileName"
fileName is just the last part of the fullName.
resTenantCode is the tenant code of the resource. It is used for the permission check where the current login user (ordinary user) can not read resource files if they are not the correct tenant. the scenario is possible because tenantCode is part of the URI.
In the frontEnd. To access resources with the right permission. all URL path has a query pattern of "fullName" and "tenantCode"
E.g. prefix=dolphinscheduler-test/tenant1/ds/resources/testDDD1/&tenantCode=tenant1/ds
In this example. fullName and tenantCode are the fullName and resTenantCode mentioned above.
Future plan:
fullname
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md