Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable READ_WRITE support for federated metastores #87

Closed
rpoluri opened this issue Jun 18, 2018 · 3 comments · Fixed by #96
Closed

enable READ_WRITE support for federated metastores #87

rpoluri opened this issue Jun 18, 2018 · 3 comments · Fixed by #96

Comments

@rpoluri
Copy link

rpoluri commented Jun 18, 2018

As a user of WD
I like to run DDL on federated metastores
So that I can run ETL againt primary metastore and update federated central datalake.

Acceptance Criteria:
ability to configure access-control-type option for federated metastores with atleast following possible values READ_ONLY, READ_WRITE.

@teabot
Copy link
Contributor

teabot commented Jun 19, 2018

Some additional information. There are a number of write operations in the metastore whose requests do not contain database/table name context, and so cannot be routed to federated instances. We believe that this is not an issue for general operation, but may be a problem if you are wanting to use certain specific Hive features. At this time we believe the following could not be supported in a writable federation model:

  • Create database: could perhaps use the prefix if that mode is enabled?
  • Function handling: create/delete/get functions
  • Type handling: create/delete/get types (although I can't see how these are surfaced in Hive)
  • Keys foreign/primary
  • Locks
  • Transactions / compact
  • Security management: roles, prinicpals, grant/revoke
  • Delegation tokens (Kerberos?)
  • Notifications
  • Config management
  • File metadata / cache

@patduin
Copy link
Contributor

patduin commented Jun 19, 2018

You can add create database to that list

@rpoluri
Copy link
Author

rpoluri commented Jun 19, 2018

Thanks for the info, our current use-case is limited to managing tables and partitions through waggle-dance federation, so should be fine with above limitations.

I also think anything that is not specific to schema should be managed by datalake owner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants