Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#5061] Basic user and group CLI #5133

Merged
merged 108 commits into from
Nov 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
b5e08c0
Added firest part of CLI code.
justinmclean Oct 2, 2024
655d133
update CLI option with output
justinmclean Oct 2, 2024
4a29c8b
Add missing license header
justinmclean Oct 2, 2024
1b12eff
Add default
justinmclean Oct 2, 2024
75de71e
Spotless on Java 11
justinmclean Oct 2, 2024
5172afa
Ignore warnings in test with Java 11
justinmclean Oct 2, 2024
0dc8856
Disable two tests on JAVA 17
justinmclean Oct 2, 2024
24dee9e
disabling two tests for now
justinmclean Oct 2, 2024
0ddbb7d
fix disabled tests
justinmclean Oct 2, 2024
4b95dd3
Still having issues on Java 17
justinmclean Oct 2, 2024
f76fa16
remove two tests as Java 17 is still giving issues
justinmclean Oct 2, 2024
8cbc605
No need to copy CLI for IT tests
justinmclean Oct 2, 2024
0ec7ecb
create catalogs and other entities
justinmclean Oct 9, 2024
fa2cdaf
remove unneeded call
justinmclean Oct 9, 2024
1e05853
Merge branch 'CLI' into CLI_II
justinmclean Oct 10, 2024
3df9f11
add more tests
justinmclean Oct 11, 2024
5e611be
add more tests
justinmclean Oct 11, 2024
68f7fa2
change version command into client and server version commands
justinmclean Oct 11, 2024
e2ebb7a
list entities and commands
justinmclean Oct 11, 2024
0e82c8f
Cache the metalake environment variable
justinmclean Oct 11, 2024
7b97750
use libs
justinmclean Oct 11, 2024
a562ed7
Merge branch 'main' into CLI
justinmclean Oct 11, 2024
47364fd
Merge branch 'CLI' into CLI_II
justinmclean Oct 12, 2024
d678b99
Trino Connector uses old deprecated CLI
justinmclean Oct 12, 2024
c201ef5
Merge branch 'CLI' into CLI_II
justinmclean Oct 12, 2024
253a2b4
fix test
justinmclean Oct 12, 2024
0ef4695
add basic user and group commands
justinmclean Oct 14, 2024
f90a086
fix spelling error
justinmclean Oct 16, 2024
3c87c3d
add examples not using metalake
justinmclean Oct 16, 2024
10af3d2
remove duplication and clarify
justinmclean Oct 16, 2024
f955db2
use client shadow jar
justinmclean Oct 16, 2024
80aef91
wrong file
justinmclean Oct 16, 2024
6d897c0
new -> 19
justinmclean Oct 17, 2024
76aa1fe
fix example command
justinmclean Oct 17, 2024
affd6ad
revert as the suggested name is not value
justinmclean Oct 17, 2024
abc2da8
simplify command options
justinmclean Oct 17, 2024
a7c2a88
Merge branch 'CLI' into CLI_II
justinmclean Oct 17, 2024
055678b
fix merge issue and tests
justinmclean Oct 17, 2024
9808022
update command options
justinmclean Oct 17, 2024
464a74b
fix imports
justinmclean Oct 17, 2024
05bc9ed
update command options
justinmclean Oct 17, 2024
d27c59a
Merge branch 'CLI_II' into CLI_III
justinmclean Oct 17, 2024
423219c
Merge branch 'main' into CLI
justinmclean Oct 17, 2024
954b351
Make executable and runnable from bin directory. update command format.
justinmclean Oct 22, 2024
808fffc
revert back to using two args and fix command format
justinmclean Oct 22, 2024
848a381
remove meatlake form name and add back metalake option
justinmclean Oct 22, 2024
2fe9a80
add Gravitino URL as an environment variable
justinmclean Oct 22, 2024
526d305
update README to mention environment variables
justinmclean Oct 22, 2024
d2ded4c
improved description of met lake name and add to options docs
justinmclean Oct 22, 2024
cb5a5d7
remove version number from alias
justinmclean Oct 22, 2024
71107fd
order in alphabetical order
justinmclean Oct 22, 2024
a6aeb42
add final where needed
justinmclean Oct 22, 2024
ce3ddc1
made private
justinmclean Oct 22, 2024
70aaaf9
use Joiner instead of StringBuilder
justinmclean Oct 22, 2024
22ca191
update commands
justinmclean Oct 23, 2024
ab52cf2
improve command line error handling
justinmclean Oct 23, 2024
bd57944
add executable shell script
justinmclean Oct 23, 2024
445cd41
put back README and make it developer focused
justinmclean Oct 23, 2024
7ea7125
support ignore option to ignore client/server version mismatch
justinmclean Oct 23, 2024
0a8844e
move list commands up one entity
justinmclean Oct 23, 2024
c342d7e
add comment
justinmclean Oct 23, 2024
d2b28af
no need for class
justinmclean Oct 23, 2024
038e1f6
fix command error handling and unwanted output
justinmclean Oct 23, 2024
ce5fa18
remove example scripts
justinmclean Oct 23, 2024
069f425
add CLI documentation
justinmclean Oct 23, 2024
5ed49fa
Merge branch 'CLI' into CLI_II
justinmclean Oct 23, 2024
4d68055
update tests
justinmclean Oct 23, 2024
f8c1f9b
update commands to use ignore parameter
justinmclean Oct 23, 2024
33efd6e
made variables final
justinmclean Oct 23, 2024
82ba035
remove user-facing content
justinmclean Oct 23, 2024
a4ac180
add URL to config file
justinmclean Oct 24, 2024
1ea9e12
list commands need to run first or warning in output occurs
justinmclean Oct 24, 2024
c3fc55c
add javadocs
justinmclean Oct 24, 2024
451a92f
Merge branch 'CLI_II' into CLI_III
justinmclean Oct 24, 2024
50edda6
add ignore to commands
justinmclean Oct 24, 2024
cb31019
make fields final
justinmclean Oct 24, 2024
dd97d26
simplify output
justinmclean Oct 24, 2024
4b60ea4
Merge branch 'main' into CLI_II
justinmclean Oct 24, 2024
20e550f
reset files
justinmclean Oct 24, 2024
eeefccc
restore test files
justinmclean Oct 24, 2024
9f073ac
add ignore to the config file
justinmclean Oct 24, 2024
64f18ef
update documentation
justinmclean Oct 24, 2024
74452c9
minor English updates
justinmclean Oct 25, 2024
c79c6e3
Merge branch 'CLI_II' into CLI_III
justinmclean Oct 25, 2024
d8c111f
add override annotation
justinmclean Oct 31, 2024
09a5d55
add expections
justinmclean Oct 31, 2024
3a64a1d
add @Override
justinmclean Oct 31, 2024
ccb5e01
update tests
justinmclean Oct 31, 2024
7ab5666
update documentation
justinmclean Oct 31, 2024
8472692
Merge branch 'CLI_II' into CLI_III
justinmclean Oct 31, 2024
ddc0711
add user and group examples
justinmclean Oct 31, 2024
366de9e
Remove different catalog options and classes and add the ability to s…
justinmclean Nov 4, 2024
e133aff
update documentation
justinmclean Nov 4, 2024
6ce9a37
Merge branch 'CLI_II' into CLI_III
justinmclean Nov 5, 2024
36596ef
fix import order
justinmclean Nov 5, 2024
8b65598
add back user and group options
justinmclean Nov 5, 2024
41fc424
update doc and option description
justinmclean Nov 5, 2024
6d5275c
Improved property support
justinmclean Nov 5, 2024
6255901
Merge branch 'main' into CLI_III
justinmclean Nov 6, 2024
2b22e09
Merge branch 'main' into CLI_III
justinmclean Nov 6, 2024
18af138
Delete clients/cli/examples.sh
justinmclean Nov 6, 2024
601c320
Delete clients/cli/src/main/java/org/apache/gravitino/cli/ParseTableC…
justinmclean Nov 6, 2024
ffea521
Update GroupDetails.java
justinmclean Nov 6, 2024
cdbad31
fix documentation
justinmclean Nov 6, 2024
033c04d
fix documentation
justinmclean Nov 6, 2024
f244010
Merge branch 'CLI_III' of https://github.com/justinmclean/gravitino i…
justinmclean Nov 6, 2024
9cf358c
fix merge issue
justinmclean Nov 7, 2024
76dd0f4
feedback from PR
justinmclean Nov 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ public class CommandEntities {
public static final String SCHEMA = "schema";
public static final String TABLE = "table";
public static final String COLUMN = "column";
public static final String USER = "user";
public static final String GROUP = "group";

private static final HashSet<String> VALID_ENTITIES = new HashSet<>();

Expand All @@ -40,6 +42,8 @@ public class CommandEntities {
VALID_ENTITIES.add(SCHEMA);
VALID_ENTITIES.add(TABLE);
VALID_ENTITIES.add(COLUMN);
VALID_ENTITIES.add(USER);
VALID_ENTITIES.add(GROUP);
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,8 @@ public class ErrorMessages {
public static final String METALAKE_EXISTS = "Metalake already exists.";
public static final String CATALOG_EXISTS = "Catalog already exists.";
public static final String SCHEMA_EXISTS = "Schema already exists.";
public static final String UNKNOWN_USER = "Unknown user.";
public static final String USER_EXISTS = "User already exists.";
public static final String UNKNOWN_GROUP = "Unknown group.";
public static final String GROUP_EXISTS = "Group already exists.";
}
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,27 @@
import org.apache.gravitino.cli.commands.CatalogDetails;
import org.apache.gravitino.cli.commands.ClientVersion;
import org.apache.gravitino.cli.commands.CreateCatalog;
import org.apache.gravitino.cli.commands.CreateGroup;
import org.apache.gravitino.cli.commands.CreateMetalake;
import org.apache.gravitino.cli.commands.CreateSchema;
import org.apache.gravitino.cli.commands.CreateUser;
import org.apache.gravitino.cli.commands.DeleteCatalog;
import org.apache.gravitino.cli.commands.DeleteGroup;
import org.apache.gravitino.cli.commands.DeleteMetalake;
import org.apache.gravitino.cli.commands.DeleteSchema;
import org.apache.gravitino.cli.commands.DeleteTable;
import org.apache.gravitino.cli.commands.DeleteUser;
import org.apache.gravitino.cli.commands.GroupDetails;
import org.apache.gravitino.cli.commands.ListCatalogProperties;
import org.apache.gravitino.cli.commands.ListCatalogs;
import org.apache.gravitino.cli.commands.ListColumns;
import org.apache.gravitino.cli.commands.ListGroups;
import org.apache.gravitino.cli.commands.ListMetalakeProperties;
import org.apache.gravitino.cli.commands.ListMetalakes;
import org.apache.gravitino.cli.commands.ListSchema;
import org.apache.gravitino.cli.commands.ListSchemaProperties;
import org.apache.gravitino.cli.commands.ListTables;
import org.apache.gravitino.cli.commands.ListUsers;
import org.apache.gravitino.cli.commands.MetalakeAuditInfo;
import org.apache.gravitino.cli.commands.MetalakeDetails;
import org.apache.gravitino.cli.commands.RemoveCatalogProperty;
Expand All @@ -55,6 +62,7 @@
import org.apache.gravitino.cli.commands.UpdateCatalogName;
import org.apache.gravitino.cli.commands.UpdateMetalakeComment;
import org.apache.gravitino.cli.commands.UpdateMetalakeName;
import org.apache.gravitino.cli.commands.UserDetails;

/* Gravitino Command line */
public class GravitinoCommandLine {
Expand Down Expand Up @@ -152,6 +160,10 @@ private void executeCommand() {
handleCatalogCommand();
} else if (entity.equals(CommandEntities.METALAKE)) {
handleMetalakeCommand();
} else if (entity.equals(CommandEntities.USER)) {
handleUserCommand();
} else if (entity.equals(CommandEntities.GROUP)) {
handleGroupCommand();
}
}

Expand Down Expand Up @@ -217,7 +229,7 @@ private void handleCatalogCommand() {
} else if (CommandActions.CREATE.equals(command)) {
String comment = line.getOptionValue(GravitinoOptions.COMMENT);
String provider = line.getOptionValue(GravitinoOptions.PROVIDER);
String properties = line.getOptionValue(GravitinoOptions.PROPERTIES);
String[] properties = line.getOptionValues(GravitinoOptions.PROPERTIES);
Map<String, String> propertyMap = new Properties().parse(properties);
new CreateCatalog(url, ignore, metalake, catalog, provider, comment, propertyMap).handle();
} else if (CommandActions.DELETE.equals(command)) {
Expand Down Expand Up @@ -304,6 +316,42 @@ private void handleTableCommand() {
}
}

/** Handles the command execution for Users based on command type and the command line options. */
protected void handleUserCommand() {
String url = getUrl();
FullName name = new FullName(line);
String metalake = name.getMetalakeName();
String user = line.getOptionValue(GravitinoOptions.USER);

if (CommandActions.DETAILS.equals(command)) {
new UserDetails(url, ignore, metalake, user).handle();
} else if (CommandActions.LIST.equals(command)) {
new ListUsers(url, ignore, metalake).handle();
} else if (CommandActions.CREATE.equals(command)) {
new CreateUser(url, ignore, metalake, user).handle();
} else if (CommandActions.DELETE.equals(command)) {
new DeleteUser(url, ignore, metalake, user).handle();
}
}

/** Handles the command execution for Group based on command type and the command line options. */
protected void handleGroupCommand() {
String url = getUrl();
FullName name = new FullName(line);
String metalake = name.getMetalakeName();
String group = line.getOptionValue(GravitinoOptions.GROUP);

if (CommandActions.DETAILS.equals(command)) {
new GroupDetails(url, ignore, metalake, group).handle();
} else if (CommandActions.LIST.equals(command)) {
new ListGroups(url, ignore, metalake).handle();
} else if (CommandActions.CREATE.equals(command)) {
new CreateGroup(url, ignore, metalake, group).handle();
} else if (CommandActions.DELETE.equals(command)) {
new DeleteGroup(url, ignore, metalake, group).handle();
}
}

/**
* Handles the command execution for Columns based on command type and the command line options.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ public class GravitinoOptions {
public static final String VALUE = "value";
public static final String PROVIDER = "provider";
public static final String PROPERTIES = "properties";
public static final String USER = "user";
public static final String GROUP = "group";
public static final String AUDIT = "audit";

/**
Expand Down Expand Up @@ -64,12 +66,13 @@ public Options options() {
options.addOption(createArgOption("V", VALUE, "property value"));
options.addOption(
createArgOption(
"g", PROVIDER, "provider one of hadoop, hive, mysql, postgres, iceberg, kafka"));
"t", PROVIDER, "provider one of hadoop, hive, mysql, postgres, iceberg, kafka"));
options.addOption(createArgOption("l", USER, "user name"));
options.addOption(createArgOption("g", GROUP, "group name"));

// Properties option can have multiple values
Option properties =
createArgOption("p", PROPERTIES, "comma separated property name/value pairs");
properties.hasArgs();
Option.builder("p").longOpt(PROPERTIES).desc("property name/value pairs").hasArgs().build();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"comma separated" is a necessary description for the properties I think, why not keep it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The option has been updated, and they no longer have to be comma separated - see the description in cli.md.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see; This change is irrelavant with current PR, now mixed with other changes; Next time please use a seperate PR, that will be helpful for review as well as code management (revert, cherry-pick, etc).

options.addOption(properties);

return options;
Expand Down
20 changes: 11 additions & 9 deletions clients/cli/src/main/java/org/apache/gravitino/cli/Properties.java
Original file line number Diff line number Diff line change
Expand Up @@ -55,19 +55,21 @@ public Properties(String delimiter, String keyValueSeparator) {
* <p>Each pair in the input string is split by the specified delimiter, and then each pair is
* further split by the key-value separator.
*
* @param input The input string containing name-value pairs.
* @param inputs An arrays of input strings containing name-value pairs.
* @return A map of entries, where each entry represents a key-value pair from the input string.
*/
public Map<String, String> parse(String input) {
public Map<String, String> parse(String[] inputs) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related with the "user and group" CLI? seems it has already been handled in previous PR, please double check whether this is a code merge issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It not a code merge issue - the way properties have been handled have been improved

HashMap<String, String> map = new HashMap<>();

// Split the input by the delimiter into key-value pairs
String[] pairs = input.split(delimiter);
for (String pair : pairs) {
// Split each key-value pair by the separator
String[] keyValue = pair.split(keyValueSeparator, 2);
if (keyValue.length == 2) {
map.put(keyValue[0].trim(), keyValue[1].trim());
for (String input : inputs) {
// Split the input by the delimiter into key-value pairs
String[] pairs = input.split(delimiter);
for (String pair : pairs) {
// Split each key-value pair by the separator
String[] keyValue = pair.split(keyValueSeparator, 2);
if (keyValue.length == 2) {
map.put(keyValue[0].trim(), keyValue[1].trim());
}
}
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.gravitino.cli.commands;

import org.apache.gravitino.cli.ErrorMessages;
import org.apache.gravitino.client.GravitinoClient;
import org.apache.gravitino.exceptions.GroupAlreadyExistsException;
import org.apache.gravitino.exceptions.NoSuchMetalakeException;

public class CreateGroup extends Command {
protected final String metalake;
protected final String group;

/**
* Create a new group.
*
* @param url The URL of the Gravitino server.
* @param ignoreVersions If true don't check the client/server versions match.
* @param metalake The name of the metalake.
* @param group The name of the group.
*/
public CreateGroup(String url, boolean ignoreVersions, String metalake, String group) {
super(url, ignoreVersions);
this.metalake = metalake;
this.group = group;
}

/** Create a new group. */
@Override
public void handle() {
justinmclean marked this conversation as resolved.
Show resolved Hide resolved
try {
GravitinoClient client = buildClient(metalake);
client.addGroup(group);
} catch (NoSuchMetalakeException err) {
System.err.println(ErrorMessages.UNKNOWN_METALAKE);
return;
} catch (GroupAlreadyExistsException err) {
System.err.println(ErrorMessages.GROUP_EXISTS);
return;
} catch (Exception exp) {
System.err.println(exp.getMessage());
return;
}

System.out.println(group + " created");
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.gravitino.cli.commands;

import org.apache.gravitino.cli.ErrorMessages;
import org.apache.gravitino.client.GravitinoClient;
import org.apache.gravitino.exceptions.NoSuchMetalakeException;
import org.apache.gravitino.exceptions.UserAlreadyExistsException;

public class CreateUser extends Command {
protected final String metalake;
protected final String user;

/**
* Create a new User.
*
* @param url The URL of the Gravitino server.
* @param ignoreVersions If true don't check the client/server versions match.
* @param metalake The name of the metalake.
* @param user The name of the user.
*/
public CreateUser(String url, boolean ignoreVersions, String metalake, String user) {
super(url, ignoreVersions);
this.metalake = metalake;
this.user = user;
}

/** Create a new user. */
@Override
public void handle() {
try {
GravitinoClient client = buildClient(metalake);
client.addUser(user);
} catch (NoSuchMetalakeException err) {
System.err.println(ErrorMessages.UNKNOWN_METALAKE);
return;
} catch (UserAlreadyExistsException err) {
System.err.println(ErrorMessages.USER_EXISTS);
return;
} catch (Exception exp) {
System.err.println(exp.getMessage());
return;
}

System.out.println(user + " created");
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.gravitino.cli.commands;

import org.apache.gravitino.cli.ErrorMessages;
import org.apache.gravitino.client.GravitinoClient;
import org.apache.gravitino.exceptions.NoSuchGroupException;
import org.apache.gravitino.exceptions.NoSuchMetalakeException;

public class DeleteGroup extends Command {

protected final String metalake;
protected final String group;

/**
* Delete a group.
*
* @param url The URL of the Gravitino server.
* @param ignoreVersions If true don't check the client/server versions match.
* @param metalake The name of the metalake.
* @param group The name of the group.
*/
public DeleteGroup(String url, boolean ignoreVersions, String metalake, String group) {
super(url, ignoreVersions);
this.metalake = metalake;
this.group = group;
}

/** Delete a group. */
@Override
public void handle() {
boolean deleted = false;

try {
GravitinoClient client = buildClient(metalake);
deleted = client.removeGroup(group);
} catch (NoSuchMetalakeException err) {
System.err.println(ErrorMessages.UNKNOWN_METALAKE);
return;
} catch (NoSuchGroupException err) {
System.err.println(ErrorMessages.UNKNOWN_GROUP);
return;
} catch (Exception exp) {
System.err.println(exp.getMessage());
return;
}

if (deleted) {
System.out.println(group + " deleted.");
} else {
System.out.println(group + " not deleted.");
}
}
}
Loading
Loading