Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{KeyVault} Track 2 mgmt-plane adoption #14150

Merged
merged 109 commits into from
Aug 21, 2020
Merged
Show file tree
Hide file tree
Changes from 105 commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
6a8684e
draft: integrate azure-mgmt-core 1.0.0b1
jsntcy Mar 17, 2020
3b7055d
remove code related to storage track 2
jsntcy Mar 18, 2020
63d7e11
use client_kwargs to replace client.config in track2
jsntcy Mar 27, 2020
34a27f8
fix style errors
jsntcy Mar 27, 2020
85d4da9
pass accurate paramters for generic update
jsntcy Mar 30, 2020
23871be
Test
bim-msft Mar 18, 2020
20cd498
Update tests
bim-msft Mar 19, 2020
fccd60d
Add tests
bim-msft Mar 20, 2020
210168e
All tests passed
bim-msft Mar 22, 2020
d1e1926
Add tests
bim-msft Mar 23, 2020
6a0b5e4
Update yaml
bim-msft Mar 24, 2020
38150e0
updated
bim-msft Mar 25, 2020
37f700f
Updated
bim-msft Mar 30, 2020
91064fb
Updated
bim-msft Apr 1, 2020
36f3549
Merge azure-core-preview
bim-msft Apr 1, 2020
1cb4823
Updated
bim-msft Apr 1, 2020
577c17d
data-plane
bim-msft Apr 2, 2020
feff5c8
Fix
bim-msft Apr 3, 2020
e6b148b
draft: integrate azure-mgmt-core 1.0.0b1
jsntcy Mar 17, 2020
9776668
remove code related to storage track 2
jsntcy Mar 18, 2020
5fd1527
use client_kwargs to replace client.config in track2
jsntcy Mar 27, 2020
175b174
fix style errors
jsntcy Mar 27, 2020
0f8cdc3
pass accurate paramters for generic update
jsntcy Mar 30, 2020
50159f1
fix test error
jsntcy Apr 2, 2020
f35deb3
pass raw=True explicitly for sdk calls
jsntcy Apr 3, 2020
c9a9a6d
remove unused import
jsntcy Apr 3, 2020
9a649d4
use azure-mgmt-core 1.0.0b2
jsntcy Apr 7, 2020
0a565fb
Support key list/list-versions/list-deleted
bim-msft Apr 8, 2020
2ec8658
Support create/set-attributes/delete
bim-msft Apr 8, 2020
4309dc9
Merge azure-core-preview
bim-msft Apr 8, 2020
f6a46b5
All key operations supported
bim-msft Apr 10, 2020
47b0c6d
Bugfix
bim-msft Apr 10, 2020
29cf8f9
Bugfix
bim-msft Apr 10, 2020
d7d530e
Merge branch 'dev' into bim_kv_track2
bim-msft Apr 13, 2020
c8df0ae
Key tests passed
bim-msft Apr 13, 2020
cec17f7
Support secret list/list-versions/list-deleted/set
bim-msft Apr 13, 2020
a26bcb5
Updated
bim-msft Apr 19, 2020
b7d7201
Test
bim-msft Mar 18, 2020
90d9c53
Update tests
bim-msft Mar 19, 2020
8162f7d
Add tests
bim-msft Mar 20, 2020
9a41cf3
All tests passed
bim-msft Mar 22, 2020
b9b5b3a
Add tests
bim-msft Mar 23, 2020
dd21273
Update yaml
bim-msft Mar 24, 2020
6c4c60d
updated
bim-msft Mar 25, 2020
c3940a2
Updated
bim-msft Mar 30, 2020
f922616
Updated
bim-msft Apr 1, 2020
2deefd6
draft: integrate azure-mgmt-core 1.0.0b1
jsntcy Mar 17, 2020
d5d47f9
remove code related to storage track 2
jsntcy Mar 18, 2020
32acf18
use client_kwargs to replace client.config in track2
jsntcy Mar 27, 2020
ca957de
fix style errors
jsntcy Mar 27, 2020
e284a5d
Updated
bim-msft Apr 1, 2020
0d63104
data-plane
bim-msft Apr 2, 2020
692dd0f
Fix
bim-msft Apr 3, 2020
b916eb0
Support key list/list-versions/list-deleted
bim-msft Apr 8, 2020
639f86d
Support create/set-attributes/delete
bim-msft Apr 8, 2020
2f8478d
draft: integrate azure-mgmt-core 1.0.0b1
jsntcy Mar 17, 2020
af3e536
remove code related to storage track 2
jsntcy Mar 18, 2020
7bef52f
use client_kwargs to replace client.config in track2
jsntcy Mar 27, 2020
d1a058b
fix style errors
jsntcy Mar 27, 2020
575f2bb
fix test error
jsntcy Apr 2, 2020
4fdfb1f
All key operations supported
bim-msft Apr 10, 2020
1153042
Bugfix
bim-msft Apr 10, 2020
d8d9681
Bugfix
bim-msft Apr 10, 2020
d2ddf5e
Key tests passed
bim-msft Apr 13, 2020
9b6973d
Support secret list/list-versions/list-deleted/set
bim-msft Apr 13, 2020
d828c8f
Merged dev
bim-msft Jun 11, 2020
1317d48
Merge dev
bim-msft Jun 30, 2020
cbc372f
Revert data-plane changes
bim-msft Jun 30, 2020
f119fe3
Test passed
bim-msft Jun 30, 2020
16bad45
Revert core changes
bim-msft Jun 30, 2020
88d24a7
Add hybrid tests
bim-msft Jul 2, 2020
cc0324d
merge dev & bump to azure-mgmt-keyvault 7.0.0b2
bim-msft Jul 23, 2020
ba6e5f2
Fix LRO
bim-msft Jul 23, 2020
b105a0f
Bug fix
bim-msft Jul 23, 2020
09dce0a
Remove no_wait
bim-msft Jul 23, 2020
2ee7034
merge dev
bim-msft Aug 13, 2020
2b12e15
bugfix
bim-msft Aug 13, 2020
e56a205
Fix tests
bim-msft Aug 13, 2020
28ee63d
Fix aio import
bim-msft Aug 13, 2020
6cbae37
bugfix
bim-msft Aug 13, 2020
afe9196
Merge branch 'dev' into bim_kv_track2_mgmt
bim-msft Aug 14, 2020
56edcd7
Bump version of msrest and msrestazure
bim-msft Aug 14, 2020
653fe0d
Merge branch 'dev' into bim_kv_track2_mgmt
bim-msft Aug 18, 2020
588b39d
add dependency
bim-msft Aug 18, 2020
52aaa41
bugfix
bim-msft Aug 18, 2020
016c2f9
Support track2 in patch_models.py
bim-msft Aug 18, 2020
200d145
Revert msrest version
bim-msft Aug 18, 2020
7009d75
bugfix
bim-msft Aug 18, 2020
fadee7d
bugfix
bim-msft Aug 18, 2020
5c7217c
bugfix
bim-msft Aug 18, 2020
4e6c448
Try to remove cache
bim-msft Aug 18, 2020
7e7066c
bump version
bim-msft Aug 18, 2020
e4ec376
updated
bim-msft Aug 19, 2020
fb74b3c
updated
bim-msft Aug 19, 2020
674aad8
bump version
bim-msft Aug 19, 2020
eeb68f9
updated
bim-msft Aug 19, 2020
061234e
merge dev
bim-msft Aug 20, 2020
d78a396
updated
bim-msft Aug 20, 2020
b311c7c
try remove cache
bim-msft Aug 20, 2020
35a6ffa
updated
bim-msft Aug 20, 2020
b20847c
updated
bim-msft Aug 20, 2020
dbe9023
updated
bim-msft Aug 20, 2020
d3fb97f
bugfix
bim-msft Aug 20, 2020
9a87853
bugfix
bim-msft Aug 20, 2020
2baeac6
Refined
bim-msft Aug 20, 2020
2ee7546
Remove *args
bim-msft Aug 20, 2020
a3807d9
Add wait command
bim-msft Aug 21, 2020
d0677ba
Refine azure_stack_wrapper
bim-msft Aug 21, 2020
18f529c
Use api-version in azure_stack_wrapper
bim-msft Aug 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 27 additions & 10 deletions scripts/sdk_process/patch_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@
from msrest.exceptions import HttpOperationError
"""

track2_header = copyright_header + b"""import msrest.serialization
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is quite hack. is there better way to do it, like feedback to sdk team, maybe add it in track2 generation as a option?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yungezz Actually the SDK is great, the problem is within this patch_models.py script. This script is totally a hack script.

from typing import Dict, List, Optional, Union
from msrest.exceptions import HttpOperationError
"""

paging_header = copyright_header + b"""from msrest.paging import Paged
"""

Expand All @@ -59,26 +64,27 @@ def parse_input(input_parameter):
return package_name, module_name


def solve_mro(models):
def solve_mro(models, track2=False):
for models_module in models:
models_path = models_module.__path__[0]
_LOGGER.info("Working on %s", models_path)
if Path(models_path, "models_py3.py").exists():

if not track2 and Path(models_path, "models_py3.py").exists():
_LOGGER.info("Skipping since already patched")
return

# Build the new files in a temp folder
with tempfile.TemporaryDirectory() as temp_folder:
final_models_path = Path(temp_folder, "models")
final_models_path.mkdir()
solve_one_model(models_module, final_models_path)
solve_one_model(models_module, final_models_path, track2=track2)

# Switch the files
shutil.rmtree(models_path)
shutil.move(final_models_path, models_path)


def solve_one_model(models_module, output_folder):
def solve_one_model(models_module, output_folder, track2=False):
"""Will build the compacted models in the output_folder"""

models_classes = [
Expand Down Expand Up @@ -110,8 +116,8 @@ def solve_one_model(models_module, output_folder):
else:
enum_file_module_name = None

write_model_file(Path(output_folder, "models_py3.py"), models_classes)
write_model_file(Path(output_folder, "models.py"), py2_models_classes)
write_model_file(Path(output_folder, "models_py3.py"), models_classes, track2=track2)
write_model_file(Path(output_folder, "models.py"), py2_models_classes, track2=track2)
write_paging_file(Path(output_folder, "paged_models.py"), paged_models_classes)
write_init(
Path(output_folder, "__init__.py"),
Expand All @@ -122,9 +128,12 @@ def solve_one_model(models_module, output_folder):
)


def write_model_file(output_file_path, classes_to_write):
def write_model_file(output_file_path, classes_to_write, track2=False):
with open(output_file_path, "bw") as write_fd:
write_fd.write(header)
if track2:
write_fd.write(track2_header)
else:
write_fd.write(header)

for model in classes_to_write:
_, model_file_path, _ = model
Expand Down Expand Up @@ -191,7 +200,7 @@ def find_models_to_change(module_name):
return [
importlib.import_module('.' + label + '.models', main_module.__name__)
for (_, label, ispkg) in pkgutil.iter_modules(main_module.__path__)
if ispkg
if ispkg and label != 'aio'
]


Expand Down Expand Up @@ -220,7 +229,15 @@ def find_autorest_generated_folder(module_prefix="azure.mgmt"):
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)

track2_packages = [
'azure.mgmt.keyvault'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will future track2 mgmt sdks need to add this line also?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yungezz Yes, I think. Anyway, this is a hack script and maybe we should totally understand it and try to deprecate it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script was involved at this point: #7061 The motivation is to resolve SDK model references. I think track 2 SDKs won't have the reference related issue.

]
prefix = sys.argv[1] if len(sys.argv) >= 2 else "azure.mgmt"
for autorest_package in find_autorest_generated_folder(prefix):
models = find_models_to_change(autorest_package)
solve_mro(models)
track2 = False
for track2_pkg in track2_packages:
if autorest_package.startswith(track2_pkg):
track2 = True
break
solve_mro(models, track2=track2)
4 changes: 4 additions & 0 deletions src/azure-cli-core/azure/cli/core/commands/arm.py
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,10 @@ def get_provisioning_state(instance):
properties = getattr(instance, 'properties', None)
if properties:
provisioning_state = getattr(properties, 'provisioning_state', None)
# some SDK, like keyvault, has 'provisioningState' under 'properties.additional_properties'
if not provisioning_state:
additional_properties = getattr(properties, 'additional_properties', {})
provisioning_state = additional_properties.get('provisioningState')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this case, is privisioningState always existing ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arrownj This is a bug in core. The provisioningState label for keyvault is under properties.provisioning_state.additional_properties, the original code was not able to read it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to add some comments here to explain it

return provisioning_state

def handler(args):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ def keyvault_command_handler(command_args):
command_args[client_arg_name] = client
if 'cmd' not in op_args:
command_args.pop('cmd')

try:
result = op(**command_args)
# apply results transform if specified
Expand Down
9 changes: 9 additions & 0 deletions src/azure-cli/azure/cli/command_modules/keyvault/_help.py
Original file line number Diff line number Diff line change
Expand Up @@ -388,3 +388,12 @@
az keyvault update --enabled-for-disk-encryption true --name MyKeyVault --resource-group MyResourceGroup
crafted: true
"""

helps['keyvault wait'] = """
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a new command? or just fix of help file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a new command? or just fix of help file?

@yungezz This is for LRO. keyvault create and keyvault purge became LRO in track 2.

type: command
short-summary: Place the CLI in a waiting state until a condition of the vault is met.
examples:
- name: Pause CLI until the vault is created.
text: |
az keyvault wait --name MyVault --created
"""
Original file line number Diff line number Diff line change
Expand Up @@ -242,13 +242,14 @@ def validate_deleted_vault_name(cmd, ns):
if not vault:
raise CLIError('No deleted vault was found with name ' + ns.vault_name)

setattr(ns, 'resource_group_name', getattr(ns, 'resource_group_name', None) or id_comps['resource_group'])

# resource_group_name must match the resource group of the deleted vault
if id_comps['resource_group'] != ns.resource_group_name:
raise CLIError("The specified resource group does not match that of the deleted vault %s. The vault "
"must be recovered to the original resource group %s."
% (vault_name, id_comps['resource_group']))
if 'purge' not in cmd.name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's purge for ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arrownj az keyvault purge doesn't require parameter resource_group. In track 1, if we pass an additional resource_group argument into purge function, it would be ignored. But in track 2, any unexpected arguments would cause a failure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code below line 245 is for automatically setting resource_group parameter.

setattr(ns, 'resource_group_name', getattr(ns, 'resource_group_name', None) or id_comps['resource_group'])

# resource_group_name must match the resource group of the deleted vault
if id_comps['resource_group'] != ns.resource_group_name:
raise CLIError("The specified resource group does not match that of the deleted vault %s. The vault "
"must be recovered to the original resource group %s."
% (vault_name, id_comps['resource_group']))


def validate_x509_certificate_chain(ns):
Expand Down
21 changes: 12 additions & 9 deletions src/azure-cli/azure/cli/command_modules/keyvault/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
def load_command_table(self, _):
mgmt_api_version = str(get_api_version(self.cli_ctx, ResourceType.MGMT_KEYVAULT))
mgmt_api_version = mgmt_api_version.replace('-', '_')

data_api_version = str(get_api_version(self.cli_ctx, ResourceType.DATA_KEYVAULT))
data_api_version = data_api_version.replace('.', '_')
data_api_version = data_api_version.replace('-', '_')
Expand Down Expand Up @@ -61,9 +62,9 @@ def load_command_table(self, _):

# Management Plane Commands
with self.command_group('keyvault', kv_vaults_sdk, client_factory=keyvault_client_vaults_factory) as g:
g.custom_command('create', 'create_keyvault',
g.custom_command('create', 'create_keyvault', supports_no_wait=True,
doc_string_source='azure.mgmt.keyvault.v' + mgmt_api_version + '.models#VaultProperties')
g.custom_command('recover', 'recover_keyvault')
g.custom_command('recover', 'recover_keyvault', supports_no_wait=True)
g.custom_command('list', 'list_keyvault')
g.show_command('show', 'get')
g.command('delete', 'delete', deprecate_info=g.deprecate(
Expand All @@ -73,21 +74,23 @@ def load_command_table(self, _):
'the soft deleted state. Please see the following documentation for additional '
'guidance.\n'
'https://docs.microsoft.com/en-us/azure/key-vault/general/soft-delete-overview'))
g.command('purge', 'purge_deleted')
g.custom_command('set-policy', 'set_policy')
g.custom_command('delete-policy', 'delete_policy')
g.command('purge', 'begin_purge_deleted', supports_no_wait=True)
g.custom_command('set-policy', 'set_policy', supports_no_wait=True)
g.custom_command('delete-policy', 'delete_policy', supports_no_wait=True)
g.command('list-deleted', 'list_deleted')
g.generic_update_command(
'update', setter_name='update_keyvault_setter', setter_type=kv_vaults_custom,
custom_func_name='update_keyvault',
doc_string_source='azure.mgmt.keyvault.v' + mgmt_api_version + '.models#VaultProperties')
doc_string_source='azure.mgmt.keyvault.v' + mgmt_api_version + '.models#VaultProperties',
supports_no_wait=True)
g.wait_command('wait')

with self.command_group('keyvault network-rule',
kv_vaults_sdk,
min_api='2018-02-14',
client_factory=keyvault_client_vaults_factory) as g:
g.custom_command('add', 'add_network_rule')
g.custom_command('remove', 'remove_network_rule')
g.custom_command('add', 'add_network_rule', supports_no_wait=True)
g.custom_command('remove', 'remove_network_rule', supports_no_wait=True)
Comment on lines +92 to +93
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in these scenarios, is there a wait command to wait util condition satisfied?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Juliehzl Good catch. Added.

g.custom_command('list', 'list_network_rules')

with self.command_group('keyvault private-endpoint-connection',
Expand All @@ -99,7 +102,7 @@ def load_command_table(self, _):
validator=validate_private_endpoint_connection_id)
g.custom_command('reject', 'reject_private_endpoint_connection', supports_no_wait=True,
validator=validate_private_endpoint_connection_id)
g.command('delete', 'delete', validator=validate_private_endpoint_connection_id)
g.command('delete', 'begin_delete', validator=validate_private_endpoint_connection_id, supports_no_wait=True)
g.show_command('show', 'get', validator=validate_private_endpoint_connection_id)
g.wait_command('wait', validator=validate_private_endpoint_connection_id)

Expand Down
Loading