Skip to content

Commit

Permalink
[k8s-extension] Release v0.4.3 (#3491)
Browse files Browse the repository at this point in the history
* Create pull.yml

* Update pull.yml

* Update azure-pipelines.yml

* Initial commit of k8s-extension

* Update pipelines file

* Update CODEOWNERS

* Update private preview pipelines

* Remove open service mesh from public release

* Update pipeline files

* Update public extension pipeline

* Change condition variable

* Add version to public preview/private preview

* Update pipelines

* Add different testing based on private branch

* Add annotations to extension model

* Update k8s-custom-pipelines.yml

* Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13)

* Update sdks with updated swagger spec

* Update version and history rst

* Reorder release history timeline

* Fix ExtensionInstanceForCreate for import

* remove py2 bdist support

* Add custom table formatting

* Remove unnecessary files

* Fix style issues

* Fix branch based on comments

* Update identity piece manually

* Don't handle defaults at the CLI level

* Remove defaults from CLI client

* Check null target namespace with namespace scope

* Update style

* Add cassandra operator and location to model

* Stage Public Version of k8s-extension 0.2.0 for official release (#15)

* Create pull.yml

* Update pull.yml

* Update azure-pipelines.yml

* Initial commit of k8s-extension

* Update pipelines file

* Update CODEOWNERS

* Update private preview pipelines

* Remove open service mesh from public release

* Update pipeline files

* Update public extension pipeline

* Change condition variable

* Add version to public preview/private preview

* Update pipelines

* Add different testing based on private branch

* Add annotations to extension model

* Update k8s-custom-pipelines.yml

* Update SDKs with Updated Swagger Spec for 2020-07-01-preview (#13)

* Update sdks with updated swagger spec

* Update version and history rst

* Reorder release history timeline

* Fix ExtensionInstanceForCreate for import

* remove py2 bdist support

* Add custom table formatting

* Remove unnecessary files

* Fix style issues

* Fix branch based on comments

* Update identity piece manually

* Don't handle defaults at the CLI level

* Remove defaults from CLI client

* Check null target namespace with namespace scope

* Update style

* Add cassandra operator and location to model

Co-authored-by: [email protected] <Action - Fork Sync>

* Remove custom pipelines file

* Update extension description, remove private const

* Update pipeline file

* Disable refs docs

* Update to include better create warning logs and remove update context (#20)

* Update to include better create warning logs and remove update context

* Remove help text for update

* Fix spelling error

* Update message

* Fix k8s-extension conflict with private version

* Fix style errors

* Fix filename

* add customization for microsoft.azureml.kubernetes (#23)

* add customization for microsoft.azureml.kubernetes

* Update release history

Co-authored-by: Yue Yu <[email protected]>
Co-authored-by: jonathan-innis <[email protected]>

* Add E2E Testing from Separate branch into internal code (#26)

* Add internal e2e testing

* Change to testing folder

* Inference CLI validation for Scoring FE (#24)

* cli validation starter

* added the call to the fe validation function

* nodeport validation not required

* test fix

Co-authored-by: Jonathan Innis <[email protected]>

* legal warning added (#27)

* Remove deprecated method logger.warn

* Update k8s-custom-pipelines.yml for Azure Pipelines

* Update k8s-custom-pipelines.yml for Azure Pipelines

* Add Azure Defender to E2E testing (#28)

* Add azure defender testing to e2e

* Remove the debug flag

* Add configuration testing

* Fix pipeline failures

* Make test script more intuitive

* Remove parameter from testing

* Fix wrong location for k8s config whl

* Fix pip upgrade issue

* Fix pip install upgrade issue

* Fix pip install issue

* delete resurce in testcase (#29)

Co-authored-by: Yue Yu <[email protected]>
Co-authored-by: Jonathan Innis <[email protected]>

* Check Provider is Registered with Subscription Before Making Requests (#18)

* Add check for KubernetesConfiguration

* Disable pylint and rename

* Update provider registration link

* Update version

* Remove extra blank line

* Fix bug in import

* only validate scoring fe when inference is enabled (#31)

* only validate scoring fe when inference is enabled

* Fix versioning

Co-authored-by: Yue Yu <[email protected]>
Co-authored-by: jonathan-innis <[email protected]>

* Provider registration case insensitive

* do not validate against scoring fe if inference is not enabled. (#33)

* do not validate against scoring fe if inference is not enabled.

* add inference enabled scenario

* refine

* increase sleeping time

* fix

Co-authored-by: Yue Yu <[email protected]>
Co-authored-by: Jonathan Innis <[email protected]>

* Add OSM as Public Preview Extension (#34)

* Add OSM as public preview extension

* Add osm testing

* Add release train to tests

* Fix failing osm test

* Upgrade pip in integration testing

* Remove ununsed import

* Fix release train check in update

* Parallelize E2E Testing (#36)

* Add OSM as public preview extension

* Add osm testing

* Update test logic to parallelize

* Fix test success checking

* Parallelize extension testing

* Better error checking logic

* Fix azureml deletion

* Fix private build (#40)

* change amlk8s to amlarc (#42)

Co-authored-by: Yue Yu <[email protected]>

* Servicebus client model changes (#44)

* Servicebus client model changes

* Fix testing script

* Update history file and pipeline

* Update min cli core version for track 2 updates

* Read SSL cert and key from files (#38)

* first sketch of the change

fixes

removed extra blank lines

changes regarding param renaming

added ssl tests

added more detail to the unit test

additional import

moved pem files out of public folder

fixed import

chenged import

changed import

unit tests fix

unit test fix

fixed unit tests

fixed unit test

unit test fix

changes int test cert and key

* test protected config

* fix test typo

* temporary changes reverted

* fixing tests

* fixed file paths

* removed accidentally added file

* changes according to review comments

* more changes according to review comments

* changes according to review comments

Co-authored-by: Jonathan Innis <[email protected]>

* Upgrade release version

* Liakaz/inference read ssl from file (#47)

* first sketch of the change

fixes

removed extra blank lines

changes regarding param renaming

added ssl tests

added more detail to the unit test

additional import

moved pem files out of public folder

fixed import

chenged import

changed import

unit tests fix

unit test fix

fixed unit tests

fixed unit test

unit test fix

changes int test cert and key

* test protected config

* fix test typo

* temporary changes reverted

* fixing tests

* fixed file paths

* removed accidentally added file

* changes according to review comments

* more changes according to review comments

* changes according to review comments

* fixed decode error

* renamed the experimental param

Co-authored-by: Jonathan Innis <[email protected]>

* Fix style issues (#51)

* Fixed scoring fe related extension param names (#49)

* fixed scoring fe related extension params

* bug fix and style fixes

* variable rename

* fixed the error type

* set cluster to prod by default

* Remove unneeded files

Co-authored-by: [email protected] <Action - Fork Sync>
Co-authored-by: yuyue9284 <[email protected]>
Co-authored-by: Yue Yu <[email protected]>
Co-authored-by: Lia Kazakova <[email protected]>
  • Loading branch information
4 people authored Jun 11, 2021
1 parent 5105f47 commit 46f8b52
Show file tree
Hide file tree
Showing 10 changed files with 132 additions and 46 deletions.
5 changes: 5 additions & 0 deletions src/k8s-extension/HISTORY.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
Release History
===============

0.4.3
++++++++++++++++++
* Add SSL support for AzureML


0.4.2
++++++++++++++++++

Expand Down
2 changes: 0 additions & 2 deletions src/k8s-extension/azext_k8s_extension/custom.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@
import json
from knack.log import get_logger

from msrestazure.azure_exceptions import CloudError

from azure.cli.core.azclierror import ResourceNotFoundError, MutuallyExclusiveArgumentError, \
InvalidArgumentValueError, CommandNotFoundError, RequiredArgumentMissingError
from azure.cli.core.commands.client_factory import get_subscription_id
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
# --------------------------------------------------------------------------------------------

# pylint: disable=unused-argument
# pylint: disable=line-too-long
# pylint: disable=too-many-locals

import copy
from hashlib import md5
from typing import Any, Dict, List, Tuple
Expand All @@ -17,22 +20,27 @@
import azure.mgmt.storage.models
import azure.mgmt.loganalytics
import azure.mgmt.loganalytics.models
from ..vendored_sdks.models import (
ExtensionInstance, ExtensionInstanceUpdate, Scope, ScopeCluster)
from azure.cli.core.azclierror import InvalidArgumentValueError
from azure.cli.core.azclierror import InvalidArgumentValueError, MutuallyExclusiveArgumentError
from azure.cli.core.commands.client_factory import get_mgmt_service_client, get_subscription_id
from azure.mgmt.resource.locks.models import ManagementLockObject
from knack.log import get_logger
from msrestazure.azure_exceptions import CloudError

from .._client_factory import cf_resources
from .PartnerExtensionModel import PartnerExtensionModel
from ..vendored_sdks.models import (
ExtensionInstance,
ExtensionInstanceUpdate,
Scope,
ScopeCluster
)

logger = get_logger(__name__)

resource_tag = {'created_by': 'Azure Arc-enabled ML'}


# pylint: disable=too-many-instance-attributes
class AzureMLKubernetes(PartnerExtensionModel):
def __init__(self):
# constants for configuration settings.
Expand Down Expand Up @@ -66,6 +74,14 @@ def __init__(self):
self.SERVICE_BUS_JOB_STATE_TOPIC = 'jobstate-updatedby-computeprovider'
self.SERVICE_BUS_JOB_STATE_SUB = 'compute-scheduler-jobstate'

# constants for enabling SSL in inference
self.sslKeyPemFile = 'sslKeyPemFile'
self.sslCertPemFile = 'sslCertPemFile'
self.allowInsecureConnections = 'allowInsecureConnections'
self.privateEndpointILB = 'privateEndpointILB'
self.privateEndpointNodeport = 'privateEndpointNodeport'
self.inferenceLoadBalancerHA = 'inferenceLoadBalancerHA'

# reference mapping
self.reference_mapping = {
self.RELAY_SERVER_CONNECTION_STRING: [self.RELAY_CONNECTION_STRING_KEY, self.RELAY_CONNECTION_STRING_DEPRECATED_KEY],
Expand Down Expand Up @@ -151,7 +167,7 @@ def __validate_config(self, configuration_settings, configuration_protected_sett
config_keys = configuration_settings.keys()
config_protected_keys = configuration_protected_settings.keys()
dup_keys = set(config_keys) & set(config_protected_keys)
if len(dup_keys) > 0:
if dup_keys:
for key in dup_keys:
logger.warning(
'Duplicate keys found in both configuration settings and configuration protected setttings: %s', key)
Expand All @@ -168,6 +184,7 @@ def __validate_config(self, configuration_settings, configuration_protected_sett
if enable_inference:
logger.warning("The installed AzureML extension for AML inference is experimental and not covered by customer support. Please use with discretion.")
self.__validate_scoring_fe_settings(configuration_settings, configuration_protected_settings)
self.__set_up_inference_ssl(configuration_settings, configuration_protected_settings)
elif not (enable_training or enable_inference):
raise InvalidArgumentValueError(
"Please create Microsoft.AzureML.Kubernetes extension instance either "
Expand All @@ -181,33 +198,63 @@ def __validate_config(self, configuration_settings, configuration_protected_sett
configuration_protected_settings.pop(self.ENABLE_INFERENCE, None)

def __validate_scoring_fe_settings(self, configuration_settings, configuration_protected_settings):
clusterPurpose = _get_value_from_config_protected_config(
'clusterPurpose', configuration_settings, configuration_protected_settings)
if clusterPurpose and clusterPurpose not in ["DevTest", "FastProd"]:
raise InvalidArgumentValueError(
"Accepted values for '--configuration-settings clusterPurpose' "
"are 'DevTest' and 'FastProd'")

feSslCert = _get_value_from_config_protected_config(
'scoringFe.sslCert', configuration_settings, configuration_protected_settings)
sslKey = _get_value_from_config_protected_config(
'scoringFe.sslKey', configuration_settings, configuration_protected_settings)
isTestCluster = _get_value_from_config_protected_config(
self.inferenceLoadBalancerHA, configuration_settings, configuration_protected_settings)
isTestCluster = str(isTestCluster).lower() == 'false'
if isTestCluster:
configuration_settings['clusterPurpose'] = 'DevTest'
else:
configuration_settings['clusterPurpose'] = 'FastProd'
feSslCertFile = configuration_protected_settings.get(self.sslCertPemFile)
feSslKeyFile = configuration_protected_settings.get(self.sslKeyPemFile)
allowInsecureConnections = _get_value_from_config_protected_config(
'allowInsecureConnections', configuration_settings, configuration_protected_settings)
self.allowInsecureConnections, configuration_settings, configuration_protected_settings)
allowInsecureConnections = str(allowInsecureConnections).lower() == 'true'
if (not feSslCert or not sslKey) and not allowInsecureConnections:
if (not feSslCertFile or not feSslKeyFile) and not allowInsecureConnections:
raise InvalidArgumentValueError(
"Provide ssl certificate and key. "
"Otherwise explicitly allow insecure connection by specifying "
"'--configuration-settings allowInsecureConnections=true'")

feIsNodePort = _get_value_from_config_protected_config(
self.privateEndpointNodeport, configuration_settings, configuration_protected_settings)
feIsNodePort = str(feIsNodePort).lower() == 'true'
feIsInternalLoadBalancer = _get_value_from_config_protected_config(
'scoringFe.serviceType.internalLoadBalancer', configuration_settings, configuration_protected_settings)
self.privateEndpointILB, configuration_settings, configuration_protected_settings)
feIsInternalLoadBalancer = str(feIsInternalLoadBalancer).lower() == 'true'
if feIsInternalLoadBalancer:

if feIsNodePort and feIsInternalLoadBalancer:
raise MutuallyExclusiveArgumentError(
"Specify either privateEndpointNodeport=true or privateEndpointILB=true, but not both.")
elif feIsNodePort:
configuration_settings['scoringFe.serviceType.nodePort'] = feIsNodePort
elif feIsInternalLoadBalancer:
configuration_settings['scoringFe.serviceType.internalLoadBalancer'] = feIsInternalLoadBalancer
logger.warning(
'Internal load balancer only supported on AKS and AKS Engine Clusters.')

def __set_up_inference_ssl(self, configuration_settings, configuration_protected_settings):
allowInsecureConnections = _get_value_from_config_protected_config(
self.allowInsecureConnections, configuration_settings, configuration_protected_settings)
allowInsecureConnections = str(allowInsecureConnections).lower() == 'true'
if not allowInsecureConnections:
import base64
feSslCertFile = configuration_protected_settings.get(self.sslCertPemFile)
feSslKeyFile = configuration_protected_settings.get(self.sslKeyPemFile)
with open(feSslCertFile) as f:
cert_data = f.read()
cert_data_bytes = cert_data.encode("ascii")
ssl_cert = base64.b64encode(cert_data_bytes).decode()
configuration_protected_settings['scoringFe.sslCert'] = ssl_cert
with open(feSslKeyFile) as f:
key_data = f.read()
key_data_bytes = key_data.encode("ascii")
ssl_key = base64.b64encode(key_data_bytes).decode()
configuration_protected_settings['scoringFe.sslKey'] = ssl_key
else:
logger.warning(
'SSL is not enabled. Allowing insecure connections to the deployed services.')

def __create_required_resource(
self, cmd, configuration_settings, configuration_protected_settings, subscription_id, resource_group_name,
cluster_name, cluster_location):
Expand All @@ -222,9 +269,8 @@ def __create_required_resource(
configuration_settings[self.AZURE_LOG_ANALYTICS_CUSTOMER_ID_KEY] = ws_costumer_id
configuration_protected_settings[self.AZURE_LOG_ANALYTICS_CONNECTION_STRING] = shared_key

if not configuration_settings.get(
self.RELAY_SERVER_CONNECTION_STRING) and not configuration_protected_settings.get(
self.RELAY_SERVER_CONNECTION_STRING):
if not configuration_settings.get(self.RELAY_SERVER_CONNECTION_STRING) and \
not configuration_protected_settings.get(self.RELAY_SERVER_CONNECTION_STRING):
logger.info('==== BEGIN RELAY CREATION ====')
relay_connection_string, hc_resource_id, hc_name = _get_relay_connection_str(
cmd, subscription_id, resource_group_name, cluster_name, cluster_location, self.RELAY_HC_AUTH_NAME)
Expand All @@ -233,9 +279,8 @@ def __create_required_resource(
configuration_settings[self.HC_RESOURCE_ID_KEY] = hc_resource_id
configuration_settings[self.RELAY_HC_NAME_KEY] = hc_name

if not configuration_settings.get(
self.SERVICE_BUS_CONNECTION_STRING) and not configuration_protected_settings.get(
self.SERVICE_BUS_CONNECTION_STRING):
if not configuration_settings.get(self.SERVICE_BUS_CONNECTION_STRING) and \
not configuration_protected_settings.get(self.SERVICE_BUS_CONNECTION_STRING):
logger.info('==== BEGIN SERVICE BUS CREATION ====')
topic_sub_mapping = {
self.SERVICE_BUS_COMPUTE_STATE_TOPIC: self.SERVICE_BUS_COMPUTE_STATE_SUB,
Expand All @@ -252,7 +297,7 @@ def __create_required_resource(

def _get_valid_name(input_name: str, suffix_len: int, max_len: int) -> str:
normalized_str = ''.join(filter(str.isalnum, input_name))
assert len(normalized_str) > 0, "normalized name empty"
assert normalized_str, "normalized name empty"

if len(normalized_str) <= max_len:
return normalized_str
Expand All @@ -267,6 +312,7 @@ def _get_valid_name(input_name: str, suffix_len: int, max_len: int) -> str:
return new_name


# pylint: disable=broad-except
def _lock_resource(cmd, lock_scope, lock_level='CanNotDelete'):
lock_client: azure.mgmt.resource.locks.ManagementLockClient = get_mgmt_service_client(
cmd.cli_ctx, azure.mgmt.resource.locks.ManagementLockClient)
Expand All @@ -275,14 +321,13 @@ def _lock_resource(cmd, lock_scope, lock_level='CanNotDelete'):
try:
lock_client.management_locks.create_or_update_by_scope(
scope=lock_scope, lock_name='amlarc-resource-lock', parameters=lock_object)
except:
except Exception:
# try to lock the resource if user has the owner privilege
pass


def _get_relay_connection_str(
cmd, subscription_id, resource_group_name, cluster_name, cluster_location, auth_rule_name) -> Tuple[
str, str, str]:
cmd, subscription_id, resource_group_name, cluster_name, cluster_location, auth_rule_name) -> Tuple[str, str, str]:
relay_client: azure.mgmt.relay.RelayManagementClient = get_mgmt_service_client(
cmd.cli_ctx, azure.mgmt.relay.RelayManagementClient)

Expand Down Expand Up @@ -370,8 +415,7 @@ def _get_service_bus_connection_string(cmd, subscription_id, resource_group_name


def _get_log_analytics_ws_connection_string(
cmd, subscription_id, resource_group_name, cluster_name, cluster_location) -> Tuple[
str, str]:
cmd, subscription_id, resource_group_name, cluster_name, cluster_location) -> Tuple[str, str]:
log_analytics_ws_client: azure.mgmt.loganalytics.LogAnalyticsManagementClient = get_mgmt_service_client(
cmd.cli_ctx, azure.mgmt.loganalytics.LogAnalyticsManagementClient)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
from azure.cli.core.commands import LongRunningOperation
from azure.cli.core.commands.client_factory import get_mgmt_service_client, get_subscription_id
from azure.cli.core.util import sdk_no_wait
from msrestazure.azure_exceptions import CloudError
from msrestazure.tools import parse_resource_id, is_valid_resource_id

from ..vendored_sdks.models import ExtensionInstance
Expand Down Expand Up @@ -104,8 +103,7 @@ def _invoke_deployment(cmd, resource_group_name, deployment_name, template, para
if cmd.supported_api_version(min_api='2019-10-01', resource_type=ResourceType.MGMT_RESOURCE_RESOURCES):
validation_poller = smc.begin_validate(resource_group_name, deployment_name, deployment)
return LongRunningOperation(cmd.cli_ctx)(validation_poller)
else:
return smc.validate(resource_group_name, deployment_name, deployment)
return smc.validate(resource_group_name, deployment_name, deployment)

return sdk_no_wait(no_wait, smc.begin_create_or_update, resource_group_name, deployment_name, deployment)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dGVzdGNlcnQ=
dGVzdGtleQ==
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
testcert
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
testkey
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# --------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------

# pylint: disable=protected-access

import os
import unittest

from azext_k8s_extension.partner_extensions.AzureMLKubernetes import AzureMLKubernetes


TEST_DIR = os.path.abspath(os.path.join(os.path.abspath(__file__), '..'))


class TestAzureMlExtension(unittest.TestCase):

def test_set_up_inference_ssl(self):
azremlk8sInstance = AzureMLKubernetes()
config = {'allowInsecureConnections': 'false'}
# read and encode dummy cert and key
sslKeyPemFile = os.path.join(TEST_DIR, 'data', 'azure_ml', 'test_key.pem')
sslCertPemFile = os.path.join(TEST_DIR, 'data', 'azure_ml', 'test_cert.pem')
protected_config = {'sslKeyPemFile': sslKeyPemFile, 'sslCertPemFile': sslCertPemFile}
azremlk8sInstance._AzureMLKubernetes__set_up_inference_ssl(config, protected_config)
self.assertTrue('scoringFe.sslCert' in protected_config)
self.assertTrue('scoringFe.sslKey' in protected_config)
encoded_cert_and_key_file = os.path.join(TEST_DIR, 'data', 'azure_ml', 'cert_and_key_encoded.txt')
with open(encoded_cert_and_key_file, "r") as text_file:
cert = text_file.readline().rstrip()
assert cert == protected_config['scoringFe.sslCert']
key = text_file.readline()
assert key == protected_config['scoringFe.sslKey']
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
# Licensed under the MIT License. See License.txt in the project root for license information.
# --------------------------------------------------------------------------------------------

import os
import unittest
# pylint: disable=line-too-long

import os
from azure.cli.testsdk import (ScenarioTest, ResourceGroupPreparer, record_only)


Expand All @@ -27,13 +27,16 @@ def test_k8s_extension(self):
'version': '0.1.0'
})

self.cmd('k8s-extension create -g {rg} -n {name} -c {cluster_name} --cluster-type {cluster_type} --extension-type {extension_type} --release-train {release_train} --version {version}', checks=[
self.check('name', '{name}'),
self.check('releaseTrain', '{release_train}'),
self.check('version', '{version}'),
self.check('resourceGroup', '{rg}'),
self.check('extensionType', '{extension_type}')
])
self.cmd('k8s-extension create -g {rg} -n {name} -c {cluster_name} --cluster-type {cluster_type} '
'--extension-type {extension_type} --release-train {release_train} --version {version}',
checks=[
self.check('name', '{name}'),
self.check('releaseTrain', '{release_train}'),
self.check('version', '{version}'),
self.check('resourceGroup', '{rg}'),
self.check('extensionType', '{extension_type}')
]
)

# Update is disabled for now
# self.cmd('k8s-extension update -g {rg} -n {name} --tags foo=boo', checks=[
Expand Down
2 changes: 1 addition & 1 deletion src/k8s-extension/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
# TODO: Add any additional SDK dependencies here
DEPENDENCIES = []

VERSION = "0.4.2"
VERSION = "0.4.3"

with open('README.rst', 'r', encoding='utf-8') as f:
README = f.read()
Expand Down

0 comments on commit 46f8b52

Please sign in to comment.