Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: update find_secret on Fabric and doc #2132

Merged
merged 15 commits into from
Nov 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion core/src/main/python/synapse/ml/core/platform/Platform.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,15 @@ def running_on_databricks():


def find_secret(secret_name, keyvault):
if running_on_synapse() or running_on_synapse_internal():
if running_on_synapse():
from notebookutils.mssparkutils.credentials import getSecret

return getSecret(keyvault, secret_name)
elif running_on_synapse_internal():
from notebookutils.mssparkutils.credentials import getSecret

keyVaultURL = f"https://{keyvault}.vault.azure.net/"
return getSecret(keyVaultURL, secret_name)
elif running_on_databricks():
from pyspark.sql import SparkSession
from pyspark.dbutils import DBUtils
Expand Down
154 changes: 154 additions & 0 deletions docs/Get Started/Set up Cognitive Services.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,160 @@
"* [Create Azure AI service (Decision, Language, Speech, Vision) resource](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesAllInOne). You can follow the steps at [Create a multi-service resource for Azure AI services](https://learn.microsoft.com/en-us/azure/ai-services/multi-service-resource?tabs=windows&pivots=azportal#create-a-new-azure-cognitive-services-resource). \n",
"* Get your Azure AI Services resource's key. After your resource is successfully deployed, select **Next Steps** > **Go to resource**. Once at the resource, you can get the key from **Resource Management** > **Keys and Endpoint**. Copy the key and paste it into the notebook. Store keys securely and do not share them. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Manage secrets with Azure Key Vault and access the secrets with find_secret"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After you create an Azure AI resource, you will obtain a resource key. You can use this resource key directly in our notebooks as a string, but we recommend to manage secrets with Azure Key Vault. Azure Key Vault is a cloud-based service that allows you to store and manage cryptographic keys, certificates, and secrets used by cloud applications and services.\n",
"\n",
"You can skip the following content if you want to use a secret string instead of Azure Key Vault. This is not recommended for production workloads."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create Azure Key Vault\n",
"\n",
"Refer to [this article](https://learn.microsoft.com/azure/key-vault/general/quick-create-portal) to create a key vault using Azure Portal"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Save secret to Azure Key Vault\n",
"\n",
"- Go to Access control (IAM) and assign **Key Vault Administrator Role** to yourself.\n",
"- Go to Secrets and choose **+ Generate/Import**, create a key with the secret value obtained from Azure AI service.\n",
"- Choose **Create**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use find_secret on Microsoft Fabric / Power BI\n",
"\n",
"- Go to Azure Key Vault, Select Access control (IAM), Select **+ Add**, Add role assignment.\n",
"- Granted the Fabric or Power BI Service Account Get permissions in the Azure Key Vault."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use find_secret on Azure Databricks\n",
"\n",
"- In the Azure Portal, find and select the Azure Key Vault Instance. Select the **Access configuration** tab under Settings. Set Permission model to Vault access policy.\n",
"- On Databricks, go to `https://<databricks-instance>#secrets/createScope`. This URL is case sensitive.\n",
" \n",
"\n",
"<img src=\"https://mmlspark.blob.core.windows.net/graphics/adb_create_secret_scope.png\" width=\"600\" style=\"float: center;\"/>\n",
"\n",
"\n",
"\n",
"- Enter the name of the secret scope and choose desired Manage Principal.\n",
"- Go to your Azure Key Vault -> Properties and find the **DNS Name** (Vault URI on Azure Key Vault) and **Resource ID**. Enter the DNS Name and Resource ID on Databricks createScope page.\n",
"\n",
"\n",
"<img src=\"https://mmlspark.blob.core.windows.net/graphics/adb_find_resource_id.png\" width=\"600\" style=\"float: center;\"/>\n",
"\n",
"\n",
"- Click the **Create** Button.\n",
" \n",
"\n",
"Refer to [this article](https://learn.microsoft.com/azure/databricks/security/secrets/secret-scopes#--create-an-azure-key-vault-backed-secret-scope) for more details.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Use find_secret on Synapse\n",
"\n",
"- Open the Synapse Studio and go to the Manage tab.\n",
"- Under External connections, select Linked services.\n",
"- To add a linked service, select New.\n",
"- Select the Azure Key Vault tile from the list and select Continue.\n",
"- Enter a linked service name and choose the key vault you want to connect to.\n",
"- Select Create\n",
"- Click Publish\n",
"\n",
"\n",
"- Go to Azure Key Vault, Select Access control (IAM), Select **+ Add**, Add role assignment.\n",
"- Choose **Key Vault Administrator** in Role blade, select Next.\n",
"- In Members blade, choose Assign access to **Managed identity**. Select members, choose the subscription your Synapse Workspace in. For Managed identity, select Synapse workspace, choose your workspace.\n",
"- Select **Review + assign**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Quick Test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "plaintext"
}
},
"outputs": [],
"source": [
"from synapse.ml.services.language import AnalyzeText\n",
"from synapse.ml.core.platform import find_secret\n",
"\n",
"ai_service_key = find_secret(\n",
" secret_name=\"ai-services-api-key\", keyvault=\"mmlspark-build-keys\"\n",
") # use your own key vault name and api name\n",
"ai_service_location = \"eastus\" # use your own AI service location\n",
"\n",
"df = spark.createDataFrame(\n",
" data=[\n",
" [\"en\", \"Dr. Smith has a very modern medical office, and she has great staff.\"],\n",
" [\"en\", \"I had a wonderful trip to Seattle last week.\"],\n",
" ],\n",
" schema=[\"language\", \"text\"],\n",
")\n",
"\n",
"entity_recognition = (\n",
" AnalyzeText()\n",
" .setKind(\"EntityRecognition\")\n",
" .setLocation(ai_service_location)\n",
" .setSubscriptionKey(ai_service_key)\n",
" .setTextCol(\"text\")\n",
" .setOutputCol(\"entities\")\n",
" .setErrorCol(\"error\")\n",
" .setLanguageCol(\"language\")\n",
")\n",
"\n",
"df_results = entity_recognition.transform(df)\n",
"display(df_results.select(\"language\", \"text\", \"entities.documents.entities\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Your result should looks like\n",
"\n",
"| language | text | entities |\n",
"|--------------|-----------|------------|\n",
"| en | Dr. Smith has a very modern medical office, and she has great staff. | [{\"category\": \"Person\", \"confidenceScore\": 0.98, \"length\": 5, \"offset\": 4, \"subcategory\": null, \"text\": \"Smith\"}, {\"category\": \"Location\", \"confidenceScore\": 0.79, \"length\": 14, \"offset\": 28, \"subcategory\": \"Structural\", \"text\": \"medical office\"}, {\"category\": \"PersonType\", \"confidenceScore\": 0.85, \"length\": 5, \"offset\": 62, \"subcategory\": null, \"text\": \"staff\"}] |\n",
"| en | I had a wonderful trip to Seattle last week. | [{\"category\": \"Event\", \"confidenceScore\": 0.74, \"length\": 4, \"offset\": 18, \"subcategory\": null, \"text\": \"trip\"}, {\"category\": \"Location\", \"confidenceScore\": 1, \"length\": 7, \"offset\": 26, \"subcategory\": \"GPE\", \"text\": \"Seattle\"}, {\"category\": \"DateTime\", \"confidenceScore\": 0.8, \"length\": 9, \"offset\": 34, \"subcategory\": \"DateRange\", \"text\": \"last week\"}] |"
]
}
],
"metadata": {
Expand Down