Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: AzureSQL masking pipeline #22

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

sumeetdas-dlpx
Copy link

@sumeetdas-dlpx sumeetdas-dlpx commented Oct 15, 2024

Problem

We want to make a DCS for Azure template that mask data available in Azure SQL DB.

Solution

Add templates for DCS for Azure Azure SQL to Azure SQL for masking.

Testing Done

Followed the build instructions and imported the template into an existing data factory.

Successfully masked a table with all supported data types and an identity column and inserted the masked data into a sink schema table using the Azure SQL masking pipeline.

Note: Currently facing an issue masking a table without identity column. Created DAAS-9463 ticket for the same. This issue is fixed in the latest commit.

@sumeetdas-dlpx sumeetdas-dlpx force-pushed the dlpx/pr/sumeetdas-dlpx/719dac7b-fcf1-4305-b21e-f5f1fb1db712 branch 2 times, most recently from 3b83aab to c11bc1f Compare October 15, 2024 05:27
@sumeetdas-dlpx sumeetdas-dlpx force-pushed the dlpx/pr/sumeetdas-dlpx/719dac7b-fcf1-4305-b21e-f5f1fb1db712 branch from c11bc1f to 7573540 Compare October 15, 2024 07:28
Copy link

@ankurs-delphix ankurs-delphix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set this in pre/post script for both sink (filtered and non filtered)

In pre SQL script

concat('SET IDENTITY_INSERT', ' ', $DF_SINK_SCHEMA, '.', $DF_SINK_TABLE, ' ', 'ON')

In post SQL script

concat('SET IDENTITY_INSERT', ' ', $DF_SINK_SCHEMA, '.', $DF_SINK_TABLE, ' ', 'OFF')

General Syntax is

SET IDENTITY_INSERT <SCHEMA>.<TABLE_NAME> <ON|OFF>

PS: I've tested this with table that has identity enabled (Auto incrementing primary key). Request you to test your flow with this enabled.

You might need to add directly in the SINK settings and reexport the template.

@sumeetdas-dlpx sumeetdas-dlpx force-pushed the dlpx/pr/sumeetdas-dlpx/719dac7b-fcf1-4305-b21e-f5f1fb1db712 branch 3 times, most recently from 6adebfc to 8179bda Compare October 21, 2024 01:02
@sumeetdas-dlpx sumeetdas-dlpx force-pushed the dlpx/pr/sumeetdas-dlpx/719dac7b-fcf1-4305-b21e-f5f1fb1db712 branch from 6272164 to 40be998 Compare October 23, 2024 04:47
@sumeetdas-dlpx sumeetdas-dlpx marked this pull request as ready for review October 23, 2024 14:27
@sumeetdas-dlpx sumeetdas-dlpx requested a review from a team as a code owner October 23, 2024 14:27
@sumeetdas-dlpx sumeetdas-dlpx force-pushed the dlpx/pr/sumeetdas-dlpx/719dac7b-fcf1-4305-b21e-f5f1fb1db712 branch from b204213 to 0da24e0 Compare October 25, 2024 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants