Skip to content

Files

Latest commit

 

History

History

soda-data-quality-checks

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 

Demo: Soda data quality checks with LocalStack Snowflake

This project illustrates how to use the Soda framework to run data quality checks against Snowflake tables, entirely on your local machine.

The code is based on the Snowflake Quickstart Guide on Data Quality Testing with Soda.

Prerequisites

To install the snow CLI with a local connection profile, follow the instructions here. It basically boils down to running this command:

snow connection add \
    --connection-name localstack \
    --user test \
    --password test \
    --account test \
    --host snowflake.localhost.localstack.cloud

Instructions

Install dependencies

Run the following command to install dependencies for the project (into a local Python virtual environment):

make install

Start LocalStack

Start the LocalStack Snowflake emulator using the following command:

DOCKER_FLAGS='-e SF_LOG=trace' \
  IMAGE_NAME=localstack/snowflake \
  DEBUG=1 \
  localstack start

Initialize the data tables

The sample application provides Makefile targets to simplify the setup process.

Run the following command to initialize the environment and seed test data into local Snowflake:

make init  

Use Soda to run data quality checks

Once the test data has been set up, we can run the Soda data quality checks via this command:

make scan

License

The code in this project is licensed under the Apache 2.0 License.