Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

MSSQL Init DB Experiments #153

Closed
wants to merge 5 commits into from
Closed

MSSQL Init DB Experiments #153

wants to merge 5 commits into from

Conversation

pattisdr
Copy link
Contributor

@pattisdr pattisdr commented Jan 10, 2022

Purpose

Experiments trying to initialize a test mssql db and populate it when we bring up the mssql container on M1.

Known issues:

  1. We can't run sql server in docker on M1 because there's no arm64 image - using azure instead
  2. While PostgreSQL lets you put an init SQL script in docker-entrypoint-initdb.d, this isn't available for mssql
  3. Sqlcmd tool not available inside the arm64 version of edge containers

Changes

Three experiments:

  1. First commit tries to create a Dockerfile where we bring up azure and then use sqlcmd to run an initialization script but it looks like I can't use sqlcmd on M1
  2. Second commit tries to bring up an additional container just to run a python script to populate database after all the other containers are up when running make integration-env. Errored with Failed to load /var/opt/mssql/mssql.conf ini file with error open /var/opt/mssql/mssql.conf: no such file or directory. Maybe this is permission issues, not sure. This solution isn't ideal because I have to reinstall a lot of things that are in the fidesops container - it would be nice to figure out how to run this python script on the fidesops container -
  3. Running the mssql_setup python function which creates a mssql database and populates when we run start_webserver. Works but not great, as we we're combining methods to populate test databases with our core method to bring up fidesops.

…e, but it looks like sqlcmd not available in arm64 version of azure edge containers.
… database, but it looks like sqlcmd not available in arm64 version of azure edge containers."

This reverts commit 8d415d9.
…python script. Tried to put minimal pieces in mssql Dockerfile to just run mssql and python script, but lots of duplication .

mssql_init_db waits until fidesops comes up, then dies. looks like mssql container has error:

mssql_example_database | 2022/01/08 22:58:33 [launchpadd] WARNING: Failed to load /var/opt/mssql/mssql.conf ini file with error open /var/opt/mssql/mssql.conf: no such file or directory
mssql_example_database | 2022/01/08 22:58:33 [launchpadd] INFO: DataDirectories =  /bin:/etc:/lib:/lib32:/lib64:/sbin:/usr/bin:/usr/include:/usr/lib:/usr/lib32:/usr/lib64:/usr/libexec/gcc:/usr/sbin:/usr/share:/var/lib:/opt/microsoft:/opt/mssql-extensibility:/opt/mssql/mlservices:/opt/mssql/lib/zulu-jre-11:/opt/mssql-tools
mssql_example_database | 2022/01/08 22:58:33 Drop permitted effective capabilities.

mssql_example_database | 2022/01/08 22:58:33 [launchpadd] INFO: Polybase remote hadoop bridge disabled
mssql_example_database | 2022/01/08 22:58:33 [launchpadd] INFO: Launchpadd is connecting to mssql on localhost:1431
mssql_example_database | 2022/01/08 22:58:33 [launchpadd] WARNING: Failed to connect to SQL because: dial tcp 127.0.0.1:1431: connect: connection refused, will reattempt connection.
…example python script. Tried to put minimal pieces in mssql Dockerfile to just run mssql and python script, but lots of duplication ."

This reverts commit ed228b3.
…not ideal at all, we should keep these test integrations separate from this main process.
@pattisdr pattisdr closed this Jan 10, 2022
@pattisdr
Copy link
Contributor Author

Seems to be solved with bringing up docker containers in detached mode including mssql one, then exec-ing into the fidesops to run the python script which creates an mssql_example db and populates, and then we do docker logs for all these containers afterwards.

integration-env: compose-build
	@echo "Bringing up main image and images for integration testing"
	@docker-compose -f docker-compose.yml -f docker-compose.integration-test.yml up -d
	@echo "Waiting 15s for integration containers to be ready..."
	@sleep 15
	@echo "Running additional setup for mssql integration tests"
	@docker exec -it fidesops python tests/integration_tests/mssql_setup.py
	@docker-compose -f docker-compose.yml -f docker-compose.integration-test.yml logs -f -t

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant