Write a script to create a full local copy of production database on dev

[jluebeck]

Allows better testing of weirdness that can happen in production environment.

[liefeld]

Made a first pass at this this morning with limited results that I am recording here. This should be doable using the mongoexport/mongoimport commands but I cannot seem to get a connection to the db from the amplicon-dev ubuntu node ( error connecting to db server: no reachable servers). Not sure if this is a mongo/documentdb version incompatibility. From within the container I can connect using pymongo from within the container. Export script is

from bson.json_util import dumps
from pymongo import MongoClient

mstr = "DB_URI_SECRET from config file"
client = MongoClient(mstr)

collections = database.collection_names()
for i, collection_name in enumerate(collections):
col = getattr(db,collections[i])
print(f”saving {collection_name}”)
collection = col.find()
jsonpath = collection_name + ".json"
with open(jsonpath, 'wb') as jsonfile:
jsonfile.write(dumps(collection))

[liefeld]

The DBs are too big to export on dev or prod nodes (prod is 145GB, disk is 80GB so it won't fit). In the meantime I added a file /home/ubuntu/AmpliconRepository-dev/mongo_shell.sh that is setup so that you can use a docker container of appropriate version to get a mongo shell into the DocumentDB database. It should be easily modifiable to use mongodump and mongorestore to make a new copy

https://www.mongodb.com/docs/manual/release-notes/4.2-compatibility/#remove-support-for-the-copydb-and-clone-commands

[jluebeck]

@liefeld - I'm not sure it needs to be the entire DB that we clone for testing purposes. How about something like all public non-deleted projects? Or all non-deleted projects for a specific user. Thoughts?

[liefeld]

use caper
db.projects.find({'private': false, 'delete': false}).forEach(function(x){
db.getSiblingDB('caper-dev')['projects'].insert(x);
});

[jluebeck]

Shelving this issue for the moment because there are no issues currently where we need to have a copy of the production database. Copy is also very large (80gb?).