-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #19 from hubmapconsortium/test-release
v2.0.0 release
- Loading branch information
Showing
20 changed files
with
1,466 additions
and
674 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,11 @@ | ||
# uuid-api for HuBMAP | ||
# HuBMAP UUID API | ||
The uuid-api service is a restful web service used to create and query UUIDs used across HuBMAP. Three types of IDs are supported: | ||
* HuBMAP UUID: Standard randomly generated 128 bit UUIDs represented as 32 hexadecimal digits. These are generated by the service. | ||
* HuBMAP DOI prefix: A randomly generated unique id that can be used to construct a HuBMAP DOI in the format ###.XXXX.###. These are optionally generated by the service. | ||
* HuBMAP Display Id: An id specified when generating a UUID and stored by the service to associate user defined ids with UUIDs. | ||
|
||
|
||
## Development and deployment environments | ||
## Docker development and deployment environments | ||
|
||
We have the following 5 development and deployment environments: | ||
|
||
|
@@ -89,4 +89,4 @@ You can also stop the running container and remove it by: | |
|
||
### Updating API Documentation | ||
|
||
The documentation for the API calls is hosted on SmartAPI. Modifying the `entity-api-spec.yaml` file and commititng the changes to github should update the API shown on SmartAPI. SmartAPI allows users to register API documents. The documentation is associated with this github account: [email protected]. Please contact Chuck Borromeo ([email protected]) if you want to register a new API on SmartAPI. | ||
The documentation for the API calls is hosted on SmartAPI. Modifying the `entity-api-spec.yaml` file and commititng the changes to github should update the API shown on SmartAPI. SmartAPI allows users to register API documents. The documentation is associated with this github account: [email protected]. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
1.7.0 | ||
2.0.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
version: "3.7" | ||
|
||
services: | ||
|
||
uuid-api: | ||
environment: | ||
- HOST_GID=${HOST_GID:-1000} | ||
- HOST_UID=${HOST_UID:-1000} | ||
init: true | ||
restart: always | ||
volumes: | ||
# Mount the VERSION file and BUILD file | ||
- "../VERSION:/usr/src/app/VERSION" | ||
- "../BUILD:/usr/src/app/BUILD" | ||
# Mount the source code to container | ||
- "../src:/usr/src/app/src" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
match (s:Sample) where s.organ starts with 'LY' set s.organ = 'LY' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
import traceback | ||
from py2neo import Graph | ||
from hubmap_commons.exceptions import ErrorMessage | ||
from hubmap_commons.properties import PropHelper | ||
from hubmap_commons import file_helper | ||
from uuid_worker import UUIDWorker | ||
import sys | ||
import os | ||
import json | ||
import hashlib | ||
import requests | ||
from shutil import copyfile | ||
|
||
SAMPLES_TO_FIX_Q = "match (s:Sample) where s.portal_metadata_upload_files is not null and not s.portal_metadata_upload_files = '[]' return s.uuid, s.portal_metadata_upload_files" | ||
|
||
class FixFileUploads: | ||
|
||
def __init__(self, property_file_name): | ||
self.props = PropHelper(property_file_name, required_props = ['neo4j.server', 'neo4j.username', 'neo4j.password', 'db.host', 'db.name', 'db.username', 'db.password', 'user.mapping', 'client.id', 'client.secret', 'old.ingest.upload.dir', 'new.ingest.upload.dir', 'uuid.api.url', 'user.nexus.token']) | ||
self.graph = Graph(self.props.get('neo4j.server'), auth=(self.props.get('neo4j.username'), self.props.get('neo4j.password'))) | ||
self.uuid_wrker = UUIDWorker(self.props.get('client.id'), self.props.get('client.secret'), self.props.get('db.host'), self.props.get('db.name'), self.props.get('db.username'), self.props.get('db.password')) | ||
self.old_ingest_upload_dir = file_helper.ensureTrailingSlash(self.props.get('old.ingest.upload.dir')) | ||
self.new_ingest_upload_dir = file_helper.ensureTrailingSlash(self.props.get('new.ingest.upload.dir')) | ||
self.uuid_api_url = file_helper.ensureTrailingSlashURL(self.props.get('uuid.api.url')) + "hmuuid" | ||
self.user_token = self.props.get('user.nexus.token') | ||
|
||
def fix_sample_file_uploads(self): | ||
samples = self.graph.run(SAMPLES_TO_FIX_Q).data() | ||
tofix = [] | ||
for sample in samples: | ||
uuid = sample['s.uuid'] | ||
file_json = sample['s.portal_metadata_upload_files'].replace("'", '"') | ||
file_info = json.loads(file_json) | ||
changes = {"uuid":uuid, "changes":[]} | ||
tofix.append(changes) | ||
for fi in file_info: | ||
change_info = {} | ||
file_path = fi['filepath'] | ||
dirs = file_path.split('/') | ||
n_dirs = len(dirs) | ||
old_rel_dir = dirs[n_dirs-3] + os.sep + dirs[n_dirs-2] | ||
|
||
change_info['copy_from'] = self.old_ingest_upload_dir + old_rel_dir + os.sep + dirs[n_dirs-1] | ||
if not os.path.exists(change_info['copy_from']): | ||
raise Exception(f'File {change_info["copy_from"]} attached to entity with uuid:{uuid} does not exist') | ||
change_info['file_uuid'] = self.__gen_file_uuid(change_info['copy_from'], uuid + os.sep + '<uuid>' + os.sep + dirs[n_dirs-1], uuid) | ||
new_rel_dir = uuid + os.sep + change_info['file_uuid'] | ||
new_rel_path = new_rel_dir + os.sep + dirs[n_dirs-1] | ||
change_info['copy_to'] = self.new_ingest_upload_dir + new_rel_path | ||
change_info['file_path'] = new_rel_path | ||
change_info['file_name'] = dirs[n_dirs-1] | ||
if not os.path.exists(self.new_ingest_upload_dir + new_rel_dir): | ||
os.makedirs(self.new_ingest_upload_dir + new_rel_dir) | ||
description = "" | ||
if 'description' in fi: | ||
change_info['description'] = fi['description'] | ||
changes['changes'].append(change_info) | ||
|
||
for fix_rcd in tofix: | ||
uuid = fix_rcd['uuid'] | ||
new_files_recd = [] | ||
for change in fix_rcd['changes']: | ||
print(f'copy {change["copy_from"]} to {change["copy_to"]}') | ||
file_rcd = {} | ||
file_rcd['filename'] = change['file_name'] | ||
file_rcd['file_uuid'] = change['file_uuid'] | ||
if 'description' in change: | ||
file_rcd['description'] = change['description'] | ||
new_files_recd.append(file_rcd) | ||
copyfile(change["copy_from"], change["copy_to"]) | ||
|
||
#update the record | ||
update_cql = "match (s:Sample {uuid:'" + uuid + "'}) set s.metadata_files = '" + json.dumps(new_files_recd) + "' return s.uuid" | ||
self.graph.run(update_cql) | ||
#print(uuid + ":" + json.dumps(new_files_recd)) | ||
#if file_path | ||
#print(f"{uuid}:{file_json}") | ||
|
||
def __gen_file_uuid(self, file_current_path, file_rel_path, parent_entity_uuid): | ||
checksum = hashlib.md5(open(file_current_path, 'rb').read()).hexdigest() | ||
filesize = os.path.getsize(file_current_path) | ||
headers = {'Authorization': 'Bearer ' + self.user_token, 'Content-Type': 'application/json'} | ||
data = {} | ||
data['entity_type'] = 'FILE' | ||
data['parent_ids'] = [parent_entity_uuid] | ||
file_info= {} | ||
file_info['path'] = file_rel_path | ||
file_info['checksum'] = checksum | ||
file_info['base_dir'] = 'INGEST_PORTAL_UPLOAD' | ||
file_info['size'] = filesize | ||
data['file_info'] = [file_info] | ||
response = requests.post(self.uuid_api_url, json = data, headers = headers, verify = False) | ||
if response is None or response.status_code != 200: | ||
raise Exception(f"Unable to generate uuid for file {file_rel_path}") | ||
|
||
rsjs = response.json() | ||
file_uuid = rsjs[0]['uuid'] | ||
return file_uuid | ||
|
||
|
||
def tfix(self): | ||
samples = self.graph.run("match (s:Sample) where not s.metadata_files is null return s.uuid, s.metadata_files").data() | ||
for sample in samples: | ||
uuid = sample['s.uuid'] | ||
file_json = sample['s.metadata_files'].replace("'", '"') | ||
file_info_arry = json.loads(file_json) | ||
replace_arry = [] | ||
for file_info in file_info_arry: | ||
if 'file_path' in file_info: | ||
f_path = file_info['file_path'] | ||
s_path = f_path.split('/') | ||
file_info['filename'] = s_path[3] | ||
print(file_info['filename']) | ||
file_info.pop('file_path') | ||
replace_arry.append(file_info) | ||
update_cql = "match (s:Sample {uuid:'" + uuid + "'}) set s.metadata_files = '" + json.dumps(replace_arry) + "' return s.uuid" | ||
self.graph.run(update_cql) | ||
|
||
try: | ||
fixer = FixFileUploads(os.path.dirname(os.path.realpath(__file__)) + "/reload.properties") | ||
fixer.fix_sample_file_uploads() | ||
|
||
except ErrorMessage as em: | ||
print(em.get_message()) | ||
exit(1) | ||
except Exception as e: | ||
exc_type, exc_value, exc_traceback = sys.exc_info() | ||
eMsg = str(e) | ||
print("ERROR Occurred: " + eMsg) | ||
traceback.print_tb(exc_traceback) | ||
exit(1) | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# Point to the MySQL db that will be loaded | ||
db.host=hubmap-mysql | ||
db.name=hm.uuid | ||
db.username=root | ||
db.password=123 | ||
|
||
#a mapping of user email addresses to user ids used when | ||
#a user id is missing in neo4j | ||
user.mapping={"[email protected]":"43e53b4a-7853-33e4-99f3-bce224a0e312","[email protected]":"5eab9484-6bec-486d-88cc-7492620a3d6c","[email protected]":"6ce3090a22-11c8-3544-cc53-b91f8d257362"} | ||
|
||
#the location of the files previously upload via the ingest portal | ||
old.ingest.upload.dir=/Users/bill/projects/hubmap/ingest-uploads-testing/uploads | ||
#location where the previously uploaded files should be copied | ||
new.ingest.upload.dir=/Users/bill/projects/hubmap/file_uploads | ||
|
||
uuid.api.url=https://uuid-api.refactor.hubmapconsortium.org/ | ||
|
||
|
||
# Point to the Neo4j database that will be used | ||
# to load the uuid database | ||
neo4j.server=bolt://hubmap-neo4j:7687 | ||
neo4j.username=neo4j | ||
neo4j.password=123 |
Oops, something went wrong.