Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix merge conflicts with master #291

Merged
merged 16 commits into from
Jul 12, 2023
Prev Previous commit
Next Next commit
Release/2.9.4 (#260)
* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Display es scores (#199)

* Include ES scores in variable results

* Round ES score to 6

* Update _version.py (#200)

* Dev version bump (#202)

* Release/2.8.0 (#198)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Update _version.py

* Update _version.py

updating version for final push to master

* Update factory.py

Adding more comments

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Release/v2.9.0 (#201)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Display es scores (#199)

* Include ES scores in variable results

* Round ES score to 6

* Update _version.py (#200)

* Update _version.py

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* Attribute mapping from node to dug element (#203)

* Release/2.8.0 (#198)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Update _version.py

* Update _version.py

updating version for final push to master

* Update factory.py

Adding more comments

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Release/v2.9.0 (#201)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Display es scores (#199)

* Include ES scores in variable results

* Round ES score to 6

* Update _version.py (#200)

* Update _version.py

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* adding more config options for node extraction

* some refactoring

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* Changed DbGaP to SPARC in the scicrunch parser (#204)

* Anvil (#207)

* Added updated anvil dataset catalog

* Added script for downloading all anvil data dicts

* Added current anvil data dictionaries to data folder to be used for indexing

* Anvil parser (#208)

* Release/2.8.0 (#198)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Update _version.py

* Update _version.py

updating version for final push to master

* Update factory.py

Adding more comments

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Release/v2.9.0 (#201)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Display es scores (#199)

* Include ES scores in variable results

* Round ES score to 6

* Update _version.py (#200)

* Update _version.py

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* anvil parser

* bump number of files test

* Update dbgap_parser.py

* Update anvil_dbgap_parser.py

change to AnVIL

* Update test_parsers.py

update test

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* Initial Kaniko build.

* Move version file definition.

* Quote env vars.

* Update env vars.

* Update env vars.

* Update env vars.

* env var changes.

* env var changes.

* env var changes.

* env var changes.

* Update DOCKER_IMAGE var.

* Update DOCKER_IMAGE var in kaniko cmd.

* Update kaniko destination line.

* Update kaniko destination line.

* Moree variable madness.

* Programatically remove quotes from version tag.

* dug dump concepts api created and tested (#229)

Co-authored-by: Nathan Braswell <nbraswell@rti.org>

* Update _version.py (#234)

* Version changes + separate build and publish.

* Semantic versioning prep.

* Add develop and master versioning and tagging.

* Bump version.

* Revert version to dug format.

* Ncpi index fix (#232)

* Renamed anvil to ncpi

* Update ncpi datasets catalog

* Modified script to download NCPI datasets into platform subfolders

* Updated NCPI integration dataset

* Removed unused variable

* Removed ncpi top level folder to spread results among subfolders

* Change output dir to data instead of ncpi subdir

* Moved NCPI subdirs into main data folder for ingest as per Yaphet's request

Co-authored-by: Alex Waldrop <awaldrop@RTI-108292.rti.ns>

* Add github creds env var.

* Fix version typo.

* Initial commit

* Reduce ephemeral storage limits and requests

* More parsers (#248)

* Release/2.8.0 (#198)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Update _version.py

* Update _version.py

updating version for final push to master

* Update factory.py

Adding more comments

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Release/v2.9.0 (#201)

* Bumping version

* support for extracting dug elements from graph (#197)

* support for extracting dug elements from graph

* adding flag for enabling dug element extraction from graph

* adding new config for node_to dug element parsing

* adding more parameters to crawler to able configuration to element extraction logic

* add tests

* add tests for crawler

Co-authored-by: Yaphetkg <yaphetkg@renci.org>

* Display es scores (#199)

* Include ES scores in variable results

* Round ES score to 6

* Update _version.py (#200)

* Update _version.py

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* consolidate dbgap format parser in single file , adds crdc and kfdrc parsers

* adding tests

* bump version

* parser when versions of studies are > 9

* test for version

* fix long text issues, and encoding errors

* nltk initialization

* change nltk approach for sliding window

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>

* update version

* remove cruft from merge

Co-authored-by: Carl Schreep <schreepc@renci.org>
Co-authored-by: Yaphetkg <yaphetkg@renci.org>
Co-authored-by: Ginnie Hench <vhench@rti.org>
Co-authored-by: Howard Lander <howard@renci.org>
Co-authored-by: Alex Waldrop <39315592+alexwaldrop@users.noreply.github.com>
Co-authored-by: Charles Bennett <cnbennett3@gmail.com>
Co-authored-by: Nathaniel Braswell <braswent6@gmail.com>
Co-authored-by: Nathan Braswell <nbraswell@rti.org>
Co-authored-by: cnbennett3 <53792237+cnbennett3@users.noreply.github.com>
Co-authored-by: Alex Waldrop <awaldrop@RTI-108292.rti.ns>
Co-authored-by: Hoid <tylerlcheek@gmail.com>
  • Loading branch information
12 people authored Aug 23, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit f6f30677b4dd1179df58a81730564a03c85f52d6
63 changes: 20 additions & 43 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
library 'pipeline-utils@master'

pipeline {
agent {
kubernetes {
@@ -18,11 +20,11 @@ spec:
requests:
cpu: "512m"
memory: "1024Mi"
ephemeral-storage: "4Gi"
ephemeral-storage: "2Gi"
limits:
cpu: "1024m"
memory: "2048Mi"
ephemeral-storage: "8Gi"
ephemeral-storage: "2Gi"
command:
- /busybox/cat
tty: true
@@ -51,6 +53,7 @@ spec:
environment {
PATH = "/busybox:/kaniko:/ko-app/:$PATH"
DOCKERHUB_CREDS = credentials("${env.CONTAINERS_REGISTRY_CREDS_ID_STR}")
GITHUB_CREDS = credentials("${env.GITHUB_CREDS_ID_STR}")
REGISTRY = "${env.REGISTRY}"
REG_OWNER="helxplatform"
REG_APP="dug"
@@ -66,19 +69,10 @@ spec:
stages {
stage('Build') {
steps {
container(name: 'kaniko', shell: '/busybox/sh') {
sh '''#!/busybox/sh
echo "Build stage"
/kaniko/executor --dockerfile ./Dockerfile \
--context . \
--verbosity debug \
--no-push \
--destination $IMAGE_NAME:$TAG1 \
--destination $IMAGE_NAME:$TAG2 \
--destination $IMAGE_NAME:$TAG3 \
--destination $IMAGE_NAME:$TAG4 \
--tarPath image.tar
'''
script {
container(name: 'kaniko', shell: '/busybox/sh') {
kaniko.build("./Dockerfile", ["$IMAGE_NAME:$TAG1", "$IMAGE_NAME:$TAG2", "$IMAGE_NAME:$TAG3", "$IMAGE_NAME:$TAG4"])
}
}
}
post {
@@ -96,34 +90,17 @@ spec:
}
stage('Publish') {
steps {
container(name: 'crane', shell: '/busybox/sh') {
sh '''
echo "Publish stage"
echo "$DOCKERHUB_CREDS_PSW" | crane auth login -u $DOCKERHUB_CREDS_USR --password-stdin $REGISTRY
crane push image.tar $IMAGE_NAME:$TAG1
crane push image.tar $IMAGE_NAME:$TAG2
if [ $BRANCH_NAME == "develop" ]; then
crane push image.tar $IMAGE_NAME:$TAG3
elif [ $BRANCH_NAME == "master" ]; then
crane push image.tar $IMAGE_NAME:$TAG3
crane push image.tar $IMAGE_NAME:$TAG4
if [ $(git tag -l "$VERSION") ]; then
echo "ERROR: Tag with version $VERSION already exists! Exiting."
else
# Recover some things we've lost:
git config --global user.email "helx-dev@lists"
git config --global user.name "rencibuild rencibuild"
grep url .git/config
git checkout $BRANCH_NAME

# Set the tag
SHA=$(git log --oneline | head -1 | awk '{print $1}')
git tag $VERSION "$SHA"
git remote set-url origin https://$GITHUB_CREDS_PSW@github.com/helxplatform/dug.git
git push origin --tags
fi
fi
'''
script {
container(name: 'crane', shell: '/busybox/sh') {
def imageTagsPushAlways = ["$IMAGE_NAME:$TAG1", "$IMAGE_NAME:$TAG2"]
def imageTagsPushForDevelopBranch = ["$IMAGE_NAME:$TAG3"]
def imageTagsPushForMasterBranch = ["$IMAGE_NAME:$TAG4"]
image.publish(
imageTagsPushAlways,
imageTagsPushForDevelopBranch,
imageTagsPushForMasterBranch
)
}
}
}
}
120 changes: 0 additions & 120 deletions bin/get_anvil_data_dicts.py

This file was deleted.

Loading