Code

Pipeline parameterize restructure#95

Code

Merged

YaphetKG merged 197 commits intodevelophelxplatform/roger:developfrom pipeline_parameterize_restructurehelxplatform/roger:pipeline_parameterize_restructureCopy head branch name to clipboardApr 18, 2024

+2,238-470

Commits

Commits on Mar 23, 2023

• roger cli preped for Merge Deploy

  

  Nathan Braswell

  committed
  290ca53

Commits on Mar 29, 2023

• Update Makefile to work with python env

  

  esurface

  committed
  1e46a36

• Update redisgraph-bulk-loader to fix issue with loading MODULE LIST

  

  esurface

  committed
  7baf7ef

• Revert "Update redisgraph-bulk-loader to fix issue with loading MODULE LIST"

  Show description for 0f5e35e
  

  esurface

  committed
  0f5e35e

Commits on Mar 31, 2023

• Finalized dev deployment of dug inside Catapult Merge, deployment yamls, code changes and configurations

  

  Nathan Braswell

  committed
  ee17823

Commits on Apr 17, 2023

• updated to reflect the Dug-Api updates to FastAPI

  

  Nathan Braswell

  committed
  135d54b

Commits on May 5, 2023

• adding multi label redis by removing 'biolink:' on nodes, edges cannot be fixed after update so they need to be solved either by changing TranQl AND Plater or forking bulk-redisgraph to allow for c…

  Show description for bb148a4
  

  Nathan Braswell

  committed
  bb148a4

Commits on May 24, 2023

• Working multi label redis nodes w/ no biolink label

  braswent

  committed
  b6ea2c7

Commits on May 25, 2023

• Latest code changes to deploy working Roger in Merge

  braswent

  committed
  5b85480

Commits on Jun 9, 2023

• biolink data move to '.' separator

  braswent

  committed
  0083b60

Commits on Jun 20, 2023

• updates to include new dug fixes, upgraded redis-bulk-loader and made changes to for biolink variables to specify it's domain with a 'biolink.'

  braswent

  committed
  c7b3a71

Commits on Jun 21, 2023

• adding test roger code

  braswent

  committed
  50f5b23

Commits on Aug 9, 2023

• removed helm deployments

  braswent

  committed
  444fd16

Commits on Sep 29, 2023

• Merge remote-tracking branch 'rti/develop' into rti-merge

  YaphetKG

  committed
  ce54b73

• change docker owner

  YaphetKG

  committed
  28fb94b

• remove core.py

  YaphetKG

  committed
  142e874

• remove dup dev config

  YaphetKG

  committed
  3b4ed65

• redis graph is not directly used removing cruft

  YaphetKG

  committed
  d6a779a

• remove print statement

  YaphetKG

  committed
  1df8ed5

• remove logging files

  YaphetKG

  committed
  e608fa1

• update requriemtns

  YaphetKG

  committed
  d2cee55

• update requriemtns

  YaphetKG

  committed
  89ba673

Commits on Oct 2, 2023

• add redis graph.py

  YaphetKG

  committed
  82161c3

• fix import error for logger

  YaphetKG

  committed
  8ef6b49

• adding es scheme and ca_path config

  YaphetKG

  committed
  d03acb0

• adding es scheme and ca_path config

  YaphetKG

  committed
  16e3d3a

• Parameterized annotate tasks with input_data_path and output_data_path

  mbacon-renci

  committed
  a84e926

Commits on Oct 3, 2023

• adding debug code

  YaphetKG

  committed
  0beba39

• removing debug

  YaphetKG

  committed
  5725045

• adding nodes args

  YaphetKG

  committed
  ae8b0fd

• adding biolink.

  YaphetKG

  committed
  a9e422f

• adding biolink.

  YaphetKG

  committed
  5c37b91

Commits on Oct 10, 2023

• Parameterized annotate tasks with input_data_path and output_data_path (#85)

  mbacon-renci

  authored
  662d153

• adding lakefs changes to roger-2.0

  YaphetKG

  committed
  1c212dc

• point avalon to vg1 branch

  YaphetKG

  committed
  f7feb78

• change avalon dep

  YaphetKG

  committed
  e335ce4

Commits on Oct 11, 2023

• update airflow

  YaphetKG

  committed
  42ab8d1

• fix avalon tag typo

  YaphetKG

  committed
  148c3f6

• update jenkins to tag version on main branch only

  YaphetKG

  committed
  f9157ac

• update jenkins to tag version

  YaphetKG

  committed
  86ad101

• update jenkins to tag version

  YaphetKG

  committed
  793e396

• psycopg2 installation

  YaphetKG

  committed
  044e4f1

• add cncf k8s req

  YaphetKG

  committed
  0996b16

• use airflow non-slim

  YaphetKG

  committed
  30476cd

• simplified for testing

  YaphetKG

  committed
  1727c56

• simplified for testing

  YaphetKG

  committed
  dee9b68

• change dag name

  YaphetKG

  committed
  1696564

• Erroneous parameter passed, should not be None

  mbacon-renci

  committed
  b749316

• Merge branch 'develop' into patch/input_output_path_params

  mbacon-renci

  committed
  ab5307b

• Catching up to roger-2.0 build

  mbacon-renci

  committed
  c3a4aa2

Commits on Oct 12, 2023

• adding pre-exec

  YaphetKG

  committed
  cda7389

• adding pre-exec

  YaphetKG

  committed
  322d888

• adding pre-exec

  YaphetKG

  committed
  0922c5a

• typo preexec

  YaphetKG

  committed
  ed3008a

• typo preexec

  YaphetKG

  committed
  0155f3e

• fix context

  YaphetKG

  committed
  d3c7c24

Commits on Oct 17, 2023

• get files from repo

  YaphetKG

  committed
  a190c65

• get files from repo

  YaphetKG

  committed
  ebc49ed

• get files from repo

  YaphetKG

  committed
  3716179

• get files from repo

  YaphetKG

  committed
  81f69a4

• First shot at moving pipeline into base class and implementing. Anvil pipeline not complete

  mbacon-renci

  committed
  84ceaf7

• Syntax fix, docker image version bump to airflow 2.7.2-python3.11

  mbacon-renci

  committed
  7ee69fe

• update storage dir

  YaphetKG

  committed
  b13c4c0

• update remove dir code

  YaphetKG

  committed
  1ca6ead

• update remove dir code

  YaphetKG

  committed
  694cc05

• remote path to *

  YaphetKG

  committed
  3341ef8

• fix input dir for annotators

  YaphetKG

  committed
  80793bb

• fix input dir for annotators

  YaphetKG

  committed
  7233742

• fix input dir for annotators

  YaphetKG

  committed
  4984dfb

• kwargs to task

  YaphetKG

  committed
  bff30a9

• kwargs to task

  YaphetKG

  committed
  58be140

• kwargs to task

  YaphetKG

  committed
  d107d53

• kwargs to task

  YaphetKG

  committed
  a56d239

Commits on Oct 18, 2023

• kwargs to task

  YaphetKG

  committed
  ecfd4e2

• kwargs to task

  YaphetKG

  committed
  08a9ac4

• kwargs to task

  YaphetKG

  committed
  92bdb18

• kwargs to task

  YaphetKG

  committed
  00dc858

• kwargs to task

  YaphetKG

  committed
  3751ed4

• kwargs to task

  YaphetKG

  committed
  eb473a7

• kwargs to task

  YaphetKG

  committed
  6cc7984

• adding branch info on lakefs config

  YaphetKG

  committed
  3825dbb

• callback push to branch

  YaphetKG

  committed
  93ad480

• back to relative import

  YaphetKG

  committed
  bef2d3f

• reformat temp branch name based on unique task id

  YaphetKG

  committed
  802c499

• add logging

  YaphetKG

  committed
  cabecb4

• add logging

  YaphetKG

  committed
  29a7d65

• convert posix path to str for avalon

  YaphetKG

  committed
  5211e6f

• add extra / to root path

  YaphetKG

  committed
  585674d

Commits on Oct 20, 2023

• New dag created using DugPipeline subclasses

  mbacon-renci

  committed
  e95303d

• EmptyOperator imported from wrong place

  mbacon-renci

  committed
  c105af6

• import and syntax fixes

  mbacon-renci

  committed
  fd1aa05

Commits on Oct 21, 2023

• utterly silly syntax error

  mbacon-renci

  committed
  55e9970

• Added anvil to default input data sets for testing purposes

  mbacon-renci

  committed
  61afe68

Commits on Oct 23, 2023

• adding / to local path

  YaphetKG

  committed
  7689203

• commit meta task args empty string

  YaphetKG

  committed
  4357fbe

• add merge logic

  YaphetKG

  committed
  d985d58

• add merge logic

  YaphetKG

  committed
  04bb9dc

• upstream task dir pull for downstream task

  YaphetKG

  committed
  79d5936

• Switched from subdag to taskgroup because latest Airflow depricated subdag

  mbacon-renci

  committed
  add4ce3

• Added BACPAC pipeline object

  mbacon-renci

  committed
  9116311

• Temporarily ignoring configuration variable for enabled datasets for testing

  mbacon-renci

  committed
  9f92ec0

• Passed dag in to create task group to see if it helps dag errors

  mbacon-renci

  committed
  813de3d

• Fixed silly syntax error

  mbacon-renci

  committed
  af79f5d

• adding input / output dir params for make kgx

  YaphetKG

  committed
  6bffb12

• Trying different syntax to make taskgroups work.

  mbacon-renci

  committed
  d39371b

• adding input / output dir params for make kgx

  YaphetKG

  committed
  c3e57ea

• Parsing, syntax, pylint fixes

  mbacon-renci

  committed
  738c3f0

• adding input / output dir params for make kgx

  YaphetKG

  committed
  c8ca1ed

Commits on Oct 24, 2023

• Added pipeline name to task group name to ensure uniqueness

  mbacon-renci

  committed
  e3bd1c8

• oops, moved something out of scope. Fixed

  mbacon-renci

  committed
  764a74a

Commits on Oct 26, 2023

• Filled out pipeline with methods from dug_utils. Needs data path changes

  mbacon-renci

  committed
  583178a

Commits on Nov 3, 2023

• Finished implementing input_data_path and output_data_path handling, pylint cleanup

  mbacon-renci

  committed
  f56a44f

• Merged lakefs-2.0 changes into restructure branch

  mbacon-renci

  committed
  64711fc

Commits on Dec 11, 2023

• Update requirements.txt

  YaphetKG

  authored
  dfe99e3

• adding toggle to avoid sending config obj

  YaphetKG

  committed
  7522a10

• adding toggle to avoid sending config obj

  YaphetKG

  committed
  a9eecd7

Commits on Dec 12, 2023

• disable to string for test

  YaphetKG

  committed
  05df5b1

• control pipelines for testing

  YaphetKG

  committed
  08ab1f0

• add self to anvil get files

  YaphetKG

  committed
  9819cf6

• add log stream to make it available

  YaphetKG

  committed
  95490ae

• typo fix

  YaphetKG

  committed
  c3fc968

Commits on Dec 13, 2023

• correcting branch id

  YaphetKG

  committed
  e66e446

• adding source repo

  YaphetKG

  committed
  7c1cb91

• adding source repo

  YaphetKG

  committed
  ccbee52

• patch name-resolver response

  YaphetKG

  committed
  ede3ba2

Commits on Dec 14, 2023

• no pass input repo and branch , if not overriden to pre-exec

  YaphetKG

  committed
  f610c79

• no pass input repo and branch , if not overriden to pre-exec

  YaphetKG

  committed
  5588623

• no pass input repo and branch , if not overriden to pre-exec

  YaphetKG

  committed
  a19acbf

• dug pipeline edit

  YaphetKG

  committed
  1cf283f

• recurisvely find recursively

  YaphetKG

  committed
  65affe6

• recurisvely find recursively

  YaphetKG

  committed
  b197d3d

• setup output path for crawling

  YaphetKG

  committed
  8864f6a

• all task functions should have input and output params

  YaphetKG

  committed
  2c01862

• adding annotation as upstream for validate index

  YaphetKG

  committed
  f09dc04

Commits on Dec 15, 2023

• revamp create task , and task wrapper

  YaphetKG

  committed
  cb5af24

• add validate concepts index task

  YaphetKG

  committed
  ba3809b

Commits on Dec 18, 2023

• adding concept validation

  YaphetKG

  committed
  4dc9ed0

• add index_variables task as dependecy for validate concepts

  YaphetKG

  committed
  0b7d0c4

• add index_variables task as dependecy for validate concepts

  YaphetKG

  committed
  34505c7

Commits on Dec 19, 2023

• await client exist

  YaphetKG

  committed
  08cf965

• await client exist

  YaphetKG

  committed
  5dd29bd

• concepts not getting picked up for indexing

  YaphetKG

  committed
  eaf238d

• concepts not getting picked up for indexing

  YaphetKG

  committed
  c6f89e4

• Merge branch 'develop' into pipeline_parameterize_restructure

  YaphetKG

  committed
  8b4aa83

• fix search elements

  YaphetKG

  committed
  ca06e0b

• converting annotation output to json

  YaphetKG

  committed
  b73602e

• json format annotation outputs

  YaphetKG

  committed
  ef6ed63

• adding support for json format elements and concepts read

  YaphetKG

  committed
  c53e377

• json back to dug objects

  YaphetKG

  committed
  31eeb45

• fixing index valriables with json objects

  YaphetKG

  committed
  cda0d6d

• indetation and new line for better change detection :?

  YaphetKG

  committed
  4bd0ee5

• indetation and new line for better change detection

  YaphetKG

  committed
  ce24e45

• treat dictionary concepts as dictionary

  YaphetKG

  committed
  af4001a

• read concepts json as a dict

  YaphetKG

  committed
  ada224d

• concepts files are actually file paths

  YaphetKG

  committed
  b6fa0bb

• debug message

  YaphetKG

  committed
  bbe946c

• make output jsonable

  YaphetKG

  committed
  9c72c1c

Commits on Dec 20, 2023

• clear up dir after commit , and delete unmerged branch even if no changes

  YaphetKG

  committed
  9b636cd

• don`t clear indexes, parallel dataset processing will be taxed

  YaphetKG

  committed
  6891596

• memory leak?

  YaphetKG

  committed
  6f0f4a5

• memory leak?

  YaphetKG

  committed
  c0250b5

• memory leak?

  YaphetKG

  committed
  0b2e103

• dumping pickles to debug locally

  YaphetKG

  committed
  0cc5eeb

Commits on Dec 21, 2023

• find out why concepts are being added to every other element

  YaphetKG

  committed
  c7e3f96

• find out why concepts are being added to every other element

  YaphetKG

  committed
  de3fd93

• pointless shuffle 🤷‍♂️

  YaphetKG

  committed
  f7d0bd8

• revert back in time

  YaphetKG

  committed
  3982b12

• back to sanitize dug

  YaphetKG

  committed
  6370afd

• output just json for annotation

  YaphetKG

  committed
  c1821e5

• adding jsonpickle

  YaphetKG

  committed
  18f3d29

• jsonpickle 🥒

  YaphetKG

  committed
  87adfe0

• unpickle for index

  YaphetKG

  committed
  daaca7a

• unpickle for validate index

  YaphetKG

  committed
  905f151

• crawling fixes

  YaphetKG

  committed
  35daa88

• crawling fixes

  YaphetKG

  committed
  8d0ca0d

• crawling validation fixes

  YaphetKG

  committed
  d246476

• fix index concepts

  YaphetKG

  committed
  8850681

• fix makekgx

  YaphetKG

  committed
  abaa33a

• adding other bdc pipelines

  YaphetKG

  committed
  ba2a7d8

• adding pipeline paramters to be able to configure per instance

  YaphetKG

  committed
  da266d6

• fix

  YaphetKG

  committed
  6a2e69c

• add input dataset for pipelines

  YaphetKG

  committed
  592a32f

• Adding README to document how to create data set-specific pipelines

  mbacon-renci

  committed
  7588a66

• catchup on base.py

  mbacon-renci

  committed
  7f8f482

• Diverge/catchup merge to add README.md

  mbacon-renci

  committed
  84cd683

• Added dbgap and nida pipelines

  mbacon-renci

  committed
  f133b86

• fix import errors

  YaphetKG

  committed
  705b3db

Commits on Dec 22, 2023

• Merge branch 'develop' into pipeline_parameterize_restructure

  YaphetKG

  authored
  f79350a

Commits on Jan 29, 2024

• annotator modules added by passing config val (#90)

  Show description for 21e4271
  

  braswent

  and

  YaphetKG

  authored
  21e4271

Commits on Feb 8, 2024

• Add heal parsers (#96)

  Show description for d3e6c53
  

  YaphetKG

  and

  braswent

  authored
  d3e6c53

Commits on Feb 13, 2024

• Add heal parsers (#97)

  Show description for 45d1182
  

  YaphetKG

  and

  braswent

  authored
  45d1182

Commits on Mar 12, 2024

• Radx pipeline (#99)

  Show description for 8dfeeb7
  YaphetKG

  authored
  8dfeeb7

• adding indexes as part of bulk loader paramters

  YaphetKG

  committed
  6852ca3

Commits on Mar 13, 2024

• fix id index cli arg

  YaphetKG

  committed
  38b8d01

Commits on Apr 18, 2024

• fix local cli

  YaphetKG

  committed
  08fb4fd

• dug latest

  YaphetKG

  committed
  5209aed

• Merge branch 'develop' into pipeline_parameterize_restructure

  YaphetKG

  authored
  9e900b8