-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add NGSPlotDB-hg19-data recipe #9157
Conversation
@bioconda/core The |
You can remove the |
Whoopsies! |
Hmm, doesn't build that way. I will probably resubmit in a different PR then?! @mbargull |
recipes/r-ngsplot-hg19/meta.yaml
Outdated
sha256: a3ad6daceec383f88faf3d3ee899f2bef37b3be2658ee9afa01e86404c0c92bd | ||
build: | ||
number: 0 | ||
rpaths: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can remove rpaths
for a data package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
recipes/r-ngsplot-hg19/meta.yaml
Outdated
- lib/R/lib/ | ||
- lib/ | ||
requirements: | ||
build: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No build
requirement needed for a data package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, removed that
|
||
# Platform-specific md5sum checks. | ||
if [[ $(uname -s) == "Linux" ]]; then | ||
if md5sum -c <<<"$MD5 $TARBALL"; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you use a sha256 hash, you'll have to use sha256sum
here.
But note that on macOS you'll likely have to use another command for this. Can't remember what, though.
The safe route might be to add a dependency on openssl
and use openssl sha256
? (E.g.,
openssl sha256 libStatGen-v$VERSION.tar.gz | grep 70a504c5cc4838c6ac96cdd010644454615cc907df4e3794c999baf958fa734b |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed back to md5
break | ||
fi | ||
else if [[ $(uname -s) == "Darwin" ]]; then | ||
if [[ $(md5 $TARBALL | cut -f4 -d " ") == "$MD5" ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sha256, see above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed to md5
recipes/r-ngsplot-hg19/meta.yaml
Outdated
package: | ||
name: 'ngsplotdb-{{ name|lower }}' | ||
version: '{{ version }}' | ||
source: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No source
needed since post-link.sh
does the job.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, removed that - but the md5 is still required / good practice right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nah, the md5
in meta.yaml
is only tied to the one in source
, hence can be removed, too.
recipes/r-ngsplot-hg19/post-link.sh
Outdated
fi | ||
|
||
# Install and clean up | ||
ngsplotdb.py install ngsplotdb_hg19_75_3.00.tar.gz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You want to use "${TARBALL}" here.
Also, if this outputs anything (progress status and such), you'll want to write that to .messages.txt
, see https://conda.io/docs/user-guide/tasks/build-packages/link-scripts.html, i.e.,
ngsplotdb.py install "${TARBALL}" > "${PREFIX}/.messages.txt"
maybe even
ngsplotdb.py install "${TARBALL}" > "${PREFIX}/.messages.txt" 2>&1
to also capture stderr
.
recipes/r-ngsplot-hg19/pre-unlink.sh
Outdated
@@ -0,0 +1 @@ | |||
ngsplotdb.py remove hg19 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Redirect output to .messages.txt
, see above.
recipes/r-ngsplot-hg19/post-link.sh
Outdated
|
||
if [[ $SUCCESS != 1 ]]; then | ||
echo "ERROR: post-link.sh was unable to download any of the following URLs with the md5sum $MD5:" | ||
printf '%s\n' "${URLS[@]}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think even this should write to .messages.txt
(see below), i.e.,
echo "ERROR: post-link.sh was unable to download any of the following URLs with the md5sum $MD5:" > "${PREFIX}/.messages.txt" 2>&1
printf '%s\n' "${URLS[@]}" > "${PREFIX}/.messages.txt" 2>&1
You can use this one, no problem. +#!/bin/bash
+outdir=$PREFIX/share/$PKG_NAME-$PKG_VERSION-$PKG_BUILDNUM
+mkdir -p $outdir
+mkdir -p $PREFIX/bin
+cp -R * $outdir ^-- please don't copy everything unconditionally but rather selectively the files you want. +#Set up links for
+for f in $outdir/*; do
+ ln -s $outdir/* $PREFIX/bin ^-- you'll want to use + fbname=$(basename "$f")
+ chmod 0755 ${PREFIX}/bin/$fbname
+done |
I took a quick peek at the file list in https://github.com/shenlab-sinai/ngsplot/archive/2.63.tar.gz. There is a |
And the files in |
Hi @mbargull ! Therefore, I'd rather say I restrict to simply add all tools in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few changes needed, but it looks nearly done 🎉.
r-ngsplot
built fine, but I'm not sure whether the tests actually succeeded. By piping to true
, you're masking the failures, I think (that is, I don't believe conda-build
runs the tests with -o pipefail
). Please remove all | true
.
If the programs return non-zero exit codes on success, a workaround is to pipe to grep -q 'some expression you know the program outputs
.
Therefore, I'd rather say I restrict to simply add all tools in bin to the path, similar to what the authors suggest in their readme . What do you think?
Seems fine to me.
recipes/r-ngsplot-hg19/meta.yaml
Outdated
- wget | ||
test: | ||
commands: | ||
- ngsplotdb.py install "${TARBALL}" > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test is run after the package is installed, i.e., after the post-link.sh
is run.
So, that ngsplotdb.py install
is already done at that point. Rather than executing it again, you should run a test that checks if the database has been successfully installed (either with some (short-running!) command from r-gnsplot
or if that's not available, at least something like test -f "${PREFIX}/some-file-path"
.)
recipes/r-ngsplot-hg19/meta.yaml
Outdated
package: | ||
name: 'ngsplotdb-{{ name|lower }}' | ||
version: '{{ version }}' | ||
md5: 172a6a93de3d1ae8a3ad6d6d79e911f7 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can remove that md5
here.
recipes/r-ngsplot-hg19/post-link.sh
Outdated
URLS=( | ||
"https://drive.google.com/uc?export=download&id=0B5hDZ2BucCI6SURYWW5XdUxnbW8" | ||
) | ||
sha256="a3ad6daceec383f88faf3d3ee899f2bef37b3be2658ee9afa01e86404c0c92bd" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MD5='172a6a93de3d1ae8a3ad6d6d79e911f7'
instead of sha256
recipes/r-ngsplot-hg19/post-link.sh
Outdated
|
||
SUCCESS=0 | ||
for URL in ${URLS[@]}; do | ||
wget -O- -q $URL > $TARBALL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The CI choked on the unquoted $URL
, I think.
wget -qO "${TARBALL}" "${URL}"
should do the trick.
recipes/r-ngsplot-hg19/post-link.sh
Outdated
done | ||
|
||
if [[ $SUCCESS != 1 ]]; then | ||
echo "ERROR: post-link.sh was unable to download any of the following URLs with the md5sum $MD5:" > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My bad, should be (appending to .messages.txt
):
echo "ERROR: post-link.sh was unable to download any of the following URLs with the md5sum $MD5:" >> "${PREFIX}/.messages.txt" 2>&1
recipes/r-ngsplot-hg19/pre-unlink.sh
Outdated
@@ -0,0 +1 @@ | |||
ngsplotdb.py remove hg19 > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, missing new line on EOF 😉
recipes/r-ngsplot-hg19/post-link.sh
Outdated
# Install and clean up | ||
ngsplotdb.py install "${TARBALL}" > "${PREFIX}/.messages.txt" 2>&1 | ||
rm $TARBALL | ||
rmdir $STAGING |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing new line on EOF 😉
recipes/r-ngsplot/build.sh
Outdated
for f in $outdir/*; do | ||
ln -s $outdir/* $PREFIX/bin | ||
for f in ${outdir}/bin/*; do | ||
ln -s ${f} ${PREFIX}/bin | ||
fbname=$(basename "$f") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can be removed now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't get it right now - why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I've been imprecise. Just meant the fbname=$(basename "$f")
part 😉
recipes/r-ngsplot/build.sh
Outdated
fbname=$(basename "$f") | ||
chmod 0755 ${PREFIX}/bin/$fbname | ||
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, missing new line on EOF 😉
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
recipes/r-ngsplot/build.sh
Outdated
@@ -4,8 +4,7 @@ mkdir -p $outdir | |||
mkdir -p $PREFIX/bin | |||
cp -R * $outdir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if it's still the case, but I know conda-build
sometimes wrote files to the cwd during building (I think conda-build.sh
or run-test.sh
or something). Those files will/might get copied too if you cp -R *
here.
recipes/r-ngsplot-hg19/post-link.sh
Outdated
@@ -1,7 +1,7 @@ | |||
#!/bin/bash | |||
FN="ngsplotdb_hg19_75_3.00.tar.gz" | |||
URLS=( | |||
"https://doc-0g-a0-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/jiksn2m41cm3dbspq0nqt102h6m6eo1o/1528221600000/01382619737792242945/*/0B5hDZ2BucCI6SURYWW5XdUxnbW8?e=download" | |||
"https://drive.google.com/uc?export=download&id=0B5hDZ2BucCI6SURYWW5XdUxnbW8" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 looks much saner 😄. galaxyproject/cargo-port#148 is probably still preferred, I think, but if that isn't merged in a timely manner, we can always package a new build for r-ngsplot-hg19
updating that URL later.
So setting this environment variable as suggested in the README won't work. Therefore, the second test fails and probably also the recipe can't be called properly. Using something like this:
Doesn't work unfortunately... Setting something via @mbargull any idea? This is the last thing required for the recipe... |
Thanks @ewels for your hint with https://github.com/bioconda/bioconda-recipes/blob/master/recipes/bcftools/1.2/build.sh, this seems to work locally now at least! |
Ok, I don't have a clue what causes this test fail... @bioconda/core anybody an idea? |
recipes/r-ngsplot/build.sh
Outdated
for f in $outdir/*; do | ||
ln -s $outdir/* $PREFIX/bin | ||
for f in ${outdir}/bin/*; do | ||
ln -s ${f} ${PREFIX}/bin | ||
fbname=$(basename "$f") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I've been imprecise. Just meant the fbname=$(basename "$f")
part 😉
recipes/r-ngsplot/build.sh
Outdated
mkdir -p $DEACTIVATE_DIR | ||
cp $RECIPE_DIR/scripts/activate.sh $ACTIVATE_DIR/ngsplot-activate.sh | ||
cp $RECIPE_DIR/scripts/deactivate.sh $DEACTIVATE_DIR/ngsplot-deactivate.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid conflicts, you should rename those to the exact name of the package, i.e., r-ngsplot-...
. You can use the ${PKG_NAME}
variable for this, see, e.g.: https://github.com/conda-forge/toolchain-feedstock/blob/bf0bfa1f06f6592eb3e6ef8989339a005cfd93ef/recipe/install-toolchain.sh#L8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done :-)
recipes/r-ngsplot-hg19/post-link.sh
Outdated
|
||
if [[ $SUCCESS != 1 ]]; then | ||
echo "ERROR: post-link.sh was unable to download any of the following URLs with the md5sum $MD5:" >> "${PREFIX}/.messages.txt" 2>&1 | ||
printf '%s\n' "${URLS[@]}" > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix my >
mistake 😉
printf '%s\n' "${URLS[@]}" >> "${PREFIX}/.messages.txt" 2>&1
recipes/r-ngsplot-hg19/post-link.sh
Outdated
fi | ||
|
||
# Install and clean up | ||
ngsplotdb.py install "${TARBALL}" > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix my >
mistake 😉
ngsplotdb.py install "${TARBALL}" >> "${PREFIX}/.messages.txt" 2>&1
recipes/r-ngsplot-hg19/pre-unlink.sh
Outdated
@@ -0,0 +1 @@ | |||
ngsplotdb.py remove hg19 > "${PREFIX}/.messages.txt" 2>&1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix my >
mistake 😉
ngsplotdb.py remove hg19 >> "${PREFIX}/.messages.txt" 2>&1
That's because of a shortcoming on how the containers are built/run, see galaxyproject/galaxy-lib#81. Choices you have for this:
|
@@ -0,0 +1,3 @@ | |||
#!/bin/bash | |||
export NGSPLOT=$PREFIX/share/$PKG_NAME-$PKG_VERSION-$PKG_BUILDNUM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, and using $PKG_NAME-$PKG_VERSION-$PKG_BUILDNUM
should not work here, I believe.
Those variable are only defined during the build/test phase or during post-link
(and others), but not when activating an environment. If the tests passed for this, that's only because they were set in that case.
You'll want to create/modify the activation script in build.sh
.
recipes/r-ngsplot/build.sh
Outdated
mkdir -p $ACTIVATE_DIR | ||
mkdir -p $DEACTIVATE_DIR | ||
cp $RECIPE_DIR/scripts/activate.sh $ACTIVATE_DIR/ngsplot-activate.sh |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In regards to the comment below for scripts/activate.sh
:
Replace
cp $RECIPE_DIR/scripts/activate.sh $ACTIVATE_DIR/ngsplot-activate.sh
by something like
echo > "${ACTIVATE_DIR}/${PKG_NAME}-activate.sh" <<EOF
#!/bin/bash
export NGSPLOT="${outdir}"
export _CONDA_SET_NGSPLOT_ENV=1
EOF
. ${outdir}
will then have been evaluated at build time (becaus of using <<EOF
with EOF
being unquoted) and replaced by the correct output path (which conda
will in turn adjust to the environment's path at install time).
Ah, I feel the urge to clarify the
The different approaches are of course not necessarily exclusive -- "hero" is always something you can try to achieve 😄. The "don't care" approach is not really preferable since it leaves the container in an incomplete/not directly usable state: The user would have to "guess" the path and set the variable themselves to make use of the container. The "defensive patch" approach might be overly wary, but it retain's the software's functionality to let the user choose/set an alternative path. Probably that variable is just used as a workaround to achieve relocatability so that functionality isn't needed if you patch and let |
Thanks for all these hints - never had that much trouble with a single recipe than this one (which is tbh in a bad shape for packaging...). I'll patch the call in the script I'm calling here in
As far as I see it, this is the best way other than the hero approach, which I don't feel qualified to do. |
So replaced the NGSPLOT ENV call in the R script. Lets hope that works fine then :-) |
Did a quick grep for |
Didn't quite work yet... debugging...
(I must've missed that comment before.) You're welcome, happy to help. Yeah it's a bit troublesome that it's not packaged as usual, but hey, you got to learn a few new tricks which is never a bad things 😉 . |
I think the last changes might make this pass. If it indeed does and you still think the patching is fine, feel free to merge! |
- (ngsplotdb.py list | grep -qF "hg19" || ngsplotdb.py list) | ||
about: | ||
home: 'https://github.com/shenlab-sinai/ngsplot' | ||
license: GPL-2.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder: does the database have another home URL? Is the database itself also licensed under the GPL-2.0
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, they are on GDrive and linked from the same Github repository - so I think its fine to use the same licence (as they didn't state explicitly another one).
Thanks for all the help - learned a lot here and will add two more databases today to finalize things here :-)
The GDrive URL is somewhat a bit crazy - submitted a PR to get this to Galaxy Cargo-Port too. Maybe that eases things...
Referencing nf-core/chipseq#2
and nf-core/chipseq#15 to keep things related.
If this one is fine, I'm trying to do the same for hg38 and mm10 to cover the most used DBs for ngsplot.