-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Submitting jobs on a remote cluster #55
Comments
Hey, there is currently no way to talk to the remote side directly. The way we use the system on a "remote" cluster is to first ssh to the login node and then install, configure and use jip there. I was in fact thinking about adding a thin "remote" layer on to so you could execute commands directly from your local machine, and we might still implement something like this, but none the less, jip needs to always be installed and available on the remote cluster/login node. The reason for this is that the jobs interact directly with the job database and do not do this through a server. This avoid starting a server on your cluster and you don't need to enable connection from and to your compute cluster. I hope it helps! If you find a need for a thin client that you can use directly from your workstation, feel free to explain your use case in a bit more here and we can see if we can do something about it. |
Thanks for the response! The use case I was thinking of is that sometimes I want to run jobs on the cluster and sometimes I want to run them on my local machine. It would be nice to manage both things from my local desktop. If you have all of the commands going over ssh, then I don’t think you need to have anything installed on the remote machine except the scripts that actually get executed (and the data, of course, but it’s beyond the scope of JIP to manage that). The scripts could just be copied to the remote server via scp, or kept in sync via rsync. I’d be happy to contribute to development to make this a part of JIP, but I first I would want to work with you on a plan for the best way to implement it. Thanks, John Didion, PhD On Oct 7, 2014, at 8:07 AM, thasso <[email protected]mailto:[email protected]> wrote: Hey, there is currently no way to talk to the remote side directly. The way we use the system on a "remote" cluster is to first ssh to the login node and then install, configure and use jip there. I was in fact thinking about adding a thin "remote" layer on to so you could execute commands directly from your local machine, and we might still implement something like this, but none the less, jip needs to always be installed and available on the remote cluster/login node. The reason for this is that the jobs interact directly with the job database and do not do this through a server. This avoid starting a server on your cluster and you don't need to enable connection from and to your compute cluster. I hope it helps! If you find a need for a thin client that you can use directly from your workstation, feel free to explain your use case in a bit more here and we can see if we can do something about it. — |
Hi John, okay I think I understand the use case and I think it would be a good idea With this in mind, what I see right now is essentially a SSH wrapper to What I am not sure about right now is how to deal with input files and if Best, 2014-10-07 14:27 GMT+02:00 John Didion [email protected]:
|
Sure, I think that workflow will definitely appeal to some users. For me, i’m more interested in having a single interface to execute jobs, whether they be local or remote, but the jobs I run locally and remotely are different (the remote jobs are pipelines for processing sequencing data, while the local jobs are typically quick analyses on the processed data). The ssh wrapper sounds like the best solution. As far as input files, I think JIP should handle syncing any JIP scripts but (at least for now) it’s up to the user to make sure the data files are in place before executing the job. To me it doesn’t seem ideal to have JIP trying to manage keeping dozens or hundreds of huge bam files in sync. As far as implementation, can you briefly describe how you would go about doing it? You’re much more familiar with the structure of the code and I want to implement this in the way that makes the most sense. Thanks John Didion, PhD On Oct 7, 2014, at 8:55 AM, thasso <[email protected]mailto:[email protected]> wrote: Hi John, okay I think I understand the use case and I think it would be a good idea With this in mind, what I see right now is essentially a SSH wrapper to What I am not sure about right now is how to deal with input files and if Best, 2014-10-07 14:27 GMT+02:00 John Didion <[email protected]mailto:[email protected]>:
— |
I’ve thought through this some more, and I think I have a good plan. The model I am working with is that there will be two separate JIP databases, one on the local machine and one on the remote cluster. For a job submitted from the local machine to the remote cluster (via the new ‘remote’ command described below), a placeholder will be inserted into the local database marking that job as a remote job. Subsequent calls to ‘jip jobs’ will fetch job information from the remote machine and update the placeholder records in the local database. That way, a user can track all job information in his local database even if some jobs are submitted locally and some are submitted remotely. New commands:
I think these changes should be fairly transparent to current users, i.e. it wouldn’t affect how they currently do things. There would have to be some kind of database migration step to upgrade the database schema for current users that want to take advantage of the new work flow, but I think SQLAlchemy has facilities for that. Please let me know if you see any problems with this approach, or if you recommend a better way of doing it. Thanks, John Didion, PhD On Oct 7, 2014, at 9:06 AM, John Didion <[email protected]mailto:[email protected]> wrote: Sure, I think that workflow will definitely appeal to some users. For me, i’m more interested in having a single interface to execute jobs, whether they be local or remote, but the jobs I run locally and remotely are different (the remote jobs are pipelines for processing sequencing data, while the local jobs are typically quick analyses on the processed data). The ssh wrapper sounds like the best solution. As far as input files, I think JIP should handle syncing any JIP scripts but (at least for now) it’s up to the user to make sure the data files are in place before executing the job. To me it doesn’t seem ideal to have JIP trying to manage keeping dozens or hundreds of huge bam files in sync. As far as implementation, can you briefly describe how you would go about doing it? You’re much more familiar with the structure of the code and I want to implement this in the way that makes the most sense. Thanks John Didion, PhD On Oct 7, 2014, at 8:55 AM, thasso <[email protected]mailto:[email protected]> wrote: Hi John, okay I think I understand the use case and I think it would be a good idea With this in mind, what I see right now is essentially a SSH wrapper to What I am not sure about right now is how to deal with input files and if Best, 2014-10-07 14:27 GMT+02:00 John Didion <[email protected]mailto:[email protected]>:
— |
Sounds good to me. I don't see any obvious flaws at the moment and we can iterate on this. If you want to start implementing this, please note that current development version of JIP can be found in the "develop" branch. I would suggest you create a pull request against that branch and I can review the changes before merging them in. |
I’ve scaled this back a bit. I decided mucking up the database with lots of pointers to jobs running on remote hosts was probably not worth the cost. Instead I’m implementing the following: export: export jobs as a json object This should be relatively quick to implement since I will be using an existing ssh library (although this will create an additional dependency). On Oct 13, 2014, at 7:00 AM, thasso [email protected] wrote:
|
Hi Thasso, I am writing a pipeline where the first step is parallel alignment of multiple sets of fastq files and the second step will be to merge the resulting BAM files. I’m not sure of how to implement this in JIP. I assume the first step is to iterate over all pairs of fastq files and call run, i.e.: p = Pipeline() But how to make those run as a single group, and to have the merge step depend on the completion of all the jobs in that group? Thanks, John On Oct 15, 2014, at 5:57 PM, John Didion [email protected] wrote:
|
Hi John, Sorry for the major delay, but now there is some time finally :) You are on the right track already with the dependencies. You can do it as you The assumption here is basically that the
In pseudo code, this would look something like this:
Please note that the expansion and job creation is on necessary if you use Also note the The final merge jobs input is just the align job. This works as long as Using the jip pipeline graph and expanding on the inputs and outputs of the I hope it was not too late and still helps. Best, |
I apologize for asking a question here, but there is no dedicated support forum. The documentation is not clear on how to actually get JIP to submit jobs on a remote cluster. In my environment, and I think that of many others, we ssh into a login node to submit jobs on the cluster. I did not see any option to configure the remote host name or login password in the cluster configuration. Thanks
The text was updated successfully, but these errors were encountered: