Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simple job running api #89

Merged
merged 149 commits into from
Jan 30, 2024
Merged
Show file tree
Hide file tree
Changes from 78 commits
Commits
Show all changes
149 commits
Select commit Hold shift + click to select a range
328eb7a
Moved the atom_data module to tcutility.data.atom
YHordijk Jan 19, 2024
e1ce16c
Added the data.functionals module which gives information about funct…
YHordijk Jan 19, 2024
d4258a4
Added the job module.
YHordijk Jan 19, 2024
4a30759
Added some more information to the ADF results. We added charge, rela…
YHordijk Jan 19, 2024
487a3c9
Added the system charge to the AMS information
YHordijk Jan 19, 2024
6993a54
Fixing ruff issues
YHordijk Jan 19, 2024
c9ba898
Added docstring for the AMSJob class
YHordijk Jan 19, 2024
d13673c
Moved molecule loading method to generic.Job
YHordijk Jan 19, 2024
551fd50
Added the CRESTJob class
YHordijk Jan 19, 2024
9d70169
Now checking if the setup was successfull
YHordijk Jan 19, 2024
d576c4e
Now checking if the setup was successfull
YHordijk Jan 19, 2024
3975b8b
Improved printing of error
YHordijk Jan 19, 2024
90ce5fa
Improved the postamble for default crest jobs
YHordijk Jan 19, 2024
57723c8
Added crest-job to init
YHordijk Jan 19, 2024
0609244
Added an example workflow for conformer generation/optimization
YHordijk Jan 19, 2024
25173ee
Missing import
YHordijk Jan 19, 2024
242a53f
fixed wrong method call
YHordijk Jan 19, 2024
b79cd9d
Fixed early exit for missing molecuel path
YHordijk Jan 19, 2024
34dc684
Added some documentation
YHordijk Jan 19, 2024
bffaf9a
Directories were switched around
YHordijk Jan 19, 2024
f6d1d5d
Added kill_on_invalid_dep when setting a dependency in case the previ…
YHordijk Jan 19, 2024
ec2bd4a
Moved geometry opt convergence criteria to AMSJob
YHordijk Jan 19, 2024
985c28e
Moved default geometry_convergence settings to AMSJob
YHordijk Jan 19, 2024
abfb43b
Added the orca job
YHordijk Jan 20, 2024
4628ed2
Removed shell=True for subprocess call
YHordijk Jan 20, 2024
c2bc128
Runfiles should now be correctly generated
YHordijk Jan 20, 2024
58bf583
Fixed incorrect reference
YHordijk Jan 20, 2024
ac07ebf
Stripping the orca_path, as it contained a \n
YHordijk Jan 20, 2024
12f8449
Testing the ORCA job now
YHordijk Jan 20, 2024
3ac36c6
Importing orcajob
YHordijk Jan 20, 2024
67166db
Fixed typo
YHordijk Jan 20, 2024
b2ade74
Added sbatch settings
YHordijk Jan 20, 2024
60a83f8
Vibrations are calculated by default for optimization
YHordijk Jan 20, 2024
eda7edb
Improved example
YHordijk Jan 20, 2024
c0d8df1
Changed append to add
YHordijk Jan 20, 2024
3fc4b89
Added preamble
YHordijk Jan 20, 2024
6fa4f00
Added preamble
YHordijk Jan 20, 2024
9090d6f
Added preamble
YHordijk Jan 20, 2024
0db222c
Testing the conformer workflow example
YHordijk Jan 22, 2024
324ff7f
Added the QCGJob class for running quantum cluster growth calculations
YHordijk Jan 22, 2024
67a2d23
Setting default ORCA method and basis sets
YHordijk Jan 22, 2024
5908a81
Added a new example job
YHordijk Jan 22, 2024
60ad299
Added a new example job for QCG
YHordijk Jan 22, 2024
96810f0
Added the QCGJob to init
YHordijk Jan 22, 2024
5bb1f63
Added ensemble_mode option
YHordijk Jan 22, 2024
429cc3d
Ensemble generation is now enabled for QCGJob
YHordijk Jan 22, 2024
4f3e051
Enabled ADF calcs
YHordijk Jan 22, 2024
0c17df6
Solvent molecules can now be given by name. tcutility will use data.m…
YHordijk Jan 22, 2024
13f0d62
Prettified the example
YHordijk Jan 22, 2024
f6dfd3d
Added a get_molecules function which returns molecules from the molec…
YHordijk Jan 22, 2024
1105314
Fixing ruff warnings
YHordijk Jan 22, 2024
e29629f
Simplified options specification for crestjob
YHordijk Jan 22, 2024
aaf77fc
Changed tabs to spaces
YHordijk Jan 22, 2024
6e257a5
Using property of CRESTJob instead of remaking the directory
YHordijk Jan 22, 2024
e368360
Fixing atom data test
YHordijk Jan 22, 2024
b126d7c
Fixed an issue with getting spin polarisation
YHordijk Jan 22, 2024
f04ce8e
Added a check for directory existence, if it does return the files in…
YHordijk Jan 22, 2024
5de1fde
Added the hold argument for Job construction. This will make the scri…
YHordijk Jan 22, 2024
cd9dfa0
Added the wait_for_job function which will wait for a job to finish b…
YHordijk Jan 22, 2024
f9d757a
Renamed hold to wait_for_finish
YHordijk Jan 22, 2024
206b300
Now number defaults to None, if it is not set we return all files in …
YHordijk Jan 22, 2024
92c2f02
Now waiting for CRESTJob to finish before submitting the DFTB and DFT…
YHordijk Jan 22, 2024
6a25576
Updated the example
YHordijk Jan 22, 2024
d89b38c
Updated the example
YHordijk Jan 22, 2024
67f07f4
Fixed logic
YHordijk Jan 22, 2024
7837cca
Fixed logic
YHordijk Jan 22, 2024
2cb2bf6
Fixed logic
YHordijk Jan 22, 2024
b6a8b4b
Fixed logic
YHordijk Jan 22, 2024
0e6e750
Fixed logic
YHordijk Jan 22, 2024
67c06ad
Debug printing
YHordijk Jan 22, 2024
511a48d
Sorting the files now
YHordijk Jan 22, 2024
2048c64
Added the ability to give fragments as a list of integer indices. Thi…
YHordijk Jan 23, 2024
39ae7ad
Fixed an issue where job would crash for ADFFragmentjobs because of m…
YHordijk Jan 23, 2024
f4cf7ca
Added a new example for running fragment calculations
YHordijk Jan 23, 2024
88f4985
Removed useless statement
YHordijk Jan 23, 2024
215a0b7
Fixed formatting and commented line
YHordijk Jan 23, 2024
b1d8af9
Steven fixed the solvent molecules
YHordijk Jan 23, 2024
6c46a71
Fixing Ruff errors
YHordijk Jan 23, 2024
d6c8bb0
Added postamble to QCG, as well as directory and file getting for ens…
YHordijk Jan 23, 2024
ba3e673
Fixed wrong reference to molecules.get_mol, which was renamed to mole…
YHordijk Jan 23, 2024
6a2df46
Added alpb option
YHordijk Jan 23, 2024
9f779c5
Added the nofix option
YHordijk Jan 23, 2024
714a231
Added error message if solvent is not given
YHordijk Jan 23, 2024
1097ff2
renamed best_conformer_path to best_ensemble_path
YHordijk Jan 23, 2024
a0b03c0
QCGJob will now attempt to get the alpb solvent name from the xyz file
YHordijk Jan 23, 2024
d855699
Removed unused files
YHordijk Jan 23, 2024
c6cc3ef
Fixed typo
YHordijk Jan 23, 2024
1551e02
Fixed missing import
YHordijk Jan 23, 2024
e23f0fd
Fixed ruff error
YHordijk Jan 23, 2024
3d8cc71
Fixed splitting of ensemble file into parts
YHordijk Jan 23, 2024
45baf6d
Ensemble xyz files are now send to ensemble/ensemble
YHordijk Jan 23, 2024
e884953
Now ignoring docs/_build
YHordijk Jan 23, 2024
884da96
Now ignoring docs/_build
YHordijk Jan 23, 2024
3c29359
Started working on documentation for job
YHordijk Jan 23, 2024
1d58455
Started working on recipes. For now added the conformer_search recipe
YHordijk Jan 23, 2024
1bebf6e
Added the tabs extension for sphinx
YHordijk Jan 23, 2024
ac0fdb9
Added new requirement for tabs in sphinx
YHordijk Jan 23, 2024
8076e6f
Update build_docs_no_deploy.yaml
YHordijk Jan 23, 2024
1bb74e5
Moved sphinx-tabs requirement from pyproject to github workflow
YHordijk Jan 23, 2024
66eb4ab
Update build_docs.yml
YHordijk Jan 23, 2024
08836db
Update build_docs_no_deploy.yaml
YHordijk Jan 23, 2024
a2213d1
Update build_docs.yml
YHordijk Jan 23, 2024
8a56098
Added copybutton to documentation for easy copying of code blocks
YHordijk Jan 24, 2024
9c952c6
Autodoc will now sort classes and functions by the order in the source
YHordijk Jan 24, 2024
ef1d996
Added some more work for the documentation for the job module
YHordijk Jan 24, 2024
0c0ecd0
Added a new function that is used to get fragments from a molecule ob…
YHordijk Jan 24, 2024
b82c31f
Improved documentation
YHordijk Jan 24, 2024
8eeca7f
Merge remote-tracking branch 'origin/68-simple-job-running-api' into …
YHordijk Jan 24, 2024
c0c4f56
Update build_docs_no_deploy.yaml
YHordijk Jan 24, 2024
3619c3d
Update build_docs_no_deploy.yaml
YHordijk Jan 24, 2024
cd8f079
Removed _build from repo
YHordijk Jan 24, 2024
0f37673
Now hiding some internal methods for Transform class
YHordijk Jan 25, 2024
0844c81
Fixed ruff complaints
YHordijk Jan 25, 2024
f9eeabb
Fixed ruff complaints
YHordijk Jan 25, 2024
7e32e79
Added two tests for the KabschTransform class
YHordijk Jan 26, 2024
1955487
Added job and results to modules rst
YHordijk Jan 26, 2024
5736aef
Added tuctility.job to the toctree and separated the API documentation
YHordijk Jan 26, 2024
c223e6c
Changed headers to sections
YHordijk Jan 26, 2024
52ea439
Added more documentation
YHordijk Jan 26, 2024
5a61d99
Added more documentation
YHordijk Jan 26, 2024
40a52d1
Now hiding the setup_job method
YHordijk Jan 26, 2024
b2e6e4a
Added an improved cache module implementing a normal caching decorato…
YHordijk Jan 27, 2024
1b1f5a1
Using the timed cache for slurm calls
YHordijk Jan 27, 2024
20500b0
Now printing sbatch output
YHordijk Jan 27, 2024
7a481cd
Now printing sbatch output
YHordijk Jan 27, 2024
5faa662
Added the sbatch command, which will submit a job to slurm and return…
YHordijk Jan 27, 2024
b58e257
Reworked run to use the new slurm.sbatch function
YHordijk Jan 27, 2024
270802b
The submit file should now be executable
YHordijk Jan 27, 2024
cb236c4
Added an extra example
YHordijk Jan 28, 2024
9bac267
Added more documentation for the generic job class
YHordijk Jan 28, 2024
2d1bd51
Improved the index of the documentation
YHordijk Jan 28, 2024
0e47063
Started working on writing the job documentation
YHordijk Jan 28, 2024
4aa91f5
Added the NMRJob class
YHordijk Jan 28, 2024
d307d68
Added a start to the examples documentation page
YHordijk Jan 28, 2024
6510f27
Now writing nics coordinates to multipole coordinates in the pre_nmr …
YHordijk Jan 28, 2024
5d45a00
Added documentation for the NMRJob
YHordijk Jan 28, 2024
8792764
Removed unused get_sbatch_command and added more documentation
YHordijk Jan 28, 2024
7768279
Added the preoptimization recipe
YHordijk Jan 28, 2024
ca9e017
Enabling overwrite will now remove the original rundir
YHordijk Jan 28, 2024
12e4fc0
Tested the preoptimize workflow
Jan 28, 2024
1297936
Fixing ruff issues
YHordijk Jan 28, 2024
9fedad3
Merge branch '68-simple-job-running-api' of github.com:TheoChem-VU/TC…
Jan 28, 2024
cd34669
Changed check for getting molecules
YHordijk Jan 28, 2024
a962a7b
Added a script to write functional information to a json file
YHordijk Jan 28, 2024
dfdbc08
Removed charge, is handled by Siebe's PR
YHordijk Jan 28, 2024
dbda4df
Improved the documentation
YHordijk Jan 28, 2024
6e0d011
Added a function to convert path-safe functional name to the correspo…
YHordijk Jan 29, 2024
693325a
Geometry optimizations now calculate the hessian before-hand with a l…
YHordijk Jan 30, 2024
a4cc2f1
removed _build
YHordijk Jan 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions examples/job/NH3BH3.xyz
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
8

N 0.00000000 0.00000000 -0.81474153 frag=Donor
B -0.00000000 -0.00000000 0.83567034 frag=Acceptor
H 0.47608351 -0.82460084 -1.14410295 frag=Donor
H 0.47608351 0.82460084 -1.14410295 frag=Donor
H -0.95216703 0.00000000 -1.14410295 frag=Donor
H -0.58149793 1.00718395 1.13712667 frag=Acceptor
H -0.58149793 -1.00718395 1.13712667 frag=Acceptor
H 1.16299585 -0.00000000 1.13712667 frag=Acceptor

frag_Donor = 1, 3, 4, 5
frag_Acceptor = 2, 6, 7, 8
20 changes: 20 additions & 0 deletions examples/job/asc.xyz
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
18

C -0.94127539 -5.26718347 -3.96828713
C -0.33238494 -3.88895689 -3.84308606
N -1.16865847 -2.97724681 -3.25523725
N -0.51040403 -1.76570770 -2.89744287
C -0.49687631 -1.56835550 -1.52557810
N 0.39218754 -0.61181473 -1.15607414
C 0.35907326 -0.02567442 0.17168965
O 0.75362497 -3.58468600 -4.28757501
S -1.50149713 -2.38418510 -0.51048053
H -0.26790316 -5.90833092 -4.53215794
H -1.90174898 -5.20338615 -4.47736764
H -1.10247419 -5.69106602 -2.97752780
H -1.72914011 -3.25140398 -2.42926272
H 0.39417088 -1.76840521 -3.37017404
H 0.80737342 -0.05086076 -1.88509623
H -0.16787989 -0.71881141 0.83076065
H -0.16753334 0.93360314 0.15490937
H 1.37850078 0.12634418 0.52991953
33 changes: 33 additions & 0 deletions examples/job/conformer_workflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from tcutility.job import ADFJob, DFTBJob, CRESTJob
import os

j = os.path.join

mol_name = 'Pentane'

with CRESTJob(wait_for_finish=True) as crest_job:
crest_job.molecule('pentane.xyz')
crest_job.md_length(0.2)
crest_job.md_temperature(1000)
crest_job.sbatch(p='tc', n=64)
crest_job.rundir = f'calculations/{mol_name}'
crest_job.name = 'CREST'

for i, xyzfile in enumerate(crest_job.get_conformer_xyz()):
with DFTBJob() as dftb_job:
dftb_job.molecule(xyzfile)
dftb_job.sbatch(p='tc', n=8)
dftb_job.optimization()
dftb_job.rundir = f'calculations/{mol_name}/DFTB'
dftb_job.name = f'conformer_{i}'
dftb_job.dependency(crest_job)

with ADFJob() as adf_job:
adf_job.molecule(dftb_job.output_mol_path)
adf_job.sbatch(p='tc', n=32)
adf_job.optimization()
adf_job.functional('BLYP-D3(BJ)')
adf_job.basis_set('TZ2P')
adf_job.rundir = f'calculations/{mol_name}/ADF'
adf_job.name = f'conformer_{i}'
adf_job.dependency(dftb_job)
12 changes: 12 additions & 0 deletions examples/job/dmso.xyz
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
10

C 1.38960000 -0.86360000 -0.19670000
S 0.00000000 0.15470000 0.37120000
O 0.00000000 1.39000000 -0.33110000
C -1.38960000 -0.86360000 -0.19670000
H 2.32850000 -0.38010000 0.07300000
H 1.33760000 -1.84480000 0.27520000
H 1.33760000 -0.97800000 -1.27940000
H -1.33770000 -1.84480000 0.27520000
H -2.32850000 -0.38010000 0.07300000
H -1.33760000 -0.97800000 -1.27940000
101 changes: 101 additions & 0 deletions examples/job/fragment_calculation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
from tcutility.job import ADFFragmentJob
from tcutility import molecule

# this example will demonstrate how you may separate two fragments from each other

'''
The NH3BH3.xyz file looks like this:

8

N 0.00000000 0.00000000 -0.81474153 frag=Donor # method 1
B -0.00000000 -0.00000000 0.83567034 frag=Acceptor
H 0.47608351 -0.82460084 -1.14410295 frag=Donor
H 0.47608351 0.82460084 -1.14410295 frag=Donor
H -0.95216703 0.00000000 -1.14410295 frag=Donor
H -0.58149793 1.00718395 1.13712667 frag=Acceptor
H -0.58149793 -1.00718395 1.13712667 frag=Acceptor
H 1.16299585 -0.00000000 1.13712667 frag=Acceptor

frag_Donor = 1, 3, 4, 5 # method 2
frag_Acceptor = 2, 6, 7, 8
'''

# a simple donor-acceptor complex with randomly ordered atoms
# the atoms have tags with them that denote the fragment they belong to
# two ways are presented. first is to put the tags on the atoms
# second is to put the tags at the bottom in a list
mol = molecule.load('NH3BH3.xyz') # load the molecule using our own function. This also loads meta-data


# ### Method 1 ### #
'''
The first method uses the atom flags defined in the xyz file.
This reads anything after the atom definition, for example it
will read the frag= part and store it as atom.flags.frag.
'''
fragment_names = set(atom.flags.frag for atom in mol) # store fragment names here
fragment_atoms = {name: [] for name in fragment_names} # store atoms here

for atom in mol:
# get the fragment the atom belongs to and add it to the list
fragment_atoms[atom.flags.frag].append(atom)

with ADFFragmentJob(test_mode=True) as job1:
# add the fragments to the job one by one
for fragment_name, atoms in fragment_atoms.items():
job1.add_fragment(atoms, fragment_name)

# set the job settings
job1.functional('BLYP')
job1.basis_set('TZ2P')


# ### Method 2 ### #
'''
This method uses the molecular flags defined below the atom definitions.
The flags are stored as mol.flags.frag_Donor and mol.flags.frag_Acceptor.
They both hold a list of integers which we will first turn into lists of atoms.
'''
frag_flags = {flag.removeprefix('frag_'): data for flag, data in mol.flags.items() if flag.startswith('frag_')}
frag_atoms = {frag: [mol[i] for i in x] for frag, x in frag_flags.items()}

with ADFFragmentJob(test_mode=True) as job2:
# add the fragments to the job one by one
for fragment_name, atoms in fragment_atoms.items():
job2.add_fragment(atoms, fragment_name)

# set the job settings
job2.functional('BLYP')
job2.basis_set('TZ2P')


# ### Method 3 ### #
'''
This method is similar to method 2, but we can now supply a molecule to the
parent job and then define the fragments based on atom indices.
'''
fragment_indices = {flag.removeprefix('frag_'): x for flag, x in mol.flags.items() if flag.startswith('frag_')}

with ADFFragmentJob(test_mode=True) as job3:
job3.molecule(mol)
for fragment_name, indices in fragment_indices.items():
job3.add_fragment(indices, fragment_name)

# set the job settings
job3.functional('BLYP')
job3.basis_set('TZ2P')


# check the fragments for each method
print('# ### Method 1 ### #')
print(job1.childjobs)
[print(frag_name, j._molecule) for frag_name, j in job1.childjobs.items()]

print('# ### Method 2 ### #')
print(job2.childjobs)
[print(frag_name, j._molecule) for frag_name, j in job2.childjobs.items()]

print('# ### Method 3 ### #')
print(job3.childjobs)
[print(frag_name, j._molecule) for frag_name, j in job3.childjobs.items()]
11 changes: 11 additions & 0 deletions examples/job/orca.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from tcutility.job import ORCAJob

with ORCAJob() as job:
job.molecule('water.xyz')
job.sbatch(p='tc', n=32, mem=20000)
job.name = 'ORCA_opt'
job.rundir = 'calculations/water'
job.add_preamble('export ORCA_FP=“/scistor/tc/dra480/bin/orca500/orca”')
job.add_preamble('export PATH=“/scistor/tc/dra480/bin/ompi411/bin:$PATH”')
job.add_preamble('export LD_LIBRARY_PATH=“/scistor/tc/dra480/bin/ompi411/lib/:$LD_LIBRARY_PATH”')
job.optimization()
19 changes: 19 additions & 0 deletions examples/job/pentane.xyz
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
17

C -3.15949327 -3.86575292 0.05080059
C -1.90503885 -3.00170042 -0.08117712
C -0.62815719 -3.84327295 0.02578951
C 0.62931287 -2.97271947 -0.08042531
C 1.90308216 -3.80822038 0.05065497
H -3.19112018 -4.61909694 -0.73782948
H -4.05227778 -3.24389872 -0.02716694
H -3.17357396 -4.37413682 1.01637198
H -1.90577930 -2.23902163 0.70606675
H -1.92042079 -2.48298781 -1.04645116
H -0.61922411 -4.59370909 -0.77263971
H -0.62231135 -4.37535085 0.98379862
H 0.61272553 -2.21090984 0.70740609
H 0.63282656 -2.45317624 -1.04532480
H 2.78158417 -3.16643350 -0.02770312
H 1.95110047 -4.56045507 -0.73823929
H 1.92918836 -4.31640544 1.01606843
11 changes: 11 additions & 0 deletions examples/job/qcg.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from tcutility.job import QCGJob

with QCGJob() as job:
job.molecule('asc.xyz')
job.solvent('dmso.xyz')
job.rundir = 'calculations/ASC'
job.name = 'QCG'
job.md_length(5)
job.md_temperature(1000)
job.sbatch(p='tc', n=32)
job.ensemble_mode('MTD')
31 changes: 31 additions & 0 deletions examples/job/screen_functionals.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from tcutility.job import ADFJob
from tcutility.data import functionals
from tcutility import results, log
from yutility import pathfunc

# run the calculations:
available_functionals = functionals.get_available_functionals()
for functional_name, functional_info in available_functionals.items():
with ADFJob(test_mode=True) as job:
# load your ams version
job.add_preamble('module load ams/2023.101')
# load your molecule
job.molecule('molecule.xyz')

# sort calculation by their category
job.rundir = f'DFT_screening/{functional_info.category}'
job.name = functional_info.path_safe_name

# slurm settings
job.sbatch(p='tc', ntasks_per_node=32)

# ADF settings
job.functional(functional_name)
job.basis_set('TZ2P')

# analyse the results
rows = []
for rundir, path_data in pathfunc.match('DFT_screening', '{category}/{functional_name}').items():
res = results.read(rundir)
rows.append(res.status.code)
log.table(rows, ['Status'])
5 changes: 5 additions & 0 deletions examples/job/water.xyz
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
3

O -1.00054733 -2.19438222 0.00000000
H -1.05209811 -2.99168159 -0.53221822
H -0.84589501 -2.43142270 0.91733008
File renamed without changes.
Loading