0.2.0 (#181)

API: - Move data objects and collections, aka rename - Improve connection checks - Rework the API - Bugfix in search - Examples in docstrings CLI: - Adding mkcoll and tree Update tutorials * Adding mkdir to CLI (#137) * Adding mkdir to CLI * pylint * Update ibridges/__main__.py Co-authored-by: qubixes <[email protected]> * Update ibridges/__main__.py Co-authored-by: qubixes <[email protected]> * Update ibridges/__main__.py Co-authored-by: qubixes <[email protected]> * Rename to mkcoll --------- Co-authored-by: qubixes <[email protected]> * Move data objects and collections (#136) * Adding moveing data objects and collections * Adding test for renaming * Renaming and adjusting tests. * Create dependabot.yml (#143) * Check for host and port before checking network (#139) * Check for host and port before checking network * Small changes --------- Co-authored-by: Staiger, Christine <[email protected]> * Update sphinx requirement from <7.0.0 to <8.0.0 (#145) * Adding IrodsPath doc * Small changes and index * Turning network_check into class method for GUI (#146) * Turning network_check into class method for GUI * Ruff * Install doc (#147) * Create install.rst * Adding install instructions * Update docs/source/install.rst Co-authored-by: qubixes <[email protected]> * Update docs/source/install.rst Co-authored-by: qubixes <[email protected]> --------- Co-authored-by: qubixes <[email protected]> * Fix docstring formatting error (#140) * Fix docstring formatting error * Add ruff to CI and fix small linting errors * Add sync documentation. (#148) * Add sync documentation. * add index * Update docs/source/sync.rst Co-authored-by: qubixes <[email protected]> * FOrmatting and small changes * Ruff --------- Co-authored-by: qubixes <[email protected]> Co-authored-by: Staiger, Christine <[email protected]> * Ipath doc (#153) * Update ipath with other documentation (#152) * Turning network_check into class method for GUI (#146) * Turning network_check into class method for GUI * Ruff * Install doc (#147) * Create install.rst * Adding install instructions * Update docs/source/install.rst Co-authored-by: qubixes <[email protected]> * Update docs/source/install.rst Co-authored-by: qubixes <[email protected]> --------- Co-authored-by: qubixes <[email protected]> * Fix docstring formatting error (#140) * Fix docstring formatting error * Add ruff to CI and fix small linting errors * Add sync documentation. (#148) * Add sync documentation. * add index * Update docs/source/sync.rst Co-authored-by: qubixes <[email protected]> * FOrmatting and small changes * Ruff --------- Co-authored-by: qubixes <[email protected]> Co-authored-by: Staiger, Christine <[email protected]> --------- Co-authored-by: qubixes <[email protected]> Co-authored-by: Staiger, Christine <[email protected]> * Formatting and cross referencing. --------- Co-authored-by: qubixes <[email protected]> Co-authored-by: Staiger, Christine <[email protected]> * Update ipath.rst * Update ipath.rst * Move functions to utils and add walk to IrodsPath (#151) * Add missing search for data_name (#161) * Add missing search for data_name * Rework API documentation (#156) * Rework API documentation * TRying to add file to resolve conflict * Rremove added file * Adding to edit * Delete file again. --------- Co-authored-by: chstaiger <[email protected]> Co-authored-by: chStaiger <[email protected]> * Bugfix in search (#162) * Don't use assert for control flow (#166) * Don't use assert for control flow * Fix __main__.py Only throw exception when needed. * Update ibridges/path.py Co-authored-by: chStaiger <[email protected]> --------- Co-authored-by: chStaiger <[email protected]> * Update iRODS path tutorial. (#165) * Update iRODS path tutorial. * Refer to tutorials for iRODS paths in RTD * Some more hints on paths in cli doc * Adjustong README * Remove old tutorial * Small edits * Update docs/source/cli.rst Co-authored-by: Terrell Russell <[email protected]> --------- Co-authored-by: Terrell Russell <[email protected]> * Sync update (#158) Rewrite of the sync_data function so that it uses the walk functionality of the IrodsPath instead of duplicated effort. Performance should be roughly the same. This update also fixes some issues with the walk functionality of the IrodsPath and makes it much faster. * Add tree for the iBridges CLI (#159) * Add tree for the iBridges CLI * Add options to tree - Ascii - depth - show-max * Make pylint happy * Add type annotations * Fix spelling errors in the README (#171) * Rewrite download/upload (#169) The main goal is code reduction and using the similarities of the download/upload and sync functionality. As a direct benefit, the upload and download functions now also have a progress bar. Some other notable changes: - `--dry-run` for upload and download - IrodsPath.absolute_path becomes IrodsPath.absolute * Cleanup tutorials. Create a First steps tutorial for the impatient. (#174) * Cleanup tutorials. Create a First steps tutorial for the impatient. * Review First steps * Update Irods-paths tutorial * Update working-with-data * Remove duplicate file --------- Co-authored-by: Raoul Schram <[email protected]> * update metadata * Fix behavior of parent property of IrodsPath (#175) * Fix behavior of parent property of IrodsPath * Fix the fix * Fix tests for new behavior * Fix tutorials (#177) * Spellchecking tutorials * Turn None time into empty string in listing tickets. (#178) * Add examples (#180) Add examples to many of the mostly used docstrings. Also contains a few tiny fixes. * Ruff --------- Co-authored-by: qubixes <[email protected]> Co-authored-by: Dawa Ometto <[email protected]> Co-authored-by: Staiger, Christine <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Terrell Russell <[email protected]> Co-authored-by: Raoul Schram <[email protected]>
iBridges-for-iRODS · May 30, 2024 · a734572 · a734572
1 parent 0c0274e
commit a734572
Show file tree

Hide file tree

Showing 45 changed files with 2,598 additions and 2,015 deletions.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -39,6 +39,10 @@ jobs:
         python -m pip install --upgrade pip
         pip install .[test]
 
+    - name: Lint with Ruff
+      run: |
+        ruff check ibridges
+
     - name: Lint with PyLint
       run: |
         pylint ibridges

diff --git a/.gitignore b/.gitignore
@@ -3,7 +3,7 @@
 irods-basicGUI/logs/*
 irods-iBridgesGui.spec
 *DS_Store
-
+docs/source/api/generated
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

diff --git a/README.md b/README.md
@@ -53,9 +53,9 @@ Below are some basic examples of the features in iBridges.
 
 ```py
 # Create an iRODS session
-from ibridges import Session
+from ibridges.interactive import interactive_auth
 
-session = Session(irods_env_path="~/.irods/irods_environment.json", password="mypassword")
+session = interactive_auth()
 
 # Upload data
 from ibridges import upload
@@ -69,16 +69,16 @@ download(session, "/irods/path", "/other/local/path")
 
 ```
 
-## Commandline interface
-To simply upload or download data you do not need to write full python program, we offer a Commandline interface
+## Command line interface
+To simply upload or download data you do not need to write full python program, we offer a command line interface
 
 - Establish a connection
 
   ```bash
   ibridges init
   ```
 
-- List a colletion
+- List a collection
 
   ```bash
   # list your home collection
@@ -88,7 +88,7 @@ To simply upload or download data you do not need to write full python program,
   ibridges list irods:~/<collection>
 
   # list a collection on a different path than your home
-  ibridges. list irods:<full_irods_path>
+  ibridges list irods:<full_irods_path>
   ```
 
 - Upload data
@@ -106,22 +106,22 @@ To simply upload or download data you do not need to write full python program,
 - Synchronise data
 
   ```bash
-      ibridges sync some_local_directory irods:~/some_collection
+  	ibridges sync some_local_directory irods:~/some_collection
   ```
 
 ## Tutorials
 ### Documentation
 - **[ReadTheDocs](https://ibridges.readthedocs.io/en/latest/)**
 
 ### Guides
-- [QuickStart](tutorials/QuickStart.ipynb)
-- [iRODS Paths](tutorials/iRODS_paths.ipynb)
-- [Data synchronisation](tutorials/Data_sync.ipynb)
+- [Manual with examples](tutorials/QuickStart.ipynb)
+- [Tutorial for iRODS paths](tutorials/01-iRODS_paths.ipynb)
+- [Data synchronisation manual](tutorials/Data_sync.ipynb)
 
 ### Beginners tutorials
 - [Setup client configuration](tutorials/01-Setup-and-connect.ipynb)
+- [iRODS Paths](tutorials/01-iRODS_paths.ipynb)
 - [Working with data](tutorials/02-Working-with-data.ipynb)
-- [iRODS and local Paths](tutorials/03-iRODS-Paths.ipynb)
 - [Metadata](tutorials/04-Metadata.ipynb)
 - [Sharing data](tutorials/05-Data-Sharing.ipynb)
 

diff --git a/docker/irods_client/tests/conftest.py b/docker/irods_client/tests/conftest.py
@@ -1,15 +1,12 @@
 import json
 from pathlib import Path
-import os
 
 import pytest
 import tomli
 
 from ibridges import Session
 from ibridges.data_operations import (
     create_collection,
-    get_collection,
-    get_dataobject,
     upload,
 )
 from ibridges.path import IrodsPath
@@ -61,6 +58,5 @@ def collection(session):
 def dataobject(session, testdata):
     ipath = IrodsPath(session, "~", "bunny.rtf")
     upload(session, testdata/"bunny.rtf", IrodsPath(session, "~"), overwrite=True)
-    data_obj = get_dataobject(session, ipath)
-    yield data_obj
+    yield ipath.dataobject
     ipath.remove()
diff --git a/docker/irods_client/tests/test_cli.py b/docker/irods_client/tests/test_cli.py
@@ -52,7 +52,7 @@ def test_upload_download_cli(session, config, testdata, tmpdir, irods_env_file,
 
     if "resc2" in config["resources"]:
         subprocess.run(["ibridges", "upload", testdata/"plant.rtf", "irods:" + str(ipath),
-                        "--overwrite", "--resource resc2"],
+                        "--overwrite", "--resource", "resc2"],
                         check=True, **pass_opts)
     else:
         subprocess.run(["ibridges", "upload", testdata/"plant.rtf", "irods:" + str(ipath),
@@ -64,9 +64,9 @@ def test_upload_download_cli(session, config, testdata, tmpdir, irods_env_file,
     assert isinstance(testdata, Path)
     if "resc2" in config["resources"]:
         subprocess.run(["ibridges", "download", "irods:~/plant.rtf", testdata/"plant2.rtf",
-                        "--resource resc2"], check=True, **pass_opts)
+                        "--resource", "resc2"], check=True, **pass_opts)
     else:
-        subprocess.run(["ibridges", "download", "irods:~/plant.rtf", testdata/"plant2.rtf"], 
+        subprocess.run(["ibridges", "download", "irods:~/plant.rtf", testdata/"plant2.rtf"],
                         check=True, **pass_opts)
     assert Path(testdata/"plant2.rtf").is_file()
 

diff --git a/docker/irods_client/tests/test_data_ops.py b/docker/irods_client/tests/test_data_ops.py
@@ -5,13 +5,10 @@
 
 from ibridges.data_operations import (
     download,
-    get_collection,
-    get_dataobject,
-    is_collection,
-    is_dataobject,
     upload,
 )
 from ibridges.path import IrodsPath
+from ibridges.util import is_collection, is_dataobject
 
 
 def _get_digest(obj_or_file):
@@ -29,29 +26,40 @@ def _check_files_equal(*files):
     return True
 
 
+def _check_count(ops, nlist):
+    for var, count, in zip(["create_collection", "create_dir", "download", "upload"],
+                          nlist):
+        assert len(ops[var]) == count
+
+
+
 def test_upload_download_dataset(session, testdata):
     ipath = IrodsPath(session, "~", "plant.rtf")
     ipath.remove()
-    upload(session, testdata/"plant.rtf", IrodsPath(session, "~"))
-    data_obj = get_dataobject(session, ipath)
+    ops = upload(session, testdata/"plant.rtf", IrodsPath(session, "~"))
+    _check_count(ops, [0, 0, 0, 1])
+    data_obj = ipath.dataobject
     assert is_dataobject(data_obj)
     assert not is_collection(data_obj)
     with pytest.raises(ValueError):
-        get_collection(session, ipath)
-    download(session, ipath, testdata/"plant.rtf.copy", overwrite=True)
+        _ = ipath.collection
+    ops = download(session, ipath, testdata/"plant.rtf.copy", overwrite=True)
     assert _check_files_equal(testdata/"plant.rtf.copy", testdata/"plant.rtf")
+    _check_count(ops, [0, 0, 1, 0])
 
 
 def test_upload_download_collection(session, testdata, tmpdir):
     ipath = IrodsPath(session, "~", "test")
     ipath.remove()
-    upload(session, testdata, ipath)
-    collection = get_collection(session, ipath)
+    ops = upload(session, testdata, ipath)
+    _check_count(ops, [2, 0, 0, 7])
+    collection = ipath.collection
     assert is_collection(collection)
     assert not is_dataobject(collection)
     with pytest.raises(ValueError):
-        get_dataobject(session, ipath)
-    download(session, ipath, tmpdir/"test")
+        ipath.dataobject
+    ops = download(session, ipath, tmpdir/"test")
+    _check_count(ops, [0, 4, 7, 0])
     files = list(testdata.glob("*"))
 
     for cur_file in files:

diff --git a/docker/irods_client/tests/test_meta.py b/docker/irods_client/tests/test_meta.py
@@ -1,9 +1,10 @@
+import irods
 import pytest
 from pytest import mark
-import irods
 
-from ibridges.meta import MetaData
 from ibridges.export_metadata import export_metadata_to_dict
+from ibridges.meta import MetaData
+
 
 @mark.parametrize("item_name", ["collection", "dataobject"])
 def test_meta(item_name, request):
@@ -64,11 +65,10 @@ def test_meta(item_name, request):
     assert ("y", "x") in meta
 
 @mark.parametrize("item_name", ["collection", "dataobject"])
-def test_export_metadata(item_name, request):
+def test_metadata_todict(item_name, request):
     item = request.getfixturevalue(item_name)
     meta = MetaData(item)
     meta.clear()
-
     # test against:
     test_dict = {
             'name': item.name,
@@ -78,7 +78,7 @@ def test_export_metadata(item_name, request):
 
     if isinstance(item, irods.data_object.iRODSDataObject):
         test_dict['checksum'] = item.checksum
-    
+
     # Add some metadata
     meta.add("x", "z", "m")
     assert "x" in meta
@@ -88,3 +88,9 @@ def test_export_metadata(item_name, request):
         assert key in test_dict
     for value in result.values():
         assert value in test_dict.values()
+
+@mark.parametrize("item_name", ["collection", "dataobject"])
+def test_metadata_export(item_name, request, session):
+    item = request.getfixturevalue(item_name)
+    res = export_metadata_to_dict(MetaData(item), session)
+    assert isinstance(res, dict)
diff --git a/docker/irods_client/tests/test_move.py b/docker/irods_client/tests/test_move.py
@@ -1,4 +1,3 @@
-import pytest
 from pytest import mark
 from ibridges.path import IrodsPath
 

diff --git a/docker/irods_client/tests/test_permissions.py b/docker/irods_client/tests/test_permissions.py
@@ -1,10 +1,8 @@
 import pytest
-from irods.exception import CAT_NO_ACCESS_PERMISSION
 from pytest import mark
 
-from ibridges.data_operations import download, upload
-from ibridges.permissions import Permissions
 from ibridges.path import IrodsPath
+from ibridges.permissions import Permissions
 
 
 @mark.parametrize("item_name", ["collection", "dataobject"])

diff --git a/docker/irods_client/tests/test_session.py b/docker/irods_client/tests/test_session.py
@@ -1,9 +1,11 @@
-import pytest
 import json
 
+import pytest
+
 from ibridges import Session
 from ibridges.interactive import interactive_auth
 
+
 def test_session_from_cached_pw(config, irods_env):
     # test only for plain irods
     if config.get("create_session_from_cached_pw", True):

diff --git a/docker/irods_client/tests/test_sync.py b/docker/irods_client/tests/test_sync.py
@@ -1,44 +1,36 @@
-from ibridges.sync import sync_data, _calc_checksum
+from ibridges.data_operations import _calc_checksum, create_collection, sync
 from ibridges.path import IrodsPath
-from ibridges.data_operations import create_collection, get_dataobject
+
 
 def test_sync_dry_run(session, testdata, capsys):
 
     ipath = IrodsPath(session, "~", "empty")
     coll = create_collection(session=session, coll_path=ipath)
+    assert len(coll.data_objects)+len(coll.subcollections)==0, "Dry run starting not empty"
 
     # upload
-    sync_data(session=session,
-         source=testdata,
-         target=ipath,
-         max_level=None,
-         dry_run=True,
-         copy_empty_folders=True)
-
-    captured = capsys.readouterr()
-    lines=sorted([x.strip() for x in captured.out.split("\n") if len(x.strip())>0])
-
-    assert lines == ['/tempZone/home/rods/empty/more_data',
-        'Will create collection(s):',
-        "Will upload from '/tmp/testdata' to '/tempZone/home/rods/empty':",
-        'beach.rtf  727',
-        'bunny.rtf  10150',
-        'example.r  182',
-        'more_data/polarbear.txt  2717',
-        'plant.rtf  992',
-        'plant.rtf.copy  992',
-        'sun.rtf  661']
+    ops = sync(session=session,
+               source=testdata,
+               target=ipath,
+               max_level=None,
+               dry_run=True,
+               copy_empty_folders=True)
 
+    assert len(ops["create_collection"]) == 1
+    assert len(ops["create_dir"]) == 0
+    assert len(ops["download"]) == 0
+    assert len(ops["upload"]) == 7
     assert len(coll.data_objects)+len(coll.subcollections)==0, "Dry run did sync"
 
+
 def test_sync_upload_download(session, testdata, tmpdir):
     ipath = IrodsPath(session, "~", "empty")
     coll = create_collection(session=session, coll_path=ipath)
 
     assert len(coll.data_objects)+len(coll.subcollections)==0, "iRODS folder not empty"
 
     # upload
-    sync_data(session=session,
+    sync(session=session,
          source=testdata,
          target=ipath,
          max_level=None,
@@ -49,7 +41,7 @@ def test_sync_upload_download(session, testdata, tmpdir):
         s_ipath = IrodsPath(session, "~", "empty", cur_file.name)
         if cur_file.is_file():
             assert s_ipath.dataobject_exists(), "File not uploaded"
-            cur_obj=get_dataobject(session=session, path=s_ipath)
+            cur_obj=s_ipath.dataobject
             assert _calc_checksum(cur_file)==(cur_obj.checksum
                 if len(cur_obj.checksum)>0
                 else cur_obj.chksum()), "Checksums not identical after upload"
@@ -58,13 +50,13 @@ def test_sync_upload_download(session, testdata, tmpdir):
     assert s_ipath.collection_exists(), "Subfolder not uploaded"
     s_ipath = IrodsPath(session, "~/empty/more_data/polarbear.txt")
     assert s_ipath.dataobject_exists(), "File in subfolder not uploaded"
-    obj = get_dataobject(session=session, path=s_ipath)
+    obj = s_ipath.dataobject
     assert _calc_checksum(testdata / "more_data" / "polarbear.txt")== \
         (obj.checksum if len(obj.checksum)>0 else obj.chksum()), \
             "Checksums not identical after upload"
 
     # download
-    sync_data(session=session,
+    sync(session=session,
         source=ipath,
         target=tmpdir,
         max_level=None,

diff --git a/docs/Makefile b/docs/Makefile
@@ -12,9 +12,12 @@ BUILDDIR      = build
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+.PHONY: help Makefile clean
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+clean:
+	rm -r $(BUILDDIR) $(SOURCEDIR)/api/generated