Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: Euro-BioImaging/BatchConvert
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.0.4
Choose a base ref
...
head repository: Euro-BioImaging/BatchConvert
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 11 commits
  • 32 files changed
  • 2 contributors

Commits on Feb 21, 2024

  1. Copy the full SHA
    426c322 View commit details
  2. Copy the full SHA
    34e41d9 View commit details

Commits on Feb 22, 2024

  1. Copy the full SHA
    b68ef0f View commit details
  2. Copy the full SHA
    9e5550c View commit details

Commits on Sep 23, 2024

  1. Copy the full SHA
    75a1679 View commit details

Commits on Nov 3, 2024

  1. multiple changes

    bugraoezdemir committed Nov 3, 2024
    Copy the full SHA
    c927a49 View commit details
  2. multiple changes

    bugraoezdemir committed Nov 3, 2024
    Copy the full SHA
    12f7946 View commit details
  3. update readme

    bugraoezdemir committed Nov 3, 2024
    Copy the full SHA
    e7e1b3a View commit details
  4. Copy the full SHA
    2126670 View commit details

Commits on Jan 13, 2025

  1. updates to OME-Zarr

    bugraoezdemir committed Jan 13, 2025
    Copy the full SHA
    11a0b84 View commit details
  2. updates

    bugraoezdemir committed Jan 13, 2025
    Copy the full SHA
    215ad50 View commit details
12 changes: 12 additions & 0 deletions .idea/BatchConvert.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 16 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -352,7 +352,7 @@ However, this command would fail to create a single OME-Zarr folder due to the n
lengths of the filenames. Instead, the files would be split into two groups based on the
filename length, leading to two separate OME-Zarrs with names:

`test_img_TRange{2-8-2}.ome.zarr` and `test_img_TRange{10-12-2}.ome.zarr`
`test_img_Ts2to8step2.ome.zarr` and `test_img_Ts10to12step2.ome.zarr`

Here is the corrected version of the folder for the above example-
```
@@ -365,7 +365,7 @@ time-series/test_img_T12
```

Executing the same command on this folder would result in a single OME-Zarr with the name:
`test_img_TRange{02-12-2}.ome.zarr`
`test_img_Ts02to12step2.ome.zarr`

**Example 2**-

@@ -383,7 +383,7 @@ A typical command to convert this folder to a single OME-Zarr would look like: \
However, the command would fail to assume these files as a single group due to the
non-uniform incrementation in the variable field of the filenames. Instead, the dataset
would be split into two groups, leading to two separate OME-Zarrs with the following names:
`test_img_TRange{2-4-2}.ome.zarr` and `test_img_TRange{5-7-2}.ome.zarr`
`test_img_Ts2to4step2.ome.zarr` and `test_img_Ts5to7step2.ome.zarr`


**Example 3**
@@ -405,8 +405,8 @@ BatchConvert will fail to assume these two channels as part of the same series a
will instead split the two channels into two separate OME-Zarrs.

The output would look like: \
`test_img_C1-TRange{1-3-1}.ome.zarr` \
`test_img_C2-TRange{1-2-1}.ome.zarr`
`test_img_C1-Ts1to3step1.ome.zarr` \
`test_img_C2-Ts1to2step1.ome.zarr`

To be able to really incorporate all files into a single OME-Zarr, the folder should have equal
number of images corresponding to both channels, as shown below:
@@ -420,7 +420,7 @@ multichannel_time-series/test_img_C2-T3
```
The same conversion command on this version of the input folder would result in a single
OME-Zarr with the name: \
`test_img_CRange{1-2-1}-TRange{1-3-1}.ome.zarr`
`test_img_Cs1to2step1-Ts1to3step1.ome.zarr`


**Example 4**
@@ -444,8 +444,8 @@ One can convert this folder with- \
`batchconvert omezarr --merge_files "input_dir/folder_with_multiple_groups" "output_path"`

BatchConvert will detect the two patterns in this folder and perform two grouped conversions.
The output folders will be named as `test_img_CRange{1-2-1}-TRange{1-2-1}.ome.zarr` and
`test_img_TRange{1-2-1}-ZRange{1-3-1}.ome.zarr`.
The output folders will be named as `test_img_Cs1to2step1-Ts1to2step1.ome.zarr` and
`test_img_Ts1to2step1-Zs1to3step1.ome.zarr`.


**Example 5**
@@ -478,8 +478,8 @@ So the following line can be used to convert this folder: \
`batchconvert omezarr --merge_files --concatenation_order ct,aa "input_dir/folder_with_multiple_groups" "output_path"`

The resulting OME-Zarrs will have the names:
`test_img_CRange{1-2-1}-TRange{1-2-1}.ome.zarr` and
`test_img_TRange{1-2-1}-ZRange{1-3-1}.ome.zarr`
`test_img_Cs1to2step1-Ts1to2step1.ome.zarr` and
`test_img_Ts1to2step1-Zs1to3step1.ome.zarr`

Note that `--concatenation_order` will override any dimension specifiers already
existing in the filenames.
@@ -512,7 +512,7 @@ One may try the following command to convert this folder:
`batchconvert omezarr --merge_files "input_dir/filenames_with_dates" "output_path"`

Since the concatenation axes are not specified, this command would try to create
a single OME-Zarr with name: `test_data_dateRange{03-04-1}.03.2023_imageZRange{1-2-1}-TRange{1-3-1}`.
a single OME-Zarr with name: `test_data_dates03to04step1.03.2023_imageZs1to2step1-Ts1to3step1`.

In order to force BatchConvert to ignore the date field, the user can restrict the concatenation
axes to the last two numeric fields. This can be done by using a command such as: \
@@ -540,3 +540,8 @@ Note that this option automatically uses the singularity profile:\








8 changes: 4 additions & 4 deletions batchconvert.sh
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ if [[ -f $TEMPPATH/.stderr ]];
rm $TEMPPATH/.stderr;
fi;

if [[ ${#error} > 0 ]];
if [[ ${#error} -gt 0 ]];
then
printf "${RED}$error${NORMAL}\n"
printf "${RED}The batchonvert command is invalid. Please try again.${NORMAL}\n"
@@ -40,7 +40,7 @@ if [[ $result == "inputpatherror" ]];
then
printf "${RED}Error: The input path does not exist.\n${NORMAL}"
exit
elif [[ ${#result} > 0 ]];
elif [[ ${#result} -gt 0 ]];
then
if [[ $process == 'parameters_shown' ]];
then
@@ -90,9 +90,9 @@ elif [[ $process == "default_param_set" ]];
printf "${GREEN}Default parameter updated.\n${NORMAL}";
elif [[ $process == 'converted' ]];
then
printf "${GREEN}Nextflow script has been created. Workflow is beginning.\n${NORMAL}" && \
chmod +x $BINPATH/run_conversion.py && \
pythonexe $SCRIPTPATH/bin/run_nextflow_cli.py && \
printf "${GREEN}Nextflow script has been created. Workflow is beginning.\n${NORMAL}"
pythonexe $SCRIPTPATH/bin/run_nextflow_cli.py
fi

if [[ -f $TEMPPATH/.process ]];
Binary file added bin/__pycache__/directory2symlink.cpython-310.pyc
Binary file not shown.
Binary file added bin/__pycache__/directory2symlink.cpython-39.pyc
Binary file not shown.
Binary file not shown.
Binary file not shown.
6 changes: 4 additions & 2 deletions bin/batchconvert
Original file line number Diff line number Diff line change
@@ -143,7 +143,7 @@ if __name__ == "__main__":

### specify the config profile
ometiff.add_argument('--profile', '-pf', default=getdef('profile', "conda"), type=str,
help="Specifies one of the six profiles: manual, conda, mamba, docker, singularity and cluster")
help="Specifies one of the six profiles: manual, conda, mamba, _docker, singularity and cluster")

### specify the work directory
ometiff.add_argument('--workdir', '-wd', default=getdef('workdir', ""), type=str,
@@ -216,7 +216,7 @@ if __name__ == "__main__":

### specify the config profile
omezarr.add_argument('--profile', '-pf', default=getdef('profile', "conda"), type=str,
help="Specifies one of the five profiles: manual, conda, mamba, docker, singularity and cluster")
help="Specifies one of the five profiles: manual, conda, mamba, _docker, singularity and cluster")

### specify the work directory
omezarr.add_argument('--workdir', '-wd', default=getdef('workdir', ""), type=str,
@@ -516,6 +516,8 @@ if __name__ == "__main__":
# print(args.input_column)
# print(args.root_column)
if ( args.in_path.endswith('.csv') or (args.in_path.endswith('.txt')) ):
if args.root_column in ("None", None, '', ""):
args.__dict__['root_column'] = "null"
assert len(args.input_column) > 0, 'The csv file must contain at least one input_column.'
os.chdir(scriptpath)
if args.keep_workdir == True:
2 changes: 1 addition & 1 deletion bin/create_hyperstack
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python
import argparse
from PatternHandler import FilelistGrouper
from pattern_manager import FilelistGrouper

if __name__ == '__main__':
parser = argparse.ArgumentParser()
38 changes: 0 additions & 38 deletions bin/csv2Symlink.py

This file was deleted.

61 changes: 61 additions & 0 deletions bin/csv2symlink.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env python

import csv, os, sys, argparse, glob, shutil
from pathlib import Path

from directory2symlink import makelinks


parser = argparse.ArgumentParser()
parser.add_argument('csv_file_path')
parser.add_argument('root_column')
parser.add_argument('relpath_column')
parser.add_argument('dest_path')
parser.add_argument('--contains', default = None)
parser.add_argument('--ignores', default = None)


if __name__ == '__main__':
args = parser.parse_args()
result_dict_list = []
csv_file_path = args.csv_file_path

dircol = args.root_column
namecol = args.relpath_column
destdir = args.dest_path

os.makedirs(destdir, exist_ok = True)

result_dict_list = []
with open(csv_file_path, 'r') as csv_file:
csv_reader = csv.DictReader(csv_file)
for row in csv_reader:
result_dict_list.append(row)

pathlist = []
for dictitem in result_dict_list:
if dircol in dictitem:
root = dictitem[dircol].replace('root/', '')
elif dircol in ("auto", None, ""):
root = os.path.dirname(csv_file_path)
else:
raise Exception(f"The specified root column '{dircol}' does not exist.")

if namecol in dictitem:
relpath = dictitem[namecol]
else:
raise Exception(f"The specified name column '{namecol}' does not exist.")

if relpath.startswith('/'):
assert dircol in ("auto", None, ""), f"A relative path cannot start with a '/'."
# relpath = relpath[1:]
fpath = relpath
else:
fpath = os.path.join(root, relpath)
pathlist.append(fpath)

makelinks(pathlist, destdir,
replace_outpath = False,
contains = args.contains,
ignores = args.ignores)

106 changes: 106 additions & 0 deletions bin/directory2symlink.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
#!/usr/bin/env python
import os, sys, argparse, glob, json, copy
import shutil
from pathlib import Path

def makelinks(inpath,
outpath,
contains = None,
ignores = None,
replace_outpath = True,
files_only = 'False' # If true, do not symlink sub-directories.
):

if isinstance(inpath, Path):
inpath = inpath.as_posix()
if isinstance(outpath, Path):
outpath = outpath.as_posix()

# print(inpath)

if isinstance(inpath, (list, tuple)):
paths = inpath
elif os.path.isfile(inpath):
paths = glob.glob(inpath)
else:
if '**' in inpath:
files_only = 'True' ## With the recursive option, do not symlink sub-directories.
paths = glob.glob(inpath, recursive = True)
elif '*' in inpath:
paths = glob.glob(inpath)
# print(inpath, paths)
else:
paths = glob.glob(os.path.join(inpath, '*'))
# print(paths)

if files_only in ('True', True):
paths = [path for path in paths if os.path.isfile(path)]

fpaths = copy.deepcopy(paths)
# print(fpaths)
fpaths_s = [fpath.split('/') for fpath in fpaths]
# print(fpaths_s)
for i, path in enumerate(fpaths_s):
fpath = ['/' if item == '' else item for item in path]
fpaths_s[i] = fpath
# print(fpaths_s)
l = len(fpaths_s[0])
shortest = fpaths_s[0]
for fpath in fpaths_s:
if len(fpath) < l:
shortest = fpath
l = len(fpath)
for i in range(l):
fields = [path[i] for path in fpaths_s]
if all([field == shortest[i] for field in fields]):
pass
else:
break

basenames = ['_'.join(item[i:]) for item in fpaths_s]
fnames = [item[-1] for item in fpaths_s]

outfiles = []
for i, basename in enumerate(basenames):
names = copy.deepcopy(fnames)
names.pop(i)
fname = fnames[i]
# print(fname)
if fname in names:
outfiles.append(os.path.join(outpath, basename).replace(' ', '_'))
else:
outfiles.append(os.path.join(outpath, fname).replace(' ', '_'))
###
filenames = [item[-1] for item in fpaths_s]

if replace_outpath:
if os.path.exists(outpath):
shutil.rmtree(outpath)
os.makedirs(outpath, exist_ok=True)

symlink = lambda source, sink: os.symlink(source, sink) if not os.path.exists(sink) else None

for path, name, outfile in zip(fpaths, filenames, outfiles):
if contains in (None, 'None', '') and ignores in (None, 'None', ''):
symlink(path, outfile)
elif ignores not in (None, 'None', ''):
if ignores not in name:
symlink(path, outfile)
elif contains not in (None, 'None', ''):
if contains in name:
symlink(path, outfile)
return outpath



if __name__ == '__main__':
scriptpath = os.path.dirname(os.path.realpath(__file__))

parser = argparse.ArgumentParser()
parser.add_argument('in_path')
parser.add_argument('--contains', default=None)
parser.add_argument('--ignores', default=None)
parser.add_argument('--files_only', default = 'False', choices = ('True', 'False'))
args = parser.parse_args()

makelinks(args.in_path, 'symlinks', contains=args.contains, ignores=args.ignores, files_only=args.files_only)
Loading