Reduce pipeline processes #298
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christian-monch
force-pushed
the
enh-reduce-pipeline-processes
branch
from
e18f550
to
113321d
Compare
christian-monch
force-pushed
the
enh-reduce-pipeline-processes
branch
from
113321d
to
6183a2c
Compare
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## maint_0.4 #298 +/- ##
==========================================
Coverage 86.27% 86.27%
==========================================
Files 88 88
Lines 4830 4830
==========================================
Hits 4167 4167
Misses 663 663 ☔ View full report in Codecov by Sentry. |
|
that's a lot of commits/work -- is it going to be merged? |
christian-monch
force-pushed
the
enh-reduce-pipeline-processes
branch
from
31d87d4
to
7cea3e2
Compare
This commit adds `build` and `twine` to `requirements-devel.txt`. It also moves sphinx-dependencies into development requirements. The datalad version is updated to >=0.17 In addition it sort the entries in `requirements-devel.txt` and `requirements.txt`.
This commit introduces AnnexedFileInfo, to hold annex-status information for a single file. To simplify handling, the dataclasses_json package is used and added to requirements Python version requirement has been set to >=3.7
This commit extends the FileInfo dataclass and derives the AnnexedFileInfo class from it. The classes hold file-information that is returned by AnnexRepo.get_content_annexinfo(), or by GitRepo.status(). It adds a parameter to pass JSON-serialized FileInfo or AnnexedFileInfo objects to the extract process via arguments, thus releaving the necessity to invoce git-annex to determine file-status.
This commit uses the --file-info parameter to provide extractors with status information about the element from which metadata should be extracted.
This commit adds code to handle repositories that do not posses an ID, usually these are plain git repositories.
This commit adds a pipeline provider and a pipeline processor with definable input output behavior. The content that should be yielded can be defined externally, the rate in which content is yielded can also be defined externally. This allows to perform repeatable performance measurements.
This commit adds information about the object-id and the processor pid of the processor and provider probes that are executed by meta-conduct
This commit adds an invocation count to the processor probe that counts the invocations on this instance of the probe.
This commit rebases the branch on maint_0.4 and adds code to check for the existence of datset IDs.
This commit fixes the reporting of datasets in traversal. There is still something to due in datasets, i.e. report "state", "gitshasum", and "prev_gitshasum"
This commit adds size information to the traverser output for non-annexed files. Because git-ls-files does not provide the size information, an additional git-ls-tree call is used to determine sizes of non-annexed files
|
Git version 2.30.2, which is currently available in debian-stable, does not support I very much like the spirit of this PR and I'm not sure whether debian-stable compatibility is a goal, so I'm just leaving this as an observation. |
This commit improves the code to handle the result of `git ls-files` properly, especially the size-attribute of directories.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes #261 partially and fixes #268
This PR modifies the dataset traverser component to send almost all information it has about the traversed dataset elements to the processors. That reduces the number of processes that the processors have to execute.