python based packages for pipeline based framework for data processing
- Free software: ISC license
- Documentation: https://pyoct.readthedocs.org.
assume you have input data as data_in, and you want to run the data
through a pipeline that is composed of a list of functions, you can run:
p = Pipeline([list of functions])
p.process(data_in)
p.close() to close the pipelinenotice that the data_in could be a iterable, so that the pipeline will run the iterable till the iterable stopiteration. In addition, internally the pipeline can use async method to enable concurrent processing.
- you need to define a function, e.g., test_func in file test_func.py.
- you need to define the input and ouput data type and format in test_func.ini
- you need to import the function and use funcwrap and argswrap to pipenize test_func
- use the Pipeline class to build and run the pipeline processing
use the config.py to configure the package to setup your enviroments this can includes: 1. use specific implementation in pypeline.impl, for example if you want to use the frame work for a specific applicaiton, e.g., OCT - you can specify a filter to load functions from a set of files - you can specify a list to load functions from a set of files 2. use specific function in pypeline.ext
same as load functions written in .py, currently the pypeline supports matlab and c/c++. please pay attention to following things: for matlab function 1. matlab function and file name should be the same. 2. you need to install matlab.engine provided by mathworks, and you need to have matlab installed on your computer 3. internally, matlab function is wrapped in pypelin.funcwrap module using matlab.engine
for c/c++ function 1. dll file is loaded 3. internally, function in dll file is wrapped in pypelin.funcwrap module using ctypes
- load multiple functions defined in a single .py file
Tools used in rendering this package: