You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The StreamingPythonExecutor streams data from Java to Python though a FIFO, but there is no support for streaming the results back to Java. Tools like NeuralNetInference need a way to propagate per-variant values that are calculated in Python, back to the original input variants. There are several possibilities:
Manually implement a post-traversal traversal that allows joining the values that have been written by Python to a file, with the original inputs, and then write the final output file. NeuralNetInference currently does this in onTraversalSuccess.
Codify this pattern as a TwoPassVariantWalker base class, similar to TwoPassReadWalker
Change NeuralNetInference and the StreamingPythonExecutor to allow bi-directional streaming. The tool would retain a batch of variants in memory, stream summary data to Python, then stream back the results, and write the "joined" values back before starting the next batch. This would work for model assumes that we never need to complete the first pass
The text was updated successfully, but these errors were encountered:
The StreamingPythonExecutor streams data from Java to Python though a FIFO, but there is no support for streaming the results back to Java. Tools like NeuralNetInference need a way to propagate per-variant values that are calculated in Python, back to the original input variants. There are several possibilities:
The text was updated successfully, but these errors were encountered: