Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Java/Python bidirectional data streaming #4316

Closed
cmnbroad opened this issue Jan 31, 2018 · 2 comments
Closed

Support Java/Python bidirectional data streaming #4316

cmnbroad opened this issue Jan 31, 2018 · 2 comments
Assignees

Comments

@cmnbroad
Copy link
Collaborator

The StreamingPythonExecutor streams data from Java to Python though a FIFO, but there is no support for streaming the results back to Java. Tools like NeuralNetInference need a way to propagate per-variant values that are calculated in Python, back to the original input variants. There are several possibilities:

  • Manually implement a post-traversal traversal that allows joining the values that have been written by Python to a file, with the original inputs, and then write the final output file. NeuralNetInference currently does this in onTraversalSuccess.
  • Codify this pattern as a TwoPassVariantWalker base class, similar to TwoPassReadWalker
  • Change NeuralNetInference and the StreamingPythonExecutor to allow bi-directional streaming. The tool would retain a batch of variants in memory, stream summary data to Python, then stream back the results, and write the "joined" values back before starting the next batch. This would work for model assumes that we never need to complete the first pass
@cmnbroad cmnbroad added this to the Engine-4.1 milestone Jan 31, 2018
@cmnbroad cmnbroad self-assigned this Jan 31, 2018
@droazen droazen modified the milestones: Engine-4.1, Engine-1Q2018 Feb 5, 2018
@droazen droazen modified the milestones: Engine-1Q2018, Engine-2Q2018 Apr 6, 2018
@droazen droazen removed this from the Engine-2Q2018 milestone Oct 4, 2018
@droazen
Copy link
Contributor

droazen commented Oct 4, 2018

@cmnbroad Is this done?

@cmnbroad
Copy link
Collaborator Author

cmnbroad commented Oct 5, 2018

Yes, now that we have TwoPassVariantWalker, we're using that for CNN (in #5175 which should go in soon).

@cmnbroad cmnbroad closed this as completed Oct 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants