[FEATURE REQ] blobURL.stageBlock should be able to get length from the Flowable

[ohadbitt]

The Api is expecting a flowable to work with - so i have a flowable which does some mappings and such on the data, which is totally logical (not everything is in files and known length) and i end up with a byteBuffer the problem is that i need to know the length in advance ?!?!
So the only way for me to know the size is by blocking and creating a new flowable, I think there is a simple solution to this that could make it much better and it is to accept a flowable of a new class which will contain a ByteBuffer and the amount of the data to read from it.
If its not possible please help me achieve a better solution.
Another solution would be to simply allow an "unsafe" mode - as it looks from the UnexpectedLengthException exception that the right length is known
Thanks!

[maggiepint]

Thanks for the feedback. We have routed to the appropriate team for follow-up. cc: @mayurid

[joshfree]

@alzimmermsft could you consider this for Preview 2 / Preview 3?

[alzimmermsft]

This will be added as a backlog item for further investigation during our Preview 3 sprint.

[rickle-msft]

Hi, @ohadbitt. Thank you for this feedback. Based on your use of Flowable, it sounds like you are using v11. You should be able to take advantage of TransferManager.uploadFromNonReplayableFlowable (excuse the name), which doesn't require a length parameter.

A few more things to consider:

• The reason we require a length parameter is because we have to set the content length header when we first send the request to the service (before we've touched the body at all). There's no way for us to get the length of a flowable before draining it, and we don't want to buffer the whole body, so we have to request it from the user. Unfortunately the service does not support chunked encoding, which would get around this.
• The UnexpectedLenthException is thrown as a result of keeping track of how much data we have read. If either the Flowable completes before we have tracked the hinted number of bytes or we have read the passed number of bytes but the Flowable has not completed, we will throw, but there is no way for us to check the length of the data and validate this until we have subscribed to the body and started to read. We included this exception as a safety check so that customers are protected against losing their data or sending data they didn't expect.
• You should be aware that the method I suggested to you is able to elide the length parameter by buffering. It does not block, so your async workflows should still be preserved, but it does do some internal buffering in order chunk up the data and determine the size of each body before sending it.
• The interface for this method should be improved when it gets added to v12. It's not there yet as we're still figuring out the best way to offer it to customers, but it will be there soon. I am thinking that it will likely become a method on BlobClient (the equivalent of BlobURL) with some name to indicate that the behavior (specifically the support for multiple subscribes which we typically require for retry support) and length of the flowable/flux are unknown.
• Because you are using v11, you should know that we are now recommending customers either stay on v8 or be prepared to upgrade to v12. The preview for v12 is out, and it looks similar to v11, though it is unfortunately another break. We have some explanation of why this was necessary coming out soon, but it should hopefully be a manageable transition with lots of benefits in the way of stability and cleanliness. If you are uncomfortable with working with a preview version or the preview version has not yet ported a feature from v11 that you need such as this one, then we will work with you in v11 in the meantime and help you upgrade when the time comes.

We would also be willing to consider your suggestion of an unsafe method or the use of this custom type if you would be able to offer some more details on those ideas.

[rickle-msft]

@ohadbitt If you are interested, you can take a look at #4935 which should address this feature request.

[rickle-msft]

@ohadbitt I am going to close this issue as the PR I referenced has been merged. If you feel this does not adequately address your feature rquest, please feel free to reopen it or post a new issue.

[ghost]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.