api ref: better explanation on disc and memory usage for read/open #1037
Conversation
jorgeorpinel
commented
Mar 8, 2020
jorgeorpinel
commented
|
For now I've implemented a SAX XML example in the doc, but let's continue the discussion on what we want to show in examples. What's the advantage of streaming files in open/read? Probably just making a big file available quickly so you can start processing it before it's all downloaded.
I don't think you'd want to show the progress of a real-time processing, or is that a major use case you guys see? |
| @@ -45,8 +45,8 @@ file can be tracked by DVC or by Git. | |||
| This function makes a direct connection to the | |||
| [remote storage](/doc/command-reference/remote/add#supported-storage-types) | |||
| (except for Google Drive), so the file contents can be streamed as they are | |||
| read. This means it does not require space on the disc to save the file before | |||
| making it accessible. The only exception is when using Google Drive as | |||
| read. This means it does not require disc space to save the file before making | |||
There was a problem hiding this comment.
are read -> is read? (similar to data, it's strange to see contents are)
There was a problem hiding this comment.
"file contents are" seems correct to me. I can change it to "file content is" though. Same meaning
to emphasize about disc and memory usage in each one
jorgeorpinel
changed the title
| [remote type](/doc/command-reference/remote/add#supported-storage-types). | ||
| (except for Google Drive), so the file contents can be streamed. Your code can | ||
| process the data [buffer](https://docs.python.org/3/c-api/buffer.html) as it's | ||
| streamed, which optimizes memory usage. |
| `dvc.api.open()` is able to stream the data download. (The `mySAXHandler` object | ||
| should handle the event-driven parsing of the document in this case.) This | ||
| increases the performance of the code (minimizing memory usage), and is | ||
| typically faster than loading the whole data into memory. |
|
Woot woot 🎉 Cc @Suor feel fee to review this port-merge. Thanks! |
