WARC filename prefix + rollover size + improved 'livestream' / truncated response support. #440
Conversation
- add a size to takeStream to ensure data is streamed in fixed chunks - fix adding WARC-Truncated header, need to check after stream is finished - ensure WARC is not rewritten after it is done, skip writing records if stream already flushed - first close browser, then finish writing WARC to ensure any truncated responses are captured - add timeout to fetcherQ to avoid never hanging on finish
- create new filename from template, interpolating '$ts' - add --warcPrefix arg for setting WARC filename prefix (default to 'rec') - create new WARC file when offset exceeds rollover size, configured via --rolloverSize
|
Will add some tests for rollover and prefix, truncated / livestream response a bit more tricky. |
| @@ -30,6 +30,8 @@ const MAX_BROWSER_TEXT_FETCH_SIZE = 25_000_000; | |||
|
|
|||
| const MAX_NETWORK_LOAD_SIZE = 200_000_000; | |||
|
|
|||
| const TAKE_STREAM_BUFF_SIZE = 1024 * 64; | |||
There was a problem hiding this comment.
Might be good to make this configurable via args so people can fetch larger streams if they want?
There was a problem hiding this comment.
Oh, this isn't a limit, just size of buffer that is fetched at a time, streaming in 64k chunks. It is then either stored in memory or flushed to disk (if the the MAX_BROWSER_* total size is reached)
There was a problem hiding this comment.
Right, BUFF_SIZE! So the limit is just the appropriate timeouts?
There was a problem hiding this comment.
Those limits are for how much could be kept in memory before being buffered to disk. However, only content that is buffered in memory can be processed through the rewriting system and served back to browser. The limits for text are higher because it might be rewritten. There is no limit (beyond the disk volume size of course) for response that is streamed and buffered to disk, however, it is not served back to the browser. In case of a live-stream audio/video, the data is not actually played in the browser as a result during capture, but is written to disk, which should be fine (only issues might be for screenshot comparison...)
Probably should explain this better somewhere!
Support for rollover size and custom WARC prefix templates:
${prefix}-${crawlId}-$ts-${this.workerid}.warc${his.gzip ? ".gz" : ""}with$tsreplaced at new file creation time with current timestampImproved support for long (non-terminating) responses, such as from live-streaming:
WARC-Truncatedheader, need to set after stream is finished to determine if its been truncatedtmp-dldir to main temp folder, outside of collection (no need to be there).