Import WARC with WebRecorder.AppImage (and Upload to Conifer.Rhizome.org) stops at 50% with "Error Encountered".

[payingattention]

Referencing another issue #111:

I couldn't open it though with https://github.com/webrecorder/webrecorder-player or https://github.com/webrecorder/pywb, it gives a record.

I can confirm that with the Import button in webrecorder.AppImage, AND with saving by "Uploading [WARC] to Collection" with https://conifer.rhizome.org/037 (https://WebRecorder.io/037 changed name), in each your tool's WARC file only get to 50% and then says "Error Encountered" in each interface.

My same https://twitter.com/prosodyContext page saved with the https://WebRecorder.net https://github.com/webrecorder/webrecorder-desktop ".AppImage" binary works (I do not mean to make you compete, multiple tools is important).

Should I stay in the original thread I referenced or is here appropriate/okay/good/better?, maybe here is specific, I do not want to ruin their conversation.

I can provide the WARCreate Google Chrome extension WARC archive(s) in question if you ask.

Originally posted by @payingattention in #111 (comment)

[machawk1]

Hi @payingattention, your feedback and experience is important is remedying the issue you are experiencing. I have yet to isolate the part of the WARC that WARCreate is creating that is causing Webrecorder to choke. It is the responsibility of WARCreate to generate valid WARCs and not the expectation of other tools to account for any violations of the spec.

While I hope Webrecorder will improve to make the processing of invalid WARCs (ping @ikreymer) still usable in their tool suite (as there might be other tools with a similar affliction as WARCreate), I will need to isolate exactly what sort of record or formatting issue is causing the issue of tools meant to make the files WARCreate generates usable.

I do not have the time to do this at the moment (WARCreate is currently a volunteer-run, unfunded project), but am committed to get it back to a working state with time. Further independent contributions or studies onto the reasons for the effects you described (and of course, PRs) are welcomed.

[machawk1]

Also, @payingattention, if you would be willing to provide the resultant WARC, this can be used as the basis for reproducibility.

[payingattention]

Thank you, the greater spec/format is drastic improvement over just trusting archive.org/webcitation.org/archive.today for user-centric evidence gathering and giving users proof, we are fortunate to be the first to know the toolability exists, so troubleshooting WARCs is actually important for me:

• 20200704064631467.warc.zip is exact same archive from screenshot given.
• 20200708172743178.warc.zip is the same site but tried again fresh today, sadly same error.

While I have limited forensics as a user, I gather by my Ubuntu Linux system's U.I. behavior, that it's not showing the official WARC mime-type thumbnail (other .warc's show a package thumb/icon) because the generated archive copy itself is not saving completely/fully, if I'm forced to guess.

I appreciate the .warc still shows basic metadata for reference in a human-readable format. Thank you for the free open source extension, I hope my WARC zips help reproduce the error, I value your time.

[ibnesayeed]

$ wget https://github.com/machawk1/warcreate/files/4892191/20200704064631467.warc.zip
$ unzip 20200704064631467.warc.zip
$ pip install warcio
$ python3
Python 3.8.2 (default, Apr 23 2020, 14:22:33) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from warcio.archiveiterator import ArchiveIterator
>>> with open('20200704064631467.warc', 'rb') as stream:
...     for record in ArchiveIterator(stream):
...         if record.rec_type == 'response':
...             print(record.rec_headers.get_header('WARC-Target-URI'))
... 
https://twitter.com/prosodyContext
    WARNING: Record not followed by newline, perhaps Content-Length is invalid
    Offset: 616195
    Remainder: b'C/1.0\r\n'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 224, in _detect_type_load_headers
    rec_headers = self.warc_parser.parse(stream, statusline)
  File "/usr/local/lib/python3.8/site-packages/warcio/statusandheaders.py", line 270, in parse
    raise StatusAndHeadersParserException(msg, full_statusline)
warcio.statusandheaders.StatusAndHeadersParserException: Expected Status Line starting with ['WARC/1.1', 'WARC/1.0', 'WARC/0.17', 'WARC/0.18'] - Found: WARC-Type: request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.8/site-packages/warcio/archiveiterator.py", line 110, in _iterate_records
    self.record = self._next_record(self.next_line)
  File "/usr/local/lib/python3.8/site-packages/warcio/archiveiterator.py", line 257, in _next_record
    record = self.loader.parse_record_stream(self.reader,
  File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 85, in parse_record_stream
    (the_format, rec_headers) = (self.
  File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 229, in _detect_type_load_headers
    raise ArchiveLoadFailed(msg + str(se.statusline))
warcio.exceptions.ArchiveLoadFailed: Invalid WARC record, first line: WARC-Type: request

>>>