jwarc failing to read Http body at the second pass of a file #80

vlofgren · 2023-12-12T20:44:12Z

I'm running into an issue where the WarcReader behaves differently when reading the same file twice. It seems there is some global state that is not appropriately cleaned up perhaps. In the second pass, http().body().isOpen() returns false, when it returned true in the first pass; and attempting to read the data fails.

I'm on jwarc 0.28.4.

This is the smallest test case I've been able to produce:

    @Test
    public void spookyActionAtADistance() throws IOException {
        Path tempFile = Files.createTempFile("test", ".warc");
        try {
            StringBuilder sb = new StringBuilder();
            
            sb.append("WARC/1.0\r\n");
            sb.append("WARC-Type: revisit\r\n");
            sb.append("WARC-Target-URI: https://www.example.com/\r\n");
            sb.append("WARC-Date: 2023-12-12T20:31:13Z\r\n");
            sb.append("WARC-Record-ID: <urn:uuid:ee40237d-5fb4-41a7-aa2e-bfb67c57f5f0>\r\n");
            sb.append("Content-Length: 134\r\n");
            sb.append("Content-Type: application/http;msgtype=response\r\n");
            sb.append("WARC-Payload-Digest: sha1:KC62VKWCVTBH5CT636VIFBP3KTLP4UZM\r\n");
            sb.append("WARC-Block-Digest: sha1:T2C4UJZP7PHK75RWHCID3GDDKWX5J5UX\r\n");
            sb.append("\r\n");
            sb.append("HTTP/1.1 200 OK\r\n");
            sb.append("Content-Type: text/html\r\n");
            sb.append("Content-Length: 45\r\n");
            sb.append("Content-Encoding: UTF-8\r\n");
            sb.append("\r\n");
            sb.append("<?doctype html><html><body>test</body></html>");

            Files.writeString(tempFile, sb.toString());

            try (var reader = new WarcReader(tempFile)) {
                for (var record : reader) {
                    WarcRevisit revisit = (WarcRevisit) record;
                    var http = revisit.http();
                    try (var body = http.body()) {
                        System.out.println(body.isOpen()); // prints true
                        body.stream().readAllBytes(); // works fine
                    }
                }
            }

            try (var reader = new WarcReader(tempFile)) {
                for (var record : reader) {
                    WarcRevisit revisit = (WarcRevisit) record;
                    var http = revisit.http();
                    try (var body = http.body()) {
                        System.out.println(body.isOpen());  // prints false
                        body.stream().readAllBytes();  // exception !!
                    }
                }
            }

        }
        finally {
            Files.deleteIfExists(tempFile);
        }
    }

Bugs fixed * Fixed ClosedChannelException when reading a WarcRevisit body after closing a previous one due to reuse of empty MessageBody. #80

ato · 2023-12-13T03:10:36Z

Oops. Thanks for reporting this! The same empty MessageBody instance was being reused which of course breaks if it gets closed.

There might be a secondary issue here in that jwarc currently assumes all revisit records have no payload but perhaps that's a false assumption.

ato · 2023-12-13T03:11:14Z

Fix released as v0.28.5. It should sync to maven central in a couple of hours.

vlofgren · 2023-12-13T12:52:22Z

Thanks for the quick fix :D

I'll work around the revisit limitations now that I know they exist.

ato closed this as completed in 35bae7b Dec 13, 2023

ato added a commit that referenced this issue Dec 13, 2023

Release 0.28.5

e50593f

Bugs fixed * Fixed ClosedChannelException when reading a WarcRevisit body after closing a previous one due to reuse of empty MessageBody. #80

ato added the bug label Dec 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jwarc failing to read Http body at the second pass of a file #80

jwarc failing to read Http body at the second pass of a file #80

vlofgren commented Dec 12, 2023

ato commented Dec 13, 2023

ato commented Dec 13, 2023

vlofgren commented Dec 13, 2023

jwarc failing to read Http body at the second pass of a file #80

jwarc failing to read Http body at the second pass of a file #80

Comments

vlofgren commented Dec 12, 2023

ato commented Dec 13, 2023

ato commented Dec 13, 2023

vlofgren commented Dec 13, 2023