Added support for reading ARC and ARC.GZ.

[thomasegense]

Support for ARC.GZ and ARC files.

Added Javascript methods for handling ZIP content, since this is not supported by the warc.io.js library.
See: webrecorder/warcio.js#5

What the new code does

ARC file parsing — ARC records have no WARC-style header block, just one plain-text line (URL IP-address date content-type length) directly followed by the raw captured HTTP response. A small streaming parser (ByteCursor + indexArcFile) walks these header lines and skips over each body by its declared length — mirroring the existing two-pass design (cheap indexing pass, then full detail fetched only when a record is clicked). Records are typed as response, filedesc (the file's own descriptor), or dns so they plug straight into the existing type-filter chips. For response records, the captured HTTP block is split into status line + headers + body and handed to the app's existing payload-decoding logic, unchanged.

GZIP/DEFLATE decoder (gunzipMultiMember and helpers) — this is the one genuinely new piece of infrastructure. .arc.gz/.warc.gz files gzip each record separately and concatenate them, but the browser's built-in DecompressionStream API is spec-restricted to a single gzip member per stream and errors on anything beyond it. Rather than pull in an external library, this is a from-scratch, dependency-free implementation of RFC 1951 (DEFLATE: Huffman decoding, LZ77 back-references, stored/fixed/dynamic blocks) plus RFC 1952 (gzip framing), that loops over concatenated members and hands back one continuous decompressed buffer. .arc.gz files are decompressed fully up front and then indexed/read exactly like a plain .arc; uncompressed .arc files are still streamed record-by-record without ever loading the whole thing into memory.

Everything else (file pickers, index.html labels/hints, the "ARC Header" pane relabeling) is plumbing to route .arc/.arc.gz selections to this new path instead of the WARC one.

[ato]

Maybe ARC support should be added to warcio.js instead of directly to WARC Explorer? It looks like warcio.js uses pako for gzip decoding so maybe the ARC support can use that instead of adding another gzip/deflate implementation?

[joncto]

Great initiative, @thomasegense! Thanks for submitting this PR.

Supporting ARC(.gz) is a good idea. (We converted our old NedLib and ARC files to WARC, so I almost forgot those formats...) I also appreciate your effort to avoid external dependencies.

I tested your code on my laptop:

• Small ARC.gz (~3MB) works really well,
• Medium size ARC.gz (~100MB) takes a bit more time and occupy more memory, but can still be handled
• Large files (~1GB) explodes the memory and results in an error.

Since non-specialists are kind of the primary target group, it would be great to have a solution that work on an "average curator's computer", ensuring the tool remains lightweight.

Would it be feasible to design the reading of ARC(.gz) in a similiar manner as warcio.js has done for WARC files, allowing for streaming or in other ways fetching only certain parts of a file or record? This way, one could index the one-line "header" and maybe even http headers when choosing an ARC file, and then handle payloads individually for each record when they are interacted with?

Jon

[ato]

While warcio.min.js doesn't expose pako directly, it does offer warcio.AsyncIterReader. If we use that we don't need to add the second gzip/deflate implementation and can also avoid keeping the whole decompressed ARC file in memory.

Here's an AI proof of concept demonstrating that. (I haven't cleaned this up or checked it in detail. This is just intended to show it's possible.)
2ab3474

[joncto]

With @ato's proof of concept, I was able to index a 1.2GB ARC.gz (1732 records) on my laptop.

I performed ten test-runs, each of them completed within a span of 8.2-8.4 seconds. So even if it takes some time (large WARC.gz files are slow as well), I didn't run out of memory :)

[thomasegense]

I am back from vacation now. What is the best way forward?
I suggest using @ato patch since it is solve the memory issue and is a complete solution for arc/warc exploration.
Custom GUI for ARC files can then be added. Also showing offsets for both arc/warc files would be really usefull.
Maybe even a lookup where you can enter an offset and jump the record.

But the long term best solution would be to have warc.io.js (https://github.com/webrecorder/warcio.js) support ARC files. I did one attempt with Claude but it failed and I lost confidence it that it would not also break something.

[thomasegense]

Alex's patch is now merge into the PR.

[joncto]

Thanks, @thomasegense!
The technical part of reading ARC seems to be solved, and it at least seem to work well on old NB ARCs.
My week is a bit hectic with focus on a couple of other projects, but I'll have a look at adapting the layout within a couple of weeks. When that is in place, I'll merge this PR and probably launch a new version that include some other adjustments as well.