Make zimHttpServer32 compatible with AVM Fritz!Box routers (freetz)

[cm8]

AVM Fritz!Box products are big endian machines, while the integers
and shorts in the zim file format are stored in little endian byte
order.

Essentially, this patch adds robustness to read the zim file format,
regardless of the endianess of the target machine that executes perl
and this script. For this, unpack("I") and unpack("s") statements
were replaced.

[cm8]

I was not aware of your "endian" branch while creating this patch, and hence it has some slight differences to the solution given there, now.

In particular you can use "<" and ">" modifiers to unpack from perl 5.10 and greater, i.e. unpack("s<"). For prior versions of perl, unpack("s", pack("S", unpack("v", $_))) cascade may be used, so you may not need uint16_int16 sub.

The modified code runs on little-endian machines as well, so there should be no need to have dedicated branches for both endianesses.

Also, decompressed xz data is piped to and processed in memory with this patch. Only $file.xz is written temporarily to /tmp. -- I also wrote a version using IPC::Run to avoid writing to /tmp at all, but found that IPC::Run is not among the standard perl modules; however IPC::Open2 (http://perldoc.perl.org/IPC/Open2.html) is, but presumably needs fork() logic to separate producer and consumer to not deadlock when used with binary data streams.

You probably do not want the avm specific README files in your master branch, but I'm not sure; just cherry-pick what you find is useful. If you had been aware of "<" and ">" modifiers in your "endian" branch, I probably would not have created this pull request.

[waynepiekarski]

Thanks for the patch, I'll have a look at it soon.

Is there some new project happening or something going on on a mailing list somewhere? Its been quiet for over a year and this week I've received 3 separate emails, which is why I did the endian patches a few days ago.

Do you think it is a good idea to do the xz in memory? I have one user who is having problems on 128 mb machines right now, and doing the decompression in memory might make it worse?

[cm8]

The mips target machine (that has 128 mb here) does not execute xz -d without enabling swap. The decompressor needs 65 mb to run and only 20 to 42 mb are available without swap.

If another 2-4 mb are written to memory with swap enabled does not really matter, however writing it to /tmp wear levels on the nand on most embedded machines. Note that swap does not necessarily swap to the nand flash built into the device, but to storage attached via usb.

Ultimately, it should be configurable whether to use /tmp or memory based approach, I think. Embedded devices differ, what works best (or "at all") on one may not be the best for another device.

Don't know why you received three mails about the topic - maybe someone relays google search history/logs to you (to which my search requests in the past week have incidentally contributed). I'm just hacking the box for curiosity, partly because kiwix-serve is not available on mips, afaik.

In my mind it would be nice to offer zim reading/serving functionality available as a package in freetz (configurable with a cgi frontend, as for similar packages) or other firmware modifying frameworks such as openwrt. Using zimHttpServer32 might be one option, porting kiwix-serve to mips another, or rewriting zimHttpServer32 to use lua instead of perl even yet another. Currently, and to my knowledge only the first option has "materialized". Related to this:

• Cross-compile kiwix-tools on MIPS kiwix/kiwix-build#48

[waynepiekarski]

I did this initial work over a year ago, and wrote some notes in my blog post http://www.tinmith.net/wayne/blog/2017/05/mediatek-perl-wikipedia.htm ... my MIPS device was the MTK7688, which also includes only 128mb of RAM. It might be that the software from a year ago was a build that used less memory, and/or using the SD card as swap space. I only use this device when I'm travelling on a plane with no WiFi, so I wasn't too worried about wear levelling. All the instructions of what I did are in the blog post above.

When I started this project, I was looking for a quick win, so picked the approach that required the least amount of work with the highest probability of success. I looked at porting kiwix-serve to OpenWRT for MIPS, but I've never worked with OpenWRT before and the build system required spending some time to learn. I saw the zimHttpServer perl script, it was only a few hundred lines of code, and I realized that perl was already available, so I could start running it straight away. I knew that I could solve the 64-bit seek issues, so I went for that. I suspect it could be possible to rewrite the whole perl script in another language and try to do the xz decompression internally in a way that doesn't require so much memory. Once you remove all the comments and documentation, the script is only a couple hundred lines of code and pretty easy to rewrite.

In the long run, implementing kiwix-serve for MIPS is probably the best strategy, since it implements everything the "correct" way, including indexing and all that. If there is someone who knows how to work with the OpenWRT build system it probably wouldn't be too difficult to port over.

[cm8]

Afaik the amount of memory used by the xz decompressor depends on parameters set when compressing data. Of course you could try to rewrite the unxz part - to put the dictionary not in RAM, but to flash or disk and then 'mmap' this to do the decompression work without the need for extra swap; but this would only re-implement swapping for a special case. I don't see a gain doing this if the kernel can do this the general purpose way.

It may be easier to repack the ZIM files with relaxed compression settings as to use less RAM when serving the files (if swap is not available or unreliable).

[kelson42]

@cm8 Sorry for cyber squatting this PR, but like I told @waynepiekarski, we would be really interested to add support for MIPS arch to kiwix-tools, see kiwix/kiwix-build#48. This is a bit of compilation work to adapt compilation of libzim/libkiwix/kiwix-tools but the result would be IMO really usefull for openWRT/Mips based devices.