tarGzExpander corrupts expanded data when tar contains more than one file #4
Comments
|
hi @martirius can you share your code ? sounds like one of the two formats (gz or tar) is using an unsupported feature, it would also be interesting to know how you proceeded to produce the .tar.gz file, and the size of its content. [edit] Temporary workaroundUse gzExpander and tarExpander separately // direct .tar.gz to file expanding is broken when tar contains multiple files
if( SPIFFS.exists("/data.tar.gz") ) {
gzExpander(SPIFFS, "/data.tar.gz", SPIFFS, "/tmp/data.tar");
tarExpander(SPIFFS, "/tmp/data.tar", SPIFFS, "/your_webroot");
SPIFFS.remove("/tmp/data.tar"); // remove temporary tar
SPIFFS.remove("/data.tar.gz"); // only expand once
} |
|
This is the tar.gz, they are html files, a css and a public pem There is a webserver on ESP32 and pages are loaded from SPIFFS In setup() i do this:
Then in server request handler i do this :
Loading files from SPIFFS worked so far before trying to having all files in a tar.gz I don't know how it was produced the tar.gz file since i didn't do it |
|
I've just clicked the link, downloaded and opened the gz file in a text editor, and I can clearly read its contents so if it's gzipped, it doesn't seem to use a strong compression level, or it has been uncompressed on the fly. So I've taken the file and copied it manually on the SPIFFS partition, then ran the tarGzExpander sketch and got this comforting result: although the ending error message seems confusing, this totally checks out.
How did your .tar.gz file get onto the SPIFFS, was it downloaded directly from the ESP32 with a HTTP Client or was it uploaded using Arduino's ESP32 Sketch Data Uploader ? Speculation : because the gzip is produced by a PHP script, the returned compression format may be different (could be depending on the client's "Accept-encoding" HTTP headers values). So if dowloading from an ESP32 HTTP Client, the If the browser does this implicitely, maybe ESP32 HTTPClient doesn't ? |
|
Thanks for investigating but the php doesn't generate every time a new data.tar.gz, it's always the same that is being downloaded. What i noticed is that you decompressed the tar.gz in the /tmp folder; i didn't do that and decompressed it in root ("/"), maybe it's a problem decompressing in root? Can you try to print on Serial the content of those files, so we can see if they are corrupted or not? |
all archive managers use a temporary folder for unpacking, I guess this is to prevent unpacked content to overwrite the archive itself while it's being read. I've quick-coded a hex viewer and I have indeed file corruption symptoms.
This sounds like a stream buffer problem, currently investigating but I'm not confident with finding a quick solution without help. Meanwhile you can still use the tar and gz expanders separately as a workaround, this test snippet does not seem to produce corrupted contents : // direct .tar.gz to file expanding is broken when tar contains multiple files
if( SPIFFS.exists("/data.tar.gz") ) {
gzExpander(SPIFFS, "/data.tar.gz", SPIFFS, "/tmp/data.tar");
tarExpander(SPIFFS, "/tmp/data.tar", SPIFFS, "/your_webroot");
SPIFFS.remove("/tmp/data.tar"); // remove temporary tar
SPIFFS.remove("/data.tar.gz"); // only expand once
} |
|
Thanks a lot, i will try the temporary proposed solution If i can help to solve the issue, let me know |
|
I still can't wrap my head around this problem, I'm thinking of removing tarGzExpander from the equation. Of course I could just add a software limitation and emit a warning/error so I can call this bug a feature :-) So I'll rename and leave this issue open for the meantime. |
tobozo
changed the title
|
hey @martirius just a heads up on the code currently available on the noram branch.
Now both methods are available
With SPIFFS some path depth checks are made (and reported if logging isn't disabled) during decompression without halting the overall process. There's also a very memory-agressive mode where gz => tar => filesystem decompression can be achieved with only 516 bytes of ram but it's soooo slow 🦥 For the real deal (fast, silent) use this code: please let me know if this worked for you so I can close this issue and produce a new release happy end of year 🍾 |
|
fixed on master, I'll produce a new release soon, closing this as solved |
Hi @tobozo ,
I'm trying to untar a tar.gz file that includes 4 files.
After decompression i see a message on Serial that says : "[ERROR]: tar expanding done!" but followed by 100% and success.
So i tried to read from SPIFFS the files expanded but they are corrupted; this is what i found printing one of files (only a portion):
document.getElementById("cdl_status").innerHTML = response["cdl_status"] == 1 ? "CDL PAIRED" : "CDL UNPAIRED"; initialized = response["uuid"] != undefinedroot.pem000664 001750 001750 00000002260 13714534531 012510 0ustar00pedupedu000000 000000
tyle="margin-left: 50px;"></span> <span style="margin-left: 10px;">WEB</span> <span id="iot_led" style="margin-left: 50px;"></span> <spanHow did this part come into the file?
What i noticed also is that some part of files are mixed (i found parts of a file in another and viceversa)
The text was updated successfully, but these errors were encountered: