Structured 3D dataset - corruptions in zip files

[ok1zjf]

Many thanks for making this dataset available!
I have downloaded the Structured3D_panorama_00.zip ... Structured3D_panorama_17.zip from the Azure cloud and found that all except 14,15,16,17 have some files corrupted inside the zip archives. I have downloaded the archives on multiple university and my private internet connections and used different applications to unzip the files (unzip, 7z in Ubuntu 22.04 and with Windows 10 archive manager). In all cases, the same errors were encountered. For example, unzipping the Structured3D_panorama_13.zip results in the following errors:

unzip -q Structured3D_panorama_13.zip -d /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/
error: invalid compressed data to inflate /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png bad CRC 406b2c0b (should be 53007390)
file #1444: bad zipfile offset (local header sig): 536390646
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png bad CRC fbc88119 (should be 35d6a845)
file #1486: bad zipfile offset (local header sig): 553273189
file #1487: bad zipfile offset (local header sig): 553279282
file #1488: bad zipfile offset (local header sig): 553279355
file #1489: bad zipfile offset (local header sig): 553279437
file #1490: bad zipfile offset (local header sig): 553279568
file #1491: bad zipfile offset (local header sig): 553279656
file #1492: bad zipfile offset (local header sig): 553519416
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png bad CRC c4090c4d (should be 02b78ee6)
file #1579: bad zipfile offset (local header sig): 586400993

and with 7z

7z x -o/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/ Structured3D_panorama_13.zip

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (306F2),ASM,AES-NI)

Scanning the drive for archives:
1 file, 12381936537 bytes (12 GiB)

Extracting archive: Structured3D_panorama_13.zip

--
Path = Structured3D_panorama_13.zip
Type = zip
Physical Size = 12381936537
64-bit = +

ERROR: Data Error : Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png
ERROR: CRC Failed : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png
ERROR: Headers Error : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_rawlight.png
ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/semantic.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/camera_xyz.txt
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/albedo.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/depth.png
ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_rawlight.png

Sub items Errors: 13
Archives with Errors: 1
Sub items Errors: 13

Could you please verify that these files are fine in the Azure cloud? If yes, then perhaps the files are corrupted only on the Azure mirror in the UK, which would be a very serious fault in Azure data management.

Kind Regards
Jiri

[bertjiazheng]

I had a quick check and found the files on Azure Cloud had already been corrupted. But it seems that only a few images have been affected. For now, you may ignore the corrupted data.

[ok1zjf]

Thank you for confirming the issue. For what it is worth, here is a list of the corrupted scenes in the Structured3D_panorama. In each scene, one or multiple files are corrupted.

00
scene_00010
scene_00013
scene_00014

01
scene_00206
scene_00211
scene_00212
scene_00213

02
scene_00403
scene_00408
scene_00409
scene_00410
scene_00411
scene_00412

03
scene_00609
scene_00612
scene_00613

04
scene_00806
scene_00809
scene_00810

05
scene_01009
scene_01010

06
scene_01209
scene_01210

07
scene_01410
scene_01411
scene_01412

08
scene_01605
scene_01607
scene_01608

09
scene_01813
scene_01814
scene_01815

11
scene_02211
scene_02212
scene_02213

12
scene_02409
scene_02411

13
scene_02601
scene_02608
scene_02609

[fengenxiang1123]

Yes, I also encountered this problem.