3D Modalities

Intro:

The dataset contains colored point clouds and textured meshes for each scanned area. 3D semantic annotations for objects and scenes are offered for both modalities, with point-level and face-level labels correspondingly. The annotations were initially performed on the point cloud and then projected onto the closest surface on the 3D mesh model. Faces in the 3D mesh that account for no projected points belong to non-annotated parts of the dataset and are labeled with a default null value. We also provide the tightest axis-aligned bounding box of each object instance and further voxelize it into a 6x6x6 grid with binary occupancy and point correspondence.

Technical details:

The dataset contains colored point clouds and textured meshes for each scanned area.

3D Point Cloud: The raw colored 3D point clouds along with both object and scene instance-level annotations per point, (tightest) axis-aligned bounding boxes and voxels with binary occupancy and point correpsondence are stored in the Area_#_PointCloud.mat file. The variables are stored in the form of nested structs:

- Area: --> name: the area name as: Area_#
		--> Disjoint-Space: struct with information on the individual spaces in the building.

In each Disjoint_Space struct:

- Disjoint_Space:	--> name: the name of that space, with per area global index (e.g. conferenceRoom_1, offie_13, etc.)
					-->AlignmentAngle: rotation angle around Z axis, to align spaces based on the CVPR2016 paper *3D Semantic Parsing of Large-Scale Indoor Spaces*. 
					--> color: a unique RGB color value [0,1] for that room, mainly for visualization purposes
					--> object: a struct that contains all objects in that space.

In each object struct:

- object:	--> name: the name of the object, wiith per space indexing* (e.g. chair-1, wall_3, clutter_13, etc.)
			--> points: the X,Y,Z coordinates of the 3D points that comprise this object
			--> RGB_color: the raw RGB color value [0,255] associated with each point.
			--> global_name: the name of the object, with per area global index**
			--> Bbox: [Xmin Xmax Ymin Ymax Zmin Zmax] of the object's boudning box
			--> Voxels: [Xmin Xmax Ymin Ymax Zmin Zmax] for each voxel in a 6x6x6 grid
			--> Voxel_Occupancy: binary occupancy per voxel (0: empty, 1: contains points)
			--> Points_per_Voxel: the object points that correspond to each voxel (in XYZ coordinates)

3D Semantic mesh:
The 3D semantic mesh is labeled with instance-level per-face annotations. The mesh is stored in semantic.obj and semantic.mtl where face labels are stored as face's material's name. In Blender, for example, the material label can be retrieved with:

mesh_material_idx = mesh.data.polygons[ face_idx ].material_index
material = mesh.data.materials[ mesh_material_idx ]
label = material.name   # final label!

The labels follow the convention class_instanceNum_roomType_roomNum_areaNum.

When using the mesh's color, the material's color should be remapped for the task at hand as the default color is designed for visualizations. One way to encode the label in the color is to set

material.diffuse_color = get_color( labels.index( material.name ) ) 

by using labels from /assets/semantic_labels.json and get_color( color ) from /assets/utils.py.