Review RGB fullfield extractor and output

[max-zilla]

Review issue for the fullfield extractor and output using stereoTop RGB inputs.

Completion criteria:

• Code review complete (includes extractor, any functions in terrautils, etc)
• Sample data output verified
• Sample metadata verified
• Changes for 2017 release implemented
• Issue(s) for changes postponed until 2018 created
• Extractor tagged/versioned
• Extractor deployed
• Final output verified/reviewed

[max-zilla]

LINKS

fullfield output dataset example
https://terraref.ncsa.illinois.edu/clowder/datasets/59b0c56f4f0ca12ea0ce160e

an output dataset includes thumbnail- and full-resolution GeoTIFFs.

outputs on ROGER are in /ua-mac/Level_1/fullfield.

[dlebauer]

This is how the full field product appears:

This issue with sun / shade is discussed in terraref/computing-pipeline#326.

[max-zilla]

@ZongyangLi I intended to run this with the darker pixel correction flag, but it doesn't look like it was actually applied, let me check and can generate corrected version after our meeting.

[max-zilla]

@craig-willis and I looked at the full field stitch after correction, and it looks the same as before to us. can you take a look @ZongyangLi ?

Output VRT, TIFFs and folders with subtiles are in /ua-mac/Level_1/fullfield/2017-07-27/

_rgb and _rgb_thumb include the darker pixel selection with split=2

_rgb_nodark and _rgb_thumb_nodark are original version without darker pixel selection.

[craig-willis]

I'm working on a way to view the large TIFF via QGIS in workbench without transferring the data. Will hopefully have a solution later today.

[max-zilla]

The darker pixel selection also results in a bad _thumb file - at first glance it looked like a single tile is being stretched across the whole field or something. I had noticed this before in tests but hoped it was just an artifact of my small test image set - the full-resolution image did not have a problem in either case.

[ZongyangLi]

@max-zilla @craig-willis

I review some output in the folder. I have some little concern after a very rough checking.

1. the base tiles created in the '/ua-mac/Level_1/fullfield/2017-07-27/unite/tiles_left/28/' folder correctly, but I didn't see other higher level tiles in other folders, there are void images in those folder.
2. after darker choice process, some tiles looks wired like the one I'm gonna show below, this is why we do not recommend to input this into a level 1 data.
   

[max-zilla]

@ZongyangLi that simplifies things if we skip for this pass, lets us focus on just the stitching (plus the darker pixel with split=2 took 44 hours).

we should look at some of the bin2tif individual images and compare their gantry position to our georeferenced position to make sure we have them in the right place - the sun patterns are kind of random in the stitched sample image above.

[ZongyangLi]

@max-zilla I downloaded first 114 raw data from /ua-mac/raw_data/stereoTop/2017-07-27/ and did the testing on my laptop. Here are some points I would like to discuss with you and @dlebauer .

1. I noticed that in 2017-07-27, stereoTop scan were in 'south-north' directory, so after downloading 114 directory of files, I could still not able to visualize two lines scan. could you please tell me more about what's it look like about 'a single tile is being stretched across the whole field or something', is the gap is in horizontal or vertical？

2. In my testing, a stitched map without sun removal looks like this:

and with a sun removal:

So when I was talking about the 'void images', I was worried about the full field map we were created on clowder is not a true 'sun removal version', they are more like a map create by folder 0 or folder 1, not from folder 'unite'

3. From my testing result, it seems to me that leaves were almost well stitched from the bottom image to the top on the edge, my output is here, please have a check.

4. @max-zilla Could you please let me know where may I have a look at the codes you are using? Or maybe we could initial a short chat to debug the codes together.

Thanks!

[max-zilla]

@ZongyangLi how do arrive at the stitched image above? looking through the code we're using, it looks like we have the united tiles directory at /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/unite/tiles_left/ with zoom levels 18-28. but when i dig into code, it isn't altering the VRT at all, so the geoTIFF is identical to before.

in your shadeRemoval script:

def main():
    src_bin_dir = '/Users/Desktop/pythonTest/stitch_map/2017-05-27'
    in_dir = '/Users/Desktop/pythonTest/rogerFS/2017-05-27/'
    out_dir = '/Users/Desktop/pythonTest/shadeRemoval/2017-05-27'
    
    create_tif_list(src_bin_dir, in_dir)
    
    darker_tiles_generator(in_dir, out_dir)

def darker_tiles_generator(in_dir, out_dir):
    
    split_num = 2
    
    tif_list = os.path.join(in_dir, 'tif_list.txt')
    if not os.path.exists(tif_list):
        return
    
    split_tif_list(tif_list, out_dir, split_num)
    
    create_diff_tiles_set(out_dir, split_num)
    
    unite_tiles_dir = os.path.join(out_dir, 'unite')
    integrate_tiles(out_dir, unite_tiles_dir, split_num)
    
    copy_missing_tiles(out_dir, unite_tiles_dir, split_num, tiles_folder_name='tiles_left')
    
    src_vrt_path = os.path.join(in_dir, 'virtualTif.vrt')
    create_unite_tiles(unite_tiles_dir, src_vrt_path)
    
    return

darker_tiles_generator() creates the "united" folder of tiles, but those aren't converted to GeoTIFF. I wonder if you are viewing the directory using a tile viewer or TMS so that it is using those files without creating a VRT?

does that make sense? I think we need a way to go from unite/tiles_left/28 (128 folders in my test output which contain 1630 JPGs each) to the GeoTIFF again.

is it possible to convert directory of tiles back to geoTIFF?

e: sounds tricky here: https://gis.stackexchange.com/questions/4798/mosaic-the-output-of-gdal2tiles

https://lists.osgeo.org/pipermail/gdal-dev/2011-January/027219.html

I tried doing something like:

find /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/unite/tiles_left/28 -name "*.png" > /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/png_list.txt 

gdalbuildvrt -input_file_list /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/png_list.txt /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/united_tiles.vrt

gdal_translate -projwin -111.9750963 33.0764953 -111.9747967 33.074485715 -outsize 10% 10% united_tiles.vrt united_tiles_thumb.tif

...but the PNG tiles have no georeference so gdalbuildvrt fails:

Warning 6: gdalbuildvrt does not support ungeoreferenced image. Skipping /home/clowder/sites/ua-mac/Level_1/fullfield/2017-07-27/unite/tiles_left/28/50723191/160376815.png

[ZongyangLi]

@max-zilla

You are right, we didn't create a new VRT at all. When @craig-willis or someone others is going to create full field image using my map tiles, I don't really know how to do this. Things in darker_tiles_generator() is only creating tiles base new images.

The way I was viewing the full field map is using google map API, the google map API will integrate all the tiles into a webpage that showing all the field map. So void images in zoom level 18-27 matters to create different zoom level's visualization.

In the function of generate_googlemaps(), the key code is:
getTileUrl: function(coord, zoom) { var bound = Math.pow(2, zoom); var y = bound-coord.y-1; return '%s' +'/' + zoom + '/' + coord.x + '/' + y + '.jpg'; },
That's the way google map api find the exactly tile. There must be some way of converting individual tiles into full field map, like the way google map doing.

[max-zilla]

We ran a new day from earlier in the season, without the darker pixel stuff:
https://terraref.ncsa.illinois.edu/clowder/datasets/59ba90ec4f0c0b27bc3cf7db

Ignore the thumbnails, they are old. The new result looks pretty great:

There's a 10% and full resolution available. When the plants are small and sun isn't glaring, it is a nice product.

@jterstriep you can see the gaps we were discussing via email. this is a screenshot of the 10% resolution raster.

[craig-willis]

@dlebauer @ZongyangLi @smarshall-bmr
We're looking at two problems with the full-field RGB image and could use your input

@smarshall-bmr The checkerboard effect in the fullfield image above appears to be an artifact of the gantry scan direction. Below is a comparison of two consecutive images on the right side of the fullfield image (2017-05-29 10:02:30.287 and 10:02:43.995). It seems that the gantry isn't capturing that last image prior to changing direction. Can you help confirm?

@ZongyangLi
The second problem is apparent in the following images. We're seeing a fair amount of duplication in the stitched images -- the same feature displaying 2-3 times. In this example, you can see the same rock in the highlighted area:

We think this is an artifact of both the stereo camera itself (per @smarshall-bmr's terraref/computing-pipeline#355), but also a possible problem with the stereo camera bounding box calculation. Can you comment?

The images here are:

• ua-mac/Level_1/rgb_geotiff/2017-05-29/2017-05-29__10-02-30-287/
• ua-mac/Level_1/rgb_geotiff/2017-05-29/2017-05-29__10-02-43-995/
• ua-mac/Level_1/rgb_geotiff/2017-05-29/2017-05-29__10-02-46-999/

[max-zilla]

note that in the first screenshot Craig posted, the gantry was moving to the right on the bottom image, then turned around and captured the top image going left as first image of next row.

[ZongyangLi]

To my understanding, this problem is hard to avoid since the FOV of camera is a taper, especially when the camera is close to the ground. When you focus on the plant, it seems to be aligned, when you focus on the ground, it turns out the artifact. Remember I did a lot of assumption without any metadata confirmed, to try to fit as much as I can to stitch well.

This is far different from a picture captured from satellite or an aerial photo. In that case, the camera is far away from the target object, and you will easily ignore the small stitching boundaries. If I was wrong in this issue, anyone please feel free to point me out.

[craig-willis]

@ZongyangLi Thanks for the additional details. Are your decisions/assumptions documented someplace (including how you came to the slope estimation and predicted plant height)? I expect that we'll just want to make sure this information is clearly documented and available to users.

[pless]

With respect to the "duplication" problem, this is something that is unavoidable without a much more complex stitching algorithm that implicitly infers the 3D structure of the ground. The justification for not going for such a rich representation is that:
(a) for the plants, since they move, it would be impossible not to have artifacts at the edges of the image, and
(b) for the ground, I judged that small stitching errors were not worth the (substantial) additional effort to build the more complete model.

[dlebauer]

Ok we can accept the current output. I think we have the caveats documenTed and next steps include the orthomosaicing terraref/computing-pipeline#355 and the aggregation of image level metrics to plot level terraref/computing-pipeline#356

I think this is ready to be deployed.

[craig-willis]

Tracking re-processing in terraref/computing-pipeline#357

[dlebauer]

Quality statement in PR terraref/extractors-stereo-rgb#11

[solmazhajmohammadi]

@smarshall-bmr @max-zilla
due to the rail issue last summer(I think), we were running the scans in the middle of the field only from east to west. Do you recall when was that? we can check the stitched from that day to confirm where the issue is coming form

[dlebauer]

Need to be able to handle many different scans / scan types. This is covered in terraref/computing-pipeline#362

will update metadata script to propagate scan script forward to identify different scans.

[craig-willis]

Stale issue, closing. We will revisit RGB fullfield review for 2018.