Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geospatial component for heightmaps (2nd try) #292

Merged
merged 15 commits into from
Mar 15, 2022

Conversation

scpeters
Copy link
Member

🎉 New feature

Second attempt at adding geospatial component.

Summary

Our first attempt at the geospatial component did not have gdal installed in macOS or Windows CI, and we didn't notice because the geospatial component is an optional dependency. This had broken ign-rendering builds, so #267 and #289 were reverted in #291. These dependency issues have been resolved in osrf/homebrew-simulation#1790, gazebo-tooling/release-tools#626, and gazebo-tooling/release-tools#624, so let's try re-adding these pull requests.

This needs to be merged in sync with downstream pull requests to ign-rendering, ign-physics, and ign-gazebo.

Test it

UNIT_Dem_TEST is the primary test added in this pull request, but this should also be tested with downstream branches.

Checklist

  • Signed all commits for DCO
  • Added tests
  • Added example and/or tutorial
  • Updated documentation (as needed)
  • Updated migration guide (as needed)
  • codecheck passed (See contributing)
  • All tests passed (See test coverage)
  • While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

jennuine and others added 2 commits January 21, 2022 08:25
Signed-off-by: Jenn Nguyen <[email protected]>
Co-authored-by: Steve Peters <[email protected]>
Co-authored-by: Ian Chen <[email protected]>
This simplifies the logic required for packaging.
Update Migration guide

Signed-off-by: Steve Peters <[email protected]>
@github-actions github-actions bot added the 🌱 garden Ignition Garden label Jan 21, 2022
@scpeters scpeters changed the title Ci matching branch/retry geospatial Geospatial component for heightmaps (2nd try) Jan 21, 2022
@codecov
Copy link

codecov bot commented Jan 21, 2022

Codecov Report

Merging #292 (2547917) into main (b960986) will increase coverage by 0.37%.
The diff coverage is 89.36%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #292      +/-   ##
==========================================
+ Coverage   76.75%   77.12%   +0.37%     
==========================================
  Files          75       76       +1     
  Lines       10328    10529     +201     
==========================================
+ Hits         7927     8121     +194     
- Misses       2401     2408       +7     
Impacted Files Coverage Δ
...nclude/ignition/common/geospatial/HeightmapData.hh 33.33% <0.00%> (ø)
...clude/ignition/common/geospatial/ImageHeightmap.hh 88.88% <ø> (ø)
geospatial/src/ImageHeightmap.cc 60.00% <ø> (ø)
geospatial/src/Dem.cc 91.45% <91.30%> (ø)
src/SystemPaths.cc 86.40% <0.00%> (+0.43%) ⬆️
graphics/src/Image.cc 70.70% <0.00%> (+1.56%) ⬆️
src/Util.cc 86.69% <0.00%> (+2.46%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b960986...2547917. Read the comment docs.

@scpeters
Copy link
Member Author

there is a test failure with dem_nodata.dem on both homebrew and windows. I'm guessing it's related to the version of gdal or proj

from windows:

59: [ RUN      ] DemTest.UnfinishedDem
59: C:\Jenkins\workspace\ign_common-pr-win\ws\ign-common\geospatial\src\Dem_TEST.cc(184): error: Expected equality of these values:
59:   7499.8281
59:   demNoData.WorldHeight()
59:     Which is: 7499.8838
59: C:\Jenkins\workspace\ign_common-pr-win\ws\ign-common\geospatial\src\Dem_TEST.cc(185): error: Expected equality of these values:
59:   14150.225
59:   demNoData.WorldWidth()
59:     Which is: 14150.134
59: [  FAILED  ] DemTest.UnfinishedDem (185 ms)

from macOS

64: [ RUN      ] DemTest.UnfinishedDem
64: /Users/jenkins/workspace/ignition_common-ci-pr_any-homebrew-amd64/ign-common/geospatial/src/Dem_TEST.cc:184: Failure
64: Expected equality of these values:
64:   7499.8281
64:   demNoData.WorldHeight()
64:     Which is: 7499.8838
64: /Users/jenkins/workspace/ignition_common-ci-pr_any-homebrew-amd64/ign-common/geospatial/src/Dem_TEST.cc:185: Failure
64: Expected equality of these values:
64:   14150.225
64:   demNoData.WorldWidth()
64:     Which is: 14150.134
64: [  FAILED  ] DemTest.UnfinishedDem (38 ms)

knowing very little about DEM files, I tried running gdalinfo dem_nodata.dem, but I can't find my way to either of those values

$ gdalinfo dem_nodata.dem 
Driver: GTiff/GeoTIFF
Files: dem_nodata.dem
Size is 65, 65
Coordinate System is:
PROJCRS["NAD27 / UTM zone 10N",
    BASEGEOGCRS["NAD27",
        DATUM["North American Datum 1927",
            ELLIPSOID["Clarke 1866",6378206.4,294.978698213898,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4267]],
    CONVERSION["UTM zone 10N",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-123,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Engineering survey, topographic mapping."],
        AREA["North America - between 126°W and 120°W - onshore. Canada - British Columbia; Northwest Territories; Nunavut; Yukon. United States (USA) - California; Oregon; Washington."],
        BBOX[34.4,-126,77.13,-120]],
    ID["EPSG",26710]]
Data axis to CRS axis mapping: 1,2
Origin = (557805.000000000000000,5122005.000000000000000)
Pixel Size = (150.923076923076934,-216.000000000000000)
Metadata:
  AREA_OR_POINT=Point
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  557805.000, 5122005.000) (122d15' 0.36"W, 46d15' 4.05"N)
Lower Left  (  557805.000, 5107965.000) (122d15' 6.53"W, 46d 7'29.19"N)
Upper Right (  567615.000, 5122005.000) (122d 7'22.25"W, 46d15' 0.79"N)
Lower Right (  567615.000, 5107965.000) (122d 7'29.48"W, 46d 7'25.94"N)
Center      (  562710.000, 5114985.000) (122d11'14.66"W, 46d11'15.06"N)
Band 1 Block=65x63 Type=Int16, ColorInterp=Gray
  NoData Value=-32767
  Unit Type: m

the "Corner Coordinates" section seems promising, but computing differences doesn't give values that match up with the test expectations of approximately 7499.8 and 14150

>>> 567615.000 - 557805.000
9810.0
>>> 5122005.000 - 5107965.000
14040.0

@jennuine
Copy link
Contributor

Per VC, there may not be a reliable way to calculate world width/height because of projections and coordinate transformations. Instead will try to set world dimensions with projected measurements since we can get meters per pixel. This is possible with already projected DEMs (projected coordinate system), need to be sure it's possible when they are in a geographic coordinate system.

There's no reason for this to be non-const, and it causes
some issues in ign-physics, so make it non-const while
it is possible to change the API.

Signed-off-by: Steve Peters <[email protected]>
@scpeters scpeters mentioned this pull request Feb 10, 2022
7 tasks
@jennuine
Copy link
Contributor

jennuine commented Feb 11, 2022

A workaround to load non-Earth DEMs fa605bd and I've created an issue regarding computing the world dimensions #311. To visually test, you can run the example from gazebosim/gz-rendering#560

I believe this PR is ready for review

@scpeters
Copy link
Member Author

I believe this PR is ready for review

looks like there's a just a few windows compiler warnings to fix

@jennuine
Copy link
Contributor

Hmmm, do you know what changed windows ci? These warnings didn't appear when we merged #267

@scpeters
Copy link
Member Author

Hmmm, do you know what changed windows ci? These warnings didn't appear when we merged #267

I think windows CI wasn't fully working when we merged #267. I believe it landed in the following build:

CMake Warning at C:/Jenkins/workspace/ign_common-ci-win/ws/install/ignition-cmake2/share/cmake/ignition-cmake2/cmake2/IgnConfigureBuild.cmake:55 (message):
   CONFIGURATION WARNINGS:
   -- Skipping component [geospatial]: Missing dependency [GDAL].
      ^~~~~ Set SKIP_geospatial=true in cmake to suppress this warning.

Signed-off-by: Jenn Nguyen <[email protected]>
@jennuine jennuine mentioned this pull request Feb 12, 2022
8 tasks
@scpeters
Copy link
Member Author

@osrf-jenkins run tests please

Copy link
Contributor

@chapulina chapulina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The examples work for me 👍 I noticed that a lot of errors and warnings are printed for the Moon's DEM, even though it seems to have been loaded correctly:

ERROR 1: PROJ: proj_create_operations: Source and target ellipsoid do not belong to the same celestial body
ERROR 6: Cannot find coordinate operations from `PROJCRS["Moon2000_npole",BASEGEOGCRS["GCS_Moon",DATUM["Moon_2000",ELLIPSOID["Moon_2000_IAU_IAG",1737400,0,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Reference_Meridian",0,ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]],CONVERSION["unnamed",METHOD["Polar Stereographic (variant B)",ID["EPSG",9829]],PARAMETER["Latitude of standard parallel",90,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8832]],PARAMETER["Longitude of origin",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8833]],PARAMETER["False easting",0,LENGTHUNIT["metre",1],ID["EPSG",8806]],PARAMETER["False northing",0,LENGTHUNIT["metre",1],ID["EPSG",8807]]],CS[Cartesian,2],AXIS["easting",south,ORDER[1],LENGTHUNIT["metre",1]],AXIS["northing",south,ORDER[2],LENGTHUNIT["metre",1]]]' to `EPSG:4326'
[Err] [Dem.cc:280] Unable to transform terrain coordinate system to WGS84 for coordinates (0,0)
[Wrn] [Dem.cc:154] Failed to automatically compute DEM size. Assuming non-Earth DEM. 

If these can be ignored, I suggest making them all warnings instead of having an error there.


My suggestions below are specific to the DEM I'm trying to load (still trying, haven't succeeded yet), but I imagine other users may run into the same issues.

double max = -ignition::math::MAX_D;
for (auto d : this->dataPtr->demData)
{
if (d > noDataValue)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing this PR with a DEM that uses NaN as the no data value, I noticed that this logic doesn't handle NaNs well, because any comparison to NaN will always return false.

How about something like this:

  for (auto d : this->dataPtr->demData)
  {
    // All comparisons to NaN return false, so guard against NaN NoData
    if (!std::isnan(noDataValue) && d <= noDataValue)
      continue;

    if (!std::isfinite(d))
      continue;

    if (d < min)
      min = d;
    if (d > max)
      max = d;
  }

{
OGRSpatialReference sourceCs;
OGRSpatialReference targetCs;
OGRCoordinateTransformation *cT;
double xGeoDeg, yGeoDeg;

// Transform the terrain's coordinate system to WGS84
char *importString = strdup(this->dataPtr->dataSet->ProjectionRef());
const char *importString
= strdup(this->dataPtr->dataSet->GetProjectionRef());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm testing this with a DEM that doesn't have a projection ref. I think we could catch this here instead of passing an empty string to importFromWkt. Something like this:

    const char *importString
        = strdup(this->dataPtr->dataSet->GetProjectionRef());
    if (importString == nullptr || importString[0] == '\0')
    {
      ignwarn << "Projection coordinate system undefined." << std::endl;
      return false;
    }

@j-rivero
Copy link
Contributor

j-rivero commented Mar 9, 2022

it looks like the windows build fails on one jenkins node and passes on the other:

* https://build.osrfoundation.org/job/ign_common-pr-win/202/ passes on https://build.osrfoundation.org/computer/win-windows_nuc.win10

* https://build.osrfoundation.org/job/ign_common-pr-win/203/ and https://build.osrfoundation.org/job/ign_common-pr-win/204/ fail on https://build.osrfoundation.org/computer/win-beefy.win10

cc @j-rivero

I've sent gazebo-tooling/release-tools#657. Bad news is that our vcpkg snapshot is too old to work (mirrors already removed the required versions from our snapshot). I don't think that the problem is a blocker to merge this PR.

Copy link
Contributor

@chapulina chapulina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm approving this, and I can open a separate PR with my suggestions.

@chapulina chapulina marked this pull request as ready for review March 15, 2022 00:36
@chapulina chapulina merged commit af583c9 into main Mar 15, 2022
@chapulina chapulina deleted the ci_matching_branch/retry_geospatial branch March 15, 2022 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🌱 garden Ignition Garden
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants