Calculate geoid offset within app for altitude MSL

[gdt]

Currently, GPSTest gets ortho height and geoid offset from NMEA. However these are usually inaccurate. (Actually, I don't think we have a report of it being done right.) That results in display HAE and ortho height, where the ortho height is off. And, the user doesn't notice that the device is off.

I'd like GPSTest to compute EGM2008 reasonably well and to display that, and compute ortho height from HAE and display that. It should also compare a decent EGM2008 approximation to the device's value and somehow show the offset.

Perhaps this should be in a height popup screen, and the AMSL height should be red if there is a discrepancy bigger than typical, (>=5m?) and green if <= 1m, with that bringing up the details.

[barbeau]

Luckily I started down this road in #22 when first starting to add ortho height to the app, and I implemented a port of the Geotools EarthGravitationalModel here that should work on Android:
https://github.com/barbeau/earth-gravitational-model

I ended up not using it and relying on NMEA instead because it was a much lighter-weight solution. But as you mention we now have clear issues with the ortho height and offsets we're getting from NMEA streams (see #503).

The main issue with using the EGM library is that we have to bundle the geoid model within the app, and I still need to figure out what the size implications on the APK are.

Here's the model data that was bundled with GeoTools:
https://github.com/barbeau/earth-gravitational-model/blob/master/src/main/resources/egm180.nor

From some poking around it seems that these are the coefficients for the EGM84 model, looking at https://en.wikipedia.org/wiki/Earth_Gravitational_Model and https://github.com/barbeau/earth-gravitational-model/blob/master/src/test/java/com/barbeaudev/geotools/referencing/operation/transform/test/EarthGravitationalModelTest.java#L35, which has passing test results that match the EGM84 output of https://geographiclib.sourceforge.io/cgi-bin/GeoidEval?input=28.0428021%2C-82.25598206&option=Submit.

@gdt Any idea where I can find similar files for EGM96 or EGM2008?

Reading the above Wikipedia page it sounds like the more recent models have more precision, which means more coefficients, which means a larger file to bundle with the app and an increased APK size.

[gdt]

I am aware of this: https://github.com/OpenTracksApp/OpenTracks

As for EGM84/96/2008, my take is that there are two orthogonal issues. One is the newer models are better, and for the current realization of WGS84, EGM2008 is right by definition. So definitely you should use EGM2008. Also, newer models are available with higher degree harmonics, but using a truncated to 1 degree newer model seems better than using an older model that is already one degree, if you decide that 1 degree is the right accuracy/space tradeoff. EGM84 in particular seems like it should not be used, especially as it is based on a datum with an origin that isn't the center of mass of the earth.

[gdt]

OpenTracks uses geographiclib data sets: https://geographiclib.sourceforge.io/html/geoid.html
It seems pretty clear that they have the 5' dataset at about 19 MB. That seems reasonable for an app intended to test GPS that is aimed at nerds.

[barbeau]

Thanks for the info! I was hoping to find a file for EGM96/2008 that's in the same format as the egm180.nor file, because that's what's compatible with earth-gravitational-model code, but so far I'm not seeing that. I suppose if we can find something that's compatible with GeoTools that should work (assuming GeoTools still uses the same format).

But, looking at OpenTracks (thanks for the pointer!), I do see an EGM2008 implementation there:

• https://github.com/OpenTracksApp/OpenTracks/blob/2843d1945099790d9022bf5d672800b9606a6ab4/src/main/java/de/dennisguse/opentracks/util/EGM2008Utils.java
• https://github.com/OpenTracksApp/OpenTracks/blob/2843d1945099790d9022bf5d672800b9606a6ab4/src/androidTest/java/de/dennisguse/opentracks/util/EGM2008UtilsTest.java

Looks like they bundle the geographiclib EGM2008 model:

• https://github.com/OpenTracksApp/OpenTracks/blob/2843d1945099790d9022bf5d672800b9606a6ab4/src/main/res/raw/egm2008_5.pgm

That file is 17.8MB, and compressed it looks like it's around 16.3MB. In contrast, the egm180.nor file for EGM84 is around 700KB uncompressed.

For reference, the current entire APK size for GPSTest is 3.3MB, so bundling the geographiclib EGM2008 model would definitely multiply the current size of GPSTest several times. This has some general performance implications for the app that I need to think through. The resulting total APK size of OpenTracks is 21MB.

An alternate design would be to avoid bundling the EGM2008 file with the app and look at downloading it after install on-demand.

[gdt]

I would be inclined to steal the opentracks code, assuming that's licensing ok (and maybe ask if not), and align with geographiclib.

I didn't really notice when opentracks got bigger (I was one of the instigators of them adding it). Yes, 20 MB is way bigger than 3 MB, but I am not sure that 20 MB is a problem.

That said, if the app offers to download it and stash it someplace, that seems fine. It would be nice if it were in one place in the sd card and all the apps could share it. This is a clue that it's missing from the OS.

Personally I'd rather blow 20 MB in storage than have an old model. I guess a question is if this would make it hard to run GPSTest on really old phones, and how many people actually want to do that anyway.

[vkbellis]

An alternate design would be to avoid bundling the EGM2008 file with the app and look at downloading it after install on-demand.

Good idea.

Another idea, and building from this idea of post install downloads, might be to regionalize and approach downloading local regions of geoid models, and not limit the model choice to just EGM2008.

Smaller regions would afford better resolution;
for example, the grid spacing for xGeoid20B is 1.66666666666666664E-002degree

Here's what that looks like (in red) compared with 5 minutes of arc (in yellow):

Tackling this issue will be fundamental in all precise location app development, not just GPSTest

That said, if the app offers to download it and stash it someplace, that seems fine. It would be nice if it were in one place in the sd card and all the apps could share it.

Another good idea. Google engineers' participation could also help early adoption of a best practice in achieving this common resource concept.

[gdt]

An alternate design would be to avoid bundling the EGM2008 file with the app and look at downloading it after install on-demand.

Good idea.

Another idea, and building from this idea of post install downloads, might be to regionalize and approach downloading local regions of geoid models, and not limit the model choice to just EGM2008.

I don't think this belongs in GPSTest. It's basically heading down the path of porting proj to Android. While that's worthy, I think it should be a project that is an Android proj port. 20 MB is just not that much data, and if somebody cares, it's not a big deal, and if they don't, they can skip. Let's not add tons of complexity for a theoretical someone who is running an Android emulator with only 4 MB of RAM on a VAX, at least until we have someone explain what their issue is. (In proj, "all the data" is over a GB now, and growing, so getting what you need and regions does make sense.)

Smaller regions would afford better resolution;
for example, the grid spacing for xGeoid20B is 1.66666666666666664E-002degree

For what GPSTest is doing, xGEOID20 is a red herring. The right answer for geoid model for navigation-type GPS receivers is EGM2008 and modulo minor updates that is going to continue to be true. So if the point is "does my phone work", that's the only thing that matters when comparing.

Tackling this issue will be fundamental in all precise location app development, not just GPSTest

The real issue is trying to use an API that says WGS84 for data in other datums. For example I use Vespucci with a direct-TCP-to-receiver with RTK in "NAD83(2011) epoch 2010.0". My setup is completely avoiding the Android API, and Vespucci doesn't have datum tagging, so I apply that to the files as I import them into GIS/OSM/whatever. Which is really pointing out that proj4Android or whatever is needed to make this easy, but that's out of scope of GPSTest.

[vkbellis]

Land surveyors don't regard improved gravity models as irrelevant or as a red herring, though I appreciate hobbyists and the like might, particularly if they don't know any better.

[gdt]

Now you are just being rude. I didn't say that the US national datums were unimportant for US users and surveying. But it is all outside the scope of GPSTest, which is about an interface which is defined in the API to be centered on WGS84, which is a different family of datums and geoid models defined for a different purpose by a different agency.

[barbeau]

Tiling the geoid model is definitely a good idea (although not trivial to implement), but I agree that geoid height really should be exposed directly as a reliable, tested, first-class citizen in the Android Locations APIs (e.g., as Location.getGeoidHeight(lat, lon)), especially if this model already exists somewhere on the device.

I added another comment to the open AOSP issue (regarding #503) with the numbers for OpenTracks and requesting the feature:
https://issuetracker.google.com/issues/191674805#comment4

[dennisguse]

Yes, OpenTracks bundles the 5deg EGM2008 from GeographicLib.

@barbeau if you need help with implementing a tiling approach model, let me know.

@gdt Nerds with style 😎

IMHO
Even if the EGM2008 makes it into AOSP, we still need some kind of fallback as old devices (everything pre-Android 12) would not receive such an update.
Or it get's bundled in to PlayServices, which is also not helpful...

[barbeau]

@dennisguse Thanks for joining the discussion, and commenting on the AOSP issue! Great job on OpenTracks too, BTW :).

Even if the EGM2008 makes it into AOSP, we still need some kind of fallback as old devices (everything pre-Android 12) would not receive such an update.

Would you be willing to move the EGM2008Utils to it's own library so it could be used by other apps?

That was the original intent behind https://github.com/barbeau/earth-gravitational-model, but it's not clear to me that EGM96 and EGM2008 data is readily available in the same format as the EGM84 data that was bundled with GeoTools (?).

Some thoughts:

• Ideally a library would support both bundling and tiling the EGM models. Bundling is obviously simpler to implement (and already exists for EGM2008 thanks to your work) while tiling certainly requires some heavy lifting, as well as a server to host the data. GitHub repo may function as a reasonable open data source for hosting the tile chunks (?).
• Should this support EGM96 as well? Or perhaps a version of EGM2008 with fewer data points? The accuracy and precision tradeoff vs model size isn't currently clear to me without more research. I know as an app developer adding another 16MB to my APK size is tough to swallow for a feature that most users won't notice. 2MB for the less precise EGM96 model is easier to swallow. Overall IMHO we should let developers made that tradeoff decision based on required accuracy/precision for their use case.
• Having this as a separate library also helps with licensing compatibility

FYI, I have an open issue for this on android-maps-utils also (for which I'm a maintainer):
googlemaps/android-maps-utils#704

If anything is added to that library, though, it would need to be relatively small in size, with developers making the decision to bundle their own larger model if desired.

[gdt]

As a first step, how about having the code in a library where the calls just error at first (no data), and then if you manually invoke "get the file" then you get the 18MB, and after that the calls work. That way most don't pay the pain of 18 MB, and those who want a geoid model get a decent one, and the people that want to have only 1 MB for their region are out of luck, but I'm not sure how many people there are that would be truly bothered by the extra. One could add in tiling later.

I'll see if there is an easy way to assess accuracy with smaller files.

I am -1 on doing anything with EGM96 as it is formally obsolete.

[dennisguse]

No problem moving the implementation from OpenTracks into a library.
Adding additional resolutions is not that big an issue (in terms of lookup) and the algorithm works on all data sources (incl. EGM96).
One issue is the downrendering of the data (splitting it is not that problematic).
We may get some help from Charles (maintainer from geographiclib) regarding the data transformation.
There should some scripts somewhere.

PS: I have very scarce knowledge in this whole domain (literally: I understood why latitude is only -90/+90 back in March).
I am just good at figuring out algorithms.

PPS: For maintenance reasons, OpenTracks does not use any external dependencies (only AndroidX and test libs).
So, I would literally copy the algorithm out (no problem doing that) and merging it into android-maps-utils sounds like a great plan!

[vkbellis]

@barbeau and @dennisguse
Possibly worth your reviewing:
https://www.eye4software.com/hydromagic/documentation/manual/utilities/geoid-file-conversion/
This free utility is used in
https://www.ardusimple.com/custom-geoid-for-orthometric-height-measurements-in-android/

Also, FWIW, I just did a quick test using this:
https://play.google.com/store/apps/details?id=cz.mjakl.trueheight&hl=en_US&gl=US
I've contacted the developer of CorrectAltitude, Michal Jakl asking if it is possible to use the app in static mode and await his reply.
As far as I could tell, CorrectAltitude requires the user to be in motion for averaging to work, but the EGM2008 values that it is showing look correct.

[barbeau]

FWIW, looks like the APK size of CorrectAltitude is 47MB. They apparently bundle the EGM 2008 NGA data.

@vkbellis If you come across more of these apps using geoid offsets could you add a comment with a link in this issue? It would probably help convince Google to make this feature a first-class citizen in the Android APIs if we can show demand. Especially if those apps bundle their own models, as Google is very interested in helping developers shrink APK sizes.

[vkbellis]

@barbeau - roger, will comply.

One agnostic geoid accounting Android app which towers above any other I've run across thus far is Mapit Spatial https://spatial.mapitgis.com/ It seems to be of good depth, though I've not had time today to properly delve into. I did pay the $15.81/year for the auto-renewing Mapit Spatial - Licence (sic) Subscription (Mapit Spatial - GIS Data Collector & Measurements) in order to allow testing.

I've reached out to the developer, Andrzej Bieniek (in the UK) alerting him of some bum links in the app; e.g., the listed URL for NGA's EGM2008, and asked if he had a known working URL for the EGM2008 data (have yet to hear back).

Initially I thought that I might have to manually convert the NGA grid file into the accepted binary format, and wait to hear back from osedok, but that wasn't necessary. The offering at https://earth-info.nga.mil/ may not be as explicitly named as the app calls for, but the downloaded archive from https://earth-info.nga.mil/ contains precisely the correct file and in the required format.

The default (embedded) geoid model that comes with the app is EGM96 1°x1° and its displayed orthometric height (H) values look reasonable.

More tests to be done using EGM2008, which installed without issue.

One really nice touch: display of the applicable EPSG code.

Related readings
https://mapitgis.com/geoid-height-extension/
https://spatial.mapitgis.com/user-guide

[barbeau]

FYI, for anyone (@dennisguse) who wants to see a reliable first-class Android API to get the H and N values directly from the platform (vs parsing NMEA sentences or bundling your own geoid model), please star my feature request to Google here:
https://issuetracker.google.com/issues/195660815

[gdt]

I wonder if you download the EPSG dataset and look at the SQL if you will find a URL. Also check the latest roj and proj-data.

NGA has reorganized their website.

• https://earth-info.nga.mil/index.php?dir=wgs84&action=wgs84
  That has a zip to download that when unpacked has the EGM2008 data in text. It has the code 2190 in the file name, which seems close to but does not match 2139.

In general I have high confidence that anytthing from GeographicLib is correct, because of the author's reputation for care and accuracy.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Please comment here if it is still valid so that we can reprioritize. Thank you!

[gdt]

hey bot still valid!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Please comment here if it is still valid so that we can reprioritize. Thank you!

[dennisguse]

There is a new API for Android 14: https://issuetracker.google.com/issues/195660815

[danieldjewell]

I'm certainly not an expert on the various correction models and such (other than a very very high level understanding of the purpose) .... But I did just come across this open-source app: https://github.com/BasicAirData/GPSLogger that has built-in support for EGM96 corrections ...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Please comment here if it is still valid so that we can reprioritize. Thank you!

[barbeau]

Not stale

[GrazianoCapelli]

As information I can confirm that BasicAirData GPS Logger implements a built-in altitude correction using the EGM96 grid.
The app is GPL3 licensed, you can easily read and use the Java code.

The EGM96 grid is downloaded from http://download.osgeo.org/proj/vdatum/egm96_15/outdated/WW15MGH.DAC
We chosen to not embed the grid to avoid regulatory problems.
The correction value is calculated using the bilinear interpolation (no spline interpolation at this time).

The class that manages the EGM Correction is EGM96.java, it is very simple to copy and use in another project.
The code that downloads the grid is in FragmentSettings.java.

PS: In the past we found your code VERY useful to implement the satellite counting using the new GnssStatus on dual freq. receivers, so thanks a lot!

[gdt]

@GrazianoCapelli Can you explain "regulatory problems"? I would expect EGM96 grids come from NGA, and are thus in the public domain. Do you mean that there are jurisdictions which prohibit distributing geoid models? Something else?

[dennisguse]

FYI OpenTracks is just switching to the Androidx AltitudeConverterCompat.

OpenTracksApp/OpenTracks@2e7a3b7

[GrazianoCapelli]

Can you explain "regulatory problems"? I would expect EGM96 grids come from NGA, and are thus in the public domain. Do you mean that there are jurisdictions which prohibit distributing geoid models? Something else?

@gdt I mean that the documentation on the NGA website doesn't explicitly mention the possibility to embed the EGM Grid into an app, thus we prefer to let the user download it at runtime.

[gdt]

In the United States, anything created by the US Government (which includes NGA) is in the public domain, meaning not subject to copyright. So there is no issue at all including it. This public domain notion runs so deep that often data/code/other-content on government web sites doesn't explicitly address copying. From a science courtesy viewpoint, just calling it EGM96 is good enough because ~everyone knows that was defined/published by NGA.