WIP: Update dependencies to work on recent node versions #300

julienr · 2020-04-10T12:36:31Z

TODO

Figure out StatusImage breakage
Remove my facebook test credentials and make the oam staging ones work

Background

I was trying to get started hacking on oam-browser and ran into some setup issues with some of the dependencies, mostly around webdriverio. It seems like the versions that oam-browser is using are not working anymore with recent node version (I'm testing on 13).

So I figured out I need to update some packages and the webdriverio ones seems to be the most problematic ones (they depend on an old fibers version that doesn't build anymore).

Closes #295

Status

I'm almost there with the integration tests running locally against oam-api (I opened hotosm/oam-api#153 for a fix for oam-api).

The only tests that seem to fail are the one simulating a click on 'View image' after upload. But is this functionality supposed to work ? Because it seems the metadata returned by oam-api doesn't contain a uuid and oam-browser is looking for a uuid, in StatusImage.js:imageClick. I am testing against the develop branch of oam-api. Guidance on this would be appreciated.

StatusImage.js issue

If I print the metadata that the imageClick function is receiving, I'm seeing this:

{"acquisition_end":"2020-04-10T10:08:05.077Z","acquisition_start":"2020-04-09T22:00:00.000Z","contact":"Bob Mary,[email protected]","platform":"satellite","provider":"42","properties":{"license":"CC-BY 4.0","sensor":"42"},"title":"test"}

This corresponds that what's stored in the uploads table of the DB, and the frontend is indeed getting that from the /uploads/{id} route of oam-api.

Now, what this function seems to expect is the actual image metadata as stored on S3 (in the XXX_meta.json file or in the metas DB table.

So I feel there is an issue there that the frontend is not calling the right oam-api route to get this metadata.

Question

Is modernizing the oam-browser packages something that is welcome ? If yes, I'll try to finish this
Are the automated regression tests supposed to pass on develop ? Looking at the commits, seems the test are broken since a while. I'd be willing to help getting them green again
If someone can help me understand better the StatusImage.js:imageClick issue, I'd be happy

…(13) This fixes a build issue due to [email protected] which doesn't seem to build anymore on recent node version. Webdriver is depending on fiber, so updating the webdriver packages fixes this. Since newer versions of webdriver have removed browser.clik in favor of $(selector).click(), this also updates the integration tests for this change. Closes hotosm#295

dakotabenjamin · 2020-04-22T18:38:51Z

Thanks so much for contributing @julienr ! As you can see there are some issues with the integration tests, which for me have been with the facebook and google logins. To address your points:

A PR to modernize the packages would be welcome for sure.
If you continue to run into issues with the integration tests, feel free to DM me on the HOTOSM Slack slack.hotosm.org. I think some changes in the FB and Google auth apis have changed. And if you think we can merge safely without the tests passing in the interim, I think that's ok for now.
I'll look into the StatusImage.js issue and get back to you.

julienr · 2020-04-26T12:05:50Z

Looking a bit more into this StatusImage issue, right after i upload an image, here is what I have in the oam-api mongodb:

> db.uploads.find().sort({"createdAt":-1})[0]
{
        "_id" : ObjectId("5ea5754cbaf14b002737be98"),
        "scenes" : [
                {
                        "contact" : null,
                        "title" : "everest",
                        "provider" : "42",
                        "platform" : "satellite",
                        "sensor" : "42",
                        "acquisition_start" : "2020-04-25T22:00:00.000Z",
                        "acquisition_end" : "2020-04-26T11:14:11.212Z",
                        "tms" : "https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true",
                        "license" : "CC-BY 4.0",
                        "tags" : "",
                        "urls" : [
                                "https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true"
                        ],
                        "images" : [
                                ObjectId("5ea5754cbaf14b002737be99")
                        ]
                }
        ],
        "user" : ObjectId("5e7f2001af7e5a00119e4fd6"),
        "createdAt" : ISODate("2020-04-26T11:49:32.518Z")
}

Now, finding the image associated with this upload (which is what's returned by the uploads/{id} endpoint, I get this

> db.images.findOne({ _id: ObjectId("5ea5754cbaf14b002737be99") })
{
        "_id" : ObjectId("5ea5754cbaf14b002737be99"),
        "url" : "https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true",
        "status" : "finished",
        "user_id" : ObjectId("5e7f2001af7e5a00119e4fd6"),
        "messages" : [
                "",
                "Downloading https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true...",
                "Transcoding 3 band(s)...",
                "Adding overviews...",
                "Creating cloud-optimized GeoTIFF...",
                "Generating thumbnail...",
                "Generating footprint...",
                "Uploading..."
        ],
        "metadata" : {
                "acquisition_end" : "2020-04-26T11:14:11.212Z",
                "acquisition_start" : "2020-04-25T22:00:00.000Z",
                "contact" : "OAM test user,[email protected]",
                "platform" : "satellite",
                "provider" : "42",
                "properties" : {
                        "license" : "CC-BY 4.0",
                        "sensor" : "42"
                },
                "title" : "everest"
        },
        "startedAt" : ISODate("2020-04-26T11:49:34.993Z"),
        "stoppedAt" : ISODate("2020-04-26T11:49:41.112Z")
}

Looking in the meta table, I think this is the corresponding metadata (although I'm unclear as to how one should find the metadata corresponding to an image. Here, the timestamps seem to match):

> db.metas.find().sort({"uploaded_at":-1})[0]                                                                                               
{                   
        "_id" : ObjectId("5ea57555baf14b002737be9b"),
        "acquisition_end" : ISODate("2020-04-26T11:14:11.212Z"),                                                                                                                                                                                                                                        "acquisition_start" : ISODate("2020-04-25T22:00:00Z"),                                                                                                                                                                                                                                          "contact" : "OAM test user,[email protected]",                                          
        "platform" : "satellite",                                                                                                               
        "provider" : "42",                                  
        "properties" : {
                "wmts" : "http://tiles.staging.openaerialmap.org/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99/wmts",
                "tms" : "http://tiles.staging.openaerialmap.org/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99/{z}/{x}/{y}",
                "thumbnail" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.png",
                "resolution" : [
                        350,
                        350
                ],
                "colorinterp" : [
                        "red",
                        "green",
                        "blue"
                ],
                "dtype" : "uint8",
                "filename" : "5ea5754cbaf14b002737be99.tif",
                "resolution_in_meters" : 350,
                "bands" : 3,
                "url" : "s3://oam-julien-oin-bucket/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.tif",
                "dimensions" : [
                        66,
                        66
                ],
                "crs" : "EPSG:32645",
                "sensor" : "42",
                "license" : "CC-BY 4.0"
        },
        "title" : "everest",
        "user" : ObjectId("5e7f2001af7e5a00119e4fd6"),
        "uuid" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.tif",
        "geojson" : ...,
        "footprint" : "POLYGON ((86.808984 27.884612, 87.044128 27.884612, 87.044128 28.093268, 86.808984 28.093268, 86.808984 27.884612))",
        "gsd" : 350,                                      
        "file_size" : 21100,                             
        "projection" : "PROJCS[\"WGS 84 / UTM zone 45N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTH
ORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHO
RITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32645\"]]",
        "meta_uri" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99_meta.json",
        "uploaded_at" : ISODate("2020-04-26T11:49:41.115Z"),
        "bbox" : [                       
                86.808984,                                                                                                 
                27.884612,                                                                                                       
                87.044128,                                                                                                                         
                28.093268        
        ],                  
        "__v" : 0          
}

I feel what's in the meta table is what's the frontend is expecting the /uploads/{id} endpoint to return.

Looking at the history of oam-api, it seems like this commit has changed the 'update image metadata' logic. Before this commit, IIUC, it was duplicating the metadata in image.metadata, but after this it's only copying some of the metadata:

hotosm/oam-api@84d3492#diff-647a467e336f16a2a502ee92eac324f2L286

I don't understand enough of the overall architecture to fully grasp the intent behind these changes, so it's very possible that I am missing something here. I could try assigning the whole meta object to image.metadata and see if it fixes the problem.

julienr · 2020-04-26T15:22:05Z

Indeed, copying the metadata in the image in oam-api seems to help with the StatusImage issue. I have a WIP fix on this branch: julienr/oam-api@818d43d (needs some oam-browser fixes as well: julienr@0eca44e)

I'm not quite sure this is the right fix though, because I guess this metadata will not be present for images in the current prod db, so this would break backward compatibility.

I feel getting these tests to pass again is going to require a number of changes. Do you know the last commits of oam-browser/oam-api for which these tests were passing ? Do you prefer that I try to fix everything at once in a big PR or we do it step-by-step in multiple small increments ?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Update dependencies to work on recent node versions #300

WIP: Update dependencies to work on recent node versions #300

julienr commented Apr 10, 2020 •

edited

Loading

dakotabenjamin commented Apr 22, 2020

julienr commented Apr 26, 2020

julienr commented Apr 26, 2020

WIP: Update dependencies to work on recent node versions #300

Are you sure you want to change the base?

WIP: Update dependencies to work on recent node versions #300

Conversation

julienr commented Apr 10, 2020 • edited Loading

TODO

Background

Status

StatusImage.js issue

Question

dakotabenjamin commented Apr 22, 2020

julienr commented Apr 26, 2020

julienr commented Apr 26, 2020

julienr commented Apr 10, 2020 •

edited

Loading