Add skyshade to raster.utah.gov from Jake's hard drive

[jacobdadams]

Summary

• Proposed Readable Name: Utah Skyshade
• Proposed Internal Dataset Name: skyshade
• Proposed External Dataset Name: Skyshade
• Proposed Data Category: Elevation
• Data Description: A Kennelly and Stewart skyshade of the entire state and surrounding areas generated from the USGS' 10m DEM
• Source Agency/Entity: UGRC
• Contact: Jacob Adams
• Update Frequency: As needed

Several audience members at the 2024 UGIC Iron Cartographer sessions wanted access to our skyshade. This is the same dataset used in Zach's basemaps but converted to 8-bit int and chunked into the same chunks as the source USGS DEMs.

I figured a porter issue is the best way to start the discussion of getting it onto raster.utah.gov.

The data should be available in

1 Check [x] all the areas where you expect the data to show up.

• Internal SGID
• Open SGID
• ArcGIS Online
• Open Data
• raster.utah.gov

The data is of high quality

• [ ] Sweeper checks have run and passed (name on 2024/00/00)
• [ ] The minimum requirements for metadata are populated (name on 2024/00/00)
• [ ] The data complies with our domain rules (name on 2024/00/00)

Where is the data source

Choose one.

• Internal SGID
• Farm from AGOL (agol item url)
• Zipfile from Jake

Action items

1. Assign a person who should complete the task by replacing name with their github @name.
2. Check [x] the box when the task is completed and add the date of completion.
3. Strike out all items that do not apply.

• [ ] Add data to the Internal SGID (name, completed: 2024/00/00)
• [ ] Configure forklift for Farm from AGOL (name, completed: 2024/00/00)
• Upload zip files to GCS (name, completed: 2024/00/00)
• Add index feature class to raster.utah.gov sources (@rkelson, completed 2024/00/00)
• Make any necessary changes to raster.utah.gov(?) (@stdavis, completed 2024/00/00)
• Update relevant gis.utah.gov data pages (@jacobdadams, completed: 2024/00/00)
• Add a SGID Index record (@jacobdadams , completed: 2024/00/00)
• [ ] Complete a SGID.META.AGOLItems record (name, completed: 2024/00/00)
  • AGOL_ITEM_ID will be populated by Forklift after it publishes the new AGOL item.
• [ ] Complete an AGOLItems_shelved record for any static or shelved item (name, completed: 2024/00/00)
• Define a record set with archives (@brigsz, completed: 2024/00/00)

🤖 Automation validation

1. Assign yourself or someone to check the item by replacing name with their github @name.
2. Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
3. Strike out all items that do not apply.

• [ ] Open SGID via cloudb (@steveoh on 2024/00/00)
• [ ] ArcGIS Online via forklift (@stdavis on 2024/00/00)
• [ ] Auditor ran successfully (@jacobdadams on 2024/00/00)
• [ ] SGID on ArcGIS (name on 2024/00/00)
• gis.utah.gov data pages (name on 2024/00/00)
• sgid-index (@steveoh on 2024/00/00)

Notification

• Twitter (@steveoh)

Group Task Assignments

1. Check [x] the box when you have assigned all the tasks relevant to your group.

• Data Team (@gregbunce)
• Dev Team (@steveoh)
• Cadastre Team (@rkelson)

[steveoh]

Discover or AGOL are the places we have available to serve this sort of data. I assume it would be tiled?

[jacobdadams]

Sorry, I should have explained more. The skyshade is a pure raster product, like a hillshade but with a significantly longer processing time (it took about a week to run on my old office desktop). The output raster needs to be styled to a user's preference, so pre-rendering wouldn't work.

The original output was something like 24gb, but I was able to get it down to about 4gb in the process of dividing it into the same sized chunks we currently serve the underlying 10m DEM in. At this size, a single download wouldn't be too much of a burden on users, so maybe adding it as a file-only AGOL item and linking to it from a datapage would be the better course of action.

[steveoh]

Yeah or GCS is a cheap option for downloads. So, what would be shown in raster.utah.gov?

[jacobdadams]

If we went the raster.utah.gov route, it'd look the same as when you go to download the USGS DEMs- a bunch of tiles that would link to zip files in GCS.

I'm intrigued by the idea of hosting it as an item in AGOL. I'd have to check the storage costs, but then we could expose it through Open Data as well.

[steveoh]

Does that mean that raster would only have a tile index? I use the preview feature but that wouldn't be available would it?

[jacobdadams]

huh, I guess I've never used the preview feature and forgot it was there. Yeah, without a served version of it somewhere, it would be like the Lidar DEMs- just an extent/tile index.

Rick can correct me if I'm wrong, but I don't think it'd take too much space/cost to make a layer on Discover, but I don't know that it's worth it just for the extent. It wouldn't be terribly useful as its own layer... maybe... I mean, I guess I could do a "default" style that I like and then we could have that as a pre-canned product in discover with the raw data available for download if desired.

[steveoh]

I guess I could do a "default" style that I like and then we could have that as a pre-canned product in discover with the raw data available for download if desired.

I like this idea

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 0 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for category.internaltablename

check	status
internal sgid	⛔
meta table
- item id	⛔
- item name	⛔
- geometry type	⛔
stewardship	⛔ Did not find category.internaltablename in the worksheet

[jacobdadams]

Rick and I chatted about this today and came up with the following plan to move forward:

1. Make a single, statewide zip available for download through a page on our website. This can be hosted on AGOL for 5 credits a month and also exposed through Hub/open data. The website page will include tips and tricks for using it.
2. Do an in-place replacement of our Hillshade layer in Discover with a cached tile version of the skyshade. This gets it out there to more people and updates a less-used service (may need to run some numbers to see the actual usage of the hillshade service)
3. At some point in the future, if needed, add the skyshade zipped tiles (same tiling scheme as the 10m DEM downloads) to raster.utah.gov under a new category in the "Step 1" box (or however it looks when we rebuild the app).

I can start step one right away, and I'll need to work with @stdavis on step two. Step three can wait until later™.

[jacobdadams]

discover service usage courtesy of Rick:

[stdavis]

So the hillshade would become the preview service for this new layer in raster.utah.gov?

[jacobdadams]

correct, if/when we get there.

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 1 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for category.internaltablename

check	status
internal sgid	⛔
meta table
- item id	⛔
- item name	⛔
- geometry type	⛔
stewardship	⛔ Did not find category.internaltablename in the worksheet

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 1 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for category.internaltablename

check	status
internal sgid	⛔
meta table
- item id	⛔
- item name	⛔
- geometry type	⛔
stewardship	⛔ Did not find category.internaltablename in the worksheet

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 1 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for category.internaltablename

check	status
internal sgid	⛔
meta table
- item id	⛔
- item name	⛔
- geometry type	⛔
stewardship	⛔ Did not find category.internaltablename in the worksheet

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 1 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔

[steveoh]

Is this a stretch goal or needing a waiting or blocked label?

[nathankota]

Added waiting label to pause Conductor notifications since this is in the FY25 Q1 I&P Sprint 6

[agrc-conductor]

conductor results for tasks - 324

check	status
@rkelson has completed 0 out of 2 tasks	⛔
@stdavis has completed 0 out of 1 tasks	⛔
@jacobdadams has completed 0 out of 2 tasks	⛔
@brigsz has completed 0 out of 1 tasks	⛔
@steveoh has completed 1 out of 3 tasks	⛔
@gregbunce has completed 0 out of 1 tasks	⛔