Best way to detect and then delete datastreams that have no observations?

[ssoper-usgs]

Hello all,

I have a system which retains the latest 120 days of data. Each day, a cleanup process leverages the filtered delete feature to remove data older than 120 days. Over time, we have accumulated a number of datastreams which no longer have any observations - I call them "empty" datastreams - can anyone recommend a good way (API call perhaps, or some other recommended approach) to detect these "empty" datastreams? I would then launch a delete api call with the datastream's id to remove it from the frost db. Perhaps there is already a cleanup mechanism in FROST-Server to periodically remove such datastreams?

Thanks for any input!

[ssoper-usgs]

My current idea is to launch a call like this one: https://<host placeholder>/Datastreams('<some datastream id')/Observations?$top=1 - and if the response comes back with no results (which looks like this: {"value":[]}, run a delete call on the datastream. I see I can also use $count in a similar way but I think counts are a bit less performant than the $top=1 call for my use case with my current frost-db configuration. Let me know if there is an even better way - thanks!

[hylkevds]

Yes, $top=1 is faster than $count. FROST-Manager has a cleanup tool that does:

EntityList<Datastream> list = service.datastreams()
                .query()
                .select("name", "id")
                .expand("Observations($select=id;$top=1)")
                .orderBy("id asc")
                .list();

So instead of fetching individual Datastreams/Observations it fetches Datastreams and expands Observations. That way it only needs one call per Datastream-page. It then deletes the Datastreams with no Observations.

Can you check what the value is for the Datastream.phenomenonTime for those empty Datastreams? If the trigger works correctly, then that should be empty. That may be an even easier way to find the empty ones.

[ssoper-usgs]

Thanks @hylkevds this is very helpful!