integration of /pin-clusters

[jmensch1]

Fixes #565

This integrates server-side pin clustering into the front end.

It also decouples the api calls so that the map data requests and the visualizations data requests don't block each other. So if the user is on the Map tab, they will see pin clusters even if the visualizations data isn't back from the server yet (and vice versa if they are on the Visualizations tab). The main logic for that is in the root saga in sagas/data.js:

export default function* rootSaga() {
  yield takeLatest(types.GET_DATA_REQUEST, getMapData);  <-- independent of vis data
  yield takeLatest(types.GET_DATA_REQUEST, getVisData);  <-- independent of map data
  yield takeLatest(uiTypes.UPDATE_MAP_POSITION, updatePinClusters);
  yield takeEvery(types.GET_PIN_INFO_REQUEST, getPinData);
  yield takeLatest(types.SEND_GIT_REQUEST, sendContactData);
}

The getMapData function makes two api calls in sequence:

1. /pin-clusters -- only the data necessary to render the pin clusters given current zoom and bounds
2. /heatmap -- returns an array of lat/longs necessary to generate the heatmap

These are done in sequence rather than in parallel because the /pin-clusters request warms up the redis cache for the /heatmap request, which will ease the load on the database.

• Up to date with dev branch
• Branch name follows guidelines
• All PR Status checks are successful
• Peer reviewed and approved

[adamkendis]

This is really good. I think we discussed this last week, but am I understanding the complete flow correctly?

1. User submits filters.
2. Postgres queried for full dataset and stored in redis cache.
3. Heatmap array of just [lat, lng]s created from above dataset and stored in redis cache.
4. Cached pin clusters based on current map zoom/bounds served to client. Done every time map is moved or zoomed.
5. Heatmap array fetched from redis cache when heatmap overlay is enabled.
6. User submits new filters and process restarts from 2.

[adamkendis]

Nice, good catch.

[jmensch1]

Hey Adam, there are some slight differences from what you described. So it goes like this:

1. User submits filters, which sends a request to /pin-clusters with the filters, zoom, and bounds.

2. Postgres is queried for requesttype, srnumber, latitude, longitude for all requests within the filters. Those four columns are stored in the redis cache. The cache key for that data is an md5 hash of the filters. Then the clusters are created based on those four fields, the zoom, and the bounds.

3. When the clusters return from the client, a request is immediately sent to /heatmap, with the same filters as before. Backend pulls latitude, longitude out of the cache (which already contains them because of the previous request), and sends them to the client. There's no modification of the cache during this step.

4. Once the lat/longs are in the client, the heatmap can render immediately whenever the overlay is enabled.

5. When user pans or zooms, a new request is sent to /pin-clusters, with the existing filters and the updated zoom and bounds. The four fields that were previously stored in the cache are pulled out, the clustering algo is run again with the new zoom and bounds, and new clusters are returned to client.

6. User submits new filters and process restarts.