285 generate parameters from sql #288

EthanMcQ-TMF · 2024-02-14T23:53:45Z

Closes #285

👩🏻‍💻 What does this PR do?

Added a new section to notifications to add an SQL query. Currently these are the same queries used for notifications

~~A parameter query is expected to return a single row with a column "parameters", as a JSON array in the same format as our JSON parameter editing in the UI~~

A parameter query is expected to return a column for each parameter, with each row being used as a parameter set

When you run the notification, a new event should generate for each defined parameter from the query result (In this example I'm just listing all the tables in my local database and the id of the first entry (There's a number of failures because I don't have data in all my tables)

This should work alongside regularly defined parameters! The list of query generated parameters and the list of user provided parameters will be merged when creating events.

In addition I've made it possible for a notification to be defined with no user defined parameters, so a notification can be created without any parameter sets (and pull exclusively from query parameters). I've added an additional "create" button for adding a new parameter when none exists already

🧪 How has/should this change been tested?

Tested locally running the queries screenshotted above ⬆️

💌 Any notes for the reviewer?

This is working, but there's definitely some improvements we could make. Currently I'm just using notification queries for fetching parameters, splitting those out to their own defined table similar to sql recipient lists is an obvious improvement.

This is also my first time writing rust so let me know if I've done anything silly there 😅

📃 Documentation

Notification Parameters - Get parameters from SQL query
[ ]

Add better error messages

and adding empty parameter sets

github-actions · 2024-02-14T23:56:26Z

Bundle size difference

Comparing this PR to main

Old size	New size	Diff
5.2 MB	5.2 MB	2.83 KB (0.05%)

EthanMcQ-TMF · 2024-02-14T23:56:31Z

frontend/packages/system/src/Notifications/Pages/Base/NotificationDetailPanel.tsx

+                if (idx === params.length) {
+                  // Allow adding empty sets from JSON
+                  params.push({});
+                  onDeleteParam(idx, 'this-is-a-hack-to-force-an-update');
+                }
+                // Iterate through all valid keys
+                for (const key of [...Object.keys(param), ...Object.keys(params[idx] || {})]) {
+                  if (param[key]) {
+                    // If we parsed the key from the JSON input, update it
+                    onUpdateParams(idx, key, param[key] || '');
+                  } else {
+                    // If the key exists on the parameter object but not the JSON, delete it
+                    onDeleteParam(idx, key);
+                  }
                }
              });
+              while (params.length > editedParams.length) {
+                onDeleteParam(editedParams.length, null);
+              }


This whole section of code is completely separate to the main issue, it just improves the behaviour of loading params from JSON. The previous implementation didn't allow for things like removing parameter sets, deleting individual keys from parameter sets, or adding empty parameter sets

EthanMcQ-TMF · 2024-02-14T23:59:46Z

frontend/packages/system/src/Notifications/Pages/Scheduled/ScheduledNotificationEditPage.tsx

+    if (parsedDraft) {
+      // Backend expects this value to be null if unedited so don't load it from config
+      parsedDraft.nextDueDatetime = null;
+    }


Not associated with main issue but fixes a bug. The current config of nextDueDatetime would always be sent alongside the rest of the data when sending an update to the config, when the backend sees this in the update query, it runs the notification. So the notification would always run on save whether you hit "Save" or "Save & Run"

EthanMcQ-TMF · 2024-02-15T00:06:07Z

frontend/packages/system/src/Notifications/Pages/Base/NotificationDetailPanel.tsx

+  const { queryParams } = useQueryParamsState();
+  const { data: queriesData } = useNotificationQueries(queryParams);
+  const selectedQuery = queriesData?.nodes.find(query => query.id === parameterQueryId);


Currently I'm just pulling all the notification queries for filling up the autocomplete selector. This will be mostly fine until we get to large numbers of queries, we might want to paginate this selection a bit better

I'm ok with grabing every thing for now.
I also didn't expect 500 parameter sets,so you never know...

jmbrunskill

Fantastic work!

Happy to merge this, but also happy if you wanted to add something to smooth the creation of the parameter queries.
Would be nice if we didn't have to have a column called parameters.
I think it would be better if just have a list of rows, each row becomes a parameter set? That would basically need to call pg_sql_query_as_json_rows I think?

jmbrunskill · 2024-02-15T04:29:10Z

backend/service/src/notification_config/parameters.rs

+    let mut sql_params: Vec<HashMap<String, serde_json::Value>>  = serde_json::from_str(&sql_params_string)
+        .map_err(|e| {
+            NotificationServiceError::InternalError(format!(
+                "Failed to parse notification sql parameters (expecting an array of params_string): {:?} - {}",
+                e, params_string
+            ))
+        })?;
+    all_params.append(&mut sql_params);


This can be written like this instead.
I'm not sure if one is strictly better than the other but in general I try to only make this mutable if they have to be.

Suggested change

let mut sql_params: Vec<HashMap<String, serde_json::Value>> = serde_json::from_str(&sql_params_string)

.map_err(|e| {

NotificationServiceError::InternalError(format!(

"Failed to parse notification sql parameters (expecting an array of params_string): {:?} - {}",

e, params_string

))

})?;

all_params.append(&mut sql_params);

let sql_params: Vec<HashMap<String, serde_json::Value>> = serde_json::from_str(&sql_params_string)

.map_err(|e| {

NotificationServiceError::InternalError(format!(

"Failed to parse notification sql parameters (expecting an array of params_string): {:?} - {}",

e, params_string

))

})?;

all_params.extend(sql_params);

Weird to me that append requires &mut, but I guess it needs the same datatype...

I think it's fine how it is but wanted to check why i needed to be mut

jmbrunskill · 2024-02-15T04:30:06Z

backend/service/src/notification_config/parameters.rs

+    ctx: &ServiceContext,
+    parameter_query_id: &String,
+) -> Result<String, NotificationServiceError> {
+    // TODO: Maybe split these to a new database table


It's quite a bit of work, and I'm not sure it's needed but could be handy, or maybe add a query type so we can filter?

jmbrunskill · 2024-02-15T04:33:48Z

backend/service/src/notification_config/parameters.rs

+
+    let parameters = match parsed_results[0].get("parameters") {
+        None => return Err(NotificationServiceError::InternalError(format!(
+                    "No 'parameters' column found in query results - {}", parameter_query_id)


If we did create a new method we could automatically run something like SELECT json_agg(q.*) as parameters from (<THEQUERY>) as q?

Then we wouldn't have to the restrictions about the column naming and parameter queries would be a bit easier to test?

jmbrunskill · 2024-02-15T04:45:26Z

backend/service/src/notification_config/update.rs

@@ -124,5 +126,8 @@ pub fn generate(
    // Note: We usually reset the next check datetime in case the schedule has changed, or something needs to be recalculated
    new_notification_config_row.next_due_datetime = next_due_datetime;

+    // We might want to clear out the parameter_query_id so it's allowed to be None


I guess we're just seeing to empty string now? It seems to be working??

This was more a comment about why it's not using the if let Some(X) check we use for other values. Since that doesn't let you un-set it back to None

jmbrunskill · 2024-02-15T04:46:49Z

frontend/packages/system/src/Notifications/Pages/Base/NotificationDetailPanel.tsx

+  const { queryParams } = useQueryParamsState();
+  const { data: queriesData } = useNotificationQueries(queryParams);
+  const selectedQuery = queriesData?.nodes.find(query => query.id === parameterQueryId);


I'm ok with grabing every thing for now.
I also didn't expect 500 parameter sets,so you never know...

jmbrunskill · 2024-02-15T04:47:37Z

frontend/packages/system/src/Notifications/Pages/Base/NotificationDetailPanel.tsx

+                    // If we parsed the key from the JSON input, update it
+                    onUpdateParams(idx, key, param[key] || '');
+                  } else {
+                    // If the key exists on the parameter object but not the JSON, delete it


Appreciate all these comments!

jmbrunskill · 2024-02-15T04:47:59Z

frontend/packages/system/src/Notifications/Pages/Scheduled/ScheduledNotificationEditPage.tsx

+    if (parsedDraft) {
+      // Backend expects this value to be null if unedited so don't load it from config
+      parsedDraft.nextDueDatetime = null;
+    }


EthanMcQ-TMF · 2024-02-15T20:51:58Z

Fantastic work!

Happy to merge this, but also happy if you wanted to add something to smooth the creation of the parameter queries. Would be nice if we didn't have to have a column called parameters. I think it would be better if just have a list of rows, each row becomes a parameter set? That would basically need to call pg_sql_query_as_json_rows I think?

Nah much easier than that, the results are already json-d, I just removed the section that expects a column called parameters and it works perfectly fine

jmbrunskill · 2024-02-15T20:56:55Z

Nah much easier than that, the results are already json-d, I just removed the section that expects a column called parameters and it works perfectly fine

Nice! Was thinking that after I added that comment, isn't it already json...

EthanMcQ-TMF added 15 commits February 7, 2024 16:08

Add parameter_query_id to notification config

26feb50

Messy first pass of backend functionality

16cbe02

Allow deleting extra parameters when updating via json

9229f6a

Basic selector for parameters query

53b56d2

Add parameter_query_id to graphql types in backend

948fd2e

Regenerate graphql schema

e85f21b

Update types to allow editing parameter query id

48eb54e

Update parameterQueryId correctly

48b231d

Working backend draft

9c91207

Add better error messages

Keep better track of the selected parameter query

caedb94

Fixup updateNotificationConfig values

7629823

Remove completed TODOs

1270b4a

Allow for no defined parameters

fee275a

Update JSON parameter updater to allow deleting individual parameters

50ad170

and adding empty parameter sets

Remove unnecessary null checking

0da54e8

EthanMcQ-TMF commented Feb 14, 2024

View reviewed changes

EthanMcQ-TMF commented Feb 15, 2024

View reviewed changes

jmbrunskill approved these changes Feb 15, 2024

View reviewed changes

EthanMcQ-TMF added 2 commits February 16, 2024 09:46

PR request - Swap append with extend

65417e6

Expect param query results as regular rows

96443c3

EthanMcQ-TMF merged commit 0ee9514 into main Feb 15, 2024

EthanMcQ-TMF deleted the 285-generate-parameters-from-sql branch February 15, 2024 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

285 generate parameters from sql #288

285 generate parameters from sql #288

EthanMcQ-TMF commented Feb 14, 2024 •

edited

Loading

github-actions bot commented Feb 14, 2024

EthanMcQ-TMF Feb 14, 2024

EthanMcQ-TMF Feb 14, 2024 •

edited

Loading

jmbrunskill Feb 15, 2024

EthanMcQ-TMF Feb 15, 2024

jmbrunskill Feb 15, 2024

jmbrunskill left a comment

jmbrunskill Feb 15, 2024

jmbrunskill Feb 15, 2024

jmbrunskill Feb 15, 2024

jmbrunskill Feb 15, 2024

EthanMcQ-TMF Feb 15, 2024

jmbrunskill Feb 15, 2024

jmbrunskill Feb 15, 2024

jmbrunskill Feb 15, 2024

EthanMcQ-TMF commented Feb 15, 2024

jmbrunskill commented Feb 15, 2024

285 generate parameters from sql #288

285 generate parameters from sql #288

Conversation

EthanMcQ-TMF commented Feb 14, 2024 • edited Loading

👩🏻‍💻 What does this PR do?

🧪 How has/should this change been tested?

💌 Any notes for the reviewer?

📃 Documentation

github-actions bot commented Feb 14, 2024

Bundle size difference

Choose a reason for hiding this comment

EthanMcQ-TMF Feb 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmbrunskill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EthanMcQ-TMF commented Feb 15, 2024

jmbrunskill commented Feb 15, 2024

EthanMcQ-TMF commented Feb 14, 2024 •

edited

Loading

EthanMcQ-TMF Feb 14, 2024 •

edited

Loading