Honor table and view schema on query

[adrinr]

Description

Add the following security checks:

1. For tables without specified columns, generate queries with the column names specified on the table instead of a global select *. Some columns might be hidden to budibase and we don't want to fetch it to ensure they are never exposed.
2. For searches with specified columns (for views, for example) we are trimming any specified column that is not part of the actual view or table schema.
3. For queries with filters, we are trimming any filter that is not present on the allowed table/view column schema. This will prevent using searches to infer some hidden column values.

Launchcontrol

Improve search row security resilience

Feature branch env

Feature Branch Link

[linear]

BUDI-8547 View search security

[mike12345567]

I think this may be too high level for this - I think the SQL layers may ignore/override this in terms of query generation. If you look at server/src/api/controllers/row/ExternalRequest.ts on line 692 there is a function called buildSqlFieldList - this comes up with a list of fields which are going to be defined as part of the SELECT <fields> in the query. SQS does something similar with its function buildInternalFieldList - these today are independent of each other as how they generate their list of fields is quite different.

We could merge these into one path through the SDK which fetches a list of fields (and relationships) to add to the QueryJson which would take into account if it was performed through a view - we'd need to pass the list of visibleTableFields down to the search implementations via the RowSearchParams.

[mike12345567]

We would also need to handle the list of relationships, found top level on the QueryJson structure under the relationships key - this would need to have relationships removed from it if the relationship column is disabled through the view (this removes the JOIN). There are again two functions which do this today, for external DBs buildExternalRelationships and for SQS buildInternalRelationships.

[mike12345567]

NOTE: none of this can be applicable to Lucene as it always returns the full row JSON, it cannot strip fields out at the database level - I wouldn't worry about this too much.

[adrinr]

After a reanalising the logic, I moved the trim to the outputProcessing. We still need to fetch all the columns for things like formulas.
This PR was meant to cover only the security around queries, so I will stick to it and I will tackle the actual SQL statement generation in a separated PR as talked on the meeting this morning

[samwho]

It's not obvious to me why the order is flipped here, a_b vs b_a. Is this intentional?

[adrinr]

It's to avoid circular loops. It's not so obvious, I am refactoring it and writing some tests for it

[mike12345567]

LGTM - very nice work!