Feat/#873 retrieve remote schemas #877

ciaranschutte · 2024-06-05T15:15:36Z

retrieval of remote schemas
responses build to full schema using gql tools (means we can use the well tested inbuilt methods without manually parsing a gql request as intermediary objects)
output is array of full schemas as GraphQlObjects

Please note this is part of the bigger schema federation work merging into feat/featured_search
The broken build issues are for some missing TS types, implicit anys
These will be addressed before merging full feature into develop branch

Co-authored-by: Anders Richardsson <[email protected]>

…arranger into feat/#872_initial_config

Co-authored-by: Anders Richardsson <[email protected]>

…eat/#873_retrieve_remote_schemas

…e_schemas

…eat/#873_retrieve_remote_schemas

justincorrigible · 2024-06-12T13:21:17Z

modules/server/src/network/index.ts

+		if (response.status === 200 && response.statusText === 'OK') {
+			// axios response "data" field, graphql response "data" field
+			const responseData = response.data?.data;
+			if (isGqlIntrospectionQuery(responseData)) {


Suggested change

if (isGqlIntrospectionQuery(responseData)) {

if (isGqlIntrospectionQuery(responseData)) {

modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

modules/server/src/network/index.ts

joneubank · 2024-06-17T16:53:48Z

modules/server/src/network/index.ts

+	// type cast because rejected errors are handled
+	const networkQueries = (await Promise.allSettled(
+		networkQueryPromises,
+	)) as PromiseFulfilledResult<FetchRemoteSchemaResult>[];


Should not use this type assertion. Promise.allSettled returns a PromiseSettledResult<T> where T is your type.

This is important to use as any rejected response will NOT have a .value property, we need to always check for the .status in each element of network queries array.

See type information for PromiseSettledResult<T>: https://github.com/microsoft/TypeScript/blob/main/src/lib/es2020.promise.d.ts#L11

joneubank · 2024-06-17T16:56:38Z

modules/server/src/network/index.ts

+
+	// build schema
+	const schemaResults = networkQueries.map((networkResult) => {
+		const { config, introspectionResult } = networkResult.value;


.value won't exist on rejected promises. We can filter those out if we only care about succeses:

const schemaResults = networkQueries .filter( (result): result is PromiseFulfilledResult<FetchRemoteSchemaResult> => result.status === 'fulfilled', ) .map((networkResult) => { const { config, introspectionResult } = networkResult.value;

that's how the code is currently handling

modules/server/src/network/index.ts

ciaranschutte · 2024-06-21T17:52:31Z

@joneubank addressed feedback. TS building. please take another look when you have some time.

joneubank

Looking strong throughout.

The axios request failure handling I think is not what you would expect to see so consider that comment and if there are changes needed.

joneubank · 2024-06-21T20:56:51Z

modules/server/src/network/gql.ts

+/**
+ * Creates a graphql query string with variables for use in a POST request
+ * @returns string
+ */


joneubank · 2024-06-21T21:05:59Z

modules/server/src/network/gql.ts

+	/**
+	 * TODO: Add typing for variables. Type will be dependant on gql endpoint being queried.
+	 */
+	variables: unknown;


Looking at how this is used, we don't really care about the specific contents of the variables. This function is doing no work to ensure that the variables expected in the query are included in variables (or vice versa). The only thing we know is we want to be able to run JSON.stringify(variables) which takes any. In practice, we expect this parameter to be an object with some variable values which themselves could be anything, and we probably don't want it to be string or number etc. So in this case I might reccomend:

variables: Record<string, any>

joneubank · 2024-06-21T21:07:47Z

modules/server/src/network/gql.ts

+ * Creates a graphql query string with variables for use in a POST request
+ * @returns string
+ */
+export const createGqlQuery = ({


Noticed this function isn't actually used in your code here, is that expected?

expected. #877 (comment)

joneubank · 2024-06-21T21:08:04Z

modules/server/src/network/gql.ts

 	// queries need to be strings when using regular http
 	const query = print(queryAST);
 	return JSON.stringify({ query, variables });
 };

-export const fetchGql = ({ url, gqlRequest }) => {
+export const fetchGql = ({ url, gqlRequest }: { url: string; gqlRequest: { query: string } }) => {


TSDoc will be wanted before feature branch is complete.

I think your TSDoc is incorrect, instead of returning an axios instance that you can use to send future requests, this sends your defined request with axios and returns the AxiosResponse (or throws error, stupid axios).

Great feedback, useful insight... but let's try keeping unhelpful personal opinions to a minimum.

joneubank · 2024-06-21T21:08:23Z

modules/server/src/network/index.ts

@@ -18,7 +19,7 @@ const isGqlIntrospectionQuery = (input: unknown): input is IntrospectionQuery =>

 type FetchRemoteSchemaResult = {
 	config: NetworkAggregationConfigInput;
-	introspectionResult: IntrospectionQuery | null;
+	introspectionResult: IntrospectionQuery | undefined;


why not just:

introspectionResult?: IntrospectionQuery

joneubank · 2024-06-21T21:08:46Z

modules/server/src/network/index.ts

+				throw new NetworkAggregationError(
+					`Unexpected data in response object. Please verify the endpoint at ${graphqlUrl} is returning a valid GQL Schema.`,


joneubank · 2024-06-21T21:16:20Z

modules/server/src/network/index.ts

+	} catch (error) {
+		console.log(`failed to retrieve schema from url: ${config.graphqlUrl}`);
+		console.error(error);
+		return { config, introspectionResult: undefined };
+	}


Axios throws errors on status 4xx and 5xx. That means that the else block you have above is behaviour that you are likely expecting to be handled here as well. You should try checking if the error here is an instance of AxiosError, and if so you can handle these error statuses in the catch block. I imagine you will want to throw your NetworkAggregationError.

joneubank · 2024-06-21T21:20:51Z

modules/server/src/network/index.ts

+		.filter(
+			(result): result is PromiseFulfilledResult<FetchRemoteSchemaResult> =>
+				result.status === 'fulfilled',
+		)


With TS 5.5 we no longer require the result is return type :D

joneubank · 2024-06-21T21:58:01Z

modules/server/src/network/index.ts

+	} catch (error) {
+		console.log(`failed to retrieve schema from url: ${config.graphqlUrl}`);
+		console.error(error);
+		return { config, introspectionResult: undefined };


PS. don't need to do introsepectionResult: undefined if you have the type defined as introspectionResult?: IntrospectionQuery

joneubank · 2024-06-26T13:03:51Z

modules/server/src/network/gql.ts

 	// queries need to be strings when using regular http
 	const query = print(queryAST);
 	return JSON.stringify({ query, variables });
 };

-export const fetchGql = ({ url, gqlRequest }) => {
+export const fetchGql = ({ url, gqlRequest }: { url: string; gqlRequest: { query: string } }) => {


I think your TSDoc is incorrect, instead of returning an axios instance that you can use to send future requests, this sends your defined request with axios and returns the AxiosResponse (or throws error, stupid axios).

joneubank · 2024-06-26T13:05:47Z

modules/server/src/network/types.ts

+// environment config
+export type NetworkAggregationConfigInput = {
+	graphqlUrl: string;
+	documentType: string;
+	displayName: string;
+};


The comment here doesn't tell me anything new. You can use TSDoc on types as well to leave a description of the type in intellisense for future devs that use it.

joneubank · 2024-06-26T13:08:43Z

modules/server/src/network/types.ts

+// additional properties after initialising network
+export type NetworkAggregationConfig = NetworkAggregationConfigInput & {
+	schema?: GraphQLSchema;
+};


This is a helpful comment but it wasn't easily understandable. I had to read the type information to understand what you mean. Maybe something like:

Suggested change

// additional properties after initialising network

export type NetworkAggregationConfig = NetworkAggregationConfigInput & {

schema?: GraphQLSchema;

};

/**

* Complete aggregation config that defines the network search setup. This includes the original

* config information plus computed fields that are generated in the network search initialization process.

*/

export type NetworkAggregationConfig = NetworkAggregationConfigInput & {

schema?: GraphQLSchema;

};

That's a nice improvement, for sure!
Naive question: am I the only one who gets bugged by the line length inconsistency?
personally, I'd chop it like this

* Complete aggregation config that defines the network search setup. * This includes the original config information plus computed fields * that are generated in the network search initialization process.

justincorrigible · 2024-06-26T13:29:22Z

modules/server/src/network/index.ts

+ * @returns
+ */
+const isGqlIntrospectionQuery = (input: unknown): input is IntrospectionQuery =>
+	!!input && typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';


wouldn't typeof input === 'object' inherently cover the !!?
(an object cannot be falsy)

Suggested change

!!input && typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';

typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';

does it matter that this function would allow input to be an array customised to include __schema?

1st point:
typeof null is object - <3 js (so '__schema' in input would break)

2nd point:
I don't think that's a concern. I'm using the getIntrospectionQuery which is part of the graphql library (conforming to spec) and this check is only happening to check response data from a GQL server.

It queries an automatically generated field in a graphql server, not a user created field.
For an array to occur the graphql spec would need to be changed.

is there a null.__schema? otherwise, !! is still redundant

joneubank · 2024-06-26T14:20:03Z

modules/server/src/network/index.ts

+					introspectionSchema !== undefined ? buildClientSchema(introspectionSchema) : undefined;
+				return { ...config, schema };
+			} catch (error) {
+				console.error('build schema error', error);


It would be nice if these messages were more descriptive. The best error messages let the reader know what was being attempted, why it failed, and (if possible) what action should be taken.

It is likely that this is about on par with other error messaging in this repo, but working to improve these as we go would be a huge help.

for sure - this is just the "retrieve schema" step
buildClientSchema is not implemented
will be keeping this in mind working on the "build schema" ticket

Ciaran Schutte and others added 30 commits June 4, 2024 22:33

initial config

e326785

add network to config template. add default values to base.json

5fc6bf7

add schema merge placeholders

d8efe90

add conditional to graphql route gen for env flag

51f190f

throw error on invalid json parse eg. empty file

3cc76e9

rename type

dc9e8b7

add vscode debug file to gitignore

60f93a2

add gql client lib

4dba960

change gql lib

e5eef63

Update modules/server/README.md

4fa70e3

Co-authored-by: Anders Richardsson <[email protected]>

add fetch field

7b4aec5

rename field to documentType

12be615

Update modules/server/configTemplates/base.json

892ca74

Co-authored-by: Anders Richardsson <[email protected]>

Merge branch 'feat/#872_initial_config' of github.com:overture-stack/…

514e375

…arranger into feat/#872_initial_config

Update modules/server/src/config/utils/getConfigFromFiles.ts

4b50e5a

Co-authored-by: Anders Richardsson <[email protected]>

Merge remote-tracking branch 'origin/feat/#872_initial_config' into f…

7a4adca

…eat/#873_retrieve_remote_schemas

Update types.ts

18630ae

Merge remote-tracking branch 'origin/feat/#872_initial_config' into f…

739b75b

…eat/#873_retrieve_remote_schemas

Update README.md

e3b5fbf

add ignore to gitignore

28566fe

rename fields, add nesting

578f5b2

remove env config to enable federated search

b660880

remove env config to enable federated search

a04db5b

remove flag from env schema

5da7740

Merge branch 'feat/#872_initial_config' into feat/#873_retrieve_remot…

d762461

…e_schemas

correct schema retrieval

61aea8a

Merge remote-tracking branch 'origin/feature/federated_search' into f…

d5a87e8

…eat/#873_retrieve_remote_schemas

fix GQL obj wrapping on schema construction

7b82c9e

example

45761b3

cleanup

7eee03b

justincorrigible reviewed Jun 12, 2024

View reviewed changes

modules/server/src/network/index.ts Outdated Show resolved Hide resolved

justincorrigible reviewed Jun 12, 2024

View reviewed changes

modules/server/src/network/index.ts Outdated Show resolved Hide resolved

ciaranschutte and others added 2 commits June 12, 2024 15:48

Update modules/server/src/network/index.ts

da5e256

Co-authored-by: Anders Richardsson <[email protected]>

add logs to thrown errors

f9d96da

ciaranschutte requested a review from justincorrigible June 13, 2024 19:40

joneubank requested changes Jun 17, 2024

View reviewed changes

Ciaran Schutte added 7 commits June 20, 2024 20:34

small feedback fixes

f741586

add typings + additional info in comments

59cc470

add better errors

da4f31e

remove unused func. renaming

c64cac7

add return type

cd8848c

remove assertion. add TS filter

18db891

change nulls to undefined

1b4f22b

ciaranschutte requested a review from joneubank June 21, 2024 17:52

joneubank reviewed Jun 21, 2024

View reviewed changes

Ciaran Schutte added 6 commits June 25, 2024 09:43

change explicit undefined to optional ts

cac6bbd

add tsdoc comment

1c386ce

change var type to loose record instead of uknown

8ae17be

if else TS

c19ff5a

if else TS

68ca18e

change to nested promise return

e67a16c

ciaranschutte requested a review from joneubank June 25, 2024 20:36

joneubank approved these changes Jun 26, 2024

View reviewed changes

justincorrigible reviewed Jun 26, 2024

View reviewed changes

joneubank reviewed Jun 26, 2024

View reviewed changes

ciaranschutte merged commit 4f62745 into feature/federated_search Jun 26, 2024
0 of 2 checks passed

ciaranschutte deleted the feat/#873_retrieve_remote_schemas branch June 26, 2024 15:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/#873 retrieve remote schemas #877

Feat/#873 retrieve remote schemas #877

ciaranschutte commented Jun 5, 2024 •

edited

Loading

justincorrigible Jun 12, 2024

joneubank Jun 17, 2024

joneubank Jun 17, 2024

ciaranschutte Jun 20, 2024

ciaranschutte commented Jun 21, 2024

joneubank left a comment

joneubank Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 21, 2024

ciaranschutte Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 26, 2024

justincorrigible Jun 26, 2024

joneubank Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 21, 2024

joneubank Jun 26, 2024

joneubank Jun 26, 2024

joneubank Jun 26, 2024

justincorrigible Jun 26, 2024

justincorrigible Jun 26, 2024

justincorrigible Jun 26, 2024

ciaranschutte Jun 26, 2024 •

edited

Loading

ciaranschutte Jun 26, 2024

justincorrigible Jun 26, 2024 •

edited

Loading

joneubank Jun 26, 2024

ciaranschutte Jun 26, 2024

	if (isGqlIntrospectionQuery(responseData)) {

	if (isGqlIntrospectionQuery(responseData)) {

		throw new NetworkAggregationError(
		`Unexpected data in response object. Please verify the endpoint at ${graphqlUrl} is returning a valid GQL Schema.`,

	!!input && typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';
	typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';

Feat/#873 retrieve remote schemas #877

Feat/#873 retrieve remote schemas #877

Conversation

ciaranschutte commented Jun 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ciaranschutte commented Jun 21, 2024

joneubank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ciaranschutte Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justincorrigible Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ciaranschutte commented Jun 5, 2024 •

edited

Loading

ciaranschutte Jun 26, 2024 •

edited

Loading

justincorrigible Jun 26, 2024 •

edited

Loading