Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Optimize Browse page queries #1236

Merged
merged 48 commits into from
Oct 31, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
6143c45
fix: Allow dataset tags to wrap
bprusinowski Oct 25, 2023
c82f421
perf: Do not query dataset counts
bprusinowski Oct 25, 2023
5621193
perf: Do not use two queries when searching for cubes
bprusinowski Oct 25, 2023
4ed5fdf
fix: Point to a correct graph to retrieve creator labels
bprusinowski Oct 25, 2023
3d23a44
perf: Use scalar for nested objects (GQL SearchCubes)
bprusinowski Oct 25, 2023
a6220e5
style: Prevent layout shift when draft tag is added
bprusinowski Oct 26, 2023
ea5501e
fix: Types
bprusinowski Oct 26, 2023
6e39852
refactor: Use scalar for SearchCube
bprusinowski Oct 26, 2023
8a11fb2
fix: Do not measure timing twice
bprusinowski Oct 26, 2023
f8fbe2f
perf: Optimize cube search query
bprusinowski Oct 26, 2023
fbba91b
fix: Order cubes by version history and by desc(iri) to select newest…
bprusinowski Oct 26, 2023
467a17a
fix: Types
bprusinowski Oct 26, 2023
15d0b76
fix: Do not filter by subthemes on the server side
bprusinowski Oct 26, 2023
e4c0278
fix: Tests
bprusinowski Oct 26, 2023
a3b4a8f
refactor: Clean up
bprusinowski Oct 26, 2023
ac5e6ed
fix: Missing text input value after refresh
bprusinowski Oct 26, 2023
3143679
fix: Type
bprusinowski Oct 26, 2023
91a298c
fix: Removing browse theme and organization filters
bprusinowski Oct 26, 2023
826ebeb
feat: Allow subthemes to be displayed when organization is second sec…
bprusinowski Oct 26, 2023
a2b4aef
fix: Selecting subthemes when both sections are filtered
bprusinowski Oct 26, 2023
bde860c
fix: Do not clear the cubes when new query is being executed
bprusinowski Oct 26, 2023
d1eff45
fix: Only update browse state if router is ready
bprusinowski Oct 26, 2023
4f65f10
docs: Update CHANGELOG
bprusinowski Oct 26, 2023
a726792
fix: Include subthemes when calculating cube search scores
bprusinowski Oct 27, 2023
95341b1
perf: Specify graph when querying themes
bprusinowski Oct 27, 2023
986e842
perf: Specify graph for subthemes
bprusinowski Oct 27, 2023
7b9c508
chore: Comment out published date datatype condition for now
bprusinowski Oct 27, 2023
4b9e50e
fix: Keep multi-lang filtering
bprusinowski Oct 27, 2023
d16b2b9
fix: Tests
bprusinowski Oct 27, 2023
6b1cedc
test: Add more E2E search tests
bprusinowski Oct 27, 2023
41353f6
fix: creativeWorkStatus has to be set
bprusinowski Oct 27, 2023
c02a7d2
style: Animate filters title
bprusinowski Oct 27, 2023
7631f7d
perf: Do not fire cubes, themes and organizations queries on dataset …
bprusinowski Oct 27, 2023
e2de8ca
fix: Do not filter publishers by language
bprusinowski Oct 27, 2023
3ed4efc
perf: Further optimize cube search query
bprusinowski Oct 30, 2023
a01b3f7
feat: Improve GQLDebugPanel
bprusinowski Oct 30, 2023
e90b2a2
fix: Search query keywords
bprusinowski Oct 30, 2023
b4e87bc
fix: Copy button placement
bprusinowski Oct 30, 2023
d8c50c2
fix: Show published and draft versions of cubes with the same version…
bprusinowski Oct 30, 2023
8d87a82
perf: Optimize search query
bprusinowski Oct 30, 2023
4652385
perf: Remove version history-related logic from SearchCubes query
bprusinowski Oct 30, 2023
e708814
perf: Optimize search cube queries by concatenating themes and sub-th…
bprusinowski Oct 30, 2023
d4117a6
fix: No results (search cubes)
bprusinowski Oct 31, 2023
67c6bdc
chore: Typo
bprusinowski Oct 31, 2023
056c640
fix: Theme filtering needs to happen in HAVING part of the query
bprusinowski Oct 31, 2023
7815ab2
perf: Do not query themes and organizations separately
bprusinowski Oct 31, 2023
0497a97
fix: Exclude topic when constructing remove URL (search filters nav i…
bprusinowski Oct 31, 2023
edb4e91
perf: Drop HAVING in favor of direct filtering
bprusinowski Oct 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions app/domain/data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,17 @@ export type GeoData = {
symbolLayer: SymbolLayer | undefined;
};

// Extracted for performance reasons.
ptbrowne marked this conversation as resolved.
Show resolved Hide resolved
export type SearchCubeCreator = {
iri: string;
label: string;
};

export type SearchCubeThemes = {
iri: string;
label: string;
}[];

const xmlSchema = "http://www.w3.org/2001/XMLSchema#";
export const parseRDFLiteral = <T = ObservationValue>(value: Literal): T => {
const v = value.value;
Expand Down
10 changes: 2 additions & 8 deletions app/graphql/queries/data-cubes.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -105,14 +105,8 @@ query SearchCubes(
description
publicationStatus
datePublished
creator {
iri
label
}
themes {
iri
label
}
creator
themes
}
}
}
Expand Down
22 changes: 11 additions & 11 deletions app/graphql/query-hooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ import { DimensionValue } from '../domain/data';
import { QueryFilters } from '../configurator';
import { Observation } from '../domain/data';
import { RawObservation } from '../domain/data';
import { SearchCubeCreator } from '../domain/data';
import { SearchCubeThemes } from '../domain/data';
import gql from 'graphql-tag';
import * as Urql from 'urql';
export type Maybe<T> = T | null;
Expand All @@ -22,6 +24,8 @@ export type Scalars = {
GeoShapes: any;
Observation: Observation;
RawObservation: RawObservation;
SearchCubeCreator: SearchCubeCreator;
SearchCubeThemes: SearchCubeThemes;
ValueIdentifier: any;
ValuePosition: any;
};
Expand Down Expand Up @@ -443,12 +447,13 @@ export type SearchCube = {
iri: Scalars['String'];
title: Scalars['String'];
description?: Maybe<Scalars['String']>;
creator?: Maybe<DataCubeOrganization>;
creator?: Maybe<Scalars['SearchCubeCreator']>;
publicationStatus: DataCubePublicationStatus;
datePublished?: Maybe<Scalars['String']>;
themes: Array<DataCubeTheme>;
themes: Scalars['SearchCubeThemes'];
};


export type SearchCubeFilter = {
type: Scalars['String'];
value: Scalars['String'];
Expand All @@ -468,6 +473,7 @@ export enum SearchCubeResultOrder {
CreatedDesc = 'CREATED_DESC'
}


export type StandardErrorDimension = Dimension & {
__typename: 'StandardErrorDimension';
iri: Scalars['String'];
Expand Down Expand Up @@ -832,7 +838,7 @@ export type SearchCubesQueryVariables = Exact<{
}>;


export type SearchCubesQuery = { __typename: 'Query', searchCubes: Array<{ __typename: 'SearchCubeResult', highlightedTitle?: Maybe<string>, highlightedDescription?: Maybe<string>, cube: { __typename: 'SearchCube', iri: string, title: string, description?: Maybe<string>, publicationStatus: DataCubePublicationStatus, datePublished?: Maybe<string>, creator?: Maybe<{ __typename: 'DataCubeOrganization', iri: string, label?: Maybe<string> }>, themes: Array<{ __typename: 'DataCubeTheme', iri: string, label?: Maybe<string> }> } }> };
export type SearchCubesQuery = { __typename: 'Query', searchCubes: Array<{ __typename: 'SearchCubeResult', highlightedTitle?: Maybe<string>, highlightedDescription?: Maybe<string>, cube: { __typename: 'SearchCube', iri: string, title: string, description?: Maybe<string>, publicationStatus: DataCubePublicationStatus, datePublished?: Maybe<string>, creator?: Maybe<SearchCubeCreator>, themes: SearchCubeThemes } }> };

export type DataCubePreviewQueryVariables = Exact<{
iri: Scalars['String'];
Expand Down Expand Up @@ -1255,14 +1261,8 @@ export const SearchCubesDocument = gql`
description
publicationStatus
datePublished
creator {
iri
label
}
themes {
iri
label
}
creator
themes
}
}
}
Expand Down
28 changes: 24 additions & 4 deletions app/graphql/resolver-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ import { DimensionValue } from '../domain/data';
import { Filters } from '../configurator';
import { Observation } from '../domain/data';
import { RawObservation } from '../domain/data';
import { SearchCubeCreator } from '../domain/data';
import { SearchCubeThemes } from '../domain/data';
import { GraphQLResolveInfo, GraphQLScalarType, GraphQLScalarTypeConfig } from 'graphql';
import { ResolvedDataCube, ResolvedObservationsQuery, ResolvedMeasure, ResolvedDimension } from './shared-types';
import { VisualizeGraphQLContext } from './context';
Expand All @@ -23,6 +25,8 @@ export type Scalars = {
GeoShapes: any;
Observation: Observation;
RawObservation: RawObservation;
SearchCubeCreator: SearchCubeCreator;
SearchCubeThemes: SearchCubeThemes;
ValueIdentifier: any;
ValuePosition: any;
};
Expand Down Expand Up @@ -444,12 +448,13 @@ export type SearchCube = {
iri: Scalars['String'];
title: Scalars['String'];
description?: Maybe<Scalars['String']>;
creator?: Maybe<DataCubeOrganization>;
creator?: Maybe<Scalars['SearchCubeCreator']>;
publicationStatus: DataCubePublicationStatus;
datePublished?: Maybe<Scalars['String']>;
themes: Array<DataCubeTheme>;
themes: Scalars['SearchCubeThemes'];
};


export type SearchCubeFilter = {
type: Scalars['String'];
value: Scalars['String'];
Expand All @@ -469,6 +474,7 @@ export enum SearchCubeResultOrder {
CreatedDesc = 'CREATED_DESC'
}


export type StandardErrorDimension = Dimension & {
__typename?: 'StandardErrorDimension';
iri: Scalars['String'];
Expand Down Expand Up @@ -669,9 +675,11 @@ export type ResolversTypes = ResolversObject<{
RelatedDimension: ResolverTypeWrapper<RelatedDimension>;
ScaleType: ScaleType;
SearchCube: ResolverTypeWrapper<SearchCube>;
SearchCubeCreator: ResolverTypeWrapper<Scalars['SearchCubeCreator']>;
SearchCubeFilter: SearchCubeFilter;
SearchCubeResult: ResolverTypeWrapper<SearchCubeResult>;
SearchCubeResultOrder: SearchCubeResultOrder;
SearchCubeThemes: ResolverTypeWrapper<Scalars['SearchCubeThemes']>;
StandardErrorDimension: ResolverTypeWrapper<ResolvedDimension>;
TemporalDimension: ResolverTypeWrapper<ResolvedDimension>;
TemporalOrdinalDimension: ResolverTypeWrapper<ResolvedDimension>;
Expand Down Expand Up @@ -710,8 +718,10 @@ export type ResolversParentTypes = ResolversObject<{
RawObservation: Scalars['RawObservation'];
RelatedDimension: RelatedDimension;
SearchCube: SearchCube;
SearchCubeCreator: Scalars['SearchCubeCreator'];
SearchCubeFilter: SearchCubeFilter;
SearchCubeResult: SearchCubeResult;
SearchCubeThemes: Scalars['SearchCubeThemes'];
StandardErrorDimension: ResolvedDimension;
TemporalDimension: ResolvedDimension;
TemporalOrdinalDimension: ResolvedDimension;
Expand Down Expand Up @@ -956,13 +966,17 @@ export type SearchCubeResolvers<ContextType = VisualizeGraphQLContext, ParentTyp
iri?: Resolver<ResolversTypes['String'], ParentType, ContextType>;
title?: Resolver<ResolversTypes['String'], ParentType, ContextType>;
description?: Resolver<Maybe<ResolversTypes['String']>, ParentType, ContextType>;
creator?: Resolver<Maybe<ResolversTypes['DataCubeOrganization']>, ParentType, ContextType>;
creator?: Resolver<Maybe<ResolversTypes['SearchCubeCreator']>, ParentType, ContextType>;
publicationStatus?: Resolver<ResolversTypes['DataCubePublicationStatus'], ParentType, ContextType>;
datePublished?: Resolver<Maybe<ResolversTypes['String']>, ParentType, ContextType>;
themes?: Resolver<Array<ResolversTypes['DataCubeTheme']>, ParentType, ContextType>;
themes?: Resolver<ResolversTypes['SearchCubeThemes'], ParentType, ContextType>;
__isTypeOf?: IsTypeOfResolverFn<ParentType, ContextType>;
}>;

export interface SearchCubeCreatorScalarConfig extends GraphQLScalarTypeConfig<ResolversTypes['SearchCubeCreator'], any> {
name: 'SearchCubeCreator';
}

export type SearchCubeResultResolvers<ContextType = VisualizeGraphQLContext, ParentType extends ResolversParentTypes['SearchCubeResult'] = ResolversParentTypes['SearchCubeResult']> = ResolversObject<{
score?: Resolver<Maybe<ResolversTypes['Float']>, ParentType, ContextType>;
cube?: Resolver<ResolversTypes['SearchCube'], ParentType, ContextType>;
Expand All @@ -971,6 +985,10 @@ export type SearchCubeResultResolvers<ContextType = VisualizeGraphQLContext, Par
__isTypeOf?: IsTypeOfResolverFn<ParentType, ContextType>;
}>;

export interface SearchCubeThemesScalarConfig extends GraphQLScalarTypeConfig<ResolversTypes['SearchCubeThemes'], any> {
name: 'SearchCubeThemes';
}

export type StandardErrorDimensionResolvers<ContextType = VisualizeGraphQLContext, ParentType extends ResolversParentTypes['StandardErrorDimension'] = ResolversParentTypes['StandardErrorDimension']> = ResolversObject<{
iri?: Resolver<ResolversTypes['String'], ParentType, ContextType>;
label?: Resolver<ResolversTypes['String'], ParentType, ContextType>;
Expand Down Expand Up @@ -1054,7 +1072,9 @@ export type Resolvers<ContextType = VisualizeGraphQLContext> = ResolversObject<{
RawObservation?: GraphQLScalarType;
RelatedDimension?: RelatedDimensionResolvers<ContextType>;
SearchCube?: SearchCubeResolvers<ContextType>;
SearchCubeCreator?: GraphQLScalarType;
SearchCubeResult?: SearchCubeResultResolvers<ContextType>;
SearchCubeThemes?: GraphQLScalarType;
StandardErrorDimension?: StandardErrorDimensionResolvers<ContextType>;
TemporalDimension?: TemporalDimensionResolvers<ContextType>;
TemporalOrdinalDimension?: TemporalOrdinalDimensionResolvers<ContextType>;
Expand Down
2 changes: 1 addition & 1 deletion app/graphql/resolvers/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ const DataCube: DataCubeResolvers = {
description: ({ data: { description } }) => description ?? null,
dateModified: ({ data: { dateModified } }) => dateModified ?? null,
datePublished: ({ data: { datePublished } }) => datePublished ?? null,
themes: ({ data: { themes } }) => themes || [],
themes: ({ data: { themes } }) => themes ?? [],
creator: ({ data: { creator } }) => creator ?? null,
dimensions: async (parent, args, context, info) => {
const source = getSource(args.sourceType);
Expand Down
7 changes: 5 additions & 2 deletions app/graphql/schema.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -323,14 +323,17 @@ type OrdinalMeasure implements Dimension {
hierarchy(sourceType: String!, sourceUrl: String!): [HierarchyValue!]
}

scalar SearchCubeCreator
scalar SearchCubeThemes

type SearchCube {
iri: String!
title: String!
description: String
creator: DataCubeOrganization
creator: SearchCubeCreator
publicationStatus: DataCubePublicationStatus!
datePublished: String
themes: [DataCubeTheme!]!
themes: SearchCubeThemes!
}

type SearchCubeResult {
Expand Down
9 changes: 6 additions & 3 deletions app/rdf/query-search.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { Literal, NamedNode } from "rdf-js";
import StreamClient from "sparql-http-client";
import ParsingClient from "sparql-http-client/ParsingClient";

import { SearchCubeCreator, SearchCubeThemes } from "@/domain/data";
import { truthy } from "@/domain/types";
import { RequestQueryMeta } from "@/graphql/query-meta";
import {
Expand Down Expand Up @@ -307,10 +308,12 @@ export const searchCubes = async ({
}

if (!parsedCube.creator && cube.creator) {
parsedCube.creator = {
const creator: SearchCubeCreator = {
iri: cube.creator,
label: cube.creatorLabel,
};

parsedCube.creator = creator;
}

if (!parsedCube.datePublished) {
Expand All @@ -326,9 +329,9 @@ export const searchCubes = async ({
}

if (cube.theme || cube.themeName) {
parsedCube.themes.push({
(parsedCube.themes as SearchCubeThemes).push({
iri: cube.theme,
name: cube.themeName,
label: cube.themeName,
});
}
}
Expand Down
4 changes: 4 additions & 0 deletions codegen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ generates:
RawObservation: "../domain/data#RawObservation"
Filters: "../configurator#QueryFilters"
GeoShape: "../domain/data#GeoShape"
SearchCubeCreator: "../domain/data#SearchCubeCreator"
SearchCubeThemes: "../domain/data#SearchCubeThemes"
app/graphql/resolver-types.ts:
plugins:
- "typescript"
Expand All @@ -35,6 +37,8 @@ generates:
RawObservation: "../domain/data#RawObservation"
Filters: "../configurator#Filters"
GeoShape: "../domain/data#GeoShape"
SearchCubeCreator: "../domain/data#SearchCubeCreator"
SearchCubeThemes: "../domain/data#SearchCubeThemes"
mappers:
DataCube: "./shared-types#ResolvedDataCube"
ObservationsQuery: "./shared-types#ResolvedObservationsQuery"
Expand Down
Loading