Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor front end Meta class and a lot of surrounding code. #1109

Merged
merged 4 commits into from
Mar 3, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions mathesar_ui/src/App.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,6 @@ export interface SchemaResponse extends SchemaEntry, TreeItem {
tables: DBObjectEntry[];
}

// TODO: Come up with a better name for representing both tables and views
export enum TabularType {
Table = 1,
View = 2,
}

export type DbType = string;

export interface FilterConfiguration {
Expand Down
9 changes: 9 additions & 0 deletions mathesar_ui/src/App.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@
import Header from '@mathesar/header/Header.svelte';
import { toast } from '@mathesar/stores/toast';
import { confirmationController } from '@mathesar/stores/confirmation';
import { currentSchemaId } from '@mathesar/stores/schemas';
import { beginUpdatingUrlWhenSchemaChanges } from './utils/routing';

// This is a bit of a hack to deal with our routing still being a patchwork of
// declarative and imperative logic. Without this call, the URL will not
// reliably set the query params when the schema changes. It actually _will_
// set the query params _sometimes_, but we weren't able to figure out why the
// behavior is inconsistent.
beginUpdatingUrlWhenSchemaChanges(currentSchemaId);
Copy link
Contributor Author

@seancolsen seancolsen Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavish This is how I'm handling that finicky routing issue we spent some time discussing yesterday.

I played with it for a while after our call. There's definitely something fishy going on which inconsistently causes the stores within TabList to update after tinro has performed client-side navigation. I tried to trace it down using the in-browser debugging tools but just ended up in a rat's nest of compiled Svelte code.

So this call to beginUpdatingUrlWhenSchemaChanges is my hacky work-around. I had to put it within a pretty top-level location in the app in order to avoid cyclical dependency linting errors. When/if we permit the user to switch databases, we'll need to implement a similar hack.

I'm interested in eventually moving Mathesar to SvelteKit (once it's stable), in which case we'll need do re-think all the routing anyway. So I'm okay with some hack in the mean time.

Copy link
Member

@pavish pavish Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm interested in eventually moving Mathesar to SvelteKit (once it's stable), in which case we'll need do re-think all the routing anyway. So I'm okay with some hack in the mean time.

I estimate this mean time to be atleast a couple years. Even then I wouldn't want to switch to SvelteKit until it has an official python adapter. Currently, if we need to switch, we'll need a node js middle layer, which is just pointless for us. There is a SvelteKit static adapter but that'll essentially be the same (if not less performant) than directly using Svelte with pre-rendered data as we do now. So, I doubt we'd ever move to SvelteKit.

I'll take a look at this issue. We need to find the cause and fix it.

Copy link
Member

@pavish pavish Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seancolsen

I tracked down the "when" of the problem:

All the callbacks of subscribers of tabs and activeTab get called everytime the TabContainer component is destroyed and re-created. This causes recalculation and updating of the query params.

To test this, in Base.svelte, you can either remove binding tabs and activeTab to TabContainer, or remove the #if condition for tabs.length > 0 for rendering the TabContainer.

After this change, you can notice that the url does not get updated when schemas are switched (except when a new schema is opened which creates a new TabList instance, in which case it is expected). The url not updating is the behaviour we expect.

Regarding the "why" of the problem:

I still have no clue.

Callback of stores are usually called whenever there is a first-subscriber and then if any dependencies change. Here, there are multiple subscribers for both tabs and activeTab which are active, I checked if subscription count goes to 0 when schema changes but that doesn't seem to be the case. So we can eliminate the first-subscriber being a reason.

Why the callback gets updated when TabContainer is destroyed and re-created could either be a problem with how we utilize these stores or most probably a bug in Svelte itself.

We need an open issue for this and try to replicate it on the Svelte REPL. If it's a svelte issue, we need to raise it on the svelte repo.

The hack:

I don't see the hack you've implemented as a hack. It's valid code that is required. The router was never at fault, we are manipulating the query params for our own custom logic. We would need to something similar even if we use SvelteKit, or basically any router.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be an issue with Svelte, or maybe it was intended but I don't see the behaviour mentioned in the docs. I opened sveltejs/svelte#7330 for this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seancolsen I think we can remove the comment which says this is a hacky way of doing things and explain why we are calling the function here.

When we have the time, it would be better if we had a call and I could explain in detail on why query params are manipulated imperatively. I think it is essential that you understand the reasoning behind it and it would be best if we document it all down in our wiki.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a SvelteKit static adapter but that'll essentially the same (if not less perfomant) than directly using Svelte with pre-rendered data as we do now.

During our sync call, I'm also curious to explore this point above more, because that definitely does not match not my understanding.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to understand from you (if you're able to explain it) why the routing behavior is inconsistent

The routing behaviour is not inconsistent, it works exactly as it is intended. The router is not at fault here. The upstream issue from svelte caused our url update method to get called which modified the query params.

Can you explain what exactly you find inconsistent?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what exactly you find inconsistent?

Before I added the call to beginUpdatingUrlWhenSchemaChanges, I was observing that sometimes Base.svelte would re-render its TabContainer child (causing the query params to update), and sometimes it would not. In the changes before this PR, it was about ~95% likely to re-render, which is why, most of the time, the query params seemed to update. But I absolutely observed it to not be updating sometimes. Then, with the changes in my PR, it was re-rendering maybe about ~5% of the time, which is what let me initially to believe that my PR had a bug which was not present in master. But (as we saw on our call, and as I later confirmed through more rigorous testing) the code in my PR would sometimes update the query params.

I suspect it has something to do with the fact that we're deciding whether to render TabContainer based on the value of $tabs, which is subject to the upstream bug. There may be some sort of race condition or inconsistent garbage collection or something there.

Copy link
Member

@pavish pavish Mar 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has to do with TabContainer being created and destroyed, since it is wrapped within an #if block. The upstream bug calls all subscribers of tabs and activeTab stores, each time TabContainer is created.

Assume you opened a table in Schema 1, which creates the TabContainer and TabList instance, which inturn updates the url. Then you open Schema 2, which contains no tables, so TabContainer gets destroyed. Note, we create a new TabList instance when a schema is opened for the first time, so the url update will happen when Schema 2 is opened.

When we switch back to Schema 1, the TabContainer gets created again, which updates the url. If you then open a table in Schema 2, switching between schemas do not update the url.

I assume in your local environment, you mostly tested by opening tables in both schemas, so you only noticed the url update happening ~5% of the time. If you hadn't you'd notice it behaves just like it did in the previous master.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Ok it sounds like there's a good chance I was not being observant enough of these small details while testing, and likely the behavior was consistent but my test scenarios were inconsistent. Thanks for explaining!

</script>

<ToastPresenter entries={toast.entries} />
Expand Down
60 changes: 45 additions & 15 deletions mathesar_ui/src/api/tables/records.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,51 @@
export interface Grouping {
/** Each string is a column name */
columns: string[];
mode: GroupingMode;
/**
* When `mode` === 'distinct', `num_groups` will always be `null`.
*
* When `mode` === 'percentile', `num_groups` will give the number of groups,
* as specified in the request params.
*/
num_groups: number | null;
ranged: boolean;
groups: Group[];
}

export type SortDirection = 'asc' | 'desc';
export interface SortingEntry {
/** column name */
field: string;
direction: SortDirection;
}
export type FilterCombination = 'and' | 'or';
export type FilterOperation = 'eq' | 'ne' | 'get_duplicates';
export interface FilterCondition {
/** column name */
field: string;
op: FilterOperation;
value: unknown;
}
type MakeFilteringOption<U> = U extends string
? { [k in U]: FilterCondition[] }
: never;
export type Filtering =
| FilterCondition[]
| MakeFilteringOption<FilterCombination>;

export interface GetRequestParams {
limit?: number;
offset?: number;
order_by?: SortingEntry[];
grouping?: Pick<Grouping, 'columns'>;
filters?: Filtering;
}

export type ResultValue = string | number | boolean | null;

export interface Result {
/** keys are column names */
[k: string]: ResultValue;
}

Expand Down Expand Up @@ -29,21 +74,6 @@ export interface Group {
result_indices: number[];
}

export interface Grouping {
/** Each string is a column name */
columns: string[];
mode: GroupingMode;
/**
* When `mode` === 'distinct', `num_groups` will always be `null`.
*
* When `mode` === 'percentile', `num_groups` will give the number of groups,
* as specified in the request params.
*/
num_groups: number | null;
ranged: boolean;
groups: Group[];
}

export interface Response {
count: number;
grouping: Grouping | null;
Expand Down
105 changes: 105 additions & 0 deletions mathesar_ui/src/component-library/common/utils/ImmutableMap.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
export default class ImmutableMap<
Key extends string | number | boolean | null,
Value,
> {
private map: Map<Key, Value>;

constructor(i: Iterable<[Key, Value]> = []) {
this.map = new Map(i);
}

/**
* This method exists to allow us to subclass this class and call the
* constructor of the subclass from within this base class.
*
* If there's a way we can use generics to avoid `any` here, we'd love to
* know.
*/
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private getNewInstance(...args: any[]): this {
// eslint-disable-next-line @typescript-eslint/no-unsafe-call, @typescript-eslint/no-explicit-any
return new (this.constructor as any)(...args) as this;
}

/**
* The value supplied here will overwrite any value that is already associated
* with `key`.
*/
with(key: Key, value: Value): this {
const map = new Map(this.map);
map.set(key, value);
return this.getNewInstance(map);
}

/**
* When the same keys exist in within the entries of this instance and the
* entries supplied, the values from the entries supplied will be used instead
* of the values in this instance. This behavior is consistent with the `with`
* method.
*/
withEntries(entries: Iterable<[Key, Value]>): this {
const map = new Map(this.map);
[...entries].forEach(([key, value]) => {
map.set(key, value);
});
return this.getNewInstance(map);
}

/**
* If `key` already exists, its corresponding value will remain. If `key` does
* not exist, then the value supplied here will be used.
*/
coalesce(key: Key, value: Value): this {
return this.has(key) ? this : this.with(key, value);
}

/**
* When the same keys exist in within the entries of this instance and the
* entries supplied, the values from this instance will be used instead of the
* values from the supplied entries. This behavior is consistent with the
* `coalesce` method.
*/
coalesceEntries(other: Iterable<[Key, Value]>): this {
const map = new Map(this.map);
[...other].forEach(([key, value]) => {
if (!this.has(key)) {
map.set(key, value);
}
});
return this.getNewInstance(map);
}

without(key: Key): this {
const map = new Map(this.map);
map.delete(key);
return this.getNewInstance(map);
}

has(key: Key): boolean {
return this.map.has(key);
}

get(key: Key): Value | undefined {
return this.map.get(key);
}

get size(): number {
return this.map.size;
}

keys(): IterableIterator<Key> {
return this.map.keys();
}

values(): IterableIterator<Value> {
return this.map.values();
}

entries(): IterableIterator<[Key, Value]> {
return this.map.entries();
}

[Symbol.iterator](): IterableIterator<[Key, Value]> {
return this.entries();
}
}
30 changes: 24 additions & 6 deletions mathesar_ui/src/component-library/common/utils/ImmutableSet.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,38 @@ export default class ImmutableSet<T extends string | number | boolean | null> {
this.set = new Set(i);
}

with(item: T): ImmutableSet<T> {
/**
* This method exists to allow us to subclass this class and call the
* constructor of the subclass from within this base class.
*
* If there's a way we can use generics to avoid `any` here, we'd love to
* know.
*/
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private getNewInstance(...args: any[]): this {
// eslint-disable-next-line @typescript-eslint/no-unsafe-call, @typescript-eslint/no-explicit-any
return new (this.constructor as any)(...args) as this;
}

with(item: T): this {
const set = new Set(this.set);
set.add(item);
return new ImmutableSet(set);
return this.getNewInstance(set);
}

union(other: ImmutableSet<T>): ImmutableSet<T> {
union(other: ImmutableSet<T>): this {
const set = new Set(this.set);
[...other.values()].forEach((value) => {
set.add(value);
});
return new ImmutableSet(set);
return this.getNewInstance(set);
}

without(item: T): ImmutableSet<T> {
without(item: T): this {
const set = new Set(this.set);
set.delete(item);
return new ImmutableSet(set);
//
return this.getNewInstance(set);
}

has(item: T): boolean {
Expand All @@ -40,4 +54,8 @@ export default class ImmutableSet<T extends string | number | boolean | null> {
valuesArray(): T[] {
return [...this.set.values()];
}

[Symbol.iterator](): IterableIterator<T> {
return this.values();
}
}
13 changes: 13 additions & 0 deletions mathesar_ui/src/component-library/common/utils/formatUtils.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { hasStringLabelProperty } from './typeUtils';

export function formatSize(sizeInBytes: number): string {
if (sizeInBytes === 0) {
return '0 B';
Expand All @@ -13,3 +15,14 @@ export function formatSize(sizeInBytes: number): string {

return `${repValue.toFixed(2)} ${repUnit}B`;
}

/**
* If the given value has a label property and it is a string, return it.
* Otherwise, return the value itself, converted to a string.
*/
export function getLabel(v: unknown): string {
if (hasStringLabelProperty(v)) {
return v.label;
}
return String(v);
}
1 change: 1 addition & 0 deletions mathesar_ui/src/component-library/common/utils/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
export { default as CancellablePromise } from './CancellablePromise';
export { default as EventHandler } from './EventHandler';
export { default as ImmutableSet } from './ImmutableSet';
export { default as ImmutableMap } from './ImmutableMap';

// Utility Functions
export * from './domUtils';
Expand Down
7 changes: 7 additions & 0 deletions mathesar_ui/src/component-library/common/utils/typeUtils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
function hasLabelProperty(v: unknown): v is { label: unknown } {
return typeof v === 'object' && v !== null && 'label' in v;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably go another step and make the 'label' key configurable. This will retain existing Select functionality as well as retain type safety. We can think about it in the upcoming Select improvements PR.

}

export function hasStringLabelProperty(v: unknown): v is { label: string } {
return hasLabelProperty(v) && typeof v.label === 'string';
}
1 change: 1 addition & 0 deletions mathesar_ui/src/component-library/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ export { default as DynamicInput } from './dynamic-input/DynamicInput.svelte';
export { default as FileUpload } from './file-upload/FileUpload.svelte';
export { default as Notification } from './notification/Notification.svelte';
export { default as Pagination } from './pagination/Pagination.svelte';
export { default as SimpleSelect } from './simple-select/SimpleSelect.svelte';
export { default as Select } from './select/Select.svelte';
export { default as TabContainer } from './tabs/TabContainer.svelte';
export { default as Tree } from './tree/Tree.svelte';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@
undefined;

// Total number of pages.
//
// TODO: @seancolsen says:
// > Refactor `pageCount` to no longer be an exported prop. We're exporting it
// > just so the parent component can access the calculation done within this
// > component. That's an unconventional flow of data. I'd rather do the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unconventional for other frameworks but the whole point of the bind keyword is to do exactly this. There are even dedicated readonly bind properties that Svelte provides. I'd say this is one of the intended ways of doing things in Svelte.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the whole point of the bind keyword is to do exactly this

I disagree. I would say the point of bind is to allow data to flow from child to parent. bind:value on an input element allows data that originates in the input element (from user input) to flow up to the parent. That's great.

This Pagination component is different. The data for pageCount originates in the parent component as total and pageSize. The pageCount value is simply a derivation of those two values, which Pagination never modifies. Thus the source-of-truth data here is being passed down. Then a derivation of that data is being passed back up. That's weird an unconventional. We have lots of bind usage in our codebase which is fine because the source of truth for the data actually originates in the child. But this is the only one I've seen which, I would say, abuses the bind pattern. Grappling with this weirdness costed me time during this refactor, which is why I suggested we clean it up.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point regarding bind keyword.

For the pageCount property however, I strongly feel that it should be the child component's property and not be passed down from the parent.

Even though pageSize and total are passed down, pageCount is still a property which is calculated within the child component inorder for it to work as expected. The source of truth for pageCount is where the calculation happens and not where it's dependencies are passed from.

We can remove the bind keyword here and dispatch an event when pageCount changes and use that to update any property on the parent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the bind keyword here and dispatch an event when pageCount changes and use that to update any property on the parent.

That would be problematic in the same way.

If I were designing this from scratch, I'd make a PaginationStats object like this:

class PaginationStats {
  readonly recordCount: number;
  readonly pageSize: number;
  readonly pageCount: number;

  constructor({
    recordCount,
    pageSize
  }: {
    recordCount: number,
    pageSize: number,
  }) {
    this.recordCount = recordCount;
    this.pageSize = pageSize;
    this.pageCount = Math.ceil(this.recordCount / this.pageSize);
  }
}

Then I'd make a stats prop on Pagination which accepts an instance of PaginationStats.

In our code currently, the StatusPane needs access to the pageCount. The only way it can get the pageCount is to render a Pagination component. That's deeply wierd to me. What if we want to take Pagination out of StatusPane but keep pageCount in StatusPane? Since StatusPane needs access to pageCount, pageCount should not be coupled to Pagination. The design above decouples pageCount from Pagination.

Copy link
Member

@pavish pavish Mar 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be problematic in the same way.

Not necessarily. pageCount would be decoupled since the parent's pageCount and the child's pageCount are different variables.

What if we want to take Pagination out of StatusPane but keep pageCount in StatusPane

We'd pass down pageCount to StatusPane.

The only way it can get the pageCount is to render a Pagination component.

It's the only current way to do it. If we absolutely need to calculate pageCount on the parent, we'd best expose a util method in the module context of the Pagination component. That way any file can do:
import { calculatePageCount } from 'Pagination.svelte'

If I were designing this from scratch, I'd make a PaginationStats object like this
Then I'd make a stats prop on Pagination which accepts an instance of PaginationStats.

This is a design pattern I'd go with only when it is absolutely essential.

It makes the Pagination component reliant on the PaginationStats object, which means that anyone wanting to use the Pagination component needs to create an instance of the PaginationStats class. Reactive updates of page and pageSize would always need creating a new object, leading to boilerplate code wherever we use this. It's one of the annoying patterns/aspects of React that made me like Svelte.

We can solve this simply by exporting some util methods, as I mentioned above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The utils pattern you suggest above is great too. It's certainly less code than my example. The calculation function would get called more times than in my proposal, but that's fine since it's not a perf concern here. I'll update my code comment to point to this conversation, and recommend the utils function approach.

// > calculation in the parent and pass it down to this component.
export let pageCount = 0;

// ARIA Label for component
Expand Down
8 changes: 8 additions & 0 deletions mathesar_ui/src/component-library/select/Select.svelte
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
<!--
@component

**NOTICE** This component will eventually be redesigned, with its props
behaving more like `SimpleSelect`.

https://github.com/centerofci/mathesar/issues/1099
-->
<script lang="ts">
import { createEventDispatcher, tick } from 'svelte';
import { Dropdown } from '@mathesar-component-library';
Expand Down
Loading