feat: option to improve Date Sorting by pre-parsing date items only o…

…nce (#1268)

* feat: option to improve Date Sorting by pre-parsing date items only once

docs/column-functionalities/sorting.md

@@ -4,6 +4,7 @@
 - [Custom Sort Comparer](#custom-sort-comparer)
 - [Update Sorting Dynamically](#update-sorting-dynamically)
 - [Dynamic Query Field](#dynamic-query-field)
+- [Pre-Parse Date Columns for better perf](#pre-parse-date-columns-for-better-perf)
 
 ### Demo
 [Demo Page](https://ghiscoding.github.io/aurelia-slickgrid/#/slickgrid/example4) / [Demo ViewModel](https://github.com/ghiscoding/aurelia-slickgrid/blob/master/packages/demo/src/examples/slickgrid/example4.ts)
@@ -137,3 +138,59 @@ queryFieldNameGetterFn: (dataContext) => {
   return dataContext.profit > 0 ? 'profitRatio' : 'lossRatio';
 },
 ```
+
+### Pre-Parse Date Columns for better perf
+##### requires v5.8.0 and higher
+
+Sorting very large dataset with dates can be extremely slow when dates formated date strings, the reason is because these strings need to first be parsed and converted to real JS Dates before the Sorting process can actually happen (i.e. US Date Format). However parsing a large dataset can be slow **and** to make it worst, a Sort will revisit the same items over and over which mean that the same date strings will have to be reparsed over and over (for example while trying to Sort a dataset of 100 items, I saw some items being revisit 10 times and I can only imagine that it is exponentially worst with a large dataset).
+
+So what can we do to make this faster with a more reasonable time? Well, we can simply pre-parse all date strings once and only once and convert them to JS Date objects. Then once we get Date objects, we'll simply read the UNIX timestamp which is what we need to Sort. The first pre-parse takes a bit of time and will be executed only on the first date column Sort (any sort afterward will read the pre-parsed Date objects).
+
+What perf do we get with pre-parsing versus regular non-parsing? The benchmark was pulled using 50K items with 2 date columns (with US date format)
+- without non-parsing: ~15sec
+- with pre-parsing: ~1.4sec (1st pre-parse) and any subsequent Date sort is about ~0.2sec => so about ~1.5sec total
+
+The summary, is that we get a 10x boost **but** not only that, we also get an extremely fast subsequent sort afterward (sorting Date objects is as fast as sorting Numbers).
+
+#### Usage
+
+You can use the `preParseDateColumns` grid option, it can be either set as either `boolean` or a `string` but there's big distinction between the 2 approaches (both approaches will mutate the dataset).
+1. `string` (i.e. set to `"__"`, it will parse a `"start"` date string and assign it as a `Date` object to a new `"__start"` prop)
+2. `boolean` (i.e. parse `"start"` date string and reassign it as a `Date` object on the same `"start"` prop)
+
+> **Note** this option **does not work** with Backend Services because it simply has no effect.
+
+For example if our dataset has 2 columns named "start" and "finish", then pre-parse the dataset,
+
+with the 1nd approach (`string`), let's use `"__"` (which is in reality a prefix) it will mutate the dataset by adding new props (where `Date` is a `Date` object)
+
+```diff
+data = [
+-  { id: 0, start: '02/28/24', finish: '03/02/24' },
+-  { id: 1, start: '01/14/24', finish: '02/13/24' },
++  { id: 0, start: '02/28/24', finish: '03/02/24', __start: Date, __finish: Date },
++  { id: 1, start: '01/14/24', finish: '02/13/24', __start: Date, __finish: Date },
+]
+```
+
+with the 2nd approach (`boolean`), it will instead mutate the dataset by overwriting the same properties
+
+```diff
+data = [
+-  { id: 0, start: '02/28/24', finish: '03/02/24' },
+-  { id: 1, start: '01/14/24', finish: '02/13/24' },
++  { id: 0, start: Date, finish: Date },
++  { id: 1, start: Date, finish: Date },
+]
+```
+
+Which approach to choose? Both have pros and cons, overwriting the same props might cause problems with the column `type` that you use, you will have to give it a try yoursel. On the other hand, with the other approach, it will duplicate all date properties and take a bit more memory usage and when changing cells we'll need to make sure to keep these props in sync, however you will likely have less `type` issues.
+
+What happens when we do any cell changes (for our use case, it would be Create/Update), for any Editors we simply subscribe to the `onCellChange` change event and we re-parse the date strings when detected. We also subscribe to certain CRUD functions as long as they come from the `GridService` then all is fine... However, if you use the DataView functions directly then we have no way of knowing when to parse because these functions from the DataView don't have any events. Lastly, if we overwrite the entire dataset, we will also detect this (via an internal flag) and the next time you sort a date then the pre-parse kicks in again.
+
+#### Can I call the pre-parse myself?
+
+Yes, if for example you want to pre-parse right after the grid is loaded, you could call the pre-parse yourself for either all items or a single item
+- all item pre-parsing: `this.sgb.sortService.preParseAllDateItems();`
+  - the items will be read directly from the DataView
+- a single item parsing: `this.sgb.sortService.preParseSingleDateItem(item);`

packages/aurelia-slickgrid/package.json

@@ -53,13 +53,13 @@
     "@aurelia/runtime": "^2.0.0-beta.21",
     "@aurelia/runtime-html": "^2.0.0-beta.21",
     "@formkit/tempo": "^0.1.2",
-    "@slickgrid-universal/common": "~5.7.0",
-    "@slickgrid-universal/custom-footer-component": "~5.7.0",
-    "@slickgrid-universal/empty-warning-component": "~5.7.0",
-    "@slickgrid-universal/event-pub-sub": "~5.7.0",
-    "@slickgrid-universal/pagination-component": "~5.7.0",
-    "@slickgrid-universal/row-detail-view-plugin": "~5.7.0",
-    "@slickgrid-universal/utils": "~5.7.0",
+    "@slickgrid-universal/common": "~5.8.0",
+    "@slickgrid-universal/custom-footer-component": "~5.8.0",
+    "@slickgrid-universal/empty-warning-component": "~5.8.0",
+    "@slickgrid-universal/event-pub-sub": "~5.8.0",
+    "@slickgrid-universal/pagination-component": "~5.8.0",
+    "@slickgrid-universal/row-detail-view-plugin": "~5.8.0",
+    "@slickgrid-universal/utils": "~5.8.0",
     "dequal": "^2.0.3",
     "sortablejs": "^1.15.3"
   },

packages/aurelia-slickgrid/src/custom-elements/aurelia-slickgrid.ts

@@ -33,6 +33,7 @@ import {
   GridStateService,
   GridStateType,
   GroupingAndColspanService,
+  isColumnDateType,
   type Observable,
   PaginationService,
   ResizerService,
@@ -62,6 +63,8 @@ import type { AureliaGridInstance, GridOption } from '../models/index';
 import { AureliaUtilService, ContainerService, disposeAllSubscriptions, TranslaterService } from '../services/index';
 import { SlickRowDetailView } from '../extensions/slickRowDetailView';
 
+const WARN_NO_PREPARSE_DATE_SIZE = 5000; // data size to warn user when pre-parse isn't enabled
+
 @customElement({
   name: 'aurelia-slickgrid',
   template: `
@@ -173,7 +176,7 @@ export class AureliaSlickgridCustomElement {
     this.filterFactory = new FilterFactory(slickgridConfig, this.translaterService, this.collectionService);
     this.filterService = new FilterService(this.filterFactory as any, this._eventPubSubService, this.sharedService, this.backendUtilityService);
     this.resizerService = new ResizerService(this._eventPubSubService);
-    this.sortService = new SortService(this.sharedService, this._eventPubSubService, this.backendUtilityService);
+    this.sortService = new SortService(this.collectionService, this.sharedService, this._eventPubSubService, this.backendUtilityService);
     this.treeDataService = new TreeDataService(this._eventPubSubService, this.sharedService, this.sortService);
     this.paginationService = new PaginationService(this._eventPubSubService, this.sharedService, this.backendUtilityService);
 
@@ -268,6 +271,8 @@ export class AureliaSlickgridCustomElement {
     if (this.gridOptions.darkMode) {
       this.setDarkMode(true);
     }
+
+    this.suggestDateParsingWhenHelpful();
   }
 
   initialization(eventHandler: SlickEventHandler) {
@@ -612,6 +617,8 @@ export class AureliaSlickgridCustomElement {
       this.grid.autosizeColumns();
       this._isAutosizeColsCalled = true;
     }
+
+    this.suggestDateParsingWhenHelpful();
   }
 
   datasetHierarchicalChanged(newHierarchicalDataset: any[] | undefined) {
@@ -745,6 +752,7 @@ export class AureliaSlickgridCustomElement {
           this.handleOnItemCountChanged(dataView.getFilteredItemCount() || 0, dataView.getItemCount() || 0);
         });
         this._eventHandler.subscribe(dataView.onSetItemsCalled, (_e, args) => {
+          this.sharedService.isItemsDateParsed = false;
           this.handleOnItemCountChanged(dataView.getFilteredItemCount() || 0, args.itemCount);
 
           // when user has resize by content enabled, we'll force a full width calculation since we change our entire dataset
@@ -1536,6 +1544,15 @@ export class AureliaSlickgridCustomElement {
     });
   }
 
+  protected suggestDateParsingWhenHelpful() {
+    if (this.dataview?.getItemCount() > WARN_NO_PREPARSE_DATE_SIZE && !this.gridOptions.preParseDateColumns && this.grid.getColumns().some(c => isColumnDateType(c.type))) {
+      console.warn(
+        '[Slickgrid-Universal] For getting better perf, we suggest you enable the `preParseDateColumns` grid option, ' +
+        'for more info visit:: https://ghiscoding.gitbook.io/slickgrid-universal/column-functionalities/sorting#pre-parse-date-columns-for-better-perf'
+      );
+    }
+  }
+
   /**
    * When the Editor(s) has a "editor.collection" property, we'll load the async collection.
    * Since this is called after the async call resolves, the pointer will not be the same as the "column" argument passed.

packages/aurelia-slickgrid/src/extensions/__tests__/slickRowDetailView.spec.ts

@@ -1,5 +1,5 @@
 import { customElement } from 'aurelia';
-import { type Column, type RowDetailViewOption, SharedService, SlickEvent, SlickEventData, SlickEventHandler, type SlickGrid, SlickRowSelectionModel } from '@slickgrid-universal/common';
+import { type Column, type RowDetailViewOption, SlickEvent, SlickEventData, SlickEventHandler, type SlickGrid, SlickRowSelectionModel } from '@slickgrid-universal/common';
 import { EventPubSubService } from '@slickgrid-universal/event-pub-sub';
 
 import type { GridOption } from '../../models/gridOption.interface';
@@ -175,7 +175,6 @@ describe('SlickRowDetailView', () => {
       gridOptionsMock.rowDetailView!.preloadViewModel = ExamplePreload;
       gridOptionsMock.rowDetailView!.viewModel = ExampleLoader;
       columnsMock = [{ id: 'field1', field: 'field1', width: 100, cssClass: 'red' }];
-      jest.spyOn(SharedService.prototype, 'slickGrid', 'get').mockReturnValue(gridStub);
       jest.spyOn(gridStub, 'getOptions').mockReturnValue(gridOptionsMock);
       jest.clearAllMocks();
       gridStub.onColumnsReordered = new SlickEvent();

packages/demo/package.json

@@ -43,15 +43,15 @@
     "@fnando/sparkline": "^0.3.10",
     "@formkit/tempo": "^0.1.2",
     "@popperjs/core": "^2.11.8",
-    "@slickgrid-universal/common": "^5.7.0",
-    "@slickgrid-universal/composite-editor-component": "^5.7.0",
-    "@slickgrid-universal/custom-tooltip-plugin": "^5.7.0",
-    "@slickgrid-universal/excel-export": "^5.7.0",
-    "@slickgrid-universal/graphql": "^5.7.0",
-    "@slickgrid-universal/odata": "^5.7.0",
-    "@slickgrid-universal/row-detail-view-plugin": "^5.7.0",
-    "@slickgrid-universal/rxjs-observable": "^5.7.0",
-    "@slickgrid-universal/text-export": "^5.7.0",
+    "@slickgrid-universal/common": "^5.8.0",
+    "@slickgrid-universal/composite-editor-component": "^5.8.0",
+    "@slickgrid-universal/custom-tooltip-plugin": "^5.8.0",
+    "@slickgrid-universal/excel-export": "^5.8.0",
+    "@slickgrid-universal/graphql": "^5.8.0",
+    "@slickgrid-universal/odata": "^5.8.0",
+    "@slickgrid-universal/row-detail-view-plugin": "^5.8.0",
+    "@slickgrid-universal/rxjs-observable": "^5.8.0",
+    "@slickgrid-universal/text-export": "^5.8.0",
     "aurelia": "^2.0.0-beta.21",
     "aurelia-slickgrid": "workspace:*",
     "bootstrap": "^5.3.3",

packages/demo/src/examples/slickgrid/example4.html

@@ -42,6 +42,9 @@ <h2>
         click.trigger="setSortingDynamically()">
   Set Sorting Dynamically
 </button>
+<button class="btn btn-outline-secondary btn-sm btn-icon" click.trigger="logItems()">
+  <span title="console.log all dataset items">Log Items</span>
+</button>
 
 <aurelia-slickgrid grid-id="grid4"
                     column-definitions.bind="columnDefinitions"

packages/demo/src/examples/slickgrid/example4.ts

@@ -20,7 +20,7 @@ import {
 function randomBetween(min: number, max: number) {
   return Math.floor(Math.random() * (max - min + 1) + min);
 }
-const NB_ITEMS = 1500;
+const NB_ITEMS = 5500;
 const URL_SAMPLE_COLLECTION_DATA = 'assets/data/collection_500_numbers.json';
 
 export class Example4 {
@@ -222,9 +222,14 @@ export class Example4 {
         ],
       },
       externalResources: [new ExcelExportService()],
+      preParseDateColumns: '__' // or true
     };
   }
 
+  logItems() {
+    console.log(this.aureliaGrid.dataView?.getItems());
+  }
+
   mockData(itemCount: number, startingIndex = 0): any[] {
     // mock a dataset
     const tempDataset: any[] = [];