feat: handle nested arrays with wildcard keys #120

mart-jansink · 2020-12-19T19:51:33Z

What: Handle nested arrays with one or more wildcard keys. As an example

const nestedObjList = [
  {aliases: [{name: {first: 'Janice'}},{name: {first: 'Jen'}}]},
  {aliases: [{name: {first: 'Fred'}},{name: {first: 'Frederic'}}]},
  {aliases: [{name: {first: 'George'}},{name: {first: 'Georgie'}}]},
]
matchSorter(nestedObjList, 'jen', {keys: ['aliases.*.name.first']})
// [{aliases: [{name: {first: 'Janice'}},{name: {first: 'Jen'}}]}]
matchSorter(nestedObjList, 'jen', {keys: ['aliases.name.first']})
// [{aliases: [{name: {first: 'Janice'}},{name: {first: 'Jen'}}]}]
matchSorter(nestedObjList, 'jen', {keys: ['aliases.0.name.first']})
// []

Why: Arrays like the above are relatively common in some applications that I'm working on. I really like the intuitive sorting of match-sorter so I'd love to be able to use this. However, using property callbacks for such a common case isn't ideal. So I thought I'd try making a pull request, hoping that this is something you might consider for merging.

How: The .reduce callback inside the getNestedValue function has gotten a lot more logic to possibly handle arrays. The calling getItemValues function now always returns an array, which simplifies the logic that returns the value a bit. Both the explicit 'aliases.*.name.first' and implicit 'aliases.name.first' wildcards are supported with this implementation; it was simpler to implement the latter while the former is clearer in use.

Checklist:

Documentation
Tests
Ready to be merged

I sadly had some trouble running npm run test:update: it complains that not all branches are covered by the test cases. However, the lines that it says are uncovered are actually called, as far as I can tell, so I couldn't fix this..

kentcdodds

Thanks for your work on this. I'm worried that this is adding a little too much magic to the library, especially when you can already work around this limitation in a pretty reasonable way using "Property Callbacks" (see the README).

mart-jansink · 2020-12-21T11:30:08Z

Thanks for your feedback. I understand the concern, but to me it seems that this library is already doing quite a bit of "magic" when it comes to sorting the results, and supporting nested keys in the first place. Of course that "magic" is well documented but so can this.

Since the applications I'm working on use the pattern of nested objects in arrays relatively often, writing similar property callbacks all the time isn't really DRY nor as performant as this can be. But if I'm the outlier in this, I'm more than happy to keep using a fork.

kentcdodds · 2020-12-22T21:44:14Z

Fair points. Luckily the added code isn't all that complex which is nice. I think this will be ok, but I'd like to see what this looks like documented first. Also, could you make sure we hit 100% test coverage with your new changes. Thanks!

codecov · 2020-12-25T22:06:24Z

Codecov Report

Merging #120 (d91842f) into master (4c2e927) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #120   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines          137       163   +26     
  Branches        31        38    +7     
=========================================
+ Hits           137       163   +26

Impacted Files	Coverage Δ
src/index.ts	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c2e927...d91842f. Read the comment docs.

mart-jansink · 2020-12-25T22:13:45Z

Thanks. The coverage is now back up to 100% again. And I've documented the changes more clearly.

While doing this I also found a few simple performance improvements that end up to matching about 20% faster. The biggest improvements were

not .concatenating arrays anymore, since they are never modified anyway there's no harm in using any given arrays directly
caching the length of the strings that are being for-looped over

I can't directly share the data that I used for benchmarking this, but I can look for something similar if you want?

kentcdodds · 2020-12-26T20:02:24Z

src/index.ts

+        return value
+      }
+
+      return value.reduce((values: Array<object | string>, arrayValue: object | null):


This looks a lot like the first part of the function. I wonder if we could use recursion instead of duplication here. Could you try it out and see whether it's any simpler?

Sure, see the last commit which is my attempt at doing so. It seems possible without any performance implications.

However, there is the downside that the implicit wildcard no longer works, except if the array is a top-level property of the object. So {favoriteIceCream: ['mint', 'chocolate']} is fine when matching 'favoriteIceCream', but {favorite: {iceCream: ['mint', 'chocolate']}} won't work with 'favorite.iceCream' and instead needs 'favorite.iceCream.*'. I personally don't care about implicit wildcards: I never added this intentionally. It just worked due to the checks that made this new feature backwards compatible with all tests. And it's these checks that caused the duplication and are now gone. Supporting nested array values without explicit wildcards again would require more duplication I'm afraid.

The README doesn't explicitly say that nested array values are supported. It says that array values are supported and nested values are supported, but nothing about combining that. But as a user I'd assume that they would be, so now this new feature would be a breaking change.

kentcdodds

I think we should be able to make this work with recursion. Care to take another whack at it?

kentcdodds · 2020-12-27T18:14:05Z

src/__tests__/index.ts

+        {favorite: {iceCream: ['birthday cake', 'rocky road', 'strawberry']}},
+      ],
+      'cc',
+      {keys: ['favorite.iceCream.*']},


I'm fine with this working, but I'd also like to support favorite.iceCream here as well. We should be able to process an item of String | Array<String> and favorite.icCream is Array<String>.

kentcdodds · 2020-12-27T18:14:31Z

src/__tests__/index.ts

-        null,
-      ],
-      'ba',
-      {keys: ['aliases.name.first']},


I don't want to support this. It's too much magic. So I'm glad that it's gone.

mart-jansink · 2020-12-27T18:22:26Z

Yeah, I'll have another go at it.

mart-jansink · 2020-12-27T19:47:13Z

How do you feel about the use of .flat()? That could solve the problem but it isn't supported by IE, see https://caniuse.com/mdn-javascript_builtins_array_flat.

kentcdodds · 2020-12-27T22:17:41Z

I'm fine with flat. Folks should have standard polyfills by now

mart-jansink · 2020-12-28T10:12:41Z

tsconfig.json

-  "extends": "./node_modules/kcd-scripts/shared-tsconfig.json"
+  "extends": "./node_modules/kcd-scripts/shared-tsconfig.json",
+  "compilerOptions": {
+    "lib": ["es2019"]


I'm not sure if this is the best way / place / solution, but without this addition type checking fails with the error Property 'flat' does not exist on type 'string[]'. Do you need to change your target library? Try changing the lib compiler option to 'es2019' or later..

mart-jansink · 2020-12-28T10:19:56Z

OK, it now supports 'favorite.iceCream' again. The validation fails because node v10 doesn't have .flat() yet. We can also use a small polyfill like [].concat.apply([],values) instead?

kentcdodds · 2020-12-28T14:18:11Z

Let's use the small polyfill idea 👍

mart-jansink · 2020-12-28T15:16:38Z

There you go. I'm kinda stumped by the typing of the result, so I had to expect a few more errors than I'd like. Also, I've added a check to see if flattening is needed because it's quite expensive for the performance.

kentcdodds

I made a couple commits to hopefully improve things a bit. I think it'll help perf, but I'm pretty sure removing the reduces and using for..of loops improves readability.

I also removed all // @ts-expect-errors in favor of casting which I think is the lesser of the two evils.

I'd love your feedback. I'm happy to make adjustments to my changes before we merge this.

Thanks again for all your work on this!

mart-jansink · 2020-12-31T11:13:34Z

That's perfect, it's much more readable indeed. I've never been a big fan of functional programming because it typically comes with a large performance penalty. But in this case I just stuck with how you had coded it.

Looking at the performance there was some degradation due to the use of for-of loops which are transpiled to support arbitrary iterators besides plain arrays. So I've refactored that once more using old-fashioned for loops, also including those in the getAllValuesToRank function.

My basic benchmark now gives the following results, comparing the last variations this branch to the master:

master x 409 ops/sec ±0.64% (86 runs sampled)
reduce x 448 ops/sec ±1.75% (83 runs sampled)
for-of x 424 ops/sec ±0.48% (85 runs sampled)
for    x 475 ops/sec ±0.34% (87 runs sampled)

So I'm perfectly happy with merging it like this.

On a side note: I can make pull request for a benchmark like this if you'd want? And, looking at my use-case of this module, I see room for more performance improvements by doing some memoization. My apps show 1000+ rows that are recursively processed multiple times while users type, or when just a few of those rows change due to server updates. Memoizing getAllValuesToRank, getHighestRanking and / or getMatchRanking would greatly reduce the amount of duplicate work that's done.

kentcdodds

Looks great to me. Thank you!

If you'd like to add a benchmark test in the future that would be welcome 👍

kentcdodds · 2020-12-31T20:30:49Z

@all-contributors please add @mart-jansink for code, tests, and docs

allcontributors · 2020-12-31T20:30:59Z

@kentcdodds

I've put up a pull request to add @mart-jansink! 🎉

kentcdodds · 2020-12-31T20:32:07Z

Thanks so much for your help! I've added you as a collaborator on the project. Please make sure that you review the other/MAINTAINING.md and CONTRIBUTING.md files (specifically the bit about the commit messages and the git hooks) and familiarize yourself with the code of conduct (we're using the contributor covenant). You might also want to watch the repo to be notified when someone files an issue/PR. Please continue to make PRs as you feel the need (you can make your branches directly on the repo rather than your fork if you want). Thanks! And welcome to the team :)

github-actions · 2020-12-31T20:34:38Z

🎉 This PR is included in version 6.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

kentcdodds reviewed Dec 20, 2020

View reviewed changes

mart-jansink force-pushed the pr/handle-nested-arrays-with-wildcard-keys branch 2 times, most recently from 2b5a576 to eb19d72 Compare December 25, 2020 21:41

mart-jansink added 4 commits December 25, 2020 22:43

feat: handle nested arrays with wildcard keys

6bd72aa

perf: only get the key attributes once

af923ff

perf: handle trivial cases directly

ac1d4ae

perf: cache the length of the string that's iterated over

0195d3e

mart-jansink force-pushed the pr/handle-nested-arrays-with-wildcard-keys branch from eb19d72 to 0195d3e Compare December 25, 2020 21:43

fix: type errors and warnings

6f827da

kentcdodds reviewed Dec 26, 2020

View reviewed changes

fix: recurse instead of duplicate, forgoing implicit wildcards

1db161b

kentcdodds reviewed Dec 27, 2020

View reviewed changes

mart-jansink commented Dec 28, 2020

View reviewed changes

fix: implicitly handle nested keys that are an array of values again

895c231

mart-jansink force-pushed the pr/handle-nested-arrays-with-wildcard-keys branch from 3030099 to 895c231 Compare December 28, 2020 15:13

kentcdodds added 2 commits December 29, 2020 16:17

refactor: remove reduces

54397ca

refactor: remove ts-expect-errors

356b2a5

kentcdodds approved these changes Dec 29, 2020

View reviewed changes

refactor: forgo for-of loops in favor of performance

d91842f

kentcdodds approved these changes Dec 31, 2020

View reviewed changes

kentcdodds merged commit b3d9d8d into kentcdodds:master Dec 31, 2020

allcontributors bot mentioned this pull request Dec 31, 2020

docs: add mart-jansink as a contributor #121

Merged

github-actions bot added the released label Dec 31, 2020

dependabot bot mentioned this pull request Mar 11, 2021

Bump match-sorter from 4.0.2 to 6.3.0 iicdii/strapi#15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: handle nested arrays with wildcard keys #120

feat: handle nested arrays with wildcard keys #120

mart-jansink commented Dec 19, 2020

kentcdodds left a comment

mart-jansink commented Dec 21, 2020

kentcdodds commented Dec 22, 2020

codecov bot commented Dec 25, 2020 •

edited

Loading

mart-jansink commented Dec 25, 2020

kentcdodds Dec 26, 2020

mart-jansink Dec 27, 2020 •

edited

Loading

kentcdodds left a comment

kentcdodds Dec 27, 2020

kentcdodds Dec 27, 2020

mart-jansink Dec 27, 2020

mart-jansink commented Dec 27, 2020

mart-jansink commented Dec 27, 2020 •

edited

Loading

kentcdodds commented Dec 27, 2020

mart-jansink Dec 28, 2020

mart-jansink commented Dec 28, 2020

kentcdodds commented Dec 28, 2020

mart-jansink commented Dec 28, 2020

kentcdodds left a comment

mart-jansink commented Dec 31, 2020

kentcdodds left a comment

kentcdodds commented Dec 31, 2020

allcontributors bot commented Dec 31, 2020

kentcdodds commented Dec 31, 2020

github-actions bot commented Dec 31, 2020

feat: handle nested arrays with wildcard keys #120

feat: handle nested arrays with wildcard keys #120

Conversation

mart-jansink commented Dec 19, 2020

kentcdodds left a comment

Choose a reason for hiding this comment

mart-jansink commented Dec 21, 2020

kentcdodds commented Dec 22, 2020

codecov bot commented Dec 25, 2020 • edited Loading

Codecov Report

mart-jansink commented Dec 25, 2020

kentcdodds Dec 26, 2020

Choose a reason for hiding this comment

mart-jansink Dec 27, 2020 • edited Loading

Choose a reason for hiding this comment

kentcdodds left a comment

Choose a reason for hiding this comment

kentcdodds Dec 27, 2020

Choose a reason for hiding this comment

kentcdodds Dec 27, 2020

Choose a reason for hiding this comment

mart-jansink Dec 27, 2020

Choose a reason for hiding this comment

mart-jansink commented Dec 27, 2020

mart-jansink commented Dec 27, 2020 • edited Loading

kentcdodds commented Dec 27, 2020

mart-jansink Dec 28, 2020

Choose a reason for hiding this comment

mart-jansink commented Dec 28, 2020

kentcdodds commented Dec 28, 2020

mart-jansink commented Dec 28, 2020

kentcdodds left a comment

Choose a reason for hiding this comment

mart-jansink commented Dec 31, 2020

kentcdodds left a comment

Choose a reason for hiding this comment

kentcdodds commented Dec 31, 2020

allcontributors bot commented Dec 31, 2020

kentcdodds commented Dec 31, 2020

github-actions bot commented Dec 31, 2020

codecov bot commented Dec 25, 2020 •

edited

Loading

mart-jansink Dec 27, 2020 •

edited

Loading

mart-jansink commented Dec 27, 2020 •

edited

Loading