feat: 🔥 First punt at priority option #223

theryansmee · 2022-03-16T17:47:11Z

Allow user to set length priority to length or unique (POC)

✅ Closes: #220

PR Checklist

Please check if your PR fulfills the following requirements:

The commit message follows our guidelines: https://github.com/ngneat/falso/blob/master/CONTRIBUTING.md#commit
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: N/A

What is the new behavior?

Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

Allow user to set length priority to length or unique (POC) ✅ Closes: #220

theryansmee · 2022-03-16T17:48:18Z

@NetanelBasal @shaharkazaz - This is a super sloppy implementation that I threw together at lunch today, what are your thoughts?

NetanelBasal · 2022-03-21T08:03:52Z

Thanks @theryansmee, @shaharkazaz will go over it asap.

shaharkazaz

🚀

packages/falso/src/lib/core/core.ts

Move randUniqueElement logic into fakFromArray. Simplified fakeFromFunction while loop logic #220

theryansmee · 2022-03-22T18:17:56Z

🚀

#223 (comment):
Yeah you are right. I'll have a think about how we can compare complex objects. With most of our entities be happy to consider it uniques as long as the id is unique.. but i'll have to look through our functions to see what other challenges we face.

I've fixed the other bits

✅ Closes: #220

packages/falso/src/lib/core/core.ts

Add default comparison functions for most likely use cases. Implimented them ✅ Closes: #220

✅ Closes: #220

theryansmee · 2022-03-25T18:12:07Z

@shaharkazaz @NetanelBasal I've still got to write tests and clean up a few bits, but I think this is mostly there now. What are your thoughts? 😊

shaharkazaz · 2022-03-28T06:23:27Z

@theryansmee I'll re-review it again today 👍

packages/falso/src/lib/core/core.ts

Add comparison function and start change over to using it ✅ Closes: #220

✅ Closes: #220

theryansmee · 2022-04-04T19:36:38Z

@shaharkazaz I've ended up creating objectIsUnique which is then used by the other checker functions around the library. Originally, I was hoping to just have a single function that got passed around by I ended up in type hell.

That said, I think this is actually quite an elegant solution.

PS
I think once we are happy with this MR, we should do some follow up work on splitting core.ts into multiple files and ensuring that it has a decent test coverage.

PPS
I'm not sure why the build it failing. Looks like we have an issue with packages somewhere down the line.

Fix snapshot error. Ensure all thrown errors throw actual errors rather than strings ✅ Closes: #220

Add missing priority doc comment ✅ Closes: #220

theryansmee · 2022-06-01T07:47:19Z

🙃 Just seen theres been commits to core.ts so i'm going to have to look at what they are and get them integrated into my work

NetanelBasal · 2022-06-01T07:50:08Z

Cool, thanks!

theryansmee · 2022-06-07T08:33:56Z

I will be fixing the conflicts and merging this later this week :)

# Conflicts: # packages/falso/src/lib/core/core.ts # packages/falso/src/lib/text.ts

✅ Closes: #220

theryansmee · 2022-06-13T12:19:32Z

@NetanelBasal - I've merged in the latest changes but now the build is failing on the setup-node@v2 step with getCacheEntry failed: Cache service responded with 503.

Any thoughts on what might be happening?

NetanelBasal · 2022-06-13T13:57:58Z

Nopehttps://stackoverflow.com/questions/72598852/getcacheentry-failed-cache-service-responded-with-503

# Conflicts: # packages/falso/src/lib/number.ts # packages/falso/src/lib/octal.ts

✅ Closes: 220

Add tests for simple array and function scenarios. Fix typo in unique checker logic ✅ Closes: 220

theryansmee · 2022-09-12T19:53:43Z

@NetanelBasal @shaharkazaz - Hi guys, sorry for the radio silence from me (Life got a little hectic). I have fixed the merge conflicts and added a few essential tests to prove priority logic works as expected with data from arrays and from functions.

I think it's ready to be reviewed again 😊

shaharkazaz · 2022-09-14T15:24:31Z

@theryansmee Hi Ryan always a pleasure to hear from you 😁
I'll re-review it today-tomorrow.

theryansmee · 2022-09-14T18:25:13Z

@theryansmee Hi Ryan always a pleasure to hear from you 😁 I'll re-review it today-tomorrow.

I'm back for good this time! haha Cheers mate 😊

shaharkazaz

This is the first cycle, I'll go in-depth over the core files, these are the first things that I saw

packages/falso/src/lib/address.ts

shaharkazaz · 2022-09-15T09:27:39Z

packages/falso/src/lib/core/core.ts

+  uniqueComparer: (
+    item: T,
+    items: T[],
+    comparisonKeys?: string[]


In the case of an object, you always pass comparisonKeys right?
If so, let's make this type stricter, and the values should also be a sub-set of T's keys.

If i'm honest I was having a little trouble with the types for this. I had a lot of issues when I tried to make this not optional. I have changed the comparisonKeys type from string[] to (keyof T)[] though

packages/falso/src/lib/core/unique-validators.ts

shaharkazaz · 2022-09-16T10:02:57Z

packages/falso/src/lib/core/unique-validators.ts

+) => boolean = (item, items) => objectIsUnique(item, items, ['id']);
+
+export const objectIsUnique: (
+  item: any,


Why any and not a generic?

Yeah I wasn't a fan of this either. On further inspection i've managed to change it to objectIsUnique<T extends Record<string, any>>. I think this should be ok for now

packages/falso/src/lib/credit-card.ts

packages/falso/src/lib/flight-details.ts

shaharkazaz · 2022-09-16T10:08:02Z

packages/falso/src/lib/json.ts

@@ -88,5 +96,10 @@ export function randJSON<Options extends RandomJSONOptions = never>(
    return generatedObject;
  };

-  return fake(factory, options);
+  return fake(factory, options, { uniqueComparer: checkUnique });


shaharkazaz · 2022-09-16T10:08:35Z

packages/falso/src/lib/nearby-gpscoordinate.ts

 }
+
+const checkUnique: (


shaharkazaz · 2022-09-16T10:16:49Z

packages/falso/src/tests/core/object-with-id-is-unique.spec.ts

+  });
+
+  describe('object id is unique', () => {
+    it('should return true', () => {


I agree with Netanel's previous comment about the naming.
Maybe something like:
it(Should pass when object ids are unique) instead of the describe. This goes to all the specs I've seen in the recent files.
@NetanelBasal thoughts?

theryansmee · 2022-09-16T10:34:06Z

This is the first cycle, I'll go in-depth over the core files, these are the first things that I saw

Cheers mate, i'll have a look through and get these sorted tomorrow 👍

Change comparison of valueOf to getTime with dateIsUnique. Combine describe block and it labels for comparison function tests ✅ Closes: 220

Change checkUnique arrow functions to function declorations. Change unique-validators from inline arrow functions to actual function. Make function more type safe ✅ Closes: 220

Strengthen type checking. Remove redundant error and test ✅ Closes: 220

Strengthen type checking for comparisonKeys ✅ Closes: 220

theryansmee · 2022-09-18T15:20:06Z

@shaharkazaz - Hello mate, I have made all of the requested changes and left a few comments 😊

packages/falso/src/lib/core/core.ts

shaharkazaz · 2022-09-19T11:00:35Z

packages/falso/src/lib/core/core.ts

+  config: FakeConfig<T>,
+  options?: Options
+): T | T[] {
+  if (!options?.length) {


You are handling the length options both here and in the fake function, please choose one place

shaharkazaz · 2022-09-19T11:01:01Z

packages/falso/src/lib/core/core.ts

+  const maxAttempts = options.length * 2;
+
+  while (items.length < options.length && attempts < maxAttempts) {
+    const item = data(0);


I think it makes sense to pass the attempts as the function's argument.

@shaharkazaz - How do you mean? To allow the user to decide the total number of attempts?

You are passing 0 to that factory function, I think it makes more sense to pass the attempts, if the factory is using the index passed than you will increase the chances of getting the same result.
@NetanelBasal WDYT?

shaharkazaz · 2022-09-19T11:01:21Z

packages/falso/src/lib/core/core.ts

+  data: T[],
+  options?: Options
+): T | T[] {
+  if (!options?.length) {


same comment as above

packages/falso/src/lib/core/core.ts

shaharkazaz · 2022-09-19T11:40:50Z

packages/falso/src/lib/core/core.ts

+  const newArray: T[] = [];
+
+  while (clonedData.length && newArray.length !== options.length) {
+    const randomIndex = getRandomInRange({


There is a hidden assumption here that the array you are given holds unique values.
Please change the code to support non-unique arrays.

@shaharkazaz - You are correct. How did I not spot that :/

I have reworked it to shuffle the data array and then pick the first n items from it. A positive of doing it this way is that the user will receive an array with a length as close as possible to their desired length. The issue is that this will be more expensive for large data sets.

export function fakeFromArray<T, Options extends FakeOptions>( data: T[], options?: Options ): T | T[] { if (options?.length === 0) { return []; } if (!options?.length) { return randElement(data); } const priority = options?.priority ?? 'length'; if (priority === 'length') { return Array.from({ length: options.length }, () => randElement(data)); } let clonedData: T[] = JSON.parse(JSON.stringify(data)); // Randomise order of data clonedData = shuffleArray<T>(clonedData); if (options.length >= clonedData.length) { return clonedData; } return clonedData.slice(0, length); } function shuffleArray<T>(array: T[]): T[] { for (let i = array.length - 1; i > 0; i--) { const randomIndex = Math.floor(random() * (i + 1)); [ array[i], array[randomIndex], ] = [ array[randomIndex], array[i], ]; } return array; }

What are your thoughts?

shaharkazaz · 2022-09-19T11:45:28Z

packages/falso/src/lib/core/core.ts

+  if (Array.isArray(data)) {
+    return fakeFromArray(data, options) as any;
+  }
+
+  return fakeFromFunction(data as FactoryFunction<T>, config, options) as any;


You have a lot of shared code here, I think I would normalize the array data source to also work on a function with a random element and just use factory-based fake for both.

I'm happy to do this I just felt like this whole fake logic was getting quite complex so keeping them separate made it easier to read for me. But i'm happy either way.

I think in hindsight, if we were starting over i'd suggest that instead of option.length property we have singular and plural function for each. E.G randId that returns a string & randIds that uses randId to generate & return a string[]. But it might be a bit late for that ;) haha

Personally I prefer the length property, adding a plural will double the API for not a good enough reason from my point of view

Fair play. It works pretty well for our users, it just adds a lot of headaches for deving haha. All good though

shaharkazaz · 2022-09-19T11:48:49Z

packages/falso/src/lib/credit-card.ts

+  return fake(factory, options, { uniqueComparer: checkUnique });
+}
+
+function checkUnique(card: CreditCard, cards: CreditCard[]): boolean {


This is repreated several times, maybe create a factory function that creates the chacker

function uniqueByKeys(keys){ return (a, b) => objectIsUnique(keys) }

@shaharkazaz - Sorry I don't know how I missed this. Do you mean something like:

export function randCreditCard<Options extends CreditCardOptions = never>( options?: Options ) { const factory: () => CreditCard = () => { const fullName = options?.fullName ?? `${randPersonTitle()} ${randFullName({ withAccents: false })}`; const brand = options?.brand ?? (randCreditCardBrand() as Brand); const dateOptions: Intl.DateTimeFormatOptions = { month: 'numeric', year: '2-digit', }; const validFrom = randPastDate({ years: 1 }).toLocaleDateString( 'en-GB', dateOptions ); const untilEnd = randFutureDate({ years: 2 }).toLocaleDateString( 'en-GB', dateOptions ); return { fullName, brand, validFrom, untilEnd, ccv: randCreditCardCVV(), number: randCreditCardNumber({ brand }), account: randAccount(), type: rand(['Credit', 'Debit']), }; }; return fake( factory, options, { // Moved logic into here rather than wrapping objectIsUnique call in new function uniqueComparer: (card, cards) => objectIsUnique(card, cards, ['number']) } ); }

shaharkazaz · 2022-09-19T11:53:20Z

packages/falso/src/lib/json.ts

+}
+
+function checkUnique(item: object, items: object[]): boolean {
+  return items.some((i) => JSON.stringify(i) === JSON.stringify(item));


Isn't this checking the opposite? Don't you need all jsons to be different?
Looking at this as a reference:

shaharkazaz · 2022-09-19T11:53:51Z

packages/falso/src/lib/nearby-gpscoordinate.ts

+}
+
+function checkUnique(coordinate: number[], coordinates: number[][]): boolean {
+  return coordinates.some((c) => c.join('') === c.join(''));


Same comment as the json unique

Co-authored-by: Shahar Kazaz <[email protected]>

Change type back to original. Move option.length short circuit checks into fakeFromFunction/fakeFromArray functions. Fix typos in checkUnique logic for randJSON & randNearbyGPSCoordinate. Add missing tests tests. ✅ Closes: 220

theryansmee · 2022-09-19T22:08:50Z

@shaharkazaz - I have made the requested changes for all the comments that I understood and left you queries everywhere else :) I hope these all make sense

theryansmee · 2022-09-27T11:00:56Z

Ok, I think all of the requested changes have been made with just a few outstanding queries left to be answered

shaharkazaz · 2022-09-27T12:24:59Z

@theryansmee I'm actually on vacation in Thailand 😀
I'll get back to it as soon as I return!
If @NetanelBasal is free he might pick this up

feat: 🔥 First punt at priority option

6160c3c

Allow user to set length priority to length or unique (POC) ✅ Closes: #220

theryansmee requested review from NetanelBasal and shaharkazaz March 16, 2022 17:47

shaharkazaz reviewed Mar 22, 2022

View reviewed changes

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

theryansmee added 2 commits March 22, 2022 17:49

Merge remote-tracking branch 'origin/main' into feat/220-unique-values

485782e

feat: 🔥 Clean up fakFromArray logic

6b57aa0

Move randUniqueElement logic into fakFromArray. Simplified fakeFromFunction while loop logic #220

theryansmee added 2 commits March 22, 2022 19:02

feat: 🔥 Replace use of random index logic with getRandomInRange

713c77d

✅ Closes: #220

feat: 🔥 Revert rand element logic for now

bbe7b0b

✅ Closes: #220

shaharkazaz reviewed Mar 23, 2022

View reviewed changes

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

theryansmee added 4 commits March 23, 2022 12:10

feat: 🔥 Allow devs to pass in comparison functions

7f69bad

Add default comparison functions for most likely use cases. Implimented them ✅ Closes: #220

feat: 🔥 Add custom comparison functions non-conforming function

77831a5

✅ Closes: #220

Merge remote-tracking branch 'origin/main' into feat/220-unique-values

96acb7e

feat: 🔥 Add json check unique function. Clean up code

af09284

✅ Closes: #220

shaharkazaz reviewed Mar 30, 2022

View reviewed changes

packages/falso/src/lib/core/core.ts Outdated Show resolved Hide resolved

Ryan Smee added 4 commits April 1, 2022 18:47

feat: 🔥 Add generic comparison function

6833592

Add comparison function and start change over to using it ✅ Closes: #220

feat: 🔥 Replace duplicate checkUnique logic with objectIsUnique

13998f6

✅ Closes: #220

feat: 🔥 Fix function name. Update file name

717bb7d

test: 🥳 Move error to snapshot

660f53a

Ryan Smee and others added 5 commits April 5, 2022 18:39

Merge remote-tracking branch 'origin/main' into feat/220-unique-values

b18c227

feat: 🔥 Fix snapshot

42a54d4

Fix snapshot error. Ensure all thrown errors throw actual errors rather than strings ✅ Closes: #220

feat: 🔥 Move snapshots folder to tests/ route

0ec65be

feat: 🔥 Clean up core file

ff6980a

feat: 🔥 Add missing priority doc comment

d57bd25

Add missing priority doc comment ✅ Closes: #220

theryansmee requested a review from shaharkazaz April 5, 2022 19:46

Ryan Smee added 2 commits June 13, 2022 12:40

Merge remote-tracking branch 'origin/main' into feat/220-unique-values

e7dac07

# Conflicts: # packages/falso/src/lib/core/core.ts # packages/falso/src/lib/text.ts

feat: 🔥 Fix merge conflicts in core

77edc54

✅ Closes: #220

Ryan Smee added 4 commits September 12, 2022 19:22

Merge remote-tracking branch 'origin/main' into feat/220-unique-values

fee5cf6

# Conflicts: # packages/falso/src/lib/number.ts # packages/falso/src/lib/octal.ts

fix: 🐞 Fix outdated tests

23f628e

✅ Closes: 220

refactor: 💡 Remove commented out code

e2fb265

✅ Closes: 220

test: 🥳 Add basic tests for core fake function

086ec7b

Add tests for simple array and function scenarios. Fix typo in unique checker logic ✅ Closes: 220

theryansmee requested a review from NetanelBasal September 14, 2022 13:41

shaharkazaz reviewed Sep 16, 2022

View reviewed changes

Ryan Smee and others added 4 commits September 18, 2022 14:09

refactor: 💡 Code review changes

2e5b650

Change comparison of valueOf to getTime with dateIsUnique. Combine describe block and it labels for comparison function tests ✅ Closes: 220

refactor: 💡 Code review changes

10cd7d0

Change checkUnique arrow functions to function declorations. Change unique-validators from inline arrow functions to actual function. Make function more type safe ✅ Closes: 220

refactor: 💡 Change request changes

274fdc9

Strengthen type checking. Remove redundant error and test ✅ Closes: 220

refactor: 💡 Code review changes

1143091

Strengthen type checking for comparisonKeys ✅ Closes: 220

theryansmee requested a review from shaharkazaz September 18, 2022 15:21

shaharkazaz reviewed Sep 19, 2022

View reviewed changes

theryansmee and others added 2 commits September 19, 2022 15:56

Update packages/falso/src/lib/core/core.ts

2a037d0

Co-authored-by: Shahar Kazaz <[email protected]>

refactor: 💡 Code review changes

8869b35

Change type back to original. Move option.length short circuit checks into fakeFromFunction/fakeFromArray functions. Fix typos in checkUnique logic for randJSON & randNearbyGPSCoordinate. Add missing tests tests. ✅ Closes: 220

theryansmee requested a review from shaharkazaz September 20, 2022 12:56

feat: 🔥 First punt at priority option #223

Are you sure you want to change the base?

feat: 🔥 First punt at priority option #223

Conversation

theryansmee commented Mar 16, 2022 • edited Loading

PR Checklist

PR Type

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

Other information

theryansmee commented Mar 16, 2022

NetanelBasal commented Mar 21, 2022

shaharkazaz left a comment

Choose a reason for hiding this comment

theryansmee commented Mar 22, 2022

theryansmee commented Mar 25, 2022

shaharkazaz commented Mar 28, 2022

theryansmee commented Apr 4, 2022 • edited Loading

theryansmee commented Jun 1, 2022

NetanelBasal commented Jun 1, 2022

theryansmee commented Jun 7, 2022

theryansmee commented Jun 13, 2022

NetanelBasal commented Jun 13, 2022

theryansmee commented Sep 12, 2022

shaharkazaz commented Sep 14, 2022

theryansmee commented Sep 14, 2022

shaharkazaz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theryansmee commented Sep 16, 2022

theryansmee commented Sep 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theryansmee Sep 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theryansmee commented Sep 19, 2022

theryansmee commented Sep 27, 2022 • edited Loading

shaharkazaz commented Sep 27, 2022 • edited Loading

theryansmee commented Mar 16, 2022 •

edited

Loading

theryansmee commented Apr 4, 2022 •

edited

Loading

theryansmee Sep 22, 2022 •

edited

Loading

theryansmee commented Sep 27, 2022 •

edited

Loading

shaharkazaz commented Sep 27, 2022 •

edited

Loading