feat(nango.yaml): dedicated package, stricter parsing, wider types support #2303

bodinsamuel · 2024-06-11T08:25:17Z

Fixes NAN-1166
Fixes https://linear.app/nango/issue/NAN-1096/add-comment-at-the-top-of-modelsts-file-to-explain-that-its-auto

🤓 Context

First I want to apologize for the length of this PR, it's way too big but it was "all or nothing" situation unfortunately.

When I started playing with types/json schema validation I wanted better output for errors at compile time:

realized the CLI had 2 different ways to parse the nango.yaml
realized nango-config.service.ts was not covering the same case that the one in cli
realized neither were not handling many types/edge cases (i.e: arrays, number, boolean, union, exotic char in model name, etc.)
then after modifying everything (took me a day), I realized it was also parsing the table sync_configs (for a reason I still don't get) so it was just breaking everything.

So I had no choice, to rewrite everything just because I wanted to cover more cases to be compatible with json schema 😵‍💫

The Good news:

we now have a dedicated package to parse nango-yaml and it's battle tested (hopefully)
we now have one less thing to remove from shared (nb: the old code is still there and it's still used by sync_configs)
There is only 13 mentions to shared left in CLI

🚜 Changes

New published package @nangohq/nango-yaml
It will be published, as soon as you validate this PR otherwise the CLI won't pass.
New output when validating nango.yaml
A lot of new edge cases, errors and warnings are now discovered. It might be breaking but at least it's explicit.
[breaking] endpoint format is enforced (verb /uri)
[breaking] runs is explicitly required
[breaking] format of NangoFlows inside model.ts
[breaking] Input and Output specifying a js value (instead of a model) now use a type interface with a unique generated name
[new] Support for unions, arrays, type array, cyclic ref, literal string, number, true, false, null, undefined, multiple extends
[fix] Disallow exotic char in model and property name
[fix] Always compile model.ts after loading nango.yaml
...not exhaustive changelist

Warning

UI won't show the appropriate model (fixing that later) and this is a breaking change for GET /flows GET /config/:key

😭 How do I review this huge PR?

Starts with: packages/nango-yaml/parser.v2.ts
You can read: packages/nango-yaml/modelsParser.ts which handle the yaml -> js types
Then packages/cli/services/model.service.ts
Optional packages/types/nangoYaml/index.ts (describe yaml then parsed yaml)
And voila, the rest is mostly side effects of having good types and slightly different objects

🧪 How do I test?

Please run the cli on your machine, I have extensively tested but it's so critical and super user dependent.

node ../nango/packages/cli/dist/index.js init 
node ../nango/packages/cli/dist/index.js generate // try to delete a script and make it re-generate it 
node ../nango/packages/cli/dist/index.js compile // check your model.ts
node ../nango/packages/cli/dist/index.js deploy dev

models:
  Test1:
    id: number
    str: "literal"
    issue: Unknown
    arr:
      - hello
      - boy

  Test2:
      id: integer
      title: string
      url: string
  Parent:
      id: integer
  Test3:
      __extends: Parent
      oui: true
      non: false
      OUI: True
      NON: False
      num: 1
      kd?: Pick<dfdk>
      nulle: null
      union: True | False
      nes:
        ted: number
   Test4:
     __string: number

Example

linear · 2024-06-13T09:38:50Z

NAN-1166 Improve nango.yaml validation

bodinsamuel · 2024-06-13T09:46:20Z

packages/nango-yaml/lib/modelsParser.ts

+        return true;
+    }
+
+    parseFields({ fields, parent }: { fields: NangoYamlModelFields; parent: string }): NangoModelField[] {


This is the main function to parse models and handle all the edge cases regarding types

bodinsamuel · 2024-06-13T09:46:49Z

packages/nango-yaml/lib/helpers.unit.test.ts

@@ -0,0 +1,154 @@
+import { expect, describe, it } from 'vitest';


Moved from shared

bodinsamuel · 2024-06-13T09:46:56Z

packages/nango-yaml/lib/helpers.ts

+    return !regQuote.test(name);
+}
+
+export function getProviderConfigurationFromPath({ filePath, parsed }: { filePath: string; parsed: NangoYamlParsed }): NangoYamlParsedIntegration | null {


moved from shared

bodinsamuel · 2024-06-13T09:47:29Z

packages/nango-yaml/lib/helpers.ts

+    .map((v) => v.toLocaleLowerCase());
+const typesWithGenerics = [...JAVASCRIPT_AND_TYPESCRIPT_TYPES.builtInObjects, ...JAVASCRIPT_AND_TYPESCRIPT_TYPES.utilityTypes];
+// Only used externally
+export function isJsOrTsType(type?: string): boolean {


I moved it before realizing it wasn't what I expected, it will be deleted as soon as I migrate the backend to use the new schema

bodinsamuel · 2024-06-13T09:47:48Z

packages/nango-yaml/lib/helpers.ts

+ * and then 1636 etc. The offset should be based on the interval and should never be
+ * greater than the interval
+ */
+export function getInterval(runs: string, date: Date): IntervalResponse | Error {


Moved from shared, with the diff that ServiceResponse does not exists and not available in CLI so I just return an Error if any

bodinsamuel · 2024-06-13T09:47:56Z

packages/nango-yaml/lib/helpers.ts

+    offset: number;
+}
+
+export function determineVersion(configData: NangoYaml): 'v1' | 'v2' {


moved from shared

bodinsamuel · 2024-06-13T09:49:14Z

packages/cli/package.json

        "copy:files": "copyfiles -u 1 lib/templates/* dist",
-        "copy:types": "cp `node -p \"require.resolve('@nangohq/shared/dist/sdk/sync.d.ts')\"` './dist/nango-sync.d.ts'",


this one is a big bet, I'm trying to load the file from user's global node_modules, should work but let's see :D

bodinsamuel · 2024-06-13T09:49:49Z

packages/cli/lib/sync.unit.cli-test.ts

-        expect(modelsFile).not.toContain(`'Other[]' | null | undefined;`);
-        expect(modelsFile).toContain(`Other[] | null | undefined;`);
-    });
-


All of this has been moved either to model.service.ts or removed because no longer relevant

khaliqgant · 2024-06-17T11:32:32Z

Could not reproduce, happy to dig if you send me the customer name in DM

This seems to happen when there is a type used in the sync instead of a sync_type

syncs:
          salesforce-contacts:
              type: full
              runs: every day
              auto_start: false
              output: Contact
              input: FieldSchema
              endpoint: /salesforce/contact
              scopes: offline_access,api
              description: |
                Syncs contacts based on a metedata defined object. Requires metadata
                Auto start false.
              webhook-subscriptions:
                - contact.update

with this configuration gives:

➜  nango-integrations git:(main) ✗ npx nango scc
nango.yaml validation failed

integrations > salesforce > syncs > salesforce-contacts
  error must NOT have additional properties [additionalProperties]
  additionalProperty: type

[object Object]
null

If a model is reference but doesn't actually exist

It seems the __string isn't handled correctly:

  Contact:
    id: string
    userId: string
    __string: string

produces:

export interface Contact {
  [key: string]: 'string';
  id: string;
  userId: string;
};

where previously it would correctly output:

export interface Contact {
  id: string;
  userId: string;
 [key: string]: string;
}

Given this:

        actions:
          salesforce-fetch-nested-fields:
              description: Fetch available fields for a nested field
              input: string
              endpoint: /salesforce/fetch-nested-fields
              output: NestedFieldSchema
              scopes: offline_access,api

The outputted model is

"input": "Anonymous_salesforce_action_salesforcefetchnestedfields_input"

which is unexpected. I see why you did it to encourage people to use models but IMO an input of a string is perfectly valid and shouldn't require a model. But maybe this is more of a product question and good for Bastien to weigh in.

Also if a model is declared but doesn't actually exist the yaml parsing continues but feel like we should catch this?

khaliqgant · 2024-06-17T11:34:29Z

When running nango scc I definitely miss getting the fields outputted -- not sure if this is coming in the piepline of your work. It used to output this:

{
        "name": "salesforce-fetch-fields",
        "type": "action",
        "models": [
          {
            "name": "FieldSchema",
            "fields": [
              {
                "name": "fields",
                "type": "Field[]"
              },
              {
                "name": "childRelationships",
                "type": "ChildField[]"
              },
              {
                "name": "userIdField",
                "type": "string | undefined | null"
              }
            ]
          },

now just gives

        {
          "name": "salesforce-fetch-fields",
          "type": "action",
          "description": "Fetch available contact fields",
          "scopes": [
            "offline_access",
            "api"
          ],
          "input": null,
          "output": [
            "FieldSchema"
          ],
          "usedModels": [
            "FieldSchema"
          ],
          "endpoint": {
            "POST": "/salesforce/fetch-fields"
          }
        }
      ]

khaliqgant · 2024-06-17T11:59:23Z

[new] Support for unions, arrays, type array, cyclic ref, literal string, number, true, false, null, undefined, multiple extends

Can you give examples of each of these?

multiple extends works nicely!

  SelectedField:
    name: string
    label: string
    type: string
    foo:
      __extends: Entity

bodinsamuel · 2024-06-17T12:20:53Z

This seems to happen when there is a type used in the sync instead of a sync_type

type was never officially supported (cf the json schema) so to me I just fixed the validation.

It seems the __string isn't handled correctly:

✅ Thanks, missed that in my tests

[object Object]

✅ Okay found the issue thanks

I definitely miss getting the fields outputted

Added the models output, but yeah I'm not keeping the raw format in memory

input of a string is perfectly valid and shouldn't require a model

It's still perfectly valid, but I need one model per input/output so that I can generate json schema (and more) in a single way. With just one way there is no need to check everywhere in the backend if a string is jsType or a model.

Also if a model is declared but doesn't actually exist the yaml parsing continues but feel like we should catch this?

That's the problem of supporting js types, there is no way to differentiate literal string and model name. So I added a warning about literal string but I don't see a better way rn

[new] Support for unions, arrays, type array, cyclic ref, literal string, number, true, false, null, undefined, multiple extends

You have in the test and PR desc.

models:
  Ref:
    id: string
  Ref2:
    name: string
  Base:
    unions: string | null | Ref | true[]
    arrUnion:
      - Ref
      - true
    literal: hello
    num: 1
    literalNull: null[]
    obj:
      nested: string
    optional?: string
    __extends: Ref,Ref2

linear · 2024-06-17T12:44:24Z

NAN-1096 Add comment at the top of `models.ts` file to explain that it's auto-generated

khaliqgant · 2024-06-17T18:25:00Z

packages/nango-yaml/lib/modelsParser.ts

+import { getNativeDataType, getPotentialTypeAlias, isDisallowedType, shouldQuote } from './helpers.js';
+import { ParserError, ParserErrorCycle, ParserErrorDataSyntax, ParserErrorExtendsNotFound, ParserErrorInvalidModelName } from './errors.js';
+
+export class ModelsParser {


Small nit but I would keep this singular, ModelParser

I see what you mean, it's like that because it's parsing the models list and contains all the models maybe ModelsPropParser would have been clearer

khaliqgant

Small nits, but works well. I was using this branch as I was writing customer scripts today 💯

This is a bit weird:

        {
          "name": "selectedFields",
          "value": [
            {
              "name": "0",
              "value": "SelectedField[]",
              "model": true,
              "optional": false
            },
            {
              "name": "1",
              "tsType": true,
              "array": false,
              "optional": false
            },
            {
              "name": "2",
              "value": null,
              "tsType": true,
              "array": false,
              "optional": false
            }
          ],
          "union": true,
          "optional": false
        }

This is when the type is like this

selectedFields: SelectedField[] | undefined | null

weird in the sense that the name is the array index, but get that this might be technically challenging.

bodinsamuel · 2024-06-17T20:38:01Z

SelectedField[] | undefined | null

Interesting, actually it should be supported. Adding this edge case, thanks a lot for the thorough testing 🙏🏻

bodinsamuel added 11 commits June 10, 2024 15:32

tmp

fb34d7c

clean up

54091a3

rollback some stuff

ada1d52

more

e13c79e

more

9b52d41

more

f296404

revert unit test

18d8596

missed this

765fa32

tmp

6f99677

working

f898991

more

382ce6b

bodinsamuel self-assigned this Jun 11, 2024

bodinsamuel added 12 commits June 11, 2024 10:46

wip

2dc3085

fix(deps): simple-oauth2 5.0.1

28154f0

Merge branch 'master' into fix/cli-harden-validation

f3f8134

tmp2

7e7d42e

create pkg

24075e0

tmp

ce06590

tmp

be76275

tmp

b5f7ae6

clean up

5eab2d5

all tests passing 😭

fa9a46c

Merge branch 'master' into fix/cli-harden-validation

53679d6

deploy working 🙀

ab7a290

bodinsamuel commented Jun 13, 2024

View reviewed changes

bodinsamuel added 4 commits June 13, 2024 12:15

endpoints

d2469a8

Merge branch 'master' into fix/cli-harden-validation

cfcb36b

more invalid stuff

5095889

handle literal object

14a805e

bodinsamuel added 2 commits June 17, 2024 12:40

Merge branch 'master' into fix/cli-harden-validation

c1dfc1a

clean up

15bde57

bodinsamuel added 3 commits June 17, 2024 14:21

review

4118f53

test

2cbb4eb

cli test

ea6cdd7

bodinsamuel requested a review from TBonnin June 17, 2024 12:42

bodinsamuel requested a review from khaliqgant June 17, 2024 12:46

TBonnin approved these changes Jun 17, 2024

View reviewed changes

khaliqgant reviewed Jun 17, 2024

View reviewed changes

khaliqgant approved these changes Jun 17, 2024

View reviewed changes

bodinsamuel added 11 commits June 17, 2024 22:39

fix array of model inside union

fe244a4

published

3e7494f

correct publish

b9b4d06

snapshot

0257414

forgot tha

13737c0

Merge branch 'master' into fix/cli-harden-validation

6433991

use import resolve

6f482ee

use polyfill

354bebf

Merge branch 'master' into fix/cli-harden-validation

a6d081b

allez on y croit

cbab0f9

Merge branch 'master' into fix/cli-harden-validation

1e22379

bodinsamuel enabled auto-merge (squash) June 18, 2024 07:16

bodinsamuel merged commit 4012b2d into master Jun 18, 2024
20 checks passed

bodinsamuel deleted the fix/cli-harden-validation branch June 18, 2024 07:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(nango.yaml): dedicated package, stricter parsing, wider types support #2303

feat(nango.yaml): dedicated package, stricter parsing, wider types support #2303

bodinsamuel commented Jun 11, 2024 •

edited

Loading

linear bot commented Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

bodinsamuel Jun 13, 2024

khaliqgant commented Jun 17, 2024 •

edited

Loading

khaliqgant commented Jun 17, 2024

khaliqgant commented Jun 17, 2024

bodinsamuel commented Jun 17, 2024

linear bot commented Jun 17, 2024

khaliqgant Jun 17, 2024

bodinsamuel Jun 17, 2024

khaliqgant left a comment

bodinsamuel commented Jun 17, 2024

		@@ -0,0 +1,154 @@
		import { expect, describe, it } from 'vitest';

		"copy:files": "copyfiles -u 1 lib/templates/* dist",
		"copy:types": "cp `node -p \"require.resolve('@nangohq/shared/dist/sdk/sync.d.ts')\"` './dist/nango-sync.d.ts'",

feat(nango.yaml): dedicated package, stricter parsing, wider types support #2303

feat(nango.yaml): dedicated package, stricter parsing, wider types support #2303

Conversation

bodinsamuel commented Jun 11, 2024 • edited Loading

🤓 Context

🚜 Changes

😭 How do I review this huge PR?

🧪 How do I test?

Example

linear bot commented Jun 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khaliqgant commented Jun 17, 2024 • edited Loading

khaliqgant commented Jun 17, 2024

khaliqgant commented Jun 17, 2024

bodinsamuel commented Jun 17, 2024

linear bot commented Jun 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khaliqgant left a comment

Choose a reason for hiding this comment

bodinsamuel commented Jun 17, 2024

bodinsamuel commented Jun 11, 2024 •

edited

Loading

khaliqgant commented Jun 17, 2024 •

edited

Loading