Required Fields V2 - Conditionally Required Fields

[nick-Ag]

This PR revamps the required property for action input fields such that a destination builder can define conditionally required fields. These builder defined conditional requirements are then compiled into a JSONSchema4 object, which is used to validate each mapping before the perform block is triggered.

SDD: Link

Conditional Field Example:

For example, the Salesforce Lead last_name field is only required when a user is creating a lead. We can now encode that in the definition of the field directly like this:

    last_name: {
      label: 'Last Name',
      description: "The lead's last name. **This is required to create a lead.**",
      type: 'string',
      ...
      required: {
        conditions: [
          {
            fieldKey: 'operation',
            operator: 'is',
            value: 'create'
          }
        ]
      },

This builder defined condition will then be compiled into a JSONSchema like this:

{
  '$schema': 'http://json-schema.org/schema#',
  type: 'object',
  additionalProperties: false,
  properties: {
    operation: {
      title: 'Operation',
	...
    },
	...
    last_name: {
      title: 'Last Name',
	...
    },
	...
  },
  required: [ 'operation' ],
  if: {
    properties: {"operation": {"const": "create"}}
  },
  then: { "required": ["last_name"] }
}

And eventually will validate Payloads such as these:
{ operation: 'create', first_name: 'nick' } // invalid - missing last_name
{ operation: 'create', last_name: 'aguilar' } // valid - required last_name is included
{ operation: 'update', first_name: 'nick' } // valid - last_name is not required

Reviewing this PR:

~2800 lines added:

• 1400 - schema-validation.test.ts.snap - this documents the JSON schema generated for each unit test. This is automated and viewing each of these generated schemas isn't important
• 1100 - schema-validation.test.ts - New unit tests which run various combinations of different conditions a builder can define
• 250 - fields-to-jsonschema.ts - The JSON Schema generation logic

Testing

Deployed successfully to staging. I have updated the Salesforce Lead action's last_name field with a conditional requirement.

As of now these conditions are not implemented in the UI, and I am only expecting new AJV error messages when the conditional requirement is not met.

The conditional requirement is enforced when operation: create but last_name is missing:

When operation: create and last_name is defined, the condition is met and event delivery succeeds:

When operation: update the condition is not enforced and last_name can be undefined:

• Added unit tests for new functionality
• Tested end-to-end using the local server
• [Segmenters] Tested in the staging environment

Merge & Deploy Timing

This PR will be a part of a series of PRs. It should not be merged or deployed before these other PRs are ready.
@joe-ayoub-segment - I will handle the final deploy of this one

app: TODO 🚧
action-cli: https://github.com/segmentio/action-cli/pull/196 💬

[joe-ayoub-segment]

hi @nick-Ag - I know this is still in draft, so just out of curiosity what will this do?

  groupConditions: {
    anyOf: ['external_id', 'user_alias', 'email', 'braze_id']
  },

Will it hide or show fields in the UI depending on which fields have a mapping populated?
Or will it perform a runtime validation which will prevent the perform() function from being invoked if none of the fields have a value?

Keen to understand the intent here as I think the second approach would be really valuable.

[nick-Ag]

hi @nick-Ag - I know this is still in draft, so just out of curiosity what will this do?

  groupConditions: {
    anyOf: ['external_id', 'user_alias', 'email', 'braze_id']
  },

Will it hide or show fields in the UI depending on which fields have a mapping populated? Or will it perform a runtime validation which will prevent the perform() function from being invoked if none of the fields have a value?

Keen to understand the intent here as I think the second approach would be really valuable.

Planning on the second approach primarily. I also want the validation to appear in the UX so that users can't save mappings where the group condition has not been met. Looking forward to showing more soon!

[github-actions]

New required fields detected

Warning

Your PR adds new required fields to an existing destination. Adding new required settings/mappings for a destination already in production requires updating existing customer destination configuration. Ignore this warning if this PR is for a new destination with no active customers in production.

The following required fields were added in this PR:

• Destination: Salesforce (Actions), Action Field(s):last_name

Add these new fields as optional instead and assume default values in perform or performBatch block.

[codecov]

Codecov Report

Attention: Patch coverage is 91.76471% with 7 lines in your changes missing coverage. Please review.

Project coverage is 79.05%. Comparing base (bb253f2) to head (488753e).

Files with missing lines	Patch %	Lines
...s/core/src/destination-kit/fields-to-jsonschema.ts	91.76%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2099      +/-   ##
==========================================
+ Coverage   78.58%   79.05%   +0.47%     
==========================================
  Files        1030     1083      +53     
  Lines       18184    20912    +2728     
  Branches     3431     4187     +756     
==========================================
+ Hits        14290    16533    +2243     
- Misses       2752     3231     +479     
- Partials     1142     1148       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maryamsharif]

Would this not break current implementation because you are making this field required in certain instances where it wasn't before?

[nick-Ag]

It won't break anything existing because we already throw an error in the case where operation: create and last_name is missing. So for event delivery it shifts the error to AJV instead.
In the UI this is currently unenforced, so users can still save a mapping where last_name is missing

[joe-ayoub-segment]

hi @nick-Ag I can't wait to start using this new feature :).
Can you tell me what happens when null, '' and undefined values are passed in to a string field which is conditionally required? Also what if the field is defined as AllowNull: true?

[joe-ayoub-segment]

Hi @nick-Ag , qq:

• does/should the generated-types.ts file change if conditions are used?
• Can you show examples how to use multiple conditions, with all/any (assuming those use-cases are supported)?

[nick-Ag]

Hi @nick-Ag , qq:

• does/should the generated-types.ts file change if conditions are used?
• Can you show examples how to use multiple conditions, with all/any (assuming those use-cases are supported)?

• The generated-types file won't change, fields which are conditionally required will be optional (ex: last_name?: string). See here
• Those cases are supported, see the unit test here for an example. I've tried to test every possible use-case I could think of in that test file

[nick-Ag]

hi @nick-Ag I can't wait to start using this new feature :). Can you tell me what happens when null, '' and undefined values are passed in to a string field which is conditionally required? Also what if the field is defined as AllowNull: true?

I'm going to make a small update to the PR to handle null values as well, thanks for pointing out this case