JSON Formatter Smartmodule

SmartModule to read values from JSON records, generate a user-define formatted string, and update the record. This SmartModule is map type, where each record-in generates a new records-out.

Input Records

Each row is an individual JSON record:

{
  "email": "[email protected]",
  "name": "Alice Liddell",
  "type": "subscribe",
  "source": "front-page"
}
{
  "email": "[email protected]",
  "name": "Bob Newland",
  "type": "use-case",
  "source": "iot",
  "description": "Tracking fleet of trucks"
}
{
  "email": "[email protected]",
  "name": "Charlie Brimmer",
  "type": "use-case",
  "source": "clickstream",
  "description": "Track user interests"
}
{
  "email": "[email protected]",
  "name": "Random Dude",
  "type": "undefined"
}

Transformation spec

The transformation spec covers 2 cases:

• match: a list of formatting rules associated with a match condition.
• default: a default formatter without a matching rule.

The default formatter is required and can be used with or without the match section. If used together, the matching statement is checked first, and default is applied if no matching record is found.

In this example, we'll use the following transformation spec:

spec:
  match:
    - key: "/type"
      value: "subscribe"
      format:
        with: ":loudspeaker: {} ({}) subscribed on {}"
        using: 
          - "/name"
          - "/email"
          - "/source"
        output: "/formatted"
    - key: "/type"
      value: "use-case"
      format:
        with: ":confetti_ball: {} ({}) wants to solve the following '{}' use-case:\n>{}"
        using: 
          - "/name"
          - "/email"
          - "/source"
          - "/description"
        output: "/formatted"
  default:
    format: 
      with: "{} ({}) submitted a request"
      using: 
        - "/name"
        - "/email"
      output: "/formatted"

NOTE: The JSON object references for look-up and output must be in JSON Pointer notation.

Output Records

JSON records augmented with formatted strings:

{
  "email": "[email protected]",
  "formatted": ":loudspeaker: Alice Liddell ([email protected]) subscribed on front-page",
  "name": "Alice Liddell",
  "source": "front-page",
  "type": "subscribe"
}
{
  "description": "Tracking fleet of trucks",
  "email": "[email protected]",
  "formatted": ":confetti_ball: Bob Newland ([email protected]) wants to solve the following 'iot' use-case:\n>Tracking fleet of trucks",
  "name": "Bob Newland",
  "source": "iot",
  "type": "use-case"
}
{
  "description": "Track user interests",
  "email": "[email protected]",
  "formatted": ":confetti_ball: Charlie Brimmer ([email protected]) wants to solve the following 'clickstream' use-case:\n>Track user interests",
  "name": "Charlie Brimmer",
  "source": "clickstream",
  "type": "use-case"
}
{
  "email": "[email protected]",
  "formatted": "Random Dude ([email protected]) submitted a request",
  "name": "Random Dude",
  "type": "undefined"
}

Build binary

Use smdk command tools to build:

smdk build

Inline Test

Use smdk to test:

smdk test --file ./test-data/input.txt -e spec='{ "match": [ { "key": "/type", "value": "subscribe", "format": { "with": ":loudspeaker: {} ({}) subscribed on {}", "using": [ "/name", "/email", "/source" ], "output": "/formatted" } }, { "key": "/type", "value": "use-case", "format": { "with": ":confetti_ball: {} ({}) wants to solve the following '{}' use-case:\n>{}", "using": [ "/name", "/email", "/source", "/description" ], "output": "/formatted" } } ], "default": { "format": { "with": "{} ({}) submitted a request", "using": [ "/name", "/email" ], "output": "/formatted" } } }'

Cluster Test

Use smdk to load the startmodule to a fluvio cluster:

smdk load 

Test using transform.yaml file:

smdk test --file ./test-data/input.txt  --transforms-file ./test-data/transform.yaml

A second default-only transforms-default.yaml file is also available:

smdk test --file ./test-data/input.txt  --transforms-file ./test-data/transform-default.yaml

NOTE: Changing the parameters inside the transforms files yield different results.

Cargo Compatible

The project has tests that can be run using cargo:

cargo build

cargo test