Webhooks

Using webhooks with Factotum

As of version 0.3.0, Factotum supports streaming job information over HTTP. No changes to the Factfile are required, to stream job updates with your existing job, use the following command:

factotum run <your Factfile> --webhook "http://my-collector.com"

As of version 0.5.0, Factotum supports varying the size of the Webhook stdout and stderr fields from the default 10_000 characters.

factotum run <your Factfile> --webhook "http://my-collector.com" --max-stdouterr-size 20000

When updates are sent to the webhook

Updates are sent when the state of the job changes:

Information that a new run has started is sent (all tasks are in the state WAITING)
As each task is started it's state changes from WAITING to RUNNING and a new update is sent to the endpoint
As each task is completed, it's state changes from RUNNING to SUCCESS (or FAILED) and a new update is sent
When Factotum has no more tasks to execute, a COMPLETED update is sent, with the final states of each task.

Update format

Factotum updates are a single self describing event:

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "self": {
    "vendor": "com.snowplowanalytics.factotum",
    "name": "job_update",
    "version": "1-0-0",
    "format": "jsonschema"
  },
  "type": "object",
  "properties": {
    "jobName": {
      "type": "string"
    },
    "runReference": {
      "type": "string"
    },
    "factfile": {
      "type": "string"
    },
    "applicationContext": {
      "type": "object",
      "properties": {
        "version": {
          "type": "string",
          "pattern": "\\d+\\.\\d+\\.\\d+-?.*"
        }
      },
      "required": [
        "version"
      ],
      "additionalProperties": false
    },
    "jobReference": {
      "type": "string"
    },
    "runState": {
      "enum": [
        "RUNNING",
        "WAITING",
        "COMPLETED",
        "FAILED"
      ]
    },
    "startTime": {
      "type": "string",
      "format": "date-time"
    },
    "runDuration": {
      "type": "string"
    },
    "taskStates": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "taskName": {
            "type": "string"
          },
          "state": {
            "enum": [
              "RUNNING",
              "WAITING",
              "COMPLETED",
              "FAILED",
              "SKIPPED"
            ]
          },
          "started": {
            "type": "string",
            "format": "date-time"
          },
          "duration": {
            "type": "string"
          },
          "stdout": {
            "type": "string"
          },
          "stderr": {
            "type": "string"
          },
          "returnCode": {
            "type": "integer",
            "maximum": 32767,
            "minimum": -32767
          },
          "error_message": {
            "type": "string"
          }
        },
        "required": [
          "taskName",
          "state"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [
    "jobName",
    "jobReference",
    "runReference",
    "runState",
    "factfile",
    "applicationContext",
    "startTime",
    "runDuration",
    "taskStates"
  ],
  "additionalProperties": false
}

Here's a sample of the JSON Factotum will POST to your webhook.

{
  "schema": "iglu:com.snowplowanalytics.factotum/job_update/jsonschema/1-0-0",
  "data": {
    "jobName": "echo order demo",
    "jobReference": "89116d60c592d62068862ef88aa5cd400e195e5d2311ddc564ab5d00b617624e",
    "runReference": "e77eb3e0377eb3462b3ecf90e3afd07034cf80da981ede9a3b8d50a2c28ec8f0",
    "factfile": "ewogICAgInNjaGVtYSI6ICJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5mYWN0b3R1bS9mYWN0ZmlsZS9qc29uc2NoZW1hLzEtMC0wIiwKICAgICJkYXRhIjogewogICAgICAgICJuYW1lIjogImVjaG8gb3JkZXIgZGVtbyIsCiAgICAgICAgInRhc2tzIjogWwogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIGFscGhhIiwKICAgICAgICAgICAgICAgICJleGVjdXRvciI6ICJzaGVsbCIsCiAgICAgICAgICAgICAgICAiY29tbWFuZCI6ICJlY2hvIiwKICAgICAgICAgICAgICAgICJhcmd1bWVudHMiOiBbICJhbHBoYSIgXSwKICAgICAgICAgICAgICAgICJkZXBlbmRzT24iOiBbXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9LAogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIGJldGEiLAogICAgICAgICAgICAgICAgImV4ZWN1dG9yIjogInNoZWxsIiwKICAgICAgICAgICAgICAgICJjb21tYW5kIjogImVjaG8iLAogICAgICAgICAgICAgICAgImFyZ3VtZW50cyI6IFsgImJldGEiIF0sCiAgICAgICAgICAgICAgICAiZGVwZW5kc09uIjogWyAiZWNobyBhbHBoYSIgXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9LAogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIG9tZWdhIiwKICAgICAgICAgICAgICAgICJleGVjdXRvciI6ICJzaGVsbCIsCiAgICAgICAgICAgICAgICAiY29tbWFuZCI6ICJlY2hvIiwKICAgICAgICAgICAgICAgICJhcmd1bWVudHMiOiBbICJhbmQgb21lZ2EhIiBdLAogICAgICAgICAgICAgICAgImRlcGVuZHNPbiI6IFsgImVjaG8gYmV0YSIgXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9CiAgICAgICAgXQogICAgfQp9Cg==",
    "applicationContext": {
      "version": "0.3.0"
    },
    "runState": "COMPLETED",
    "startTime": "2016-10-13T13:46:49.001109575+00:00",
    "runDuration": "PT1.847529443S",
    "taskStates": [
      {
        "duration": "PT0.002371592S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.001385064+00:00",
        "state": "COMPLETED",
        "stdout": "alpha",
        "taskName": "echo alpha"
      },
      {
        "duration": "PT0.002340387S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.004452530+00:00",
        "state": "COMPLETED",
        "stdout": "beta",
        "taskName": "echo beta"
      },
      {
        "duration": "PT0.001891810S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.007247325+00:00",
        "state": "COMPLETED",
        "stdout": "and omega!",
        "taskName": "echo omega"
      }
    ]
  }
}

Field glossary

Field	Notes
`schema`	Self describing event wrapper
`data`	Self describing event wrapper
`data.jobName`	The name of the job, as it appears in the Factfile
`data.jobReference`	An ID unique to the Factfile for this job
`data.runReference`	A globally unique ID for this run
`data.factfile`	A base64 encoded copy of the Factfile that's running
`data.applicationContext.version`	The version of Factotum that's executing the job
`data.runState`	The current state of the job. This can be `WAITING`,`RUNNING`,`COMPLETED` or `FAILED`
`data.startTime`	The time the job started, in RFC 3339 format
`data.runDuration`	The running time of the job so far in RFC 3339 duration format
`data.taskStates`	An array of information on the state of each task
`data.taskStates[_].taskName`	The name of the task, as it appears in the Factfile
`data.taskStates[_].state`	The current state of the task. This can be `WAITING`,`RUNNING`,`COMPLETED`,`FAILED` or `SKIPPED`
`data.taskStates[_].started`	Optional. The RFC 3339 start time of the task
`data.taskStates[_].duration`	Optional. The RFC 3339 duration of the task
`data.taskStates[_].stdout`	Optional. The output of the task to `stdout`
`data.taskStates[_].stderr`	Optional. The output of the task to `stderr`
`data.taskStates[_].returnCode`	Optional. The return code of the task
`data.taskStates[_].errorMessage`	Optional. The reason the task failed, or was skipped

Retry behaviour

If your endpoint is unavailable or doesn't return 200, Factotum will attempt to send each update three times with a random back-off period of up to one minute between.

Updates are not cached to disk, and if the endpoint cannot be reached during a run, it will not be re-sent in subsequent runs.

Provide feedback

Saved searches