Skip to content

signal-ai/tap-outreach

 
 

Repository files navigation

tap-outreach

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

  • Pulls raw data from Outreach API
  • Outputs the schema for each resource
  • Incrementally pulls data based on the input state

Fork Status

This fork currently:

Allows the relationship value to take precedence over the attribute value by ignoring the exit on the condition the field is also an attribute at (sync.py line 230)

if fk_field_name in record_flat:
    raise Exception(
        '`{}` exists as both an attribute and generated relationship name'.format(fk_field_name))

Streams

accounts

call_dispositions

call_purposes

calls

content_categories

duties

events

mailboxes

mailings

opportunities

personas

prospects

stages

tasks

teams

users

Quick Start

  1. Install

    Clone this repository, and then install using setup.py. We recommend using a virtualenv:

    > virtualenv -p python3 venv
    > source venv/bin/activate
    > python setup.py install
    OR
    > cd .../tap-outreach
    > pip install .
  2. Here is what the config should look like

    {
        "start_date": "2019-01-01T00:00:00Z",
        "client_id": <CLIENT_ID>,
        "client_secret": <CLIENT_SECRET>,
        "redirect_uri": <REDIRECT_URI>,
        "refresh_token": <REFRESH_TOKEN>,
        "quota_limit": <QUOTA_LIMIT>,
        "page_size": <PAGE_SIZE>
    }

    The following are required values: start_date, client_id, client_secret, redirect_uri, refresh_token The following are optional values: quota_limit, page_size

    Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

      {
        "bookmarks": {
          "stream_1": "2020-01-04T00:00:00.000000Z",
          "stream_2": 1576266090,
          ...
        }
      }
  3. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

    tap-outreach --config config.json --discover > catalog.json

    See the Singer docs on discovery mode here.

  4. Run the Tap in Sync Mode (with catalog) and write out to state file

    For Sync mode:

    > tap-outreach --config config.json --catalog catalog.json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To load to json files to verify outputs:

    > tap-outreach --config config.json --catalog catalog.json | target-json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    To pseudo-load to Stitch Import API with dry run:

    > tap-outreach --config config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
  5. Test the Tap

    While developing the Outreach tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

    > pylint tap_outreach -d broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring

    The pylint disables that Stitch uses are:

    broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring
    

    To check the tap, install singer-check-tap and verify working:

    > tap-outreach --config config.json --catalog catalog.json | singer-check-tap > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

    Check tap resulted in the following:

    The output is valid.
    It contained 8240 messages for 16 streams.
    
        16 schema messages
    8108 record messages
        116 state messages
    
    Details by stream:
    +-----------------------------+---------+---------+
    | stream                      | records | schemas |
    +-----------------------------+---------+---------+
    | **ENDPOINT_A**              | 23      | 1       |
    +-----------------------------+---------+---------+

Copyright © 2020 Stitch

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%