This is an example custom Jitterbit Harmony connector that interacts with Dropbox, developed using the Dropbox Java SDK API and the Jitterbit Harmony Connector SDK (v1.0.0).
IT IS NOT FOR PRODUCTION USE. It is intended as a starting point for developing custom Harmony connectors. These instructions assume that you are familiar with Jitterbit Harmony, Jitterbit Harmony Cloud Studio, Jitterbit Harmony Private Agents, and Java development. If you are not familiar with Jitterbit Harmony, please see our documentation for more information.
Complete documentation on the Dropbox connector and its activities is included in the docs directory. See its index page for information on how to use the Dropbox connector inside Cloud Studio.
The documentation can also be used as a template for preparing the documentation of a custom connector.
-
Register with the Dropbox Developer Program, create an application, and retrieve its
app-keyandaccess-tokenkeys. -
Register with Jitterbit Harmony and register a Private Agent.
-
Install Java SE Development Kit 8.
-
Install Maven.
-
Install Docker (optional, but highly recommended).
-
Install Postman (optional, but highly recommended).
-
Download the Jitterbit Harmony Docker Private Agent image using
$ docker pull jitterbit/agent:latest -
Clone this example (see Cloning the Dropbox Connector Repository), which includes a copy of the Connector SDK library.
- Connector: A collection of activities that interacts with a system. As part of the connector's packaging, it
provides a file which declares the UI of the interface for entering user-provided input to create an authenticated
connection. A connector must implement the
JitterbitConnectorinterface. - Connection: Authenticated access to a data resource that has been configured using a connector.
- Activity: An interaction with a connection that is added to a Harmony operation and configured as a source or
target. An activity represents the smallest unit of execution within an operation. It matches to a node within a Cloud
Studio operation. Every activity needs to implement a
JitterbitActivityinterface, with these two parts:Configuration: Performed by an end-user using the Cloud Studio UI.Execution: Executes an activity according to the user configuration.
- Endpoint: A specific connection and its activities. An endpoint represents a connection configured to a specific system, with endpoint information provided by an end-user using the Cloud Studio UI.
- Payload: Represents a request or response (input or output) of an activity during execution; maps to a
Payloadinterface. - Request or Response: The request or response of an activity developed with the Connector SDK (v1.0.0) will be in XML format only.
adapter.json: A JSON file containing the declarative Cloud Studio UI associated with an endpoint and the activities that make up a connector. As the developer, you provide the parameters that need to be configured and the types associated with each parameter. For more complex interfaces, there are a set of widgets available as part of the Cloud Studio UI.
These terms are slightly different than those used in the Cloud Studio documentation, as that documentation is designed for end-users of Cloud Studio rather than developers of connectors.
Follow these steps to register, build, and run the Dropbox connector:
- Registering the Connector with Jitterbit Harmony
- Cloning the Dropbox Connector Repository
- Building and Packaging
- Deploying and Running
All custom connectors need to be registered with Jitterbit Harmony prior to being built.
You'll need to obtain from Jitterbit Harmony, for the connector, these values:
- Connector key
- Connector secret
- Endpoint entity ID
- Starting activity ID
- Ending activity ID
To obtain these, use the Postman collection included in this example at
postman/JitterbitConnectorManagerAPI.collection.json.
Import the environment variables from postman/jitterbit-env-variable-postman.json into Postman and associate them
with the collection before running the REST API.
Update the email, password, and connector name environment variables appropriately. The connector name and version should
not require changing. (In any case, the value the connector name is registered under must match the value in the
adapter.json used to describe the UI.)
Now, from within Postman, first run the Log In User and then run the Register a Connector APIs.
(The other APIs in the collection can be used to list and delete existing connectors that you have registered, and to confirm the validity of a connector's key and secret.)
The response returned by the Register a Connector API contains the key-values that are needed for the connector's
manifest to complete the registration of the connector. You should receive a response such as:
{
"status": true,
"operation": "Add connector",
"key": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"secret": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"endpointEntityId": "20101",
"functionEntityStartId": "20102",
"functionEntityEndId": "20200"
}As can be seen from this example, your connector has been registered, has an entity ID of 20101, and can have
activity entity IDs ranging from 20102 through 20200.
Register a Connector API |
Manifest File Keys |
|---|---|
key |
Jitterbit-Connector-Key |
secret |
Jitterbit-Connector-Secret |
endpointEntityId |
Jitterbit-Connector-Endpoint-EntityTypeId |
functionEntityStartId |
Jitterbit-Activity-EntityTypeId-<first_activity> |
functionEntityEndId |
Jitterbit-Activity-EntityTypeId-<last_activity> |
The key and secret returned will become, as shown in the table above, the Jitterbit-Connector-Key and
Jitterbit-Connector-Secret in the manifest for the container.
The connector has been allocated an endpoint ID and a range of activity (function) endpoint IDs, from the first to the last. In the case of the Dropbox connector, which has four activities (Fetch File, Get File, Process File, and Put File), you would assign individual IDs to each activity such that they fall inside the assigned range. A maximum of 99 activities can be assigned to an individual connector.
You can use these commands to clone and start in this repository:
$ git clone https://github.com/jitterbit/dropbox-connector.git
$ cd dropbox-connector
As part of the build and packaging process, the bundle JAR that contains the connector implementation will need to
include attributes specific to the connector(s) within the MANIFEST.MF file. The key-values obtained in the
registration process need to be included in the manifest, as described above. For additional details, check the
PACKAGING.md.
Additionally, tests are run as part of the build. For the tests to run, you will need the Dropbox app-key and
access-token keys that you obtained earlier. They must be defined in the build environment as the test uses those
keys.
Skipping the details, and to directly build the Dropbox connector, run these commands (after substituting in your
values for the APP_KEY and ACCESS_TOKEN environment variables in this example):
$ export APP_KEY=<app-key>
$ export ACCESS_TOKEN=<access-token>
$ mvn jaxb2:xjc compile install
The Docker image of the Private Agent allows you to mount a host directory to a container directory such as
/mnt/connectors. To deploy the Dropbox Connector JAR after it has been built, you can run the agent as a Docker
container with the build target directory mounted as the connector directory. Within the Docker container, the agent
will scan the /mnt/connectors directory for JAR files that contain a connector implementation (only one connector
implementation is allowed in each JAR file).
For example:
$ docker run -ti --name=jitterbit-agent -p 3000:3000 \
-v "$(pwd)/dropbox-connector/target:/mnt/connectors" \
--env-file "$(pwd)/private-agent.env" \
jitterbit/agent:latest
where private-agent.env contains the environment variables used for connecting to the Jitterbit Harmony Platform.
Substitute the appropriate values:
HARMONY_USERNAME="<email-address>"
HARMONY_PASSWORD="<password>"
HARMONY_ORG="<your-organization-name>"
HARMONY_AGENT_GROUP="<your-agent-group-name>"
HARMONY_AGENT="<your-agent-name>"In a different command prompt (or Power Shell, for Windows OS systems) you can check that the container is up by using:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
x3160xx34840 jitterbit/agent:latest "./bootstrap.sh ./bin…" 2 minutes ago Up 2 minutes 3000/tcp, 46908-46909/tcp, 46912/tcp, 0.0.0.0:3000->30000/tcp jitterbit-agent
NOTE: Once the container is up and passes the health check, you will need to log in to the Harmony Portal and then open the Management Console to check if the agent has been registered and is running with the platform. Check the Harmony documentation for details.
In a different command prompt (or Power Shell, for Windows OS systems), you can access the container using a command to open a shell in the Docker image:
$ docker exec -ti jitterbit-agent /bin/bash
root@b3160de34840:/opt/jitterbit#
At this point, you have full access to the agent container. You can stop and restart the agent using the
/opt/jitterbit/bin/jitterbit utility. For a help message, from within the container use:
# /opt/jitterbit/bin/jitterbit help
From within the container, tailing the Tomcat logs to check the connector logs:
# tail -f tomcat/logs/catalina.out
27-Feb-2018 01:37:34.937 INFO [localhost-startStop-1] org.jitterbit.server.connector.ConnectorContextServletListener.contextInitialized Servlet Context /axis
2018-02-27 01:37:34,940 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorLoader:44 - Loading connectors...
2018-02-27 01:37:34,942 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorLoader:125 - Connector Directory: /opt/jitterbit/Connectors
2018-02-27 01:37:34,977 WARN com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:70 - Could not load Jitterbit Connector via manifest.mf; no Main-Class or Jitterbit-Connector-Factory-Class attributes defined.
2018-02-27 01:37:34,977 WARN com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:71 - Falling back to annotation system
2018-02-27 01:37:34,978 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:204 - Temporary Directory for undeploying connectors bundle: /opt/jitterbit/tomcat/temp
2018-02-27 01:37:35,144 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:149 - Loading annotated class java.lang.Class
2018-02-27 01:37:35,144 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:161 - Register Activity Factory with id: 10014 name: fetch factory: org.jitterbit.connector.dropbox.activities.FetchFileActivity$FetchFileActivityFactory
2018-02-27 01:37:35,145 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:161 - Register Activity Factory with id: 10011 name: get factory: org.jitterbit.connector.dropbox.activities.GetFileActivity$GetFileActivityFactory
2018-02-27 01:37:35,145 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:161 - Register Activity Factory with id: 10012 name: put factory: org.jitterbit.connector.dropbox.activities.PutFileActivity$PutFileActivityFactory
2018-02-27 01:37:35,145 INFO com.jitterbit.integration.server.api.ws.connectorframework.ConnectorClassLoader:161 - Register Activity Factory with id: 10013 name: syncUpFile factory: org.jitterbit.connector.dropbox.activities.GetFileActivity$GetFileActivityFactory
27-Feb-2018 01:37:35.163 INFO [localhost-startStop-1] org.jitterbit.connector.sdk.BaseJitterbitConnector.onInit onInit() connector name: DropBox
If you see messages similar to the above in the logs, it means that the Dropbox connector has been successfully started. You can now log in to the Harmony Portal, open Cloud Studio, and begin using the connector in projects.
Once you open a project within the organization that the Private Agent is associated with, you should see the Dropbox connector in the palette. You can then configure the connector as any other connector in Cloud Studio.
Restarting the Private Agent from within the container:
# /opt/jitterbit/bin/jitterbit restart
You can also restart individual components that make up the agent using the same utility. Help for the utility is available using:
# /opt/jitterbit/bin/jitterbit help
You can remote debug the connector using an IDE (Eclipse or IntelliJ) that attaches to the Docker agent container.
To do so, add an environment variable, either in an environment file passed to the Docker command or by adding an option
-e to the Docker command:
-e "CATALINA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'"
Add the port 5005 to the Docker command by adding an option -p 5005:5005 to the command.
Start the agent as described above, and then you should be able to connect your debugging session to the agent on
port 5005.
If you have support issues, questions, or comments about either the example Dropbox connector or the Connector SDK, get in touch with our expert support team by submitting a support case.
See Getting Support.
Copyright © 2018-2020 Jitterbit, Inc.
Licensed under the JITTERBIT MASTER SUBSCRIPTION AGREEMENT (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
https://www.jitterbit.com/cloud-eula
See the License for the specific language governing permissions and limitations under the License.