Connecting to salesforce

[danwwilson]

I'm attempting to make use of this tap and am struggling to make things work at present. I've created a salesforce connected app following these instructions. I did make the callback URL https://127.0.0.1

In the config.json file I have the following:

{
  "client_id": "my consumer key (salesforce language)",
  "client_secret": "my consumer secret (salesforce language)",
  "refresh_token": "abc123 (unchanged from README example)",
  "start_date": "2000-01-01T00:00:00Z",
  "api_type": "BULK",
  "select_fields_by_default": true
}

When I run the code tap-salesforce --config config.json --discover I get the following messages:

INFO Attempting login via OAuth2
INFO Starting new login timer
CRITICAL 400 Client Error: Bad Request for url: https://login.salesforce.com/services/oauth2/token

I'm really interested in making singer.io connect to salesforce, but can't seem to get passed connecting. Some guidance would be really appreciated.

Cheers
Dan

[KAllan357]

Hi @danwwilson, without more information from the error, I'm not sure I know exactly what's going on.

If I'm reading correctly, you are using "abc123" as the value for your refresh token? I think this would be incorrect - what you want is the value that comes back during the Oauth flow. It should be in the response under the credentials key.

Another option is to add some more debugging to the code to see the exact error message that accompanies the 400. I would expect to see more text, but you could add a breakpoint or more logging output in this area: https://github.com/singer-io/tap-salesforce/blob/master/tap_salesforce/salesforce/__init__.py#L286-L298

[sokser]

I am having the same issue, but using the value taken from "Initial Access Token Value" in the connected app setup.

[robinj]

Quick and dirty way to solve this

1. Go to the following url:
   https://[test|login].salesforce.com/services/oauth2/authorize?response_type=code&client_id=[your client id]&redirect_uri=https%3A%2F%2Ftest.salesforce.com%2Fservices%2Foauth2%2Fsuccess

After authorising your application, you will be redirected to another url. There will be a code parameter in the query string: this is your authorization_code for the next step

2. Paste this into a terminal:

curl --request POST \
  --url https://[test|login].salesforce.com/services/oauth2/token \
  --form redirect_uri=https://test.salesforce.com/services/oauth2/success \
  --form 'client_id=[your client id]
' \
  --form 'client_secret=[your client secret]
' \
  --form grant_type=authorization_code \
  --form code=[your authorisation code]== \
  --form =

You will now have your refresh token for use in this tap.

[aaronsteers]

I'm trying to follow these instructions also but I'm not clear on what the prerequisites are for getting auth correctly configured.

1. Do I actually need to create an "app" to get this working?
2. Does the app need to be created under the same organization as will be used by the tap?

I don't appear to have permissions to create an app under my organization (I'm not a salesforce admin) and I'm not clear if an app created at developer.salesforce.com will suffice.

Any guidance would be much appreciated.

[alexpdp7]

I just went through this, this is a copy/paste of my docs:

Salesforce development environments

You can register for a fully featured Salesforce developer account with access to a developer instance.

Once inside, go to Setup:

In the left-hand tree, select PLATFORM TOOLS / Apps / App Manager

New Connected App

Fill:

Connected App Name: <whatever>
API Name: <whatever>
Contact Email: your email
Enable OAuth Settings: enabled
Callback URL: https://example.com
Selected OAuth Scopes: all

With those settings you will be redirected to a web containing your Consumer Key and your Consumer Secret.

Get a refresh token

Open in your browser:

https://<your instance url>/services/oauth2/authorize?response_type=code&client_id=<your client id>&redirect_uri=https://example.com

Click allow, get the “code” from the URL (without %3D)

curl -X POST 'https://<your instance url>/services/oauth2/token?code=<code>&grant_type=authorization_code&client_id=<client_id>&client_secret=<client_secret>&redirect_uri=https%3A%2F%2Fexample.com%0A' | python -m json.tool

[aaronsteers]

@alexpdp7 - I was blocked by this before and went a different route - using instead the normal user/account-based authentication which is (as of last check) only supported in the Meltano fork of this tap.

Can you confirm for me one thing - after following the above instructions, does this only give your new "app" access to the specific dev/test salesforce instance? Specifically, it appeared based upon my previous research that you might still need some level of "admin access" on the salesforce instance you want to tap into, which for many uses cases may not feasible.

Any additional info you could provide on access-level permissions needed to be granted to the custom app within the "tapped" salesforce instance, and what process is needed to provide those permissions, would be very helpful. Thanks!

[alexpdp7]

I don't think I can help you there much, I'm just toying around with developer instances (registered via their developer website like anyone can do)- I expect in this case I'll only gain access to the test instance. You can do the same and then test on a real instance. I have the feeling that it might be possible to do adequate testing without spending money...

[samrogers226]

Has anyone here gotten past this point and is now runing into CRITICAL 401 Client Error: Unauthorized for url: https://<salesforce host>/services/data/v41.0/sobjects?

I can't seem to figure out what I'm doing wrong here.

[ps2goat]

I know this response is late, but may help others.

@aaronsteers - You still need a user to authenticate with Salesforce. The connected app seems to be a way to be able to revoke access for a specific purpose. E.g., sign into a site using Twitter, then within Twitter you could remove that app's access to your data. Your "app" that logs in as your user will have the permissions you assign to the app within Salesforce. Yes, it is per org instance. When you create the connected app, the app will be assigned a client ID unique to your org.

If using a test instance, I found the config needs to have is_sandbox set to true. This points the login url to test.salesforce.com instead of login.salesforce.com.

@samrogers226 - I did not see that error message. I did, however, add my IP to the list of allowed IP Addresses before attempting the initial sync. Instead of doing this, you can relax your IP restrictions by clicking Manage on your connected app, click Edit Policies, and select the new value from the IP Relaxation dropdown.

These are the two policies I have enabled:

• Access and manage your data (api)
• Perform requests on your behalf at any time (refresh_token, offline_access)

Also note that the discovery phase didn't automatically select any objects for me. I had to dig into the singer-python codebase and best practices to figure out what to do and where.

Inside the stream object's metadata array will be one object where the breadcrumb is an empty array. The sibling metadata property in that object is where you should add a property of "selected": true. Doing this will mark the object as selected for replication. (You may have to specify a replication key or something if the table doesn't have one-- I only got my first object replicating yesterday. I'm still looking into the state.json aspect-- It is built up and added to during replication, but I don't see it being output anywhere for the next run.)

Here's a brief example schema from the discovery phase:

{
    "streams": [
        "stream": "<object name>",
        "tap_stream_id": "<object name>",
        "schema": {
            // omitted for brevity
        },
        // this is a weird schema, in my opinion. I'd have pulled the `selected` type info
        // up into a property called "config" or "settings".
        //  instead, you have to look for the metadata object with empty breadcrumbs.
        "metadata": { 
                {
                    "breadcrumb": [
                        "properties",
                        "Id"
                    ],
                    "metadata": {
                        "inclusion": "available",
                        "selected-by-default": true
                    }
                },
                {
                    "breadcrumb": [],
                    "metadata": {
                        "valid-replication-keys": [
                            "SystemModstamp"
                        ],
                        "table-key-properties": [
                            "Id"
                        ],
                      // this is what was missing after my discovery phase.
                      "selected": true
                    }
                }

It is a pain to get an access token when there isn't much in the docs. I'm used to using the Machine-to-Machine connection from Auth0, so I was surprised that I ended up having to log in as an actual user and convert the access code to an access token, then get the refresh token. I haven't tested a lot, but it appears that refresh tokens are good for 24 hours. If your app isn't constantly running, or at least run once per <24 hours, you'll have to go through the process again. So +1 for that PR that's been sitting there for a year that will allow user name + password login, instead.

Anyway, this article walks through how to get the refresh token you need. It uses curl, but Postman or something similar can also be used. https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b

[BenjMaq]

Has anyone here gotten past this point and is now runing into CRITICAL 401 Client Error: Unauthorized for url: https://<salesforce host>/services/data/v41.0/sobjects?

I can't seem to figure out what I'm doing wrong here.

Hey @samrogers226 , were you able to figure that out? I'm facing the exact same issue now... Thanks

EDIT: I was able to connect and run in discover mode using the SOAP connection method. This is not (yet) merged into the public repo, but #65 by @cenxiao is working well for me. I had to update a few things but overall it's great. For those struggling with the above error, I'd suggest you give a try to SOAP.

[BenjMaq]

Also, for anyone who's struggling to authenticate, I've found the last answer in this thread gave me the solution.

curl -v -L \
 -F 'response_type=token' \
 -F 'client_id=<consumer key of connected app>' \
 -F 'redirect_uri=https://login.salesforce.com/services/oauth2/success' \
 -F 'scope=refresh_token <any other scopes you need>' \
 -F 'prompt=consent' \
 https://login.salesforce.com/services/oauth2/authorize

This returns a 302 redirect response. Just put the redirect URL returned into your browser, and it will take you to a Salesforce login followed by an OAuth opt-in page. Click ok, and then your access token and your refresh token appears in the URI in the browser tab.