force.com export gets LIMIT_EXCEEDED exception

[lpoulain]

Running force.com export gives the following error:

ERROR: LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 5000

I guess my org is too large, but I'm not sure how to export just the metadata for a single SObject.

[ddollar]

/cc @ryanbrainard @dcarroll @quintonwall

What can we do to export such a large org? Can the export be paginated somehow?

[lpoulain]

For what it's worth, I'm just trying to get the metadata for one SObject.

[ddollar]

Are you looking for the XML metadata? Do you want to use it externally or run it back through force import ?

[lpoulain]

Run back through force import. The enhancement I'd like to work on (if I am able to rebuild the project) is to look at permissions (OLS, FLS) and harmonize them.

[ddollar]

That sounds great. I'd be happy to answer any questions you have along the way.

Let's leave this issue open to track the LIMIT_EXCEEDED error because it seems like a legitimate problem for large orgs. If you'd like to create another issue/pull request to track your enhancement we can discuss there.

[pfeilbr]

TLDR: Don't try to solve within this tool

This is a general salesforce problem and should be solved at that level. Salesforce doesn't provide a pagination option as of now, but the other tools that deal with this pull subsets of the metadata types per request and merge into the final full directory structure on disk. The issue is that you need to have a backoff retry mechanism to get under LIMIT_EXCEEDED, but each async request takes time and you could be waiting a while. The other bigger issue and the reason salesforce needs to solve for it is because the metadata could change between requests, and you end up with an inconsistent full metadata set.

This problem is only going to get worse with this tool. The barrier to entry to create metadata has been reduced with this tool, which leads to experimentation / more metadata (which is great), and users most likely won't clean up. At the point they decided to provide a file representation of the metadata sitting in a database, they should've considered this. The full metadata state of any org should be able to be downloaded as an archive file at any time.

[dcarroll]

You are correct, the metadata api is not intended to get everything. It is why there are "on platform" ways to cut a sandbox, create change sets and so on. This kind of tool, and the usefulness of it, were not anticipated. I will gladly relate a request for pagination or some other strategy for getting the full set of metadata, through apis, to the engineering team.

[danielsokolowski]

Has there been any progress on this API limit pagination idea?

[cwarden]

Merging the profiles after multiple exports should be feasible. See https://github.com/ForceCLI/force-md/blob/b8b2154b3bc6f28302cdbfeb990126bb47e597f4/cmd/profile/merge.go#L40