Feature request: Bulk API 2.0 Query

[douglascayers]

Is your feature request related to a problem? Please describe.

• I need to query or count records in very large objects (hundreds of thousands to millions of records).
• The force:data:soql:query returns a max of 50,000 records and if there's more than that then the command doesn't show me results but rather gives error:

(node:6739) Warning: The query result is missing 738431 records due to a 50000 record limit. Increase the number of records returned by setting the config value "maxQueryLimit" or the environment variable "SFDX_MAX_QUERY_LIMIT" to 738431 or greater than 50000.

• Even if I increase the environment variable "SFDX_MAX_QUERY_LIMIT" to a sufficiently large value, I get the same error.

What are you trying to do

• I would like to use force:data:bulk:query or similar command to conveniently use the Bulk API 2.0 Query API.

Describe the solution you'd like

• A new command, force:data:bulk:query, with similar parameters as force:data:soql, to provide a convenient abstraction over the effort to set up and poll for results of a bulk query job.
• The effort to create a bulk query job and poll for results, then download the results would be encapsulated in this new command.
• To the end-user, usage would be no different than force:data:soql except the knowledge it's running a bulk query.

Parameters:

--apiversion      -- override the api version used for api requests made by t
--json            -- format output as json
--loglevel        -- logging level for this command invocation
--perflog         -- get API performance data
--query           -- SOQL query to execute
--resultformat    -- result format emitted to stdout; --json flag overrides t
--targetusername  -- username or alias for the target org; overrides default 
--wait            -- minutes to synchronously wait for results (think force:package:install --wait)

Describe alternatives you've considered

• Use another tool, like Data Loader, or write my own code using curl or jsforce library. These are not preferred because I'd like to script the bulk queries without developing something new myself, and be able to take advantage of orgs authorized with the CLI so that I don't have to fret about creating authorization headers.

Thanks

[github-actions]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[uip-robot-zz]

This issue has been linked to a new work item: W-9997550

[jclark-dot-org]

Please include an option to submit a Bulk Api 2.0 Tooling query as well - MetadataComponentDependency is kinda useless without Bulk API support.

[WillieRuemmele]

🎉 salesforcecli/plugin-data#328

🥳 https://github.com/forcedotcom/cli/blob/main/releasenotes/sfdx/README.md#71640-aug-18-2022-stable-rc