Feature: Allow user to configure the Operation polling interval

[jmdobry]

Right now Operations poll every 0.5 seconds. Unfortunately for some APIs the GetOperation rpc call counts against the user's quota. In the case of Speech#startRecognition when the user uploads a large amount of audio, use of operation.on('complete', ... can result in hundreds of API calls, decimating the user's quota (which isn't that big).

google-cloud-node should allow the user to configure the polling interval, or even use exponential backoff.

[jmdobry]

For some APIs, like Speech, we can solve this by upgrading to the latest gax-nodejs and using its LRO implementation, which relies on exponential backoff for polling by default, and already supports configurable backoff settings.

[kbyatnal]

Thank you for opening this issue.

I was just about to do it myself, having spent all day today finding the problem and bringing attention to it on the google speech discussion thread.

Could I get a timeline of when this will be fixed? I'm currently working on something time-sensitive and I'm debating between waiting for this to be fixed or switching to the python library (where this doesn't seem to be a problem).

[jmdobry]

@kbyatnal What version @google-cloud/speech are you using?

[kbyatnal]

@jmdobry
0.5.0

[jmdobry]

Are you reading the file into memory yourself or just passing the file path string to startRecognition?

[kbyatnal]

I'm passing it the File object (from Google Cloud Storage/Bucket).

I actually saw the discussion thread regarding this topic and have implemented the temporary workaround. It seems to be working for now.

[jmdobry]

Right, but if you're using 0.5.0 then the operation polling should actually be using exponential backoff with a multiplier of 1.3, and that's still eating up your quota?

[kbyatnal]

I don't think the exponential multiplier is in place for this version.

I made 3 requests to the API yesterday before I started getting quota errors, so I checked my console and saw that it had registered 2000+ hits for the API.

[jmdobry]

Try this:

return speech.api.Speech({
  config: speechConfig,
  audio: {
    uri: 'gs://YOUR_BUCKET_NAME/YOUR_FILE_NAME'
  }
}, {
  longrunning: {
    // Try fiddling with the first two numbers
    initialRetryDelayMillis: 100, // This is the initial delay
    retryDelayMultiplier: 1.3, // This is the multiplier that increases the delay over time
    maxRetryDelayMillis: 60000,
    initialRpcTimeoutMillis: null,
    rpcTimeoutMultiplier: null,
    maxRpcTimeoutMillis: null,
    totalTimeoutMillis: 600000
  }
}, (err, operation, apiResponse) => {
  if (err){
    return cb(err);
  }

  operation
    .on('complete', function (response) {
      console.log(response.results[0].alternatives[0]);
    });
});

[stephenplusplus]

@kbyatnal it might help to double-check the version of google-gax being used. If you do a full un- and re-install of @google-cloud/speech, it'll pull down 0.10.4, which has the expo logic built in:

$ npm cache clean
$ npm install --save @google-cloud/speech
$ npm ls google-gax
[email protected] /Users/stephen/dev/play/play-1481768017
└─┬ @google-cloud/[email protected]
  └── [email protected]

[kbyatnal]

@stephenplusplus

Thanks for the suggestion! I updated to 0.10.4, but now I'm getting the following error while running the code:

/Users/kbyatnal/Desktop/nodejs_tutorial/node_modules/google-gax/lib/longrunning.js:273 var previousMetadataBytes = Buffer.from ? Buffer.from("") : new Buffer(""); ^ TypeError: this is not a typed array.

Here is my code:

`
const speechConfig = {
encoding: "LINEAR16",
sampleRate: 8000
};

return speech.startRecognition(file, speechConfig, (err, operation, apiResponse) => {
if(err){
return cb(err);
}
console.log("waiting");
operation.on("error", (err) => {return console.log(err);});
operation.on("complete", (transcript) => {
console.log(transcript);
});
});
`

The file is a Google Cloud Storage File object.

Edit:

In fact, updating to this new version of google-gax seems to have broken the code which was working yesterday (w/ manual polling) and gives the same error. I can't get the speech transcription to work at all now.

[jmdobry]

cc @jmuk

[jmuk]

cc @landrito

[jmuk]

Sorry for that. The error message might look like protobuf.js decodes the data as a typed array while our code assumes it's a NodeJS Buffer? 😞

@landrito -- please look into this, and please try using Speech API with your patch before sending PRs.

[landrito]

I'll look into it. Sorry about that!

[jmdobry]

Seems like Speech system tests in this repo would have caught that no?

[landrito]

@kbyatnal Which version of node are you using?

[landrito]

googleapis/gax-nodejs#90 should solve this problem!

It appears I was using a method in Node that does not work for node versions >4.0.x but <4.5.x and does not work for node versions >5.0.x and <5.10.x.

@jmdobry I suspect this is why it slipped through the system tests.

[kbyatnal]

@landrito

I'm on node version 5.5.0 right now.

[landrito]

@kbyatnal I just published google-gax 0.10.5 which should fix the issue!

[kbyatnal]

@landrito

Awesome, that part seems to be fixed now!

I guess this brings us back to the original issue. I just tried calling the async method in the API with a 1 minute raw audio file and I got the "quota exceeded issue". I checked my API console and that one request resulted in a quota usage of 486 requests, which leads me to believe that the exponential backoff isn't working. This is with the latest google-gax version 0.10.5.

[landrito]

@kbyatnal Can you try calling the async method on google-gax version 0.10.6. I pushed a bug-fix that should solve the exponential backoff problem.

[landrito]

The quota exceeded issue is resolved with 10.6.0.

@kbyatnal, just want to make sure you try 0.10.6 before closing this issue.

[stephenplusplus]

Let's call it resolved, but please let us know if anything is not functioning as expected.