Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve callable function for long running requests. #1635

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

taeold
Copy link
Contributor

@taeold taeold commented Nov 8, 2024

In #1634, we introduced new API to return response chunks for callable functions. This means that callable function response my be long-running.

To ensure that long-running requests are handled well, we are adding 2 new features:

  1. heartbeat: Some network proxies may close request connection when no response data is sent for some time interval. By default, callable function will send a heartbeat ping every 30 seconds. Our client SDKs will ignore these pings. Users can disable heartbeat for streaming response by setting heartbeatSeconds configuration to null.

  2. AbortSignal: response object of the callable handler will now include signal property which the developer can use to check and see if a client connection has been dropped. They can forward this signal property to fetch-like APIs so that underlying calls would be terminated properly.

e.g.

exports.generateText = onCall(
  { 
    heartbeatSeconds: 10 // (Default: 30. Set to 'null' to disable heartbeat. )
  },
  async (request, response) => {
    const prompt = request.data.text;
    const result = await model.generateContentStream(
      prompt,
      { signal: response.signal }
    );

    for await (const chunk of result.stream) {
      if (response.acceptsStreaming) {
        // if the client did not request a streaming response,
        // response.write() is a no-op.
        response.write(chunk.text())
      }
    }
    return (await response()).text()
  }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant