Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update deepgram endpointing #145

Merged
merged 4 commits into from
Feb 6, 2024
Merged

Update deepgram endpointing #145

merged 4 commits into from
Feb 6, 2024

Conversation

seanmuirhead
Copy link
Contributor

Notes:

  • Added speech_final to SpeechEvent. Defaulted to False because looks like it's being used elsewhere.
  • Return speech_final for each SpeechEvent.
  • Add utterance_end_ms for more control over VAD

Relevant Docs:
https://developers.deepgram.com/docs/understanding-end-of-speech-detection
https://developers.deepgram.com/docs/understand-endpointing-interim-results

@CLAassistant
Copy link

CLAassistant commented Feb 1, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

theomonnom
theomonnom approved these changes Feb 1, 2024
View reviewed changes
Copy link
Member

@theomonnom theomonnom left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks for contributing!

@Vibrat
deepgram: Add min_silence_duration to deepgram client.
bd32d09 
deepgram controls vad by endpointing parameter, this fix allows to configure min_silence_duration in agents layer
@davidzhao davidzhao mentioned this pull request Feb 4, 2024
Closed
@Vibrat
Copy link
Contributor

Vibrat commented Feb 5, 2024

Hi @seanmuirhead,

I come from a similar PR #148.

We are sharing some changes on the codebase. I think the end result of both endpointing and interim is to have speech_final and is_final as the final output. Would be nice if we can rebase my change onto your PR so that I can close my ticket and we continue the discussion here

@seanmuirhead
Copy link
Contributor Author

Hi @Vibrat,

Yes, I was thinking the same thing! Will do that by EOD PST.

@davidzhao
Copy link
Member

what do you think about exposing end_of_speech as the field instead of speech_final? It seems a bit more generic to signal the end of input.. and also trying to avoid having both speech_final and is_final.

@seanmuirhead
Copy link
Contributor Author

@davidzhao Ya I agree, much cleaner that way. Will include in rebase

@seanmuirhead
Copy link
Contributor Author

@davidzhao @Vibrat done! Removed utterance_end_ms according to Deepgram docs that were updated ~6hrs ago. Two notes for comment:

  1. @Vibrat 's min_silence_duration sound good for the attribute name?
  2. Deepgram has endpointing cast as a string, which is why we've done it here, but int works just as fine for the record.

davidzhao
davidzhao approved these changes Feb 6, 2024
View reviewed changes
Copy link
Member

@davidzhao davidzhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good! we'll get it merged and update the other plugins to behave the same way.

@davidzhao davidzhao merged commit 604d7e3 into livekit:main Feb 6, 2024
1 check passed
askable-nic added a commit to askableorg/aimod-poc-livekit-agent that referenced this pull request Feb 8, 2024
@askable-nic
Squashed commit of the following:
a9da4d9 
commit 5fab872
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 17:18:48 2024 -0800

    Kitt to handle empty deepgram text + elevenlabs bug fix (livekit#155)

    hotfix

commit 7661824
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 16:25:35 2024 -0800

    Bump deepgram version (livekit#154)

commit ddc168e
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 16:10:52 2024 -0800

    Fixes to Elevenlabs (livekit#152)

commit 5555526
Author: David Zhao <[email protected]>
Date:   Tue Feb 6 09:12:38 2024 -0800

    Update KITT and other plugins to use end_of_speech field (livekit#153)

    * Update KITT and other plugins to use end_of_speech field

    Tested with KITT. It significantly improves the end of speech behavior so that
    we are giving it a 1s wait before starting to process user input.

    * ruff on 3.10

    * use ruff action

    * fixed ruff

commit a98a7c1
Author: Paul Lockett <[email protected]>
Date:   Tue Feb 6 00:02:51 2024 -0800

    update tts init to export StreamAdapter (livekit#149)

commit 604d7e3
Author: Sean Muirhead <[email protected]>
Date:   Mon Feb 5 21:35:55 2024 -0800

    Update deepgram endpointing  (livekit#145)

    * deepgram: Add min_silence_duration to deepgram client.

    deepgram controls vad by endpointing parameter, this fix allows to configure min_silence_duration in agents layer

    * add utterance_end_ms and speech_final to Deepgram plugin

    * add utterance_end_ms and speech_final to Deepgram plugin

    * expose speech_final as end_of_speech

    ---------

    Co-authored-by: Lam Nguyen <[email protected]>
askable-nic added a commit to askableorg/aimod-poc-livekit-agent that referenced this pull request Feb 22, 2024
@askable-nic
Squashed commit of the following:
17209f3 
commit 5fab872
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 17:18:48 2024 -0800

    Kitt to handle empty deepgram text + elevenlabs bug fix (livekit#155)

    hotfix

commit 7661824
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 16:25:35 2024 -0800

    Bump deepgram version (livekit#154)

commit ddc168e
Author: Neil Dwyer <[email protected]>
Date:   Tue Feb 6 16:10:52 2024 -0800

    Fixes to Elevenlabs (livekit#152)

commit 5555526
Author: David Zhao <[email protected]>
Date:   Tue Feb 6 09:12:38 2024 -0800

    Update KITT and other plugins to use end_of_speech field (livekit#153)

    * Update KITT and other plugins to use end_of_speech field

    Tested with KITT. It significantly improves the end of speech behavior so that
    we are giving it a 1s wait before starting to process user input.

    * ruff on 3.10

    * use ruff action

    * fixed ruff

commit a98a7c1
Author: Paul Lockett <[email protected]>
Date:   Tue Feb 6 00:02:51 2024 -0800

    update tts init to export StreamAdapter (livekit#149)

commit 604d7e3
Author: Sean Muirhead <[email protected]>
Date:   Mon Feb 5 21:35:55 2024 -0800

    Update deepgram endpointing  (livekit#145)

    * deepgram: Add min_silence_duration to deepgram client.

    deepgram controls vad by endpointing parameter, this fix allows to configure min_silence_duration in agents layer

    * add utterance_end_ms and speech_final to Deepgram plugin

    * add utterance_end_ms and speech_final to Deepgram plugin

    * expose speech_final as end_of_speech

    ---------

    Co-authored-by: Lam Nguyen <[email protected]>
parshvadaftari pushed a commit to parshvadaftari/agents that referenced this pull request Jun 23, 2024
@shahafabileah
Docs to explain how outbound calls work (livekit#145)
5315642 
Adapted from this discussion:
https://discord.com/channels/1079125925163180093/1110708807883034646/1110708812979118183
SuJingnan pushed a commit to SuJingnan/agents that referenced this pull request Nov 26, 2024
@seanmuirhead @Vibrat
Update deepgram endpointing (livekit#145)
1a47d3d 
* deepgram: Add min_silence_duration to deepgram client.

deepgram controls vad by endpointing parameter, this fix allows to configure min_silence_duration in agents layer

* add utterance_end_ms and speech_final to Deepgram plugin

* add utterance_end_ms and speech_final to Deepgram plugin

* expose speech_final as end_of_speech

---------

Co-authored-by: Lam Nguyen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@theomonnom theomonnom theomonnom approved these changes

@davidzhao davidzhao davidzhao approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@seanmuirhead @CLAassistant @Vibrat @davidzhao @theomonnom