[bounty] speaker identification (next steps)

[louis030195]

WIP

also i'm curious how we could associate frames to specific person (speaker) somehow (and thus screen text) but this is lower priority

/bounty 200

(TBD what is exactly the things to do)

[linear]

MED-291 [bounty] speaker identification (next steps)

[louis030195]

kinda tihkning about google photo ui

i think we need similar for speaker identification over long ranges, eg "listen to this voice, is it john? can you tell me the name?"

i bet they compute embeddings of face and group together and tune based on user feedback

[louis030195]

@EzraEllette what do you mean by "Attempt to use LLM for identification through meeting context"

we already do this in meeting page

[algora-pbc]

💎 $200 bounty • Screenpi.pe

Steps to solve:

1. Start working: Comment /attempt #695 with your implementation plan
2. Submit work: Create a pull request including /claim #695 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @EzraEllette	Nov 18, 2024, 9:46:17 PM	WIP

[EzraEllette]

@EzraEllette what do you mean by "Attempt to use LLM for identification through meeting context"

we already do this in meeting page

When this function is called, It should now be able to update speaker information in the database with user permission.

[EzraEllette]

/attempt #695

Algora profile	Completed bounties	Tech	Active attempts	Options
@EzraEllette	11 mediar-ai bounties	Rust, TypeScript, JavaScript & more		Cancel attempt

[NicodemPL]

Limitless uses emails/names from calendar events. Might be good idea to implement calendar to meetings at this stage.

Other idea - OCR from meeting frames to find and confirm names. We already have this data (OCR)

[louis030195]

@EzraEllette fyi build is down because of sqlite extension i feel like

https://github.com/mediar-ai/screenpipe/actions/runs/11904908266/job/33174510422
https://github.com/mediar-ai/screenpipe/actions/runs/11904908266/job/33174511134

[louis030195]

Limitless uses emails/names from calendar events. Might be good idea to implement calendar to meetings at this stage.

Other idea - OCR from meeting frames to find and confirm names. We already have this data (OCR)

yes to 2

the screen (and mic) is the universal interface that contains all apps - we do not need any integration if we just use pixels and LLMs well

let's try without calendar first and if it's too hard we can think about it but it involves many things (auth, cloud API, db etc) that are not core to screenpipe

[EzraEllette]

@EzraEllette fyi build is down because of sqlite extension i feel like

https://github.com/mediar-ai/screenpipe/actions/runs/11904908266/job/33174510422

https://github.com/mediar-ai/screenpipe/actions/runs/11904908266/job/33174511134

It looks like knfc. I will debug when I get a chance