Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: add lms for data post #8222

Merged
merged 16 commits into from
Feb 5, 2024
Merged

docs: add lms for data post #8222

merged 16 commits into from
Feb 5, 2024

Conversation

lostmygithubaccount
Copy link
Member

Description of changes

work in progress but code is finalized IMO

follow-up post for local "open source" LMs needed

covers:

  • use cases for language models w/ data
  • demonstration of each use case

Issues closed

@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 4, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 4, 2024
@lostmygithubaccount
Copy link
Member Author

forgot to wrap some lines -- want to get feedback, then finish all the prose/cleanup

@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 4, 2024

Docs preview: https://pr-8222-0a45074f511d1caafd39d4d9dd8ac07a623a0df6--ibis-quarto.netlify.app

@lostmygithubaccount lostmygithubaccount added docs-preview Add this label to trigger a docs preview docs Documentation related issues or PRs labels Feb 4, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 4, 2024
@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 4, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 4, 2024
@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 4, 2024

Docs preview: https://pr-8222-0c877433b8b66f39ca384045b7796aebd14ec742--ibis-quarto.netlify.app

1 similar comment
@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 4, 2024

Docs preview: https://pr-8222-0c877433b8b66f39ca384045b7796aebd14ec742--ibis-quarto.netlify.app

@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 5, 2024

Docs preview: https://pr-8222-dd2fe1c5298f36a5cbc1333782f422ef98d13069--ibis-quarto.netlify.app

@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 5, 2024

Docs preview: https://pr-8222-9c10acc865428636bff9a3d18dcab95d7f60b595--ibis-quarto.netlify.app

@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
chiphuyen
chiphuyen reviewed Feb 5, 2024
View reviewed changes
docs/posts/lms-for-data/index.qmd

## Natural language processing

This includes tasks like:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it's powerful to be able to say: extract sentiment of each row, then do a join/groupby, all in GPU (instead of doing sentiment on gpus and join/groupby on cpus)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're not there with Ibis just yet! soon...

chiphuyen
chiphuyen reviewed Feb 5, 2024
View reviewed changes
docs/posts/lms-for-data/index.qmd Outdated

We can think of three approaches to analytical code with language models:

1. Use LMs in an analytic subroutine
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like an agent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, just what's shown above -- basically using LLMs in UDFs

@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 5, 2024

Docs preview: https://pr-8222-4ef57f72ca0de9ca6eb29de1c395f8138bbd8715--ibis-quarto.netlify.app

@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 5, 2024

Docs preview: https://pr-8222-f49fcd130b5222c0606b54e84facb467a3f3ab75--ibis-quarto.netlify.app

@lostmygithubaccount lostmygithubaccount added the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Feb 5, 2024
@lostmygithubaccount
Copy link
Member Author

@cpcloud this should be good to merge!

@ibis-docs-bot
Copy link

ibis-docs-bot bot commented Feb 5, 2024

Docs preview: https://pr-8222-f71a1387499df4a18fdd9afff2cd64255953aea9--ibis-quarto.netlify.app

jitingxu1
jitingxu1 approved these changes Feb 5, 2024
View reviewed changes
Copy link
Contributor

@jitingxu1 jitingxu1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

docs/posts/lms-for-data/index.qmd Show resolved Hide resolved
docs/posts/lms-for-data/index.qmd Outdated Show resolved Hide resolved
docs/posts/lms-for-data/index.qmd
- summarization
- translation
- question answering

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is embedding or encoding good use cases in the future?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could but I haven't really understood why these are useful (as opposed to storing text)

docs/posts/lms-for-data/index.qmd Outdated
```

1. Import Ibis, the data engineering toolkit
2. Import Marvin, the AI engineering toolkit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe some confusion here, "our team" means ibis. I thought something like langchain.

I believe our team already invested lots of time on the toolkit, I am not very sure how popular marvin is.

@lostmygithubaccount
Copy link
Member Author

argh let me fix some of those last things before we merge

@lostmygithubaccount
Copy link
Member Author

ready to merge

@cpcloud cpcloud merged commit 8f35010 into ibis-project:main Feb 5, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jitingxu1 jitingxu1 jitingxu1 approved these changes

@cpcloud cpcloud cpcloud approved these changes

@chiphuyen chiphuyen Awaiting requested review from chiphuyen

Assignees
No one assigned
Labels
docs Documentation related issues or PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lostmygithubaccount @cpcloud @chiphuyen @jitingxu1