Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add finance advisor spanner demo #901

Merged
merged 86 commits into from
Aug 12, 2024
Merged
Show file tree
Hide file tree
Changes from 80 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
a89eec7
initial commit
anirbanbagchi1979 Jul 22, 2024
fc071d3
Readme updates
anirbanbagchi1979 Jul 23, 2024
d645c45
readme updates
anirbanbagchi1979 Jul 23, 2024
396cf2c
modififed network param in app,yaml
anirbanbagchi1979 Jul 23, 2024
12edd21
env file added
anirbanbagchi1979 Jul 24, 2024
7c3d8d8
Env Added
anirbanbagchi1979 Jul 24, 2024
e6fa4bd
modifications for CloudRun Deployment
anirbanbagchi1979 Jul 24, 2024
568c169
modifications on formatting
anirbanbagchi1979 Jul 24, 2024
a6e81fa
Added embedding vector data update scripts
anirbanbagchi1979 Jul 24, 2024
3a4f6cc
more cleanup readme
anirbanbagchi1979 Jul 24, 2024
14d416e
Pylance Cleanup
anirbanbagchi1979 Jul 24, 2024
d9ad2a8
renamed the graph file
anirbanbagchi1979 Jul 24, 2024
29ca3a2
fix: Graph file name
anirbanbagchi1979 Jul 24, 2024
5eb2bd7
fix: Graph Viz
anirbanbagchi1979 Jul 24, 2024
e654e18
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 25, 2024
c538a76
fix: finance-advisor-spanner: Removes App Engine Deployment Options
anirbanbagchi1979 Jul 25, 2024
87b85ac
fix: finance-advisor-spanner remove references to App Engine
anirbanbagchi1979 Jul 25, 2024
50536fc
fix: finance-advisor-spanner Fixed app.yaml reference to Home.py
anirbanbagchi1979 Jul 25, 2024
2351317
fix: finance-advisor-spanner Env file changd to TMP and added instruc…
anirbanbagchi1979 Jul 25, 2024
ad981ab
fix: finance-advisor-spanner Remove .env file
anirbanbagchi1979 Jul 25, 2024
ab8d4af
Merge branch 'main' into finance-advisor-spanner
holtskinner Jul 25, 2024
de55c14
Add terms to allowlist
holtskinner Jul 25, 2024
d286f8b
fix: finance-advisor-spanner Issues from 901 code review resolved
anirbanbagchi1979 Jul 26, 2024
e92088e
fix: finance-advisor-spanner 901 fixes
anirbanbagchi1979 Jul 26, 2024
9aaeb23
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 26, 2024
f76f97e
fix: finance-advisor-spanner lint errros resolved
anirbanbagchi1979 Jul 26, 2024
6385a25
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 26, 2024
228f223
fix: finance-advisor-spanner lint code fixes
anirbanbagchi1979 Jul 26, 2024
a1142d7
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 26, 2024
614743a
Merge branch 'main' into finance-advisor-spanner
holtskinner Jul 29, 2024
2a155db
Merge branch 'main' into finance-advisor-spanner
holtskinner Jul 31, 2024
e5032f9
Merge branch 'main' into finance-advisor-spanner
holtskinner Jul 31, 2024
4e77493
fix: finance-advisor-spanner 901 fixes
anirbanbagchi1979 Jul 31, 2024
aa06c25
fix: finance-advisor-spanner lint errors resovled
anirbanbagchi1979 Jul 31, 2024
061caf4
Merge branch 'main' into finance-advisor-spanner
holtskinner Jul 31, 2024
3eeed3c
Add sqlfluff ignore
holtskinner Jul 31, 2024
5d8ebfb
Update lint ignore for Spanner SQL Files
holtskinner Jul 31, 2024
7e1b519
Update filterregexexclude
holtskinner Jul 31, 2024
8d9cfb1
fix: finance-advisor-spanner fixed lint errors
anirbanbagchi1979 Aug 1, 2024
5da3e42
fix: finance-advisor-spanner remove single quotes linting error
anirbanbagchi1979 Aug 1, 2024
dfe6098
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 1, 2024
400abd5
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 1, 2024
d31eb9e
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 1, 2024
cff7c08
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 1, 2024
b6fb915
fix: finance-advisor-spanner more lint changed
anirbanbagchi1979 Aug 1, 2024
e63af94
fix: finance-advisor-spanner increasing reuse
anirbanbagchi1979 Aug 1, 2024
bf7e377
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 1, 2024
8fd61d8
fix: finance-advisor-spanner
anirbanbagchi1979 Aug 1, 2024
23a218d
fix: finance-advisor-spanner
anirbanbagchi1979 Aug 1, 2024
518b08b
Update gemini/sample-apps/finance-advisor-spanner/pages/3_Graph_Visua…
holtskinner Aug 5, 2024
277611a
Update gemini/sample-apps/finance-advisor-spanner/pages/4_Exposure_Ch…
holtskinner Aug 5, 2024
c0910a0
Apply suggestions from code review
holtskinner Aug 5, 2024
354a689
Merge branch 'main' into finance-advisor-spanner
holtskinner Aug 5, 2024
a24f0fe
Merge branch 'main' into finance-advisor-spanner
holtskinner Aug 6, 2024
fcfdeba
Update gemini/sample-apps/finance-advisor-spanner/README.md
anirbanbagchi1979 Aug 9, 2024
8e5ae28
Update gemini/sample-apps/finance-advisor-spanner/README.md
anirbanbagchi1979 Aug 9, 2024
df3b448
Update gemini/sample-apps/finance-advisor-spanner/pages/1_Asset_Searc…
anirbanbagchi1979 Aug 9, 2024
16b2c90
Update gemini/sample-apps/finance-advisor-spanner/pages/2_Semantic_Se…
anirbanbagchi1979 Aug 9, 2024
a0c06ac
fix: finance-advisor-spanner code fixes
anirbanbagchi1979 Aug 9, 2024
2eadd1a
Update gemini/sample-apps/finance-advisor-spanner/pages/1_Asset_Searc…
anirbanbagchi1979 Aug 9, 2024
8ddd3b2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
a142aa2
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 9, 2024
40ce8e2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
4a02ae7
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 9, 2024
b32f72e
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
ab50f62
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 9, 2024
e4c509a
fix: took care of common code
anirbanbagchi1979 Aug 9, 2024
e93fdd9
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
b5b4690
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
anirbanbagchi1979 Aug 9, 2024
acf257d
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
990c75e
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
anirbanbagchi1979 Aug 9, 2024
7d65c5a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
9c40654
fix: duplicate function definition
anirbanbagchi1979 Aug 9, 2024
b75666b
fix: rename Home to home
anirbanbagchi1979 Aug 9, 2024
92cbc2d
Rename Home.py to home.py
anirbanbagchi1979 Aug 9, 2024
5ed68b2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
8bebddc
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 9, 2024
519b013
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2024
fc80634
Merge branch 'finance-advisor-spanner' of https://github.com/GoogleCl…
gcf-owl-bot[bot] Aug 9, 2024
d77384d
Merge branch 'main' into finance-advisor-spanner
holtskinner Aug 12, 2024
5bce199
fix: change to google-cloud-spanner
anirbanbagchi1979 Aug 12, 2024
505a525
fix: lint errors
anirbanbagchi1979 Aug 12, 2024
ff05f9c
fix: code cleanup
anirbanbagchi1979 Aug 12, 2024
a44ba03
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 12, 2024
ea496e0
fix: code formatted
anirbanbagchi1979 Aug 12, 2024
3d52fbb
Fix spelling error - "queryies"
holtskinner Aug 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
/generative-ai/vision/getting-started @iamthuya @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/use-cases/intro_multimodal_use_cases.ipynb @saeedaghabozorgi @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/sample-apps/genwealth/ @paulramsey @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/sample-apps/finance-advisor-spanner/ @anirbanbagchi1979 @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/use-cases/applying-llms-to-data/analyze-poster-images-in-bigquery/poster_image_analysis.ipynb @aliciawilliams @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/use-cases/retail/product_attributes_extraction.ipynb @tianli @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/use-cases/retrieval-augmented-generation/RAG_Based_on_Sensitive_Data_Protection_using_Faker.ipynb @ainaomotayo @GoogleCloudPlatform/generative-ai-devrel
Expand Down
29 changes: 29 additions & 0 deletions .github/actions/spelling/allow.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
AFX
ANZ
APIENTRY
APSTUDIO
Adidas
Expand All @@ -7,6 +8,7 @@ Arborio
Arepa
Arsan
Ashish
Aspeed
Autechre
Autorater
BITCODE
Expand Down Expand Up @@ -54,6 +56,7 @@ FLX
FMWK
FPDF
FTPS
Finvest
Firestore
Fishburne
Flatform
Expand All @@ -69,6 +72,7 @@ Genkit
Gisting
Glickman
Googlers
HDFC
HIDPI
HMO
HREDRAW
Expand All @@ -78,11 +82,13 @@ Hickson
Hmmm
Hogwarts
Hubmann
ICICI
INFOPLIST
IVF
Jang
Jedi
Joji
KNN
Kaelen
Kaggle
Kamradt
Expand All @@ -104,12 +110,17 @@ Lottry
MSCHF
MSGSEND
MTL
Maarten
Mahindra
Mamah
Mandiri
Memegen
Mewgler
Mosher
NARI
NCCREATE
NDEBUG
NGRAMS
NMT
NOMINMAX
NOZORDER
Expand All @@ -123,7 +134,9 @@ Onone
PDFs
PLOTLYENV
Parmar
Persero
Phaidon
Pharma
QPM
Qwiklabs
RAGAS
Expand All @@ -132,7 +145,9 @@ ROOTSPAN
RRF
RTN
RYDE
Resona
Rizzoli
Robeco
SDKROOT
SEK
SEO
Expand All @@ -149,6 +164,7 @@ Schwimmer
Selam
Sestero
Shazeer
Shenzhou
Simpsons
Siri
Sketchfab
Expand All @@ -157,11 +173,14 @@ Storrer
Strappy
Surampudi
TARG
TOKENLIST
TPU
TPUs
Tadao
Tafel
Tbk
Tbl
Tencent
Testables
Tianli
Topolino
Expand All @@ -170,6 +189,7 @@ Tribbiani
Tricyle
UDFs
USERDATA
Unimicron
Urs
Uszkoreit
VFT
Expand All @@ -187,6 +207,7 @@ Wehn
Wnd
Womens
XXE
Zijin
Zscaler
Zuercher
aadd
Expand All @@ -204,6 +225,7 @@ astype
autoptr
autosxs
backticks
bagchi
barmode
barpolar
baxis
Expand All @@ -224,6 +246,7 @@ colab
coloraxis
colorbar
colorway
colvis
colwidth
constexpr
corpuses
Expand Down Expand Up @@ -261,6 +284,7 @@ forno
freedraw
freopen
fromarray
fts
fulltext
funtion
gboolean
Expand Down Expand Up @@ -312,6 +336,7 @@ iphoneos
ipykernel
ipynb
isa
itables
iterrows
jegadesh
jetbrains
Expand Down Expand Up @@ -344,13 +369,16 @@ mec
meme
memes
metadatas
mgrs
miranda
mpn
nbconvert
nbfmt
nbformat
ncols
ndarray
ngram
ngrams
nlp
nmade
nmilitary
Expand Down Expand Up @@ -398,6 +426,7 @@ pymupdf
pypdf
pyplot
pysftp
pyvis
qubit
qubits
ragas
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/linter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ jobs:
env:
DEFAULT_BRANCH: main
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FILTER_REGEX_EXCLUDE: 'gemini/sample-apps/finance-advisor-spanner/.*\.sql'
JAVASCRIPT_DEFAULT_STYLE: prettier
LOG_LEVEL: WARN
SHELLCHECK_OPTS: -e SC1091 -e 2086
Expand Down
10 changes: 10 additions & 0 deletions gemini/sample-apps/finance-advisor-spanner/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM python:3.12

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["streamlit", "run", "home.py", "--server.enableCORS", "false", "--browser.serverAddress", "0.0.0.0", "--browser.gatherUsageStats", "false", "--server.port", "8080"]
128 changes: 128 additions & 0 deletions gemini/sample-apps/finance-advisor-spanner/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Finvest Spanner Demo App

**Authors:** [Anirban Bagchi](https://github.com/anirbanbagchi1979) and [Derek Downey](https://github.com/dtest)

<img align="right" style="padding-left: 10px;" src="https://storage.googleapis.com/github-repo/generative-ai/sample-apps/finance-advisor-spanner/images/Finvest-white.jpg" width="35%" alt="Finvest Logo">

Consider a modern financial services company where I am a financial advisor. Finding the right financial investments can be challenging because of the complex nature of investments from structured data such as expense ratios, fund returns, to complex data such as asset holdings, their industry sectors, and more unstructured data, such as investment philosophy and client's investment goals. Let me show you how Spanner makes this process easy by combining these diverse data structures into a single multi-model platform.

The client wants me to find assets for funds in North America and Europe that invest in derivatives. I select North America and Europe and put in derivatives as my search term. Spanner runs a relational and text search to return a list of funds.

Next, the client wants to narrow this list to specific fund managers. I don't know the exact name, so I put in Liz Peters, and Spanner performs a fuzzy match(Full Text Search - Substring Match) of the name Liz Peters to find funds managed by Elizabeth Peterson.

Among these funds, the client wants to choose from socially responsible funds. Next, I check the box for vector search, and now I can see ESG funds because Spanner performed a KNN vector search to match the search term "socially responsible" with "environmental, social and governance".

Finally, before I recommend a fund, I also want to check the exposure to a particular sector. This can be complex because funds can invest in other funds, called fund of funds which makes it hard to compute this. Spanner performs a graph search using this asset knowledge graph. By traversing the funds and its holdings which could also be funds and their holdings, Spanner can compute the client's exposure to a particular sector. I can see the funds that have exposure of 20% or more in the technology sector.

With the power of Spanner's multimodel support, I can run complex workloads on a single database for relational, analytical, text and vector use cases with virtually unlimited scale, five nines of availability—including enterprise security and governance for mission critical workloads.

This demo highlights [Spanner](https://cloud.google.com/spanner), integration with [Vertex AI LLMs](https://cloud.google.com/model-garden?hl=en) for both embeddings and text completion models. You will learn how Spanner can help with use cases where you run Full Text Search, Approximate Nearest Neighbor search and vector similarity search.

## Tech Stack

The Finvest Spanner demo application was built using:

- [Spanner](https://cloud.google.com/spanner)
- [Vertex AI](https://cloud.google.com/vertex-ai?hl=en) LLMs ([textembeddings-gecko@004](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings) )
- [Cloud Run](https://cloud.google.com/run)
- [Dataflow](https://cloud.google.com/dataflow?)
- [Streamlit](https://streamlit.io/)

## Deploying the Finvest Spanner Demo Application

1. Login to the [Google Cloud Console](https://console.cloud.google.com/).

2. [Create a new project](https://developers.google.com/maps/documentation/places/web-service/cloud-setup) to host the demo and isolate it from other resources in your account.

3. [Switch](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects) to your new project.

4. [Activate Cloud Shell](https://cloud.google.com/shell/docs/using-cloud-shell) and confirm your project by running the follow4ng commands. Click **Authorize** if prompted.

```bash
gcloud auth list
gcloud config list project
```

5. Clone this repository and navigate to the project root:

```bash
cd
git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai/gemini/sample-apps/finance-advisor-spanner/
```

6. Create a Spanner instance
<https://console.cloud.google.com/spanner/instances/new>

> Note the instance Name

7. Import the data into the Spanner instance
<https://cloud.google.com/spanner/docs/import#import-database>
The bucket which has the Spanner export is in this public GCS Bucket

`https://storage.googleapis.com/github-repo/generative-ai/sample-apps/finance-advisor-spanner/spanner-fts-mf-data-export/`

> Note the Database Name

The import process will run and import the database into a new Spanner database.

8. Run Additional DDL statements for the database to have all the necessary components.
The DDL statements are in [Schema-Operations.sql](./Schema-Operations.sql) file in this directory.

Change the endpoint as per your project and the spanner instance location

```sql
ALTER MODEL EmbeddingsModel SET OPTIONS (
endpoint = '//aiplatform.googleapis.com/projects/'YOUR PROJECT ID HERE'/locations/'YOUR SPANNER INSTANCE LOCATION HERE'/publishers/google/models/text-embedding-003'
)
;
```

Next run the rest of DDL statements without any change

9. In Cloud Shell:

Open `.env` file in the same directory using vi or other Editor

Edit the following fields with the instance name from Step 6 and database name from Step 7

```bash
instance_id='YOUR INSTANCE ID'
database_id='YOUR DATABASE ID'
```

10. Now Build & Deploy the application:

Build:

```bash
gcloud builds submit --tag gcr.io/'YOUR PROJECT ID HERE'/finance-advisor-app
```

Deploy:

```bash
gcloud run deploy finance-advisor-app --image gcr.io/'YOUR PROJECT ID HERE'/finance-advisor-app --platform managed --region 'YOUR SPANNER REGION' --allow-unauthenticated
```

### Troubleshooting

### Frontend

The frontend application is Streamlit running on CloudRun

## Purpose and Extensibility

The purpose of this repository is to help you provision an isolated demo environment that highlights the Full Text Search, Semantic Search and Graph capabilities of Spanner. While the ideas in this repository can be extended for many real-world use cases, the demo code itself is overly permissive and has not been hardened for security or reliability. The sample code in this repository is provided on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, and it should NOT be used for production use cases without doing your own testing and security hardening.

## Clean Up

Be sure to delete the resources you no longer need when you're done with the demo. If you created a new project for the lab as recommended, you can delete the whole project using the command below in your Cloud Shell session (NOT the pgadmin VM).

**DANGER: Be sure to set PROJECT_ID to the correct project, and run this command ONLY if you are SURE there is nothing in the project that you might still need. This command will permanently destroy everything in the project.**

```bash
# Set your project id
PROJECT_ID='YOUR PROJECT ID HERE'
gcloud projects delete ${PROJECT_ID}
```
Loading
Loading