Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add a code sample for creating a kmeans model #267

Merged
merged 40 commits into from
Feb 27, 2024
Merged
Changes from 10 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
69fe5d7
k-means code sample
Dec 6, 2023
5fb1d4f
formatting
Dec 11, 2023
523255f
added test
Dec 11, 2023
3bb267a
docs: add code sampke for creating kmeans model
Dec 11, 2023
73d2a46
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Dec 11, 2023
b3c0578
license header + region tags added
Dec 12, 2023
c25aeb5
Resolved conflicts
Dec 12, 2023
db9f439
Update samples/snippets/create_kmeans_model_test.py
SalemJorden Dec 12, 2023
e7bd5ef
code corrections resolved
Dec 13, 2023
2a7d575
resolved merge conflicts
Dec 13, 2023
809ed05
code corrections commit 1
Dec 18, 2023
5dba2b9
descriptions of geospatial analysis functions
Dec 18, 2023
2207941
explantions revised for clarity
Dec 18, 2023
5e00a3c
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Dec 18, 2023
7c64227
Update samples/snippets/create_kmeans_model_test.py
SalemJorden Dec 19, 2023
11678e0
code corrections
Dec 19, 2023
f95cd9f
code revision
Dec 19, 2023
0df2dec
code changes
Dec 20, 2023
06a2490
revisions
Dec 21, 2023
1a9f7d9
expected output previews
Dec 26, 2023
464cf1c
revisions
Dec 26, 2023
d03f46c
Merge remote-tracking branch 'origin/main' into Salem
Dec 26, 2023
019e243
tests passing, expected output characters >80
Dec 26, 2023
72174f9
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Dec 26, 2023
50a447d
Merge branch 'main' into Salem
tswast Dec 26, 2023
ac348bf
column wrapping
Dec 27, 2023
7ce5337
merge
Dec 27, 2023
29b2e1f
Merge branch 'main' into Salem
SalemJorden Jan 30, 2024
7762f0f
Merge branch 'main' into Salem
SalemJorden Jan 30, 2024
479a828
Merge branch 'main' into Salem
SalemJorden Feb 5, 2024
1572ddd
reset session before running code smaples
SalemJorden Feb 5, 2024
3d77ddd
Update samples/snippets/create_kmeans_model_test.py
SalemJorden Feb 6, 2024
505b790
predict function added to tutorial
SalemJorden Feb 7, 2024
4505c5c
replaced project_id with model_id
SalemJorden Feb 9, 2024
cad2185
Merge branch 'main' into Salem
SalemJorden Feb 23, 2024
3ab8220
reformatting
SalemJorden Feb 23, 2024
816881c
Merge branch 'Salem' of https://github.com/googleapis/python-bigquery…
SalemJorden Feb 23, 2024
9b382d6
Merge branch 'main' into Salem
tswast Feb 23, 2024
ae9a362
reformat
SalemJorden Feb 26, 2024
5eb59ec
Merge branch 'main' into Salem
tswast Feb 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions samples/snippets/create_kmeans_model_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Copyright 2023 Google LLC
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's 2024 now. These headers should reflect when the text was first written.

Suggested change
# Copyright 2023 Google LLC
# Copyright 2024 Google LLC

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

def test_kmeans_sample():
# [START bigquery_dataframes_bqml_kmeans]
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved
import bigframes.pandas as bpd
import bigframes
from bigframes import dataframe
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved
import bigframes.pandas as bpd
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved
import datetime
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved

#Load data from BigQuery
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved
h = bpd.read_gbq("bigquery-public-data.london_bicycles.cycle_hire", h.rename(
columns = {"start_station_name": "station_name", "start_station_id": "station_id"}
))
s = bpd.read_gbq(
tswast marked this conversation as resolved.
Show resolved Hide resolved
"""
SELECT
id,
ST_DISTANCE(
ST_GEOGPOINT(s.longitude, s.latitude),
ST_GEOGPOINT(-0.1, 51.5)
) / 1000 AS distance_from_city_center
FROM
`bigquery-public-data.london_bicycles.cycle_stations` s
""" )

# transform data into queryable format
SalemJorden marked this conversation as resolved.
Show resolved Hide resolved
sample_time = datetime.datetime(2015, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc)
sample_time2 = datetime.datetime(2016, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc)

h = h.loc[(h["start_date"] >= sample_time) & (h["start_date"] <= sample_time2)]

h.start_date.dt.dayofweek.map(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just runs the mapping and discards the results. You're going to need to save this somewhere. You probably want something like:

h = h.assign(
    isweekday=h.start_date.dt.dayofweek.map(
# ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected.

{
0: "weekday",
1: "weekday",
2: "weekday",
3: "weekday",
4: "weekday",
5: "weekend",
6: "weekend",
}
)

#merge dataframes h and s
merged_df = h.merge(
right=s,
how="inner",
left_on="station_id",
right_on="id",
)
# Create new dataframe variable from merge: 'stationstats'
stationstats = merged_df.groupby("station_name").agg(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's actually a mistake in the SQL version of this tutorial. We actually want to groupby both "station_name" and "isweekday" like the actual CREATE MODEL query is doing, otherwise there's no point in having computed "isweekday" in the first place IMO:

  SELECT
    station_name,
    isweekday,
    AVG(duration) AS duration,
    COUNT(duration) AS num_trips,
    MAX(distance_from_city_center) AS distance_from_city_center
  FROM
    hs
  GROUP BY
    station_name, isweekday)

https://cloud.google.com/bigquery/docs/kmeans-tutorial#run_the_query_2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added "isweekday" to groupby() function.

{"duration": ["mean", "count"], "distance_from_city_center": "max"}
)
# [END bigquery_dataframes_bqml_kmeans]
tswast marked this conversation as resolved.
Show resolved Hide resolved


# [START bigquery_dataframes_bqml_kmeans_fit]

# import the KMeans model from bigframes.ml to cluster the data
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

from bigframes.ml.cluster import KMeans

cluster_model = KMeans(n_clusters=4)
tswast marked this conversation as resolved.
Show resolved Hide resolved
cluster_model = cluster_model.fit(stationstats).to_gbq(cluster_model)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do a to_gbq() here to save the model to a permanent location.

It should look very similar to the getting started tutorial:

# The model.fit() call above created a temporary model.
# Use the to_gbq() method to write to a permanent location.
model.to_gbq(
your_model_id, # For example: "bqml_tutorial.sample_model",
replace=True,
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to_gbq() to save the model.

# [END bigquery_dataframes_bqml_kmeans_fit]

# [START bigquery_dataframes_bqml_kmeans_predict]

tswast marked this conversation as resolved.
Show resolved Hide resolved
# Use 'contains' function to find all entries with string "Kennington".
stationstats = stationstats.str.contains("Kennington")

#Predict using the model
result = cluster_model.predict(stationstats)

tswast marked this conversation as resolved.
Show resolved Hide resolved
# [END bigquery_dataframes_bqml_kmeans_predict]

assert result is not None
Loading