Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: search statistics #1616

Merged
merged 56 commits into from
Sep 2, 2023
Merged
Changes from 8 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
b491836
experimental tweaks
chalmerlowe Jul 19, 2023
0cbb7f4
feat: adds two search statistics classes and property
chalmerlowe Jul 20, 2023
1dbf528
removes several personal debugging sentinels
chalmerlowe Jul 20, 2023
9695712
Merge branch 'main' into feat-search-statistics
chalmerlowe Jul 20, 2023
fcf7012
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 20, 2023
ae992bf
Merge branch 'feat-search-statistics' of https://github.com/googleapi…
gcf-owl-bot[bot] Jul 20, 2023
3815371
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 20, 2023
48086bd
Merge branch 'feat-search-statistics' of https://github.com/googleapi…
gcf-owl-bot[bot] Jul 20, 2023
5b14d5d
adds tests
chalmerlowe Aug 9, 2023
9689a71
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 9, 2023
6af7d6f
cleans up conflict
chalmerlowe Aug 10, 2023
53437f2
adds comment
chalmerlowe Aug 10, 2023
4fd77ba
adds some type hints, adds a test for SearchReasons
chalmerlowe Aug 11, 2023
bb2b52c
cleans up some comments
chalmerlowe Aug 11, 2023
e728883
Merge branch 'main' into feat-search-statistics
chalmerlowe Aug 11, 2023
966ddb1
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 11, 2023
e9fca23
Merge branch 'feat-search-statistics' of https://github.com/googleapi…
gcf-owl-bot[bot] Aug 11, 2023
ba2eb65
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 11, 2023
a0a0e5b
Merge branch 'feat-search-statistics' of https://github.com/googleapi…
gcf-owl-bot[bot] Aug 11, 2023
852514f
Update tests/unit/job/test_query_stats.py
chalmerlowe Aug 11, 2023
2a14bc4
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 11, 2023
8493c3a
Merge branch 'main' into feat-search-statistics
chalmerlowe Aug 15, 2023
7991e4d
updated type checks to be isinstance checks per linter
chalmerlowe Aug 15, 2023
62f9bcc
update linting
chalmerlowe Aug 15, 2023
705084b
Update tests/unit/job/test_query_stats.py
chalmerlowe Aug 15, 2023
4871339
Update tests/unit/job/test_query_stats.py
chalmerlowe Aug 15, 2023
93af6d9
experiments with some tests that are failing
chalmerlowe Aug 17, 2023
b0196c1
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 17, 2023
711d752
Fix linting
chalmerlowe Aug 18, 2023
ccf87a9
update package verification approach
chalmerlowe Aug 18, 2023
5bc0082
update pandas installed version constant
chalmerlowe Aug 18, 2023
0089524
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 18, 2023
fe23c18
remove unused package
chalmerlowe Aug 18, 2023
9bb8f14
set pragma no cover
chalmerlowe Aug 18, 2023
ddd86bd
adds controls to skip testing if pandas exceeds 2.0
chalmerlowe Aug 18, 2023
c55ba91
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 18, 2023
dc150bd
adds pragma no cover to a simple check
chalmerlowe Aug 18, 2023
7d68a8a
add checks against pandas 2.0 on system test
chalmerlowe Aug 18, 2023
2e438d3
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 18, 2023
93c345b
Merge branch 'main' into feat-search-statistics
chalmerlowe Aug 21, 2023
0423fdc
Merge branch 'main' into feat-search-statistics
chalmerlowe Aug 23, 2023
496ab6d
experiments with some tests that are failing
chalmerlowe Aug 17, 2023
392db37
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 17, 2023
a30f23c
resolves merge conflict
chalmerlowe Aug 24, 2023
c1dfc35
resolves merge conflict
chalmerlowe Aug 24, 2023
5a2f268
resolve conflicts
chalmerlowe Aug 24, 2023
744d932
resolve merge conflicts
chalmerlowe Aug 24, 2023
81f8aab
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 24, 2023
861eb98
updates due to faulty confict resolution
chalmerlowe Aug 24, 2023
20ec9e3
adds docstrings to two classes
chalmerlowe Aug 24, 2023
5659dd3
corrects formatting
chalmerlowe Aug 24, 2023
dd1b749
Update tests/unit/job/test_query_stats.py
chalmerlowe Aug 24, 2023
e056787
Update tests/unit/job/test_query_stats.py
chalmerlowe Aug 24, 2023
5ebcbb0
updates default values and corrects mypy errors
chalmerlowe Aug 28, 2023
faafac2
corrects linting
chalmerlowe Aug 28, 2023
9ab5c25
Update google/cloud/bigquery/job/query.py
chalmerlowe Sep 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion google/cloud/bigquery/job/query.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,39 @@ def from_api_repr(cls, stats: Dict[str, str]) -> "DmlStats":
return cls(*args)


class SearchReason(typing.NamedTuple):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the underlying type is IndexUnusedReason, why the name change?

Also, I'd expect property definitions for this type, and I'm not seeing them here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding naming:
I simply mirrored naming conventions that were already present in the code:

BiEngineStats
BiEngineReason

My take was that SearchReason and SearchStats specifically highlights the broader purpose/use of the Reason (i.e. this reason is related to the Search function):

If we think we get greater clarity by calling the class IndexUnusedReason, I am game to do that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on what you mean by property definitions?

Are to talking about expanding on the docstring and parameter definitions? If yes, that is yet to be done. At this second, I am focused on does the code work/seem reasonable/have adequate tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly mean getters with the "@Property" annotation for referencing fields in the message types. As these are primarily readonly fields setters shouldn't be necessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shollyman I got the pandas references out.
I will see about renaming the underlying type.
Did you come across any ideas on whether we need/want to add @properties when using a typing.NamedTuple

"""docstring"""

code: str
message: str
baseTable: TableReference
indexName: str

@classmethod
def from_api_repr(cls, reason):
code = reason.get("code")
message = reason.get("message")
baseTable = reason.get("baseTable")
indexName = reason.get("indexName")

return cls(code, message, baseTable, indexName)


class SearchStats(typing.NamedTuple):
"""docstring"""

mode: str
reason: list

@classmethod
def from_api_repr(cls, stats: Dict[str, Any]):
mode = stats.get("indexUsageMode", "INDEX_USAGE_MODE_UNSPECIFIED")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, we want to avoid the library injecting values not present in the API response, so returning the default value here may not be more desirable than empty/none.

reason = [
SearchReason.from_api_repr(r) for r in stats.get("indexUnusedReasons")
]
return cls(mode, reason)


class ScriptOptions:
"""Options controlling the execution of scripts.

Expand Down Expand Up @@ -724,7 +757,6 @@ def to_api_repr(self) -> dict:
Dict: A dictionary in the format used by the BigQuery API.
"""
resource = copy.deepcopy(self._properties)

# Query parameters have an addition property associated with them
# to indicate if the query is using named or positional parameters.
query_parameters = resource["query"].get("queryParameters")
Expand Down Expand Up @@ -858,6 +890,13 @@ def priority(self):
"""
return self.configuration.priority

@property
def search_stats(self):
chalmerlowe marked this conversation as resolved.
Show resolved Hide resolved
"""docstring"""

stats = self._job_statistics().get("searchStatistics")
return SearchStats.from_api_repr(stats)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to handle the not-present case, e.g. if stats is not None:


@property
def query(self):
"""str: The query text used in this query job.
Expand Down