[fix] Enforce the QueryResult.df to be a pandas.DataFrame object (Phase I) #8935
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
john-bodley
commented
Jan 8, 2020
| def _add_filter_from_pre_query_data( | ||
| self, df: Optional[pd.DataFrame], dimensions, dim_filter | ||
| ): | ||
| def _add_filter_from_pre_query_data(self, df: pd.DataFrame, dimensions, dim_filter): |
There was a problem hiding this comment.
The return type from client.export_pandas() is never None per here.
john-bodley
changed the title
john-bodley
commented
Jan 8, 2020
| @@ -439,11 +439,7 @@ def get_df_payload(self, query_obj=None, **kwargs): | |||
| and self.status != utils.QueryStatus.FAILED | |||
| ): | |||
| try: | |||
| cache_value = dict( | |||
| dttm=cached_dttm, | |||
| df=df if df is not None else None, | |||
There was a problem hiding this comment.
df if df is not None else None is merely df.
john-bodley
force-pushed
the
john-bodley--fix-empty-data-frame
branch
from
732f1d1
to
aaba7da
Compare
john-bodley
changed the title
willbarrett
approved these changes
Jan 8, 2020
| @@ -439,11 +439,7 @@ def get_df_payload(self, query_obj=None, **kwargs): | |||
| and self.status != utils.QueryStatus.FAILED | |||
| ): | |||
| try: | |||
| cache_value = dict( | |||
| dttm=cached_dttm, | |||
| df=df if df is not None else None, | |||
john-bodley
commented
Jan 8, 2020
| @@ -1379,7 +1377,7 @@ def query(self, query_obj: Dict) -> QueryResult: | |||
|
|
|||
| if df is None or df.size == 0: | |||
| return QueryResult( | |||
| df=pd.DataFrame([]), | |||
| df=pd.DataFrame(), | |||
john-bodley
commented
Jan 8, 2020
| @@ -1155,6 +1153,9 @@ def process_data(self, df, aggregate=False): | |||
| if fd.get("granularity") == "all": | |||
| raise Exception(_("Pick a time granularity for your time series")) | |||
|
|
|||
| if df.empty: | |||
There was a problem hiding this comment.
Short circuiting as pivot_table will throw an exception if the pandas.DataFrame is empty.
villebro
approved these changes
Jan 8, 2020
| @@ -77,6 +77,8 @@ | |||
| "size", | |||
| ] | |||
|
|
|||
| VizData = Optional[Union[List[Any], Dict[Any, Any]]] | |||
There was a problem hiding this comment.
Nice, we should really leverage alias types more.
| @@ -439,11 +439,7 @@ def get_df_payload(self, query_obj=None, **kwargs): | |||
| and self.status != utils.QueryStatus.FAILED | |||
| ): | |||
| try: | |||
| cache_value = dict( | |||
| dttm=cached_dttm, | |||
| df=df if df is not None else None, | |||
john-bodley
changed the title
12 tasks
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CATEGORY
Choose one
SUMMARY
In the past we have discussed whether we should use an empty
pandas.DataFrameobject orNoneto represent no-data. Previouslydfattribute of theQueryResultclass could either beNoneor apandas.DataFrameobject and was inconsistency defined, i.e., the Druid native connector used the former whereas the SQL connector used the later. This was problematic as theNonetype wasn't always handled, e.g. here and here would both throw an exception ifdfwasNone.This PR ensures that the
dfattribute is always apandas.DataFrameobject which can be empty.Additionally I’ve also added some basic typing to help enforce a non-optional
pandas.DataFrameand also provided a short circuit when processing of data for theNVD3TimeSeriesVizclass if the data-frame is empty.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TEST PLAN
CI.
ADDITIONAL INFORMATION
REVIEWERS
to: @etr2460 @michellethomas @mistercrunch @serenajiang @villebro @willbarrett