Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optim get schema and get class name in shards into 1 rdd operation #5573

Merged
merged 4 commits into from
Aug 31, 2022
+19 −8
Merged

optim get schema and get class name in shards into 1 rdd operation #5573

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
20bf9a3
merge get schema and get class name in shards into 1 rdd operation
dding3 Aug 29, 2022
c46832d
Merge branch 'upstream_main' into optimshards
dding3 Aug 31, 2022
1505736
fix comments
dding3 Aug 31, 2022
c011929
fix style check failure
dding3 Aug 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions python/orca/src/bigdl/orca/data/shard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -598,21 +598,32 @@ def utility_func(x, func, *args, **kwargs):
def get_schema(self):
if 'schema' in self.type:
return self.type['schema']
else:
if self._get_class_name() == 'pandas.core.frame.DataFrame':
import pandas as pd
columns, dtypes = self.rdd.map(lambda x: (x.columns, x.dtypes)).first()
self.type['schema'] = {'columns': columns, 'dtypes': dtypes}
return self.type['schema']
return None

if 'class_name' not in self.type\
or self.type['class_name'] == 'pandas.core.frame.DataFrame':
class_name, schema = self._get_schema_class_name()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only call _get_schema_class_name if class_name does not exist, or is pandas dataframe

self.type['class_name'] = class_name
self.type['schema'] = schema
return self.type['schema']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to return None otherwise

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #5558


def _get_class_name(self):
if 'class_name' in self.type:
return self.type['class_name']
else:
self.type['class_name'] = self._for_each(get_class_name).first()
class_name, schema = self._get_schema_class_name()
self.type['class_name'] = class_name
self.type['schema'] = schema
return self.type['class_name']

def _get_schema_class_name(self):
def func(x):
class_name = get_class_name(x)
schema = None
if class_name == 'pandas.core.frame.DataFrame':
schema = {'columns': x.columns, 'dtypes': x.dtypes}
return (class_name, schema)
return self.rdd.map(lambda x: func(x)).first()


class SharedValue(object):
def __init__(self, data):
Expand Down