Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigTable: Add data app profile id #5275

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
3edca4c
Provide new auto-generated layer for Bigtable.
Jan 4, 2018
a677959
Merge branch 'master' into bigtable-regen
Jan 4, 2018
d540349
Add retry for read rows
Feb 14, 2018
4aa5702
Fix line lengths
Feb 14, 2018
f7d4c69
Add parameter start_inclusive to _create_row_request
Feb 14, 2018
28892ce
Update unit test case for yield rows retry
Feb 14, 2018
8d141f0
Fix line lengths
Feb 14, 2018
902f11b
Edit table.py as requested changes
Feb 16, 2018
4afabb1
Added code review changes from @jonparrott.
Feb 19, 2018
79698b1
Fix parameter name on _RetryableReadRows class
Feb 22, 2018
9b778be
Add retry for Deadline Exceeded on read rows
Feb 23, 2018
cafc5ad
Fix line over-indented
Feb 23, 2018
f974752
Fix line over-indented
Feb 23, 2018
f7242f4
Merge commit '3edca4c3299e85d4346dae49232d540a2d34002c' of https://gi…
Feb 23, 2018
180d1d5
Merge commit 'a67795951bc9f490265eda7cf511e17d2cc70921' of https://gi…
Feb 23, 2018
f2e39e0
Working on gapic integration
Feb 23, 2018
13adfd6
Refactore yield_rows retry
Feb 23, 2018
ca69a2f
1. Moved last_scanned_row_key into YieldRowsData.
Feb 27, 2018
02c154e
Fix line over-indented
Feb 27, 2018
b6da7fa
Fix row_data.py
Feb 27, 2018
f1a2d92
Change row.py as requested
Feb 27, 2018
65c1c4b
Create new test case for yield_rows iterator failure
Feb 28, 2018
170a173
Fix for start_inclusive on retry read rows.
Mar 1, 2018
5da3fd3
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
Mar 1, 2018
f7a58b5
Change bigtable_pb2 imports to use from gapic library.
Mar 2, 2018
c8f664b
Merge branch 'feature/read_rows_retry' of https://github.com/zakons/g…
Mar 6, 2018
d5f0d98
Working on gapic
Mar 6, 2018
32df7bd
Integrate bigtable GAPIC client
Mar 8, 2018
77ec950
Refactoring unit tests
Mar 27, 2018
efbf2f5
Refactor to use GAPIC client
Mar 27, 2018
4702e3c
Put back _generated files
Mar 27, 2018
cf3fb72
Fix _generated files
Mar 27, 2018
1ffda67
Fix line too long and remove unused variables
Mar 27, 2018
124de59
Fix instance.py and its unittest cases
Mar 27, 2018
0f69800
Add admin client.
Mar 30, 2018
022bd3f
Fix the gRPC stub paths.
Mar 30, 2018
5378926
Merge commit '022bd3f21fa98ea08b6f1af71a9afec71c524805' of https://gi…
Apr 2, 2018
5575990
Update bigtable admin client pb files and imports
Apr 4, 2018
ee98e67
Remove old client and related modules
Apr 4, 2018
37235c7
Remove _generated
Apr 4, 2018
fdc6db0
Fix test cases and update table.py
Apr 5, 2018
feb7d44
Remove unused code from test_table.py
Apr 5, 2018
aaa050e
Add grpc google iam v1 on setup.py on bigtable
Apr 6, 2018
2887c50
Update doc strings
Apr 9, 2018
eb69eff
Update docstrings
Apr 10, 2018
32a16c4
Change on routing_header to use to_grpc_metadata
Apr 11, 2018
078ca2b
Fix test cases on test_bigtable_instance_admin_client_pb2.py
Apr 11, 2018
c4a4482
Fix test cases on test_bigtable_instance_admin_client_pb2.py
Apr 11, 2018
07c4073
Fix test cases on test_bigtable_instance_admin_client_pb2.py
Apr 11, 2018
ee095bc
Fix line indention and remove unused variables
Apr 11, 2018
35b63d8
Fix missing whitespace after
Apr 11, 2018
70935f3
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
Apr 11, 2018
f715b80
Update docs for bigtable changes
Apr 11, 2018
28426ad
Fix Unexpected indentation on table.py
Apr 11, 2018
e04b6b9
Add license header
Apr 13, 2018
16650cb
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
Apr 13, 2018
2ae35af
Refactore GAPIC to use like old client
Apr 17, 2018
87dbe14
Fix unit test cases and refactor code for GAPIC client
Apr 18, 2018
3b69b0b
Fix credential issue with test cases
Apr 18, 2018
cc7413f
Remove _data_stub from Client
Apr 18, 2018
3a462d3
Fix coverage issues
Apr 18, 2018
0c8238c
Fix bigtable docs
Apr 19, 2018
ef47441
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
Apr 19, 2018
b9a7d8d
Fix Definition list ends without a blank line
Apr 19, 2018
ff3316a
Fix docstring on list_instances
Apr 19, 2018
a74ff23
Fix the doc string on list_instances
Apr 19, 2018
e7c64ae
Fix Definition list ends without a blank line
Apr 19, 2018
a8d6f83
Remove custer doc from docs
Apr 19, 2018
b67ab6a
Added cluster
Apr 23, 2018
634f3b4
Add cluster.rst bigtable docs
Apr 23, 2018
480f26b
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
Apr 23, 2018
a9b6d90
Add cluster in toctree
Apr 23, 2018
ab91e64
Fix required changes on doc strings
aneepct Apr 30, 2018
8bb0816
Fix required changes on doc strings
aneepct Apr 30, 2018
3702901
Merge branch 'master' of https://github.com/zakons/google-cloud-pytho…
aneepct Apr 30, 2018
5458f8e
Add app profile methods on instance.py
aneepct Apr 30, 2018
aad86b2
Add app_profile_id on table.py and refactore on instance.py
aneepct Apr 30, 2018
967ced9
Create test cases for app_profile
aneepct May 1, 2018
248593b
Fix multiple spaces after operator
aneepct May 1, 2018
eab58c2
Fix inline comment
aneepct May 1, 2018
c9ca959
Fix docstrings
aneepct May 1, 2018
bf0176a
add drop_key_range
aneepct May 2, 2018
284d723
Add timeout on truncate and drop_by_prefix
aneepct May 2, 2018
015ce18
Fix too many blank lines
aneepct May 2, 2018
db9405e
Rebase with master
aneepct May 3, 2018
7522bbd
Add app_profile create and update parameter
aneepct May 8, 2018
9cdf4ad
Fix branch coverage for update app profile
aneepct May 8, 2018
0abc775
Remove admin app profile methods
aneepct May 9, 2018
6808aea
Remove unused imports
aneepct May 9, 2018
ef64863
Rebase with master and remove truncate and drop_by_prefix from this PR
aneepct May 22, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 31 additions & 12 deletions bigtable/google/cloud/bigtable/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,15 @@ class Table(object):

:type instance: :class:`~google.cloud.bigtable.instance.Instance`
:param instance: The instance that owns the table.

:type: app_profile_id: str
:param app_profile_id: (Optional) The unique name of the AppProfile.
"""

def __init__(self, table_id, instance):
def __init__(self, table_id, instance, app_profile_id=None):

This comment was marked as spam.

This comment was marked as spam.

self.table_id = table_id
self._instance = instance
self._app_profile_id = app_profile_id

@property
def name(self):
Expand Down Expand Up @@ -227,8 +231,9 @@ def read_row(self, row_key, filter_=None):
:raises: :class:`ValueError <exceptions.ValueError>` if a commit row
chunk is never encountered.
"""
request_pb = _create_row_request(self.name, row_key=row_key,
filter_=filter_)
request_pb = _create_row_request(
self.name, row_key=row_key, filter_=filter_,
app_profile_id=self._app_profile_id)
client = self._instance._client
rows_data = PartialRowsData(client._table_data_client._read_rows,
request_pb)
Expand Down Expand Up @@ -276,7 +281,8 @@ def read_rows(self, start_key=None, end_key=None, limit=None,
"""
request_pb = _create_row_request(
self.name, start_key=start_key, end_key=end_key, filter_=filter_,
limit=limit, end_inclusive=end_inclusive)
limit=limit, end_inclusive=end_inclusive,
app_profile_id=self._app_profile_id)
client = self._instance._client
return PartialRowsData(client._table_data_client._read_rows,
request_pb)
Expand Down Expand Up @@ -310,7 +316,7 @@ def yield_rows(self, start_key=None, end_key=None, limit=None,
"""
request_pb = _create_row_request(
self.name, start_key=start_key, end_key=end_key, filter_=filter_,
limit=limit)
limit=limit, app_profile_id=self._app_profile_id)
client = self._instance._client
generator = YieldRowsData(client._table_data_client._read_rows,
request_pb)
Expand Down Expand Up @@ -347,7 +353,8 @@ def mutate_rows(self, rows, retry=DEFAULT_RETRY):
sent. These will be in the same order as the `rows`.
"""
retryable_mutate_rows = _RetryableMutateRowsWorker(
self._instance._client, self.name, rows)
self._instance._client, self.name, rows,
app_profile_id=self._app_profile_id)
return retryable_mutate_rows(retry=retry)

def sample_row_keys(self):
Expand Down Expand Up @@ -383,7 +390,7 @@ def sample_row_keys(self):
"""
client = self._instance._client
response_iterator = client._table_data_client.sample_row_keys(
self.name)
self.name, app_profile_id=self._app_profile_id)
return response_iterator


Expand All @@ -403,10 +410,11 @@ class _RetryableMutateRowsWorker(object):
)
# pylint: enable=unsubscriptable-object

def __init__(self, client, table_name, rows):
def __init__(self, client, table_name, rows, app_profile_id=None):
self.client = client
self.table_name = table_name
self.rows = rows
self.app_profile_id = app_profile_id
self.responses_statuses = [None] * len(self.rows)

def __call__(self, retry=DEFAULT_RETRY):
Expand Down Expand Up @@ -468,7 +476,8 @@ def _do_mutate_retryable_rows(self):
return self.responses_statuses

mutate_rows_request = _mutate_rows_request(
self.table_name, retryable_rows)
self.table_name, retryable_rows,
app_profile_id=self.app_profile_id)
responses = self.client._table_data_client._mutate_rows(
mutate_rows_request, retry=None)

Expand Down Expand Up @@ -496,7 +505,8 @@ def _do_mutate_retryable_rows(self):


def _create_row_request(table_name, row_key=None, start_key=None, end_key=None,
filter_=None, limit=None, end_inclusive=False):
filter_=None, limit=None, end_inclusive=False,
app_profile_id=None):
"""Creates a request to read rows in a table.

:type table_name: str
Expand Down Expand Up @@ -528,6 +538,9 @@ def _create_row_request(table_name, row_key=None, start_key=None, end_key=None,
:param end_inclusive: (Optional) Whether the ``end_key`` should be
considered inclusive. The default is False (exclusive).

:type: app_profile_id: str
:param app_profile_id: (Optional) The unique name of the AppProfile.

:rtype: :class:`data_messages_v2_pb2.ReadRowsRequest`
:returns: The ``ReadRowsRequest`` protobuf corresponding to the inputs.
:raises: :class:`ValueError <exceptions.ValueError>` if both
Expand All @@ -551,6 +564,8 @@ def _create_row_request(table_name, row_key=None, start_key=None, end_key=None,
request_kwargs['filter'] = filter_.to_pb()
if limit is not None:
request_kwargs['rows_limit'] = limit
if app_profile_id is not None:
request_kwargs['app_profile_id'] = app_profile_id

message = data_messages_v2_pb2.ReadRowsRequest(**request_kwargs)

Expand All @@ -563,7 +578,7 @@ def _create_row_request(table_name, row_key=None, start_key=None, end_key=None,
return message


def _mutate_rows_request(table_name, rows):
def _mutate_rows_request(table_name, rows, app_profile_id=None):
"""Creates a request to mutate rows in a table.

:type table_name: str
Expand All @@ -572,12 +587,16 @@ def _mutate_rows_request(table_name, rows):
:type rows: list
:param rows: List or other iterable of :class:`.DirectRow` instances.

:type: app_profile_id: str
:param app_profile_id: (Optional) The unique name of the AppProfile.

:rtype: :class:`data_messages_v2_pb2.MutateRowsRequest`
:returns: The ``MutateRowsRequest`` protobuf corresponding to the inputs.
:raises: :exc:`~.table.TooManyMutationsError` if the number of mutations is
greater than 100,000
"""
request_pb = data_messages_v2_pb2.MutateRowsRequest(table_name=table_name)
request_pb = data_messages_v2_pb2.MutateRowsRequest(
table_name=table_name, app_profile_id=app_profile_id)
mutations_count = 0
for row in rows:
_check_row_table_name(table_name, row)
Expand Down
23 changes: 20 additions & 3 deletions bigtable/tests/unit/test_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,8 @@ def _read_row_helper(self, chunks, expected_result):
request_pb = object() # Returned by our mock.
mock_created = []

def mock_create_row_request(table_name, row_key, filter_):
def mock_create_row_request(table_name, row_key, filter_,
app_profile_id=None):
mock_created.append((table_name, row_key, filter_))
return request_pb

Expand Down Expand Up @@ -502,6 +503,7 @@ def mock_create_row_request(table_name, **kwargs):
'filter_': filter_obj,
'limit': limit,
'end_inclusive': False,
'app_profile_id': None
}
self.assertEqual(mock_created, [(table.name, created_kwargs)])

Expand Down Expand Up @@ -1146,12 +1148,14 @@ def test_do_mutate_retryable_rows_mismatch_num_responses(self):
class Test__create_row_request(unittest.TestCase):

def _call_fut(self, table_name, row_key=None, start_key=None, end_key=None,
filter_=None, limit=None, end_inclusive=False):
filter_=None, limit=None, end_inclusive=False,
app_profile_id=None):
from google.cloud.bigtable.table import _create_row_request

return _create_row_request(
table_name, row_key=row_key, start_key=start_key, end_key=end_key,
filter_=filter_, limit=limit, end_inclusive=end_inclusive)
filter_=filter_, limit=limit, end_inclusive=end_inclusive,
app_profile_id=app_profile_id)

def test_table_name_only(self):
table_name = 'table_name'
Expand Down Expand Up @@ -1234,6 +1238,19 @@ def test_with_limit(self):
)
self.assertEqual(result, expected_result)

def test_with_app_profile_id(self):
table_name = 'table_name'
limit = 1337
app_profile_id = 'app-profile-id'
result = self._call_fut(table_name, limit=limit,
app_profile_id=app_profile_id)
expected_result = _ReadRowsRequestPB(
table_name=table_name,
rows_limit=limit,
app_profile_id=app_profile_id
)
self.assertEqual(result, expected_result)


def _ReadRowsRequestPB(*args, **kw):
from google.cloud.bigtable_v2.proto import (
Expand Down