Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firestore: add Watch Support #6191

Merged
merged 164 commits into from
Nov 12, 2018
Merged
Changes from 1 commit
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
4c51dac
groundwork for firestore
crwilcox Jun 12, 2018
972b62f
Use chemelnucfin sample (pythonification of nodejs) as base
crwilcox Jun 18, 2018
a66ce73
syntactic and style fixes
crwilcox Jun 18, 2018
b06c55b
Merge branch 'master' of github.com:GoogleCloudPlatform/google-cloud-…
crwilcox Jun 27, 2018
1390eb8
hold work
crwilcox Jun 27, 2018
45f48d6
use helper for_document
crwilcox Jun 27, 2018
4758b6b
Staging changes to firestore for watch. still incomplete but returnin…
crwilcox Jul 3, 2018
ff7aad7
returning watch result now
crwilcox Jul 10, 2018
c597669
broken currently, but nearing handling of multiple documents in colle…
crwilcox Jul 10, 2018
048b5fa
small fixes. seems mostly working
crwilcox Jul 11, 2018
a2fdd18
fix variable name, remove things from doc map on remove
crwilcox Jul 11, 2018
421b841
merge crwilcox firestore-watch branch into a recent branch from master
mcdonc Aug 14, 2018
6a0294e
improve doc strings and comment out yet to be done methods
crwilcox Aug 14, 2018
fd889d5
remove fstrings for 2.7 compat
mcdonc Aug 14, 2018
83755b8
be more specific on snapshot type
crwilcox Aug 14, 2018
eb8a48c
Merge remote-tracking branch 'crwilcox/firestore-watch' into firestor…
mcdonc Aug 14, 2018
a327dfe
unit tests for watch module
mcdonc Aug 20, 2018
a26699d
these must be staticmethods; improve performance in _affects_target
mcdonc Aug 20, 2018
cf85c92
make system test for watch pass
mcdonc Aug 20, 2018
9b0a4c2
containment check instead of iteration
mcdonc Aug 20, 2018
5bd6374
fix filter update test
mcdonc Aug 20, 2018
481ae83
tests for various helper methods
mcdonc Aug 21, 2018
a5c78a2
Improve rpc_done and add early query support
crwilcox Aug 21, 2018
4ce2939
Merge branch 'firestore-watch' of github.com:mcdonc/google-cloud-pyth…
crwilcox Aug 21, 2018
4301130
add more tests
mcdonc Aug 21, 2018
e00884b
Merge remote-tracking branch 'crwilcox/firestore-watch' into firestor…
mcdonc Aug 21, 2018
623e635
add tests for close
mcdonc Aug 21, 2018
27de7be
not reraising in except broke tests
mcdonc Aug 21, 2018
c665d73
compute_snapshot_ordering test still fails but fails later than it us…
mcdonc Aug 21, 2018
86628fd
remove incorrect comment
mcdonc Aug 21, 2018
23eaad5
parent is a fq path
crwilcox Aug 21, 2018
065e988
Merge branch 'firestore-watch' of github.com:mcdonc/google-cloud-pyth…
crwilcox Aug 21, 2018
9db3d18
Merge remote-tracking branch 'crwilcox/firestore-watch' into firestor…
mcdonc Aug 21, 2018
0d9de3c
fix and add test for _reset_docs
mcdonc Aug 21, 2018
199aaf1
Merge branch 'firestore-watch' of github.com:mcdonc/google-cloud-pyth…
crwilcox Aug 21, 2018
2a0723a
undo mistaken push
mcdonc Aug 22, 2018
f2b7c13
Merge remote-tracking branch 'crwilcox/firestore-watch' into firestor…
mcdonc Aug 22, 2018
889f350
appease the linter
mcdonc Aug 22, 2018
698e512
idiom
mcdonc Aug 22, 2018
3a70102
enable collection watches
crwilcox Aug 22, 2018
ed329dd
Merge branch 'firestore-watch' of github.com:mcdonc/google-cloud-pyth…
crwilcox Aug 22, 2018
2301a0e
undo spurious changes
mcdonc Aug 22, 2018
cca772e
add unfinished test
mcdonc Aug 22, 2018
48fac6d
modify the way we compute document references to support query and co…
crwilcox Aug 23, 2018
66b6071
add system tests for each variety of watch
crwilcox Aug 24, 2018
67a609f
fix most unit tests, 3 still fail
mcdonc Aug 24, 2018
d598f32
merge and apply
crwilcox Aug 24, 2018
c6ae725
expected time of test was not the same as read time, so false failure
crwilcox Aug 24, 2018
87ccc87
tests passing
crwilcox Aug 24, 2018
725f4a4
make the datetime.datetime returned non-naive and assume it's in UTC
mcdonc Aug 25, 2018
ee12608
depends directly on pytz now
mcdonc Aug 25, 2018
6aac664
100pct statement coverage for watch and test_watch
mcdonc Aug 28, 2018
97368c9
just cutnpaste this i guess
mcdonc Aug 28, 2018
53d6745
coverage for collection and bidi modules
mcdonc Aug 28, 2018
1e73ab7
coverage for document and query methods added
mcdonc Aug 28, 2018
6ade323
100 percent branch coverage
mcdonc Aug 29, 2018
2ee71fe
appease linter
mcdonc Aug 29, 2018
92e98d7
should return object on snapshot watching. This is needed to unsubscribe
crwilcox Sep 19, 2018
cf52f51
Merge branch 'master' into firestore-watch
crwilcox Sep 19, 2018
213169e
Remove use of deprecated assert
crwilcox Sep 20, 2018
abd5c97
Fix bug in deletion of document from map (using wrong key)
crwilcox Sep 20, 2018
3ed821e
startings of ordering
crwilcox Sep 28, 2018
2f3cbc7
update tests
crwilcox Oct 5, 2018
9290511
complete implementation of order
crwilcox Oct 8, 2018
f5734e9
remove commented code areas in order
crwilcox Oct 8, 2018
78a62a6
refactor order
crwilcox Oct 8, 2018
925495c
refactor order compare_objects
crwilcox Oct 8, 2018
4316fd1
add system test for ordering (currently failing for non-ordering
crwilcox Oct 8, 2018
220ba99
add system test for query and verify order
crwilcox Oct 10, 2018
30972d9
Improve test for order
crwilcox Oct 10, 2018
d549145
Add comparator to kw creation of dummyquery, fix old test, move compa…
crwilcox Oct 10, 2018
1abc70d
flake8 fixes
crwilcox Oct 10, 2018
ab862d7
coverage + tests
crwilcox Oct 10, 2018
be3584f
Properly decode document snapshot data and fix tests
crwilcox Oct 11, 2018
abdfe40
coverage
crwilcox Oct 11, 2018
2af7db7
noxfile from master
crwilcox Oct 11, 2018
a11f8eb
Merge branch 'master' into firestore-watch
crwilcox Oct 11, 2018
4c52c55
Delete Untitled-1
crwilcox Oct 11, 2018
dd9a15f
Merge branch 'master' into firestore-watch
crwilcox Oct 16, 2018
17cd042
modify to use bidi in api-core. one test has started to fail
crwilcox Oct 16, 2018
f1d079d
get tests passing (intermittently) on 2.7,3.5,3.6
mcdonc Oct 18, 2018
e0cbda4
fix failing test_order tests when left and right were dictionaries of…
mcdonc Oct 19, 2018
fb8142c
groundwork for firestore
crwilcox Jun 12, 2018
a5dbe62
Use chemelnucfin sample (pythonification of nodejs) as base
crwilcox Jun 18, 2018
8a5bd5f
syntactic and style fixes
crwilcox Jun 18, 2018
e296b22
hold work
crwilcox Jun 27, 2018
9499c89
use helper for_document
crwilcox Jun 27, 2018
e9855fd
Staging changes to firestore for watch. still incomplete but returnin…
crwilcox Jul 3, 2018
82a4a06
returning watch result now
crwilcox Jul 10, 2018
a83ebde
broken currently, but nearing handling of multiple documents in colle…
crwilcox Jul 10, 2018
1d1be53
small fixes. seems mostly working
crwilcox Jul 11, 2018
80c6bea
fix variable name, remove things from doc map on remove
crwilcox Jul 11, 2018
799fac9
improve doc strings and comment out yet to be done methods
crwilcox Aug 14, 2018
77eea6a
remove fstrings for 2.7 compat
mcdonc Aug 14, 2018
0db6a9d
be more specific on snapshot type
crwilcox Aug 14, 2018
374775b
unit tests for watch module
mcdonc Aug 20, 2018
36e431b
these must be staticmethods; improve performance in _affects_target
mcdonc Aug 20, 2018
4f08ac3
make system test for watch pass
mcdonc Aug 20, 2018
d0bffa9
containment check instead of iteration
mcdonc Aug 20, 2018
3de5310
fix filter update test
mcdonc Aug 20, 2018
40e8b0b
tests for various helper methods
mcdonc Aug 21, 2018
1853297
Improve rpc_done and add early query support
crwilcox Aug 21, 2018
1e77943
add more tests
mcdonc Aug 21, 2018
2724132
add tests for close
mcdonc Aug 21, 2018
d7258c8
not reraising in except broke tests
mcdonc Aug 21, 2018
1396b74
compute_snapshot_ordering test still fails but fails later than it us…
mcdonc Aug 21, 2018
1465bc2
remove incorrect comment
mcdonc Aug 21, 2018
39b97db
parent is a fq path
crwilcox Aug 21, 2018
39e429f
fix and add test for _reset_docs
mcdonc Aug 21, 2018
397c6d6
undo mistaken push
mcdonc Aug 22, 2018
95a485a
appease the linter
mcdonc Aug 22, 2018
3424284
idiom
mcdonc Aug 22, 2018
429f86e
enable collection watches
crwilcox Aug 22, 2018
00de5d3
undo spurious changes
mcdonc Aug 22, 2018
4b96582
add unfinished test
mcdonc Aug 22, 2018
bcecb25
modify the way we compute document references to support query and co…
crwilcox Aug 23, 2018
05db07c
add system tests for each variety of watch
crwilcox Aug 24, 2018
dfaacc7
fix most unit tests, 3 still fail
mcdonc Aug 24, 2018
b59b12b
merge and apply
crwilcox Aug 24, 2018
2a19765
expected time of test was not the same as read time, so false failure
crwilcox Aug 24, 2018
299fd94
tests passing
crwilcox Aug 24, 2018
9081664
make the datetime.datetime returned non-naive and assume it's in UTC
mcdonc Aug 25, 2018
9489ec1
depends directly on pytz now
mcdonc Aug 25, 2018
be51be8
100pct statement coverage for watch and test_watch
mcdonc Aug 28, 2018
854d051
just cutnpaste this i guess
mcdonc Aug 28, 2018
1059f2c
coverage for collection and bidi modules
mcdonc Aug 28, 2018
346e5d6
coverage for document and query methods added
mcdonc Aug 28, 2018
6cb3e85
100 percent branch coverage
mcdonc Aug 29, 2018
7c73721
appease linter
mcdonc Aug 29, 2018
7019fd5
should return object on snapshot watching. This is needed to unsubscribe
crwilcox Sep 19, 2018
6f36b6f
Fix bug in deletion of document from map (using wrong key)
crwilcox Sep 20, 2018
23d467d
startings of ordering
crwilcox Sep 28, 2018
c8e293a
update tests
crwilcox Oct 5, 2018
6cd0ac6
complete implementation of order
crwilcox Oct 8, 2018
41ef649
remove commented code areas in order
crwilcox Oct 8, 2018
6be3422
refactor order
crwilcox Oct 8, 2018
b1c42e9
refactor order compare_objects
crwilcox Oct 8, 2018
59f725b
add system test for ordering (currently failing for non-ordering
crwilcox Oct 8, 2018
6888e8a
add system test for query and verify order
crwilcox Oct 10, 2018
f77616c
Improve test for order
crwilcox Oct 10, 2018
cc445f1
Add comparator to kw creation of dummyquery, fix old test, move compa…
crwilcox Oct 10, 2018
85c199b
flake8 fixes
crwilcox Oct 10, 2018
f75a49c
coverage + tests
crwilcox Oct 10, 2018
0125c87
Properly decode document snapshot data and fix tests
crwilcox Oct 11, 2018
cdc6594
coverage
crwilcox Oct 11, 2018
36e8772
noxfile from master
crwilcox Oct 11, 2018
738d938
Delete Untitled-1
crwilcox Oct 11, 2018
9c2ff84
modify to use bidi in api-core. one test has started to fail
crwilcox Oct 16, 2018
3f8e0c8
get tests passing (intermittently) on 2.7,3.5,3.6
mcdonc Oct 18, 2018
4a892a7
fix failing test_order tests when left and right were dictionaries of…
mcdonc Oct 19, 2018
cb74bf7
fix transports layer use in watch
crwilcox Oct 19, 2018
d42f8c5
try adding wait before doc setting. System tests failing on CI but no…
crwilcox Oct 19, 2018
138a6f1
merge from crwilcox/firestore-watch and fix conflicts
mcdonc Oct 20, 2018
3add82d
linting
mcdonc Oct 20, 2018
b484098
make a bit more idiomatic
mcdonc Oct 20, 2018
e14421b
more idiomatic, and remove special casing of -0.0 and 0.0 in compare_…
mcdonc Oct 20, 2018
e089dac
lint fixes
crwilcox Oct 22, 2018
6e588f3
Merge branch 'firestore-watch' of github.com:mcdonc/google-cloud-pyth…
crwilcox Oct 22, 2018
136a881
Start using self.close where previously using self._consumer.stop
crwilcox Oct 26, 2018
195814b
lint
crwilcox Oct 26, 2018
34b4da3
clean up comments
crwilcox Nov 7, 2018
5622f3c
alias unsubscribe to close
crwilcox Nov 7, 2018
67c9a47
move rpc to internal attr
crwilcox Nov 7, 2018
8900be2
Merge branch 'master' into firestore-watch
crwilcox Nov 9, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add system tests for each variety of watch
  • Loading branch information
crwilcox committed Aug 24, 2018
commit 66b6071ffba1e8cf1356f6d18aa84983eaa4cf00
5 changes: 4 additions & 1 deletion firestore/google/cloud/firestore_v1beta1/collection.py
Original file line number Diff line number Diff line change
@@ -398,7 +398,10 @@ def on_snapshot(collection_snapshot):
# Terminate this watch
collection_watch.unsubscribe()
"""
Watch.for_query(query_mod.Query(self), callback, document)
Watch.for_query(query_mod.Query(self),
callback,
document.DocumentSnapshot,
document.DocumentReference)


def _auto_id():
2 changes: 1 addition & 1 deletion firestore/google/cloud/firestore_v1beta1/document.py
Original file line number Diff line number Diff line change
@@ -471,7 +471,7 @@ def on_snapshot(document_snapshot):
# Terminate this watch
doc_watch.unsubscribe()
"""
Watch.for_document(self, callback, __import__(__name__))
Watch.for_document(self, callback, DocumentSnapshot, DocumentReference)


class DocumentSnapshot(object):
5 changes: 4 additions & 1 deletion firestore/google/cloud/firestore_v1beta1/query.py
Original file line number Diff line number Diff line change
@@ -627,7 +627,10 @@ def on_snapshot(query_snapshot):
# Terminate this watch
query_watch.unsubscribe()
"""
Watch.for_query(self, callback, document)
Watch.for_query(self,
callback,
document.DocumentSnapshot,
document.DocumentReference)


def _enum_from_op_string(op_string):
27 changes: 17 additions & 10 deletions firestore/google/cloud/firestore_v1beta1/watch.py
Original file line number Diff line number Diff line change
@@ -160,7 +160,8 @@ def __init__(self,
target,
comparator,
snapshot_callback,
document_module,
document_snapshot_cls,
document_reference_cls,
BackgroundConsumer=None, # FBO unit testing
ResumableBidiRpc=None, # FBO unit testing
):
@@ -180,15 +181,16 @@ def __init__(self,
snapshot was obtained.
# TODO: Go had an err here and node.js provided size.
# TODO: do we want to include either?
document_module: instance of the Document module
document_snapshot_cls: instance of DocumentSnapshot
document_reference_cls: instance of DocumentReference
"""
self._document_reference = document_reference
self._firestore = firestore
self._api = firestore._firestore_api
self._targets = target
self._comparator = comparator
self.DocumentSnapshot = document_module.DocumentSnapshot
self.DocumentReference = document_module.DocumentReference
self.DocumentSnapshot = document_snapshot_cls
self.DocumentReference = document_reference_cls
self._snapshot_callback = snapshot_callback
self._closing = threading.Lock()
self._closed = False
@@ -304,7 +306,7 @@ def unsubscribe(self): # XXX should this be aliased to close?

@classmethod
def for_document(cls, document_ref, snapshot_callback,
snapshot_class_instance):
snapshot_class_instance, reference_class_instance):
"""
Creates a watch snapshot listener for a document. snapshot_callback
receives a DocumentChange object, but may also start to get
@@ -313,8 +315,10 @@ def for_document(cls, document_ref, snapshot_callback,
Args:
document_ref: Reference to Document
snapshot_callback: callback to be called on snapshot
snapshot_class_instance: instance of snapshot cls to make
snapshot_class_instance: instance of DocumentSnapshot to make
snapshots with to pass to snapshot_callback
reference_class_instance: instance of DocumentReference to make
references

"""
return cls(document_ref,
@@ -326,10 +330,12 @@ def for_document(cls, document_ref, snapshot_callback,
},
document_watch_comparator,
snapshot_callback,
snapshot_class_instance)
snapshot_class_instance,
reference_class_instance)

@classmethod
def for_query(cls, query, snapshot_callback, snapshot_class_instance):
def for_query(cls, query, snapshot_callback, snapshot_class_instance,
reference_class_instance):
query_target = firestore_pb2.Target.QueryTarget(
parent=query._client._database_string,
structured_query=query._to_protobuf(),
@@ -342,7 +348,8 @@ def for_query(cls, query, snapshot_callback, snapshot_class_instance):
},
document_watch_comparator,
snapshot_callback,
snapshot_class_instance)
snapshot_class_instance,
reference_class_instance)

def _on_snapshot_target_change_no_change(self, proto):
_LOGGER.debug('on_snapshot: target change: NO_CHANGE')
@@ -700,7 +707,7 @@ def _affects_target(self, target_ids, current_id):
return True

return current_id in target_ids

def _current_size(self):
"""
Returns the current count of all documents, including the changes from
95 changes: 84 additions & 11 deletions firestore/tests/system.py
Original file line number Diff line number Diff line change
@@ -763,12 +763,12 @@ def test_watch_document(client, cleanup):
sleep(1)

# Setup listener
def on_response(*arg):
on_response.called_count += 1
def on_snapshot(docs, changes, read_time):
on_snapshot.called_count += 1

on_response.called_count = 0
on_snapshot.called_count = 0

doc_ref.on_snapshot(on_response)
doc_ref.on_snapshot(on_snapshot)

# Alter document
doc_ref.set({
@@ -778,34 +778,107 @@ def on_response(*arg):
})

sleep(1)
if on_response.called_count != 1:
raise AssertionError("Failed to get exactly one document change")

for _ in range(10):
if on_snapshot.called_count == 1:
return
sleep(1)

if on_snapshot.called_count != 1:
raise AssertionError(
"Failed to get exactly one document change: count: " +
str(on_snapshot.called_count))


def test_watch_collection(client, cleanup):
db = client
doc_ref = db.collection(u'users').document(
u'alovelace' + unique_resource_id())
collection_ref = db.collection(u'users')
def on_snapshot(snapshot):
for doc in snapshot.documents:
print(u'{} => {}'.format(doc.id, doc.to_dict()))

# Initial setting
doc_ref.set({
u'first': u'Jane',
u'last': u'Doe',
u'born': 1900
})

# Setup listener
def on_snapshot(docs, changes, read_time):
on_snapshot.called_count += 1

on_snapshot.called_count = 0

# def on_snapshot(docs, changes, read_time):
# for doc in docs:
# print(u'{} => {}'.format(doc.id, doc.to_dict()))

collection_ref.on_snapshot(on_snapshot)

sleep(1)

doc_ref.set({
u'first': u'Ada',
u'last': u'Lovelace',
u'born': 1815
})

sleep(1)

for _ in range(10):
if on_snapshot.called_count == 1:
return
sleep(1)

if on_snapshot.called_count != 1:
raise AssertionError(
"Failed to get exactly one document change: count: " +
str(on_snapshot.called_count))

# CM: had to stop here, this test is totally unfinished, trying to
# formalize
# https://gist.github.com/crwilcox/ce05f3857adc7a0ed86ffbd039b1a035


def test_watch_query(client, cleanup):
db = client
doc_ref = db.collection(u'users').document(
u'alovelace' + unique_resource_id())
query_ref = db.collection(u'users').where("first", "==", u'Ada')

# Initial setting
doc_ref.set({
u'first': u'Jane',
u'last': u'Doe',
u'born': 1900
})

sleep(1)

# Setup listener
def on_snapshot(docs, changes, read_time):
on_snapshot.called_count += 1
print("docs: " + docs)
print("changes: " + changes)
print("read_time: " + read_time)

on_snapshot.called_count = 0

query_ref.on_snapshot(on_snapshot)

# Alter document
doc_ref.set({
u'first': u'Ada',
u'last': u'Lovelace',
u'born': 1815
})

# CM: had to stop here, this test is totally unfinished, trying to formalize
# https://gist.github.com/crwilcox/ce05f3857adc7a0ed86ffbd039b1a035
for _ in range(10):
if on_snapshot.called_count == 1:
return
sleep(1)

if on_snapshot.called_count != 1:
raise AssertionError(
"Failed to get exactly one document change: count: " +
str(on_snapshot.called_count))