Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add pooled grpc transport #748

Merged
merged 134 commits into from
Apr 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
8c5fd7d
moved previous client to deprecated, added files for new client
daniel-sanche Mar 3, 2023
393860c
added table api skeleton
daniel-sanche Mar 3, 2023
11b3493
fixed missing files
daniel-sanche Mar 3, 2023
1ac0b7a
updated import paths
daniel-sanche Mar 3, 2023
cf981d8
updated unit tests
daniel-sanche Mar 3, 2023
5ea2bc3
updated system tests
daniel-sanche Mar 3, 2023
84fd9c3
ran blacken
daniel-sanche Mar 3, 2023
43b17dd
added additional unimplemented files
daniel-sanche Mar 3, 2023
32e8e45
improved __init__
daniel-sanche Mar 3, 2023
65cb219
stricter type checks
daniel-sanche Mar 3, 2023
c8b8a5a
ran blacken
daniel-sanche Mar 3, 2023
ff835ec
removed sample implementation from BigtableExceptionGroup
daniel-sanche Mar 3, 2023
3207ef5
fixed circular import issues
daniel-sanche Mar 4, 2023
3b3d720
added deprecation warning
daniel-sanche Mar 4, 2023
fa29ba1
updated warning messages
daniel-sanche Mar 4, 2023
4e792a5
fixed lint issues
daniel-sanche Mar 4, 2023
35e8a58
added submodule for gapic fork
daniel-sanche Mar 6, 2023
dfab801
pulled in upstream proto updates
daniel-sanche Mar 6, 2023
72f7d0e
added pooled transport class
daniel-sanche Mar 6, 2023
be3de7a
added functions to get and replace channels in pool
daniel-sanche Mar 6, 2023
f6c7f36
added client init implementation
daniel-sanche Mar 6, 2023
44e76c1
added channel management to table object
daniel-sanche Mar 6, 2023
c4d537e
refactoring channel management
daniel-sanche Mar 7, 2023
99a49a4
ping new channel before replacement
daniel-sanche Mar 7, 2023
bd4fb5e
made channel pool public
daniel-sanche Mar 7, 2023
6b13c29
moved channel refresh logic into shared client
daniel-sanche Mar 8, 2023
60841c5
call ping and warm on register_instance
daniel-sanche Mar 8, 2023
0a3086c
fixed typo
daniel-sanche Mar 8, 2023
08c3c42
renamed function
daniel-sanche Mar 8, 2023
05e10cd
added comments
daniel-sanche Mar 8, 2023
895093f
removed TypeAlis annotation
daniel-sanche Mar 8, 2023
1684274
changed sequence import
daniel-sanche Mar 8, 2023
75d276a
updated warning tests
daniel-sanche Mar 8, 2023
92752c0
updated doc snippet imports
daniel-sanche Mar 8, 2023
79c82c3
updated docs
daniel-sanche Mar 8, 2023
f3b7fbd
disabled coverage for skeleton code
daniel-sanche Mar 8, 2023
aa37a31
fixed cover change
daniel-sanche Mar 8, 2023
17d731f
adjusted coverage setup
daniel-sanche Mar 9, 2023
a0a5c57
ran blacken
daniel-sanche Mar 9, 2023
64a05d8
changed cover value in noxfile
daniel-sanche Mar 9, 2023
741147d
updated fork
daniel-sanche Mar 9, 2023
005900c
added pool transport to tests
daniel-sanche Mar 9, 2023
d58fc74
Merge branch 'fresh-client' into add_new_transport
daniel-sanche Mar 9, 2023
9983e18
fixed issues in tests
daniel-sanche Mar 9, 2023
bfeb546
got gapic tests passing
daniel-sanche Mar 9, 2023
dba7a3c
reworked the client to take instance at init
daniel-sanche Mar 10, 2023
3ae6722
moved background setup back into client init, with warning if no async
daniel-sanche Mar 11, 2023
b9f2b0d
improved comment
daniel-sanche Mar 11, 2023
e483370
Merge branch 'v3' into add_new_transport
daniel-sanche Mar 14, 2023
136a8fe
Merge branch 'main' into v3
daniel-sanche Mar 15, 2023
af86f6b
Merge branch 'v3' into add_new_transport
daniel-sanche Mar 15, 2023
cbe7062
moved instance tracking back into Table
daniel-sanche Mar 16, 2023
909f889
fixed lint issues
daniel-sanche Mar 16, 2023
5efa0ac
fixed blacken
daniel-sanche Mar 16, 2023
2b322e2
Merge branch 'v3' into add_new_transport
daniel-sanche Mar 23, 2023
d4904d7
changed generation arguments
daniel-sanche Mar 23, 2023
6917593
improved close functionality
daniel-sanche Mar 24, 2023
7b5ecbb
updated submodule
daniel-sanche Mar 24, 2023
c0616dd
reverted some generated changes
daniel-sanche Mar 24, 2023
983d4c7
reverted to protoc generation
daniel-sanche Mar 24, 2023
c19658a
got tests passing
daniel-sanche Mar 24, 2023
c2d0da0
added new test file for client
daniel-sanche Mar 25, 2023
8b54a30
set up transport in client
daniel-sanche Mar 25, 2023
0dd981b
implemented tests for underlying transport
daniel-sanche Mar 25, 2023
b0ecd3c
added some tests
daniel-sanche Mar 25, 2023
e47551f
added manage channels tests
daniel-sanche Mar 28, 2023
96d526b
added more tests
daniel-sanche Mar 28, 2023
e997892
client needs active event loop; reorganized tests around that
daniel-sanche Mar 28, 2023
5c86f57
reordered some things
daniel-sanche Mar 28, 2023
3bc4131
simpified client setup
daniel-sanche Mar 28, 2023
3c4e0b6
added test for veneer headers
daniel-sanche Mar 28, 2023
197bf95
added super init test
daniel-sanche Mar 28, 2023
9d8122b
added subclass generator to gapic template
daniel-sanche Mar 28, 2023
2632b70
finished table tests
daniel-sanche Mar 28, 2023
d4e052b
ran blacken
daniel-sanche Mar 28, 2023
1aa694b
got tests working
daniel-sanche Mar 28, 2023
e2d4bd5
fixed type
daniel-sanche Mar 28, 2023
a91362f
reverted rest client
daniel-sanche Mar 28, 2023
d80a8c0
fixed rest tests
daniel-sanche Mar 28, 2023
b888ee8
converted tests to pytest
daniel-sanche Mar 29, 2023
94c1187
added client closure
daniel-sanche Mar 29, 2023
4c02e6c
ran blacken
daniel-sanche Mar 29, 2023
d65b432
use paramaterize in tests
daniel-sanche Mar 29, 2023
4b63d87
improved some tests
daniel-sanche Mar 29, 2023
4ccc421
went back to init without event loop raising warning
daniel-sanche Mar 29, 2023
8001240
removed __del__
daniel-sanche Mar 29, 2023
7c9cea7
changed warning type
daniel-sanche Mar 29, 2023
3bbebea
ran blacken
daniel-sanche Mar 29, 2023
8bff9d0
improved task naming
daniel-sanche Mar 29, 2023
4ae2146
fixed style issues
daniel-sanche Mar 29, 2023
b9dc2f7
got 3.7 tests working
daniel-sanche Mar 30, 2023
19036d8
fixed style issue
daniel-sanche Mar 30, 2023
8a22d15
implement pool as custom grpc channel
daniel-sanche Mar 31, 2023
38e5662
did some restructuring
daniel-sanche Apr 1, 2023
5155800
got some tests working
daniel-sanche Apr 1, 2023
522f7fa
improved tests
daniel-sanche Apr 2, 2023
74029c9
updated template
daniel-sanche Apr 3, 2023
7f2be30
got tests passing
daniel-sanche Apr 3, 2023
2b044ce
removed metadata
daniel-sanche Apr 4, 2023
1743098
added sleep between swwapping and closing channels
daniel-sanche Apr 4, 2023
e5fa4b6
ran blacken
daniel-sanche Apr 4, 2023
8955ec5
got tests working
daniel-sanche Apr 4, 2023
002bc5f
fixed lint issue
daniel-sanche Apr 4, 2023
65f0d2f
fixed tests
daniel-sanche Apr 4, 2023
dbf19c9
holds a gapic client instead of inherits from it
daniel-sanche Apr 5, 2023
9f3e0c5
added comment
daniel-sanche Apr 5, 2023
a0620ea
added random noise to refresh intervals
daniel-sanche Apr 5, 2023
1486d5a
Merge branch 'v3' into add_new_transport
daniel-sanche Apr 6, 2023
28d5a7a
fixed lint issues
daniel-sanche Apr 6, 2023
70fbff9
reduced size of template by making subclass
daniel-sanche Apr 7, 2023
383d8eb
reverted unintentional gapic generation changes
daniel-sanche Apr 7, 2023
018fe03
updated submodule
daniel-sanche Apr 7, 2023
f0403e7
changed warning stack level
daniel-sanche Apr 13, 2023
cb23d32
update docstring
daniel-sanche Apr 19, 2023
bc31ab8
update docstring
daniel-sanche Apr 19, 2023
f54dfde
fix typo
daniel-sanche Apr 19, 2023
46cfc49
docstring improvements
daniel-sanche Apr 19, 2023
573bbd1
made creating table outside loop into error
daniel-sanche Apr 19, 2023
4f2657d
make tables own active instances, and remove instances when tables close
daniel-sanche Apr 19, 2023
59955be
added pool_size and channels as public properties
daniel-sanche Apr 19, 2023
377a8c9
fixed typo
daniel-sanche Apr 19, 2023
8a29898
simplified pooled multicallable
daniel-sanche Apr 20, 2023
50aa5ba
ran blacken
daniel-sanche Apr 20, 2023
42a52a3
associate ids with instances, instead of Table objects
daniel-sanche Apr 20, 2023
50dc608
reverted pooled multicallable changes
daniel-sanche Apr 20, 2023
b116755
pass scopes to created channels
daniel-sanche Apr 21, 2023
ec5eb07
added basic ping system test
daniel-sanche Apr 21, 2023
55cdcc2
keep both the names and ids in table object
daniel-sanche Apr 21, 2023
9e3b411
pull project details out of env vars
daniel-sanche Apr 21, 2023
ab43138
restructured test_client
daniel-sanche Apr 21, 2023
cb1884d
changed how random is mocked
daniel-sanche Apr 21, 2023
9a89d74
ran black
daniel-sanche Apr 21, 2023
1e62c71
added system test code to create instance if not present
daniel-sanche Apr 24, 2023
d70c685
ran black
daniel-sanche Apr 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "gapic-generator-fork"]
path = gapic-generator-fork
url = [email protected]:googleapis/gapic-generator-python.git
1 change: 1 addition & 0 deletions gapic-generator-fork
Submodule gapic-generator-fork added at b26cda
325 changes: 299 additions & 26 deletions google/cloud/bigtable/client.py

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions google/cloud/bigtable_v2/services/bigtable/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
from .transports.base import BigtableTransport, DEFAULT_CLIENT_INFO
from .transports.grpc import BigtableGrpcTransport
from .transports.grpc_asyncio import BigtableGrpcAsyncIOTransport
from .transports.pooled_grpc_asyncio import PooledBigtableGrpcAsyncIOTransport
from .transports.rest import BigtableRestTransport


Expand All @@ -67,6 +68,7 @@ class BigtableClientMeta(type):
_transport_registry = OrderedDict() # type: Dict[str, Type[BigtableTransport]]
_transport_registry["grpc"] = BigtableGrpcTransport
_transport_registry["grpc_asyncio"] = BigtableGrpcAsyncIOTransport
_transport_registry["pooled_grpc_asyncio"] = PooledBigtableGrpcAsyncIOTransport
_transport_registry["rest"] = BigtableRestTransport

def get_transport_class(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from .base import BigtableTransport
from .grpc import BigtableGrpcTransport
from .grpc_asyncio import BigtableGrpcAsyncIOTransport
from .pooled_grpc_asyncio import PooledBigtableGrpcAsyncIOTransport
from .rest import BigtableRestTransport
from .rest import BigtableRestInterceptor

Expand All @@ -27,12 +28,14 @@
_transport_registry = OrderedDict() # type: Dict[str, Type[BigtableTransport]]
_transport_registry["grpc"] = BigtableGrpcTransport
_transport_registry["grpc_asyncio"] = BigtableGrpcAsyncIOTransport
_transport_registry["pooled_grpc_asyncio"] = PooledBigtableGrpcAsyncIOTransport
_transport_registry["rest"] = BigtableRestTransport

__all__ = (
"BigtableTransport",
"BigtableGrpcTransport",
"BigtableGrpcAsyncIOTransport",
"PooledBigtableGrpcAsyncIOTransport",
"BigtableRestTransport",
"BigtableRestInterceptor",
)
Loading