Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimize retries #854

Merged
merged 89 commits into from
Aug 17, 2023
Merged
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
18650af
added function for populating shard
daniel-sanche Jul 24, 2023
df05171
Merge branch 'v3' into benchmarks_prod
daniel-sanche Jul 24, 2023
4db25b3
got table population working
daniel-sanche Jul 24, 2023
28cc1f4
finished table population
daniel-sanche Jul 24, 2023
9e97ac0
pulled out table setup fixtures from system tests
daniel-sanche Jul 25, 2023
52550cd
get system tests working
daniel-sanche Jul 25, 2023
a191236
added some basic throughput code
daniel-sanche Jul 25, 2023
31dbbd9
count operations
daniel-sanche Jul 25, 2023
eac206a
added splits to table
daniel-sanche Jul 25, 2023
ea771de
extracted cluster config
daniel-sanche Jul 25, 2023
2eda0ac
chanted test name
daniel-sanche Jul 25, 2023
f0649fd
improved benchmark output
daniel-sanche Jul 25, 2023
fbf2636
fixed population to use new format
daniel-sanche Jul 25, 2023
21df0c2
added sharded scan benchmark
daniel-sanche Jul 25, 2023
91ef008
more accurate time tracking
daniel-sanche Jul 25, 2023
f91750f
added script to deploy to GCE
daniel-sanche Jul 26, 2023
0236f85
limit to one qualifier
daniel-sanche Jul 26, 2023
fd94350
added separate test for point reads
daniel-sanche Jul 26, 2023
82ecaf2
made row size configurable
daniel-sanche Jul 26, 2023
a35f985
don't timeout benchmarks at deadline
daniel-sanche Jul 26, 2023
a440d76
added fastapi benchmark
daniel-sanche Jul 26, 2023
d0fcdb9
added driver for fastapi class
daniel-sanche Jul 26, 2023
0bb70ef
delete VM on complete; improved formatting
daniel-sanche Jul 27, 2023
3f3f59a
improved fastapi benchmark
daniel-sanche Jul 27, 2023
5a6ffad
run tests concurrently
daniel-sanche Jul 27, 2023
604071f
saved test_system changes
daniel-sanche Jul 27, 2023
3686ef6
added profling to point reads
daniel-sanche Jul 27, 2023
c42f596
add duration to run script
daniel-sanche Jul 27, 2023
e2ac258
build async generator manually
daniel-sanche Jul 28, 2023
1228c8a
lazy parse cell data
daniel-sanche Jul 28, 2023
1a3a8e7
added profiling to scans
daniel-sanche Jul 28, 2023
0659161
removed iscouroutine check
daniel-sanche Jul 28, 2023
ce2b733
added legacy benchmark
daniel-sanche Jul 28, 2023
4714e8a
added thread to legacy tests
daniel-sanche Jul 28, 2023
3111543
cut out row adapter
daniel-sanche Jul 28, 2023
c717a0b
lazy load rows
daniel-sanche Jul 28, 2023
b5cb253
simplify generator layers for non-stremaing
daniel-sanche Jul 29, 2023
aa4bed4
change pool size
daniel-sanche Jul 29, 2023
613b032
disabled lazy loading (75k rps)
daniel-sanche Jul 29, 2023
8e37967
only check for last_scanned when there are no chunks (81k rps)
daniel-sanche Jul 30, 2023
6775e2b
refactored chunk parsing (80k rps)
daniel-sanche Jul 31, 2023
537f4ff
removed subclass for row
daniel-sanche Jul 31, 2023
58b3fbb
added igor's row merger
daniel-sanche Jul 31, 2023
a7478ab
use stream by default
daniel-sanche Aug 7, 2023
689f072
use our models
daniel-sanche Aug 7, 2023
107b8c5
wrapped in object
daniel-sanche Aug 7, 2023
d76a7dc
updated row merger for acceptance tests
daniel-sanche Aug 8, 2023
1d9a475
improve performance
daniel-sanche Aug 8, 2023
37bc3a0
removed HasField from family
daniel-sanche Aug 8, 2023
ea95dcc
added request revision logic
daniel-sanche Aug 8, 2023
9b60693
added back split cell verifications
daniel-sanche Aug 8, 2023
24de29e
got all acceptance tests passing
daniel-sanche Aug 8, 2023
9c9ba5e
track last yielded key
daniel-sanche Aug 8, 2023
5d49ad8
track reamining count
daniel-sanche Aug 8, 2023
5ecfa3e
added retries
daniel-sanche Aug 8, 2023
13603ea
added exception conversion
daniel-sanche Aug 8, 2023
b40e1cb
moved into single function
daniel-sanche Aug 8, 2023
0f58066
Fixed style issues
daniel-sanche Aug 9, 2023
07341da
Merge branch 'v3' into optimize_read_rows_2
daniel-sanche Aug 9, 2023
90b85b3
fixed row and helpers tests
daniel-sanche Aug 9, 2023
4298e38
fixed some read rows tests
daniel-sanche Aug 9, 2023
62f1bf7
comments and cleanup
daniel-sanche Aug 9, 2023
98510c3
fixed issues with conformance tests
daniel-sanche Aug 9, 2023
4cf4aab
improved proto conversion
daniel-sanche Aug 9, 2023
0f0ccee
fixed read_rows tests
daniel-sanche Aug 10, 2023
42f44da
fixed tests
daniel-sanche Aug 10, 2023
3d1804c
optimizing query class
daniel-sanche Aug 10, 2023
b0882af
keep filter and limit out of proto storage
daniel-sanche Aug 10, 2023
2f2286d
fixed tests
daniel-sanche Aug 10, 2023
fbe298e
skip check when not necessary
daniel-sanche Aug 10, 2023
2a82343
experiment: yield in row batches
daniel-sanche Aug 10, 2023
4e6dd6f
Revert "experiment: yield in row batches"
daniel-sanche Aug 10, 2023
daa7a59
removed benchmarks dir
daniel-sanche Aug 10, 2023
e91d693
fixed type annotation
daniel-sanche Aug 10, 2023
cb28451
added slots to query
daniel-sanche Aug 10, 2023
1dd4d0c
use separate instances for each run
daniel-sanche Aug 14, 2023
6d2dab6
made _ReadRowOperation into slots
daniel-sanche Aug 14, 2023
309405a
clean up style issues
daniel-sanche Aug 14, 2023
e36d70b
removed commented test
daniel-sanche Aug 14, 2023
eb4bcfa
use optimizations from retries
daniel-sanche Aug 15, 2023
f6e8cd2
removed layer of wrapping
daniel-sanche Aug 15, 2023
5037891
fixed tests
daniel-sanche Aug 15, 2023
ea5b4f9
updated retryable stream submodule
daniel-sanche Aug 15, 2023
9456196
updated dependency version
daniel-sanche Aug 16, 2023
b9f8cdf
added todo
daniel-sanche Aug 16, 2023
fa0734c
Merge branch 'v3' into optimize_retries
daniel-sanche Aug 17, 2023
3323a66
moved exception builder into own method
daniel-sanche Aug 17, 2023
76d6df7
updated constraints
daniel-sanche Aug 17, 2023
bf669ad
fixed lint issues
daniel-sanche Aug 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
get system tests working
daniel-sanche committed Jul 25, 2023
commit 52550cd9792ee9873506a047c8b3e652b3133a32
2 changes: 1 addition & 1 deletion benchmarks/conftest.py
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Import fixtures for setting up table from setup_fixtures.py
Import pytest fixtures for setting up table from system test directory.
"""
import sys
import os
4 changes: 2 additions & 2 deletions tests/system/data/conftest.py → tests/system/conftest.py
Original file line number Diff line number Diff line change
@@ -12,13 +12,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Import fixtures for setting up table from setup_fixtures.py
Import pytest fixtures for setting up table for data client system tests
"""
import sys
import os
script_path = os.path.dirname(os.path.realpath(__file__))
sys.path.append(script_path)

pytest_plugins = [
"setup_fixtures",
"data.setup_fixtures",
]