Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'test_paginate_files': eventual consistency failure #2217

Closed
tseaver opened this issue Aug 29, 2016 · 1 comment · Fixed by #2293
Closed

'test_paginate_files': eventual consistency failure #2217

tseaver opened this issue Aug 29, 2016 · 1 comment · Fixed by #2293
Assignees
@dhermes
Labels
api: storage Issues related to the Cloud Storage API.

Comments

@tseaver
Copy link
Contributor

tseaver commented Aug 29, 2016

From https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/155830290#L832-L838:

======================================================================
FAIL: test_paginate_files (storage.TestStorageListFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 248, in test_paginate_files
    self.assertTrue(iterator.next_page_token is not None)
AssertionError: False is not true
@tseaver tseaver added api: storage Issues related to the Cloud Storage API. flaky labels Aug 29, 2016
@dhermes
Copy link
Contributor

dhermes commented Sep 9, 2016

@tseaver At the end of the day this is an eventual consistency error (surprising we've never seen it before). The values come from

response = self.bucket.list_blobs(max_results=2).get_next_page_response()

where we expect there to be 3 filenames:

FILENAMES = ('CloudLogo1', 'CloudLogo2', 'CloudLogo3')

So iterator.next_page_token is not None will be true if at the time the backend knew about all 3 files. But if not, then we see this failure.

@dhermes dhermes self-assigned this Sep 9, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Sep 9, 2016
@dhermes
Retry entire test_paginate_files() storage system test.
733a5ac 
This is not just "taking the easy way out", it's really
the most appropriate fix since there are so many
assertions that can fail due to eventual consistency. (For
example, asking for 2 blobs should have a next page when 3
are in the bucket, but this may break down due to eventual
consistency.)

Fixes googleapis#2217.

Also

- restoring test_second_level() to pre-googleapis#2252 (by retrying
  the entire test case)
- restoring test_list_files() to pre-googleapis#2181 (by retrying
  the entire test case)
- adding retries around remaining test cases that call
  list_blobs(): test_root_level_w_delimiter(),
  test_first_level() and test_third_level()
- adding a helper to empty a bucket in setUpClass() helper
  (which also uses list_blobs()) in both TestStoragePseudoHierarchy
  and TestStorageListFiles
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Sep 9, 2016
@dhermes
Retry entire test_paginate_files() storage system test.
da6b9eb 
This is not just "taking the easy way out", it's really
the most appropriate fix since there are so many
assertions that can fail due to eventual consistency. (For
example, asking for 2 blobs should have a next page when 3
are in the bucket, but this may break down due to eventual
consistency.)

Fixes googleapis#2217.

Also

- restoring test_second_level() to pre-googleapis#2252 (by retrying
  the entire test case)
- restoring test_list_files() to pre-googleapis#2181 (by retrying
  the entire test case)
- adding retries around remaining test cases that call
  list_blobs(): test_root_level_w_delimiter(),
  test_first_level() and test_third_level()
- adding a helper to empty a bucket in setUpClass() helper
  (which also uses list_blobs()) in both TestStoragePseudoHierarchy
  and TestStorageListFiles
@dhermes dhermes mentioned this issue Sep 9, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhermes dhermes

Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Retry entire test_paginate_files() storage system test. dhermes/google-cloud-python
2 participants
@tseaver @dhermes