Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix the PollingBatch with the current version of yagna #302

Merged
merged 10 commits into from
Apr 7, 2021
Merged

fix the PollingBatch with the current version of yagna #302

merged 10 commits into from
Apr 7, 2021

Conversation

shadeofblue
Copy link
Contributor

No description provided.

@shadeofblue shadeofblue self-assigned this Mar 29, 2021
@shadeofblue shadeofblue requested a review from a team March 29, 2021 17:13
azawlocki
azawlocki suggested changes Mar 30, 2021
View reviewed changes
yapapi/rest/activity.py Outdated Show resolved Hide resolved
@mfranciszkiewicz
Copy link
Contributor

There are 2 issues here:

  1. timeout was unintentionally removed from the spec and has been restored in ya-client
  2. in case of this particular endpoint, timeout currently defaults to 0 and the call will return immediately (I don't think this was intended). This behaviour should be changed in yagna to use a fallback 5 s value.

azawlocki
azawlocki approved these changes Apr 6, 2021
View reviewed changes
Copy link
Contributor

@azawlocki azawlocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mfranciszkiewicz mfranciszkiewicz mentioned this pull request Apr 7, 2021
Merged
azawlocki
azawlocki suggested changes Apr 7, 2021
View reviewed changes
Copy link
Contributor

@azawlocki azawlocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've realised, thanks to @filipgolem, that with _request_timeout set, get_exec_batch_result() could raise asyncio.TimeoutError, not ApiException, and therefore we should also catch that exception.

yapapi/rest/activity.py
@@ -167,7 +167,7 @@ async def __aiter__(self) -> AsyncIterator[events.CommandEventContext]:
raise BatchTimeoutError()
try:
results: List[yaa.ExeScriptCommandResult] = await self._api.get_exec_batch_results(
self._activity_id, self._batch_id, timeout=timeout
self._activity_id, self._batch_id, _request_timeout=timeout
)
except ApiException as err:
Copy link
Contributor

@azawlocki azawlocki Apr 7, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to ApiException we should also catch asyncio.TimeoutError here and raise BatchTimeoutError instead.

Copy link
Contributor

@azawlocki azawlocki Apr 7, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also shorten the amount of sleep when no new results are available, at the end of this method. It's now 10 seconds:

yapapi/yapapi/rest/activity.py

Lines 196 to 198 in 157e69e

if not any_new:
delay = min(10, max(0, self.seconds_left()))
await asyncio.sleep(delay)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's three seconds in yajsapi.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to ApiException we should also catch asyncio.TimeoutError here and raise BatchTimeoutError instead.

To avoid (tcp) connection closed errors, I limited maximum timeout to 5 seconds and added continue when timeouts happen.

@shadeofblue shadeofblue changed the base branch from master to b0.5 April 7, 2021 11:58
@shadeofblue shadeofblue changed the base branch from b0.5 to master April 7, 2021 11:59
azawlocki
azawlocki approved these changes Apr 7, 2021
View reviewed changes
Copy link
Contributor

@azawlocki azawlocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well, good job!

@shadeofblue shadeofblue merged commit 0cc5b18 into master Apr 7, 2021
@shadeofblue shadeofblue deleted the blue/polling-batch-timeout-argument branch April 7, 2021 14:39
shadeofblue added a commit that referenced this pull request Apr 7, 2021
@shadeofblue @filipgolem
fix the PollingBatch with the current version of yagna (#302)
665b0bd 
* fix the PollingBatch with the current version of yagna
* use aiohttp's general `_request_timeout`
* Set maximum client-side timeout to 5 seconds, continue on client-side and server-side timeouts.
* Decrease timeout

Co-authored-by: Filip <[email protected]>
shadeofblue added a commit that referenced this pull request Apr 7, 2021
@shadeofblue
Merge pull request #315 from golemfactory/0.5/fix-polling-batch
233b783 
fix the PollingBatch with the current version of yagna (based on #302)
azawlocki pushed a commit that referenced this pull request Apr 8, 2021
@shadeofblue @filipgolem @azawlocki
fix the PollingBatch with the current version of yagna (#302)
578ec24 
* fix the PollingBatch with the current version of yagna
* use aiohttp's general `_request_timeout`
* Set maximum client-side timeout to 5 seconds, continue on client-side and server-side timeouts.
* Decrease timeout

Co-authored-by: Filip <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@filipgolem filipgolem filipgolem left review comments

@azawlocki azawlocki azawlocki approved these changes

@mfranciszkiewicz mfranciszkiewicz Awaiting requested review from mfranciszkiewicz

Assignees

@azawlocki azawlocki

@shadeofblue shadeofblue

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shadeofblue @mfranciszkiewicz @azawlocki @filipgolem