Support asyncio

[jamesls]

This is a tracking issue for the feature request of supporting asyncio in botocore, originally asked about here: #452

There's no definitive timeline on this feature, but feel free to +1 (thumbs up 👍) this issue if this is something you'd like to see. Also, if you have any additional information about asyncio you'd like to share (even just about your specific use case) feel free to chime in.

[jab]

Thanks for logging this! Currently I'm only calling list_distributions(..) and create_distribution(..) on a boto3.client('cloudfront') instance, which I know is a small fraction of the boto3 API, but it'd be nice if those calls cooperatively yielded in between making the request and receiving the response so that other coroutines in my app could run in the meantime. (I could always just roll my own client using aiohttp too since I'm using so little of the API, but figured it was worth asking about anyway.)

In case it helps, aiodns provides an example of supporting asyncio on Python 2.6+ (see https://github.com/saghul/aiodns#python-versions).

If boto3 was already written with the assumption of blocking rather than asynchronous I/O throughout, I'm not sure how disruptive a change this would be.

[Jc2k]

I'm not that familiar with the internals of botocore, but I think the right approach is to create a custom version of endpoint.py and client.py that used aiohttp under the hood and yielded as you want.

I would probably start by subclassing Endpoint and making an aiohttp version of _send_request and _get_request. I'd also subclass ClientCreator and override _create_api_method (that is where the calls to the endpoint come from). I haven't yet figured out how i'd get from a service object to a client created with my ClientCreator subclass.

Does that sound right, @jamesls? I might have a go...

[rdbhost]

I've started a port of botocore to asyncio, at:

https://github.com/rdbhost/botocore

So far, the S3 integration tests pass, as do about 2/3 of the unit tests. I hope to have all tests converted and passing by a week from now.

The port depends on yieldfrom.requests and yieldfrom.urllib3 , asyncio ports of the requests and urllib3 libraries.

[jab]

@rdbhost Awesome! Thanks for working on this. Definitely interested to follow your progress.
@jamesls / botocore maintainers, had a chance to check this out / evaluate for merge potential?

[koliyo]

+1

[AlexNigl]

+1

[rdbhost]

This issue has gotten a couple of +1s in the last few days, so I thought I
would point out that the work is done, at:

https://github.com/rdbhost/yieldfromBotocore

It implements an asyncio version of botocore, not boto3. I may eventually
convert boto3, but my own needs have been satisfied by botocore, so other
things now have priority.

David

On Thu, Jun 18, 2015 at 6:59 AM, AlexNigl [email protected] wrote:

+1

—
Reply to this email directly or view it on GitHub
#458 (comment).

[amatthies]

+1

[jmehnle]

@rdbhost, thanks for tackling that! It seems, however, that https://github.com/rdbhost/yieldfromBotocore is 271 commits behind upstream. Are you actively maintaining it?

[rdbhost]

My intention has been to do a merge when upstream reached v1.0; that seems to have happened without my noticing.

I will be merging commits from upstream, this weekend.

[r39132]

+1 @jamesls are you planning on pulling rdbhost's changes into botocore itself? We are very interested in this feature.

[rdbhost]

It is not a very promising candidate for merging back into botocore.

The changes are numerous, and the changes needed to make botocore functional within asyncio make it non-functional outside asyncio. I expect it to be a seperate product indefinitely, with a parallel API, meaning an API as similar as possible within the asyncio constraints.

[jamesls]

One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.

[jmehnle]

Trollius (asyncio port to Python 2.x) is complete and stable.

[harai]

+1

[jettify]

I have working port of botocore for asyncio: https://github.com/jettify/aiobotocore using aiohttp for async http requests. I am trying to reuse as much botocore code as possible, so I patched only several classes and just import rest of the code as result library has few hundreds lines of code. And this approach helps to keep up with upstream, but obvious downside, I rely on internal interfaces which is subject of change for new libs. API almost the same as botocore just yield from or awaite (python 3.5) should be added before calls.

For now I am using aiobotocore with s3 and ported almost all s3 test, except I need to work more on pagination since it is not easy to implement iterator protocol with yield from.

[mpaolini]

+1 integrating aiobotocore might be doable

[balihoo-gens]

+1

[mikeplavsky]

+1

[cgst]

+1

[thomascellerier]

+1

[lsbardel]

OK guys, I've written a production ready, and hopefully useful to the community, asyncio extension to botocore. The library is currently included in https://github.com/quantmind/pulsar-cloud in the asyncbotocore module but it is self-contained and could be striped out if needed.

The API is the same as botocore but with the addition of http_session keyword which is an optional asyncio compatible HTTP client (it must expose the _loop attribute) and API similar to requests.
As far as I know there are two of such clients:

• pulsar http client
• aiohttp http client (@asvetlov does aihttp client/session have the _loop attribute?)

If used with pulsar, the library can also use greenlet in an implicit asynchronous fashion. Check https://github.com/quantmind/pulsar-cloud for more info

By the way, thanks to @jettify for the initial effort from which I leveraged from

Feedbacks welcome!

[asvetlov]

@lsbardel yes, aiohttp.ClientSession has _loop but I don't encourage using private attributes.
BTW we use https://github.com/jettify/aiobotocore as aiohttp-based library.

[lsbardel]

@asvetlov cool, I understand the private attribute thing, but the _loop attribute should be (almost) a standard for asyncio objects ;-) maybe we should ask guido.

I would be happy to use https://github.com/jettify/aiobotocore has low level async botocore library but currently it does not work with pulsar http client. At the moment it clearly requires aiohttp while cloud.asyncbototore does not require an asyncio library as such.

So maybe we should converge to a library that allows for different http clients?

[lsbardel]

By the way all that is python 3.4 or above

[asvetlov]

@lsbardel At asyncio active development times Guido was against public loop attribute, he motivated it as "user always may pass explicit loop if needed". I don't think the decision has changed now.

[nadad]

+1

[jmorris0x0]

+1

[gainskills]

+1

[GeekEast]

let's make python javascript AGAIN!

[AmitArie]

Any updates about this issue?

[tim-finnigan]

Hi all, thanks for your patience. I brought up this up for discussion with the team and the consensus was that async support won't be considered until the next major version. Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.

[flying-sheep]

Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.

Could you please clarify what you mean by that? Wouldn’t “creating a longer term async solution for botocore users” be basically the same thing as a rewrite of botocore?

[stalkerg]

@tim-finnigan is it means you started rewriting the botocore? Can you collaborate with aiobotocore instead?

[Andrew-Chen-Wang]

Redis corporate took over the entire Python redis ecosystem. If a community botocore was written supporting both sync and async(io), would AWS consider adopting it?

[stalkerg]

@tim-finnigan do you have any update?

[takeda]

@stalkerg this ticket was opened 8 years ago, there's a ticket (aws-cloudformation/cloudformation-coverage-roadmap#196 (comment)) created 4 years ago to add support for stupid tags (something that should maybe take an hour).

I wouldn't hold my breath.

There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.

[flying-sheep]

We’re interested in first class support because aiobotocore is (partially by necessity) a bad citizen of the Python package ecosystem.

• It pins its dependency botocore, which one shouldn’t do because it messes up dependency resolution.
• It does that because it’s extremely hacky and reaches deep into the internals of botocore, so each version of aiobotocore only supports one botocore version.
• And that’s of course because botocore’s public API doesn’t lend itself to make an async version of itself.

So unless aiobotocore becomes unnecessary because botocore gains async support, we’re stuck with a (as said multiple times necessary) hack. aioboto3 and aiobotocore should become unnecessary one day, and we should look forward to that day.

[dacevedo12]

Relevant aio-libs/aiobotocore#1017 (comment)

Funny, he removed it. Basically said that they're not expecting this to happen before 2024, and it will likely be a major version.

[stalkerg]

There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.

not anymore they rewrote a lot of parts and did async/await drill down. But yes, they still have a dependence on the botocore.

[ricardodns]

+1

[stalkerg]

@tim-finnigan sorry for ping you again, but do you have any updates? Any roadmaps or just strategy? It's literally difficult to do business with AWS predictable right now.
As I understand botocore/boto was initially designed for scripts, CLI tools (aws-cli like) and etc and nobody designed it for full and heavy web-services. AsyncIO basically has no sense for such goals and requires a new SDK, but because you have no resources, you just ignore it (and because aiobotocore exists). Even another AWS team tried to use aiobtocore, and it was a shame, from my point of view.
I believe the community and your customers (not small!) want an update about this topic at least once a year.

[AddisonG]

Who's looking forward to celebrating the 10th birthday of this issue with me in Feb?

I think it's safe to say that this isn't going to happen, given that AWS is happy to exploit both the open source community's goodwill and their willingness to compensate for boto's shortcomings. See you all in another decade.

[dacevedo12]

To keep the conversation alive, check awslabs/aws-crt-python#558

I'm glad the community was even able to come up with the aioboto libraries (shout out to its maintainers 🥇 ), boto is a beast of its own.

Maybe Amazon is aiming to standardize everything on CRT?

[gokturkDev]

i simply can't believe this...