Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

try to transform throttle into a reusable transformation #3409

Closed
wants to merge 6 commits into from
Closed

try to transform throttle into a reusable transformation #3409

wants to merge 6 commits into from

Conversation

mbaechler
Copy link

I tried to make it more readable but failed because of reactor/reactor-core#1099

If anybody has an idea ...

chibenwa
chibenwa reviewed May 29, 2020
View reviewed changes
...compute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java Outdated
.window(runningOptions.getUsersPerSecond(), Duration.ofSeconds(1))
.throttle(users)
return Iterators.toFlux(usersRepository.list())
.transform(ReactorUtils.Throttler.<Username>throttler().elements(runningOptions.getUsersPerSecond()).per(Duration.ofSeconds(1)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent

chibenwa
chibenwa reviewed May 29, 2020
View reviewed changes
server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java Outdated
.throttle(originalFlux)
ImmutableList<Integer> ongoingProcessingUponComputationStart = originalFlux
.transform(ReactorUtils.<Integer>throttle().elements(windowMaxSize).per(windowDuration))
.flatMap(longRunningOperation)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain to me why throttleDownStreamConcurrencyShouldNotExceedWindowMaxSize still works?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it probably doesn't but maybe you have an idea of why if you put this comment? I couldn't find why it broke

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I understand it is that transform don't limit downstream concurrency, hence I was really curious to know if that test pass.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot from 2020-06-08 10-19-08
from https://projectreactor.io/docs/core/release/reference/

How did you saw it's doing anything differently than inline operators?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your operation don't wrap the downstream flatMap request were we do actually perform the operation we want to throttle. If the prossessing is slower than the rate of the throttling you would end up doing more concurrent operations than what you would like, unless you apply some form of limitations to it.

That's what the test enforces.

Also, from what I understand the downstream flatMap needs to be part of the 'extracted operator chain'.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A syntax addressing this might be (suggestion)

originalFlux
    .transform(ReactorUtils<Integer>throttle()
        .elements(windowMaxSize)
        .per(windowDuration)
        .forOperation(longRunningOperation))

@Arsnael
Copy link
Member

Arsnael commented Jun 2, 2020

I know it's a draft, but still... No ticket related?

nvduc91
nvduc91 reviewed Jun 2, 2020
View reviewed changes
server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java Outdated
Preconditions.checkArgument(!duration.isZero(), "'windowDuration' must be strictly positive");
return flux -> flux
.windowTimeout(elements, duration)
.zipWith(Flux.interval(DELAY, duration), 1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you make more clear of 1, days or minus or second?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concurrency. if we can make it work, we'll refine the PR

@chibenwa
Copy link
Member

chibenwa commented Jun 9, 2020

I pushed a fixup with my above proposal.

I was surprised to get a buffer overrun error due to the prefect on the interval (that one #3409 (comment)) , hence added one more test to ensure we are "buffer filling proof".

Feel free to keep it or discard it.

@chibenwa chibenwa added this to the Polish 2020-06 milestone Jun 12, 2020
@Arsnael
Copy link
Member

Arsnael commented Jun 15, 2020 

[3ed77ddea478a599e63c52f902c0d99be634a38f] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project apache-james-mailbox-tools-indexer: Compilation failure
[3ed77ddea478a599e63c52f902c0d99be634a38f] [ERROR] /james-project/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java:[280,28] cannot find symbol
[3ed77ddea478a599e63c52f902c0d99be634a38f] [ERROR]   symbol:   variable Throttler
[3ed77ddea478a599e63c52f902c0d99be634a38f] [ERROR]   location: class org.apache.james.util.ReactorUtils
[3ed77ddea478a599e63c52f902c0d99be634a38f] [ERROR] -> [Help 1]

@Arsnael Arsnael force-pushed the throttle-operator branch from 3ed77dd to d2245f5 Compare June 15, 2020 04:42
@Arsnael
Copy link
Member

Arsnael commented Jun 15, 2020

There was rebase issue with master, so I rebased and tried to fix the issue

@chibenwa
Copy link
Member

docker: Error response from daemon: No such container: git-publish-throttle-operator-0eed264e-6c14-4b87-bd90-9874fa98616d.

Can be considered 🍏

test this please

@Arsnael
Copy link
Member

Arsnael commented Jun 17, 2020

Log output lost

test this please

@chibenwa
Copy link
Member

Reschedule build: test this please

@chibenwa
Copy link
Member 

[d2245f5750b18af22418b148f7adeb6dc1e710e8] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project rabbitmq-jmap-draft-integration-testing: There was a timeout or other error in the fork -> [Help 1]<

Seems unrelated

test this please

@chibenwa
Copy link
Member

docker: Error response from daemon: No such container: git-publish-throttle-operator-43d0eb15-3934-4577-97b2-12381c0ee569.

test this please

@chibenwa
Copy link
Member

docker: Error response from daemon: No such container: git-publish-throttle-operator-4f297f9f-af9e-4ad3-8509-f9d580a3cf80.

(be patient)

test this please

@chibenwa chibenwa marked this pull request as ready for review June 21, 2020 04:12
@chibenwa chibenwa self-assigned this Jun 24, 2020
@chibenwa chibenwa added waiting_merge We are about to merge this! and removed cross-review needed labels Jun 24, 2020
@Arsnael
Copy link
Member

Arsnael commented Jun 25, 2020

Still missing a relating ticket though?

@chibenwa
Copy link
Member

JAMES-3184 seems appropriate

@chibenwa
Copy link
Member

Merged

@chibenwa chibenwa closed this Jun 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nvduc91 nvduc91 nvduc91 left review comments

@chibenwa chibenwa chibenwa approved these changes

@Arsnael Arsnael Arsnael approved these changes

@rouazana rouazana rouazana approved these changes

Assignees

@chibenwa chibenwa

Labels
waiting_merge We are about to merge this!
Projects
None yet
Milestone
Polish 2020-06
Development

Successfully merging this pull request may close these issues.
5 participants
@mbaechler @Arsnael @chibenwa @rouazana @nvduc91