Hystrix timeout + rx retry() = deadlock

[costimuraru]

We seem to have gotten into a situation where we're stuck in a deadlock. The use case:
Setup:
[] Convert the HystrixCommand into an Observable
[] Wrap this observable with a retry(x times)
When:
[] HystrixCommand times out
Then:
[] Instead of retrying the same command x times, a deadlock occurs.

Below you can see the code that exhibits this deadlock.

io.reactivex:rxjava:1.1.1
com.netflix.hystrix:hystrix-core:1.4.23

public class Command extends HystrixCommand<String> {

    public Command() {
        super(Setter.withGroupKey(asKey("test"))
                .andThreadPoolPropertiesDefaults(
                        HystrixThreadPoolProperties.Setter()
                                .withCoreSize(10)
                )
                .andCommandPropertiesDefaults(
                        HystrixCommandProperties.Setter()
                                .withExecutionTimeoutInMilliseconds(10)
                ));
    }

    @Override
    protected String run() throws Exception {
        Thread.sleep(200);
        return "response";
    }
}

public class CommandWithRetryTest {

    public static void main(String[] Args) {
        String result = Observable.just(1)
                .flatMap(i -> new Command()
                                .toObservable()
                                .retry(2) // This triggers the deadlock.
                )
                .toBlocking()
                .first(); // ===> deadlock
    }
}

[mattrjacobs]

@costimuraru You shouldn't use retry() on an Observable that Hystrix produces. The Hystrix model is that each command object is single-use. Therefore, if a command fails once, it will never produce anything different.

That said, a deadlock is pretty harsh, so I'll see why this is happening and if there's anything that Hystrix can do to produce an Observable that makes this error more tolerable.

[mattrjacobs]

I believe that this got fixed by #1396. The subscribe-after-unsubscribe case was not handled well before. In #1370, it manifested as a memory leak. In this case, as a deadlock. I added a unit test in #1406 to show that this is fixed in master.