Time waste for constructing distributions in Distributions.jl

[xukai92]

We've discussed that the time used to constructed distributions in Distributions.jl is one of the bottleneck for Turing. To be more specific, even the same distribution object (same type of ditribtions with same parameters) is created for multiple times.

[xukai92]

Some quick experiments:

a = 0
@time for _ = 1:1000 a += logpdf(Normal(0, 1), 1) end
# => 0.009169 seconds (5.08 k allocations: 113.769 KB)

a = 0
lp(x) = logpdf(Normal(0, 1), x)
@time for _ = 1:1000 a += lp(1) end
# => 0.000058 seconds (2.00 k allocations: 31.250 KB)

Update

It seems the experiment above is incorrect.

@time Normal(0, 1)
# => 0.009785 seconds (14.99 k allocations: 571.074 KB)

@time Normal(0, 1)
# => 0.000041 seconds (5 allocations: 192 bytes)

@time Normal(0, 1)
# => 0.000152 seconds (5 allocations: 192 bytes)

It seems that Distributions.jl cache it somehow?

[xukai92]

I guess that's the reason why Turing runs so slow when the number of data point increases

[xukai92]

Possible solutions are

• translate distributions to our own code of logpdf
• cache distributions
  • We've actually already "cached" distributions in VarInfo. Maybe we can improve VarInfo to do that?

[xukai92]

Zoubin proposed today that we'd better somehow "overwrite" the common distributions using our own code first, with other distributions still able to call the Distributions.jl package ones.

[xukai92]

Distribution object is immutable so this issue doesn't exist.