Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Julia 1.0/1.1 can't handle deep nested structs #31663

Closed
felipenoris opened this issue Apr 10, 2019 · 8 comments · Fixed by #31680
Closed

Julia 1.0/1.1 can't handle deep nested structs #31663

felipenoris opened this issue Apr 10, 2019 · 8 comments · Fixed by #31680
Assignees
Labels
compiler:inference Type inference display and printing Aesthetics and correctness of printed representations of objects. regression Regression in behavior compared to a previous version

Comments

@felipenoris
Copy link
Contributor

felipenoris commented Apr 10, 2019

I'm migrating code from v0.6 to v1.0/v1.1 , and I found this issue that breaks Julia.

abstract type AbstractNode end                                                                                                                                                                                     
                                                                                                                                                                                                                   
struct Node{N1<:AbstractNode, N2<:AbstractNode} <: AbstractNode                                                                                                                                                    
    a::N1                                                                                                                                                                                                          
    b::N2                                                                                                                                                                                                          
end                                                                                                                                                                                                                
                                                                                                                                                                                                                   
struct Leaf <: AbstractNode                                                                                                                                                                                        
end                                                                                                                                                                                                                
                                                                                                                                                                                                                   
function gen_nodes(qty::Integer) :: AbstractNode                                                                                                                                                                   
    @assert qty > 0                                                                                                                                                                                                
    result = Leaf()                                                                                                                                                                                                
                                                                                                                                                                                                                   
    for i in 1:qty                                                                                                                                                                                                 
        result = Node(result, Leaf())                                                                                                                                                                              
    end                                                                                                                                                                                                            
                                                                                                                                                                                                                   
    return result                                                                                                                                                                                                  
end

Running on Julia v1.0 or v1.1, it runs ok for 10 nested elements, but it never finishes if I try to nest about 50 elements. In the example below, I had to use CTRL+C to cancel.

julia> include("issue.jl")                                                                                                                                                                                         
gen_nodes (generic function with 1 method)                                                                                                                                                                                                                                                                                                                                                                                            

julia> gen_nodes(10)                                                                                                                                                                                               
Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Leaf,Leaf},Leaf},Leaf}(Node{Node{Leaf,Leaf},Leaf}(Node{Leaf,Leaf}(Leaf(), Leaf()), Leaf()), Leaf()), Leaf()), Leaf()), Leaf()), Leaf()), Leaf()), Leaf()), Leaf())                                                                                                                                                                                                                                                                                                                

julia> gen_nodes(50)                                                                                                                                                                                               ^C^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT                                                                                                                                                                 
Internal error: encountered unexpected error in runtime:                                                                                                                                                           InterruptException()                                                                                                                                                                                               is_derived_type at ./compiler/typelimits.jl:32                                                                                                                                                                     is_derived_type at ./compiler/typelimits.jl:57                                                                                                                                                                     
is_derived_type at ./compiler/typelimits.jl:68                                                                                                                                                                     
is_derived_type at ./compiler/typelimits.jl:57 
is_derived_type at ./compiler/typelimits.jl:57                                                                                                                                                                     
...
is_derived_type_from_any at ./compiler/typelimits.jl:77                                                                                                                                                            
type_more_complex at ./compiler/typelimits.jl:199                                                                                                                                                                  limit_type_size at ./compiler/typelimits.jl:20                                                                                                                                                                     jfptr_limit_type_size_5660.clone_1 at /home/felipenoris/local/julia-1.1.0/lib/julia/sys.so (unknown line)                                                                                                          jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219                                                                                                                                        
abstract_call_method at ./compiler/abstractinterpretation.jl:300                                                                                                                                                   
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:85

The same code runs without issues on Julia v0.6, even if I increase to 1000 nested elements.

julia> include("issue.jl")
gen_nodes (generic function with 1 method)

julia> gen_nodes(50)
Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}(Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Node{Leaf,Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf},Leaf}
...

I would say that this is a major issue for any code that deals with trees or similar data structures.

@JeffBezanson JeffBezanson added the regression Regression in behavior compared to a previous version label Apr 10, 2019
@felipenoris felipenoris changed the title Julia 1.0/1.1 can't handle deep nested structs (regression) Julia 1.0/1.1 can't handle deep nested structs Apr 10, 2019
@felipenoris
Copy link
Contributor Author

I did a few more tests, and it looks like Julia is able to construct the value, but it can't print it.
If I define

Base.show(io::IO, n::AbstractNode) = print(io, "a node")

it succeeds:

julia> c = gen_nodes(1000)
a node

julia> c.a
a node

But my original issue arised from a type that had customized show method. So it looks like that the problem is not in the default show method, but something more general.

@felipenoris felipenoris changed the title Julia 1.0/1.1 can't handle deep nested structs Julia 1.0/1.1 can't print deep nested structs Apr 10, 2019
@fredrikekre fredrikekre added the display and printing Aesthetics and correctness of printed representations of objects. label Apr 10, 2019
@JeffBezanson JeffBezanson added the compiler:inference Type inference label Apr 10, 2019
@JeffBezanson
Copy link
Member

Possibly related to #31572?

@felipenoris
Copy link
Contributor Author

felipenoris commented Apr 10, 2019

Yes. I'm having other side effects beyond printing, related to structs with many type parameters. I'm trying to isolate the problem, but it looks like related to #31572.

@JeffBezanson JeffBezanson self-assigned this Apr 10, 2019
JeffBezanson added a commit that referenced this issue Apr 10, 2019
- avoid exponential search in `is_derived_type` when parameters are
  used as field types
- avoid inferring `show_default`
- improve a fast path in subtyping
JeffBezanson added a commit that referenced this issue Apr 11, 2019
- avoid exponential search in `is_derived_type` when parameters are
  used as field types
- avoid inferring `show_default`
- improve a fast path in subtyping
@felipenoris
Copy link
Contributor Author

felipenoris commented Apr 12, 2019

This example exposes the problem regarding other side effects beyond printing:

abstract type AbstractNode end

struct Node{N1<:AbstractNode, N2<:AbstractNode} <: AbstractNode
    a::N1
    b::N2
end

struct Leaf <: AbstractNode
end

function gen_nodes(qty::Integer) :: AbstractNode
    @assert qty > 0
    result = Leaf()

    for i in 1:qty
        result = Node(result, Leaf())
    end

    return result
end

count_leafs(l::Leaf) = 1
count_leafs(n::Node) = count_leafs(n.a) + count_leafs(n.b)

# this function call triggers the problem and does not return
count_leafs(gen_nodes(50))

@felipenoris felipenoris changed the title Julia 1.0/1.1 can't print deep nested structs Julia 1.0/1.1 can't handle deep nested structs Apr 12, 2019
@felipenoris
Copy link
Contributor Author

This is great! 8a6271b fixes both the printing and also this new example I came up with.

@felipenoris
Copy link
Contributor Author

felipenoris commented Apr 12, 2019

Just for the record, count_leafs(n::Node) in this case would be better defined as count_leafs(@nospecialize(n::Node)) to avoid problems, but the point here is just to expose the issue.

With @nospecialize you can increase n until you hit a stackoverflow error. Without it, you can hit the long inference problem again with n at 5000 or so, which is ok.

@vtjnash
Copy link
Member

vtjnash commented Apr 12, 2019

I would say that this is a major issue for any code that deals with trees or similar data structures.

FWIW, this usually isn’t an issue for data-structures since trees are usually parameterized on their element type, not the tree size and shape

@felipenoris
Copy link
Contributor Author

felipenoris commented Apr 12, 2019

@vtjnash, yes, you're right. this only happens on highly parametrized structs. But keep in mind this is a minimal working example of the issue, which is not useful by itself.

@KristofferC KristofferC mentioned this issue Apr 15, 2019
58 tasks
JeffBezanson added a commit that referenced this issue Apr 16, 2019
- avoid exponential search in `is_derived_type` when parameters are
  used as field types
- avoid inferring `show_default`
- improve a fast path in subtyping
KristofferC pushed a commit that referenced this issue Apr 20, 2019
- avoid exponential search in `is_derived_type` when parameters are
  used as field types
- avoid inferring `show_default`
- improve a fast path in subtyping

(cherry picked from commit b2b35e9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:inference Type inference display and printing Aesthetics and correctness of printed representations of objects. regression Regression in behavior compared to a previous version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants