Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crystal tool unreachable doesn't seem to work with a default Lucky app out of the box #14034

Closed
jwoertink opened this issue Dec 1, 2023 · 13 comments · Fixed by #15065
Closed
Labels
kind:bug A bug in the code. Does not apply to documentation, specs, etc. topic:tools:unreachable

Comments

@jwoertink
Copy link
Contributor

When I generate a brand new Lucky app, then try to run this command, I get this error

❯ crystal tool unreachable -s src/app.cr 
Parse:                             00:00:00.000047490 (   0.76MB)
Semantic (top level):              00:00:00.676927559 ( 154.28MB)
Semantic (new):                    00:00:00.003794976 ( 154.28MB)
Semantic (type declarations):      00:00:00.041002824 ( 170.28MB)
Semantic (abstract def check):     00:00:00.022827716 ( 186.28MB)
Semantic (restrictions augmenter): 00:00:00.012279238 ( 186.28MB)
Semantic (ivars initializers):     00:00:00.339022827 ( 218.34MB)
Semantic (cvars initializers):     00:00:00.016675461 ( 218.34MB)
Semantic (main):                   00:00:00.584360733 ( 282.46MB)
Semantic (cleanup):                00:00:00.000670768 ( 282.46MB)
Semantic (recursive struct check): 00:00:00.001140204 ( 282.46MB)

Macro runs:
 - /home/jeremy/Sites/zzz_test/lib/lucky/src/run_macros/generate_asset_helpers.cr: reused previous compilation (00:00:00.003366501)
Invalid memory access (signal 11) at address 0x7ffd98cb8e98
[0xe83369] ???
[0xe8332a] ???
[0x3b7250a] ???

❯ crystal -v
Crystal 1.10.1 [c6f3552f5] (2023-10-13)

LLVM: 15.0.7
Default target: x86_64-unknown-linux-gnu
@jwoertink jwoertink added the kind:bug A bug in the code. Does not apply to documentation, specs, etc. label Dec 1, 2023
@jwoertink
Copy link
Contributor Author

Looks like this is probably related to #13879 I just tried against master and got a stack overflow

❯ ~/Development/crystal/lang/bin/crystal tool unreachable src/app.cr 
Using compiled compiler at /home/jeremy/Development/crystal/lang/.build/crystal
Stack overflow (e.g., infinite or very deep recursion)
[0x55e605f4f0a6] *Exception::CallStack::print_backtrace:Nil +118 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605f25666] ~procProc(Int32, Pointer(LibC::SiginfoT), Pointer(Void), Nil) +310 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x7f0cb9842520] ?? +139692628780320 in /lib/x86_64-linux-gnu/libc.so.6
[0x55e6072653fa] GC_clear_stack_inner +26 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e607265414] GC_clear_stack_inner +52 in /home/jeremy/Development/crystal/lang/.build/crystal (8 times)
[0x55e60726546b] GC_clear_stack +59 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e607261aeb] GC_generic_malloc_many +1083 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e607270dbf] GC_malloc_kind +191 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605fe3586] *GC::malloc_atomic<UInt64>:Pointer(Void) +6 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605fe357e] *GC::malloc_atomic<UInt32>:Pointer(Void) +14 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605f5bead] *String#byte_slice?<Int32, Int32>:(String | Nil) +733 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605f5bb93] *String#byte_slice<Int32, Int32>:String +35 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e606070328] *Path#parents:Array(Path) +2888 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724ec4b] *Crystal::UnreachableVisitor#match_path?<String>:Bool +43 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724f6bc] *Crystal::UnreachableVisitor#interested_in<(Crystal::Location | Nil)>:(Bool | Nil) +60 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724f3c0] *Crystal::UnreachableVisitor#check_def<Crystal::Def+>:(Set(Crystal::Def+) | Nil) +160 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724fcce] *Crystal::UnreachableVisitor#track_unused_defs<Crystal::ModuleType+>:Nil +334 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724f7d2] *Crystal::UnreachableVisitor#process_type<Crystal::NamedType+>:Nil +50 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724f9ea] *Crystal::UnreachableVisitor#process_type<Crystal::NamedType+>:Nil +586 in /home/jeremy/Development/crystal/lang/.build/crystal (32661 times)
[0x55e60724f132] *Crystal::UnreachableVisitor#process_type<Crystal::Program>:Nil +450 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60724e6df] *Crystal::UnreachableVisitor#process<Crystal::Compiler::Result>:Array(Crystal::Def+) +79 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60714b301] *Crystal::Command#unreachable:Nil +2321 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e607146a7a] *Crystal::Command#tool:(Bool | Nil) +602 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60713ef58] *Crystal::Command#run:(Bool | Nil) +1480 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60713e869] *Crystal::Command::run<Array(String)>:(Bool | Nil) +25 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e60713e82d] *Crystal::Command::run:(Bool | Nil) +29 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605f01ed5] __crystal_main +2613 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e606101486] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +6 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e6061013fa] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +58 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x55e605f0f4d6] main +6 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x7f0cb9829d90] ?? +139692628680080 in /lib/x86_64-linux-gnu/libc.so.6
[0x7f0cb9829e40] __libc_start_main +128 in /lib/x86_64-linux-gnu/libc.so.6
[0x55e605f013d5] _start +37 in /home/jeremy/Development/crystal/lang/.build/crystal
[0x0] ???

@cippaciong
Copy link

cippaciong commented Jan 31, 2024

Hello, I stumbled upon this issue while looking for a similar one, and I wanted to let you know that the same problem happens also with crystal tool implementations and crystal tool context.

Here's an example with the Lucky Clover tutorial:

$ crystal tool implementations -c src/pages/fortunes/index_page.cr:2:20 src/app.cr
Stack overflow (e.g., infinite or very deep recursion)
[0x64b33e590e86] ?? +110721007947398 in crystal
[0x64b33e5907ea] ?? +110721007945706 in crystal
[0x71a465f4a710] ?? +124950899107600 in /usr/lib/libc.so.6
[0x64b33e678462] ?? +110721008895074 in crystal
[0x64b33e678467] ?? +110721008895079 in crystal (130853 times)
[0x64b33e654868] ?? +110721008748648 in crystal
[0x64b33e5aadf0] ?? +110721008053744 in crystal
[0x64b33dbbae98] __crystal_main +37720 in crystal
[0x64b33dbc0cc9] main +73 in crystal
[0x71a465f33cd0] ?? +124950899014864 in /usr/lib/libc.so.6
[0x71a465f33d8a] __libc_start_main +138 in /usr/lib/libc.so.6
[0x64b33dbb1a65] _start +37 in crystal
[0x0] ???
$ crystal tool context -c src/pages/fortunes/index_page.cr:2:20 src/clover.cr
Stack overflow (e.g., infinite or very deep recursion)
[0x555d2b5dfe86] ?? +93858647899782 in crystal
[0x555d2b5df7ea] ?? +93858647898090 in crystal
[0x7d718820a710] ?? +137926568617744 in /usr/lib/libc.so.6
[0x555d2b6a9ba2] ?? +93858648726434 in crystal
[0x555d2b6a9ba7] ?? +93858648726439 in crystal (130885 times)
[0x555d2b6a6608] ?? +93858648712712 in crystal
[0x555d2b6a1fe1] ?? +93858648694753 in crystal
[0x555d2b5f9df0] ?? +93858648006128 in crystal
[0x555d2ac09e98] __crystal_main +37720 in crystal
[0x555d2ac0fcc9] main +73 in crystal
[0x7d71881f3cd0] ?? +137926568525008 in /usr/lib/libc.so.6
[0x7d71881f3d8a] __libc_start_main +138 in /usr/lib/libc.so.6
[0x555d2ac00a65] _start +37 in crystal
[0x0] ???

@straight-shoota
Copy link
Member

That's very helpful information. If all those tools are affected, it might be an error in the compiler when configured to build without codegen. I don't expect that to have any effect on the semantic stage, though 😕
So this needs more investigation. Can anyone provide a direct source repository where this reproduces?

@cippaciong
Copy link

cippaciong commented Feb 1, 2024

Thanks a lot for having a look at it already. I just pushed my code here: https://github.com/cippaciong/lucky-clover-tutorial

You should be able to reproduce it with one of my examples above: crystal tool implementations -c src/pages/fortunes/index_page.cr:2:20 src/app.cr

@straight-shoota
Copy link
Member

So with that repo I'm getting an error that's seemingly introduced by some macro:

There was a problem expanding macro 'load_manifest'

Code in src/app.cr:4:1

 4 | Lucky::AssetHelpers.load_manifest "public/mix-manifest.json"
     ^
Called macro defined in lib/lucky/src/lucky/asset_helpers.cr:11:3

 11 | macro load_manifest(manifest_file = "")

Which expanded to:

 > 1 | Manifest at public/mix-manifest.json does not exist
       ^
Error: unknown token: '\e'

Reproduction steps:

git clone https://github.com/cippaciong/lucky-clover-tutorial
cd lucky-clover-tutorial
shards install
crystal tool implementations -c src/pages/fortunes/index_page.cr:2:20 src/app.cr

@jwoertink
Copy link
Contributor Author

You may need to run ./script/setup first. It's because it's expecting the assets to all be built. Or maybe comment out the Lucky::AssetHelpers.load_manifest in src/app.cr 🤔

@cippaciong
Copy link

@straight-shoota I removed public/mix-manifest.json from .gitignore and pushed it. You should be able to reproduce the error with those exact commands now.

@straight-shoota
Copy link
Member

A non-release compiler has some more valuable debug info:

$ crystal tool implementations -c src/pages/fortunes/index_page.cr:2:20 src/app.cr
Using compiled compiler at /home/johannes/src/crystal-lang/crystal/.build/crystal
Stack overflow (e.g., infinite or very deep recursion)
[0x55cdf7250eb6] *Exception::CallStack::print_backtrace:Nil +118 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf7226946] ~procProc(Int32, Pointer(LibC::SiginfoT), Pointer(Void), Nil) +310 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x7fbe7751a520] ?? +140456022353184 in /lib/x86_64-linux-gnu/libc.so.6
[0x55cdf88c5c37] *Crystal::ImplementationsVisitor +7 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf88c5f04] *Crystal::ImplementationsVisitor +724 in /home/johannes/src/crystal-lang/crystal/.build/crystal (7695 times)
[0x55cdf88c590a] *Crystal::ImplementationsVisitor +506 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf88c56a6] *Crystal::ImplementationsVisitor +54 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf88c549a] *Crystal::ImplementationsVisitor#process<Crystal::Compiler::Result>:Crystal::ImplementationResult +74 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf87be982] *Crystal::Command#implementations:Nil +1650 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf87b6ee0] *Crystal::Command#tool:(Bool | Nil) +624 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf87ae749] *Crystal::Command#run:(Bool | Crystal::Repl::Value | Nil) +1865 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf87ade4b] *Crystal::Command::run<Array(String)>:(Bool | Crystal::Repl::Value | Nil) +43 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf87addbe] *Crystal::Command::run:(Bool | Crystal::Repl::Value | Nil) +46 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf719b160] __crystal_main +2736 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf7448513] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +19 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf744847a] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +58 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x55cdf71a8736] main +6 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x7fbe77501d90] ?? +140456022252944 in /lib/x86_64-linux-gnu/libc.so.6
[0x7fbe77501e40] __libc_start_main +128 in /lib/x86_64-linux-gnu/libc.so.6
[0x55cdf719a5e5] _start +37 in /home/johannes/src/crystal-lang/crystal/.build/crystal
[0x0] ???

@cippaciong
Copy link

@straight-shoota hmm interesting but I didn't get if I should already be able to infer anything from that. Did it give you a clue of what might be the issue here? To me it's pretty unintelligible unfortunately.

@straight-shoota
Copy link
Member

No, I got no clue from that. I tried to dig a bit more, but didn't get far.

Just left that comment to have a trail for further analysis as it has a bit more information than the previous traces.

@zw963
Copy link
Contributor

zw963 commented Feb 24, 2024

We can reduce the code base to make the clue more clearly.

@Blacksmoke16
Copy link
Member

Blacksmoke16 commented Oct 5, 2024

From a quick look I did this seems to be originating from:

type.types?.try &.each_value do |inner_type|
process_type(inner_type)
end

Here's some information and how you can see the loop.

type        # => Bool::Lucky
type.class  # => Crystal::NonGenericModuleType
type.types? # => {"ColumnType" => Bool::Lucky::ColumnType,
 "Criteria" => Bool::Lucky::Criteria(T, V)}
type        # => Bool::Lucky::ColumnType
type.class  # => Crystal::AliasType
type.types? # => {"Lucky" => Bool::Lucky}
type        # => Bool::Lucky
type.class  # => Crystal::NonGenericModuleType
type.types? # => {"ColumnType" => Bool::Lucky::ColumnType,
 "Criteria" => Bool::Lucky::Criteria(T, V)}
type        # => Bool::Lucky::ColumnType
type.class  # => Crystal::AliasType
type.types? # => {"Lucky" => Bool::Lucky}

Ultimately was able to reduce it down to:

struct Bool
  module Lucky
    alias ColumnType = Bool
  end
end

Which produces:

Stack overflow (e.g., infinite or very deep recursion)
[0x5c93e8d22576] *Exception::CallStack::print_backtrace:Nil +118 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e8cecf36] ~procProc(Int32, Pointer(LibC::SiginfoT), Pointer(Void), Nil) +310 in /home/george/dev/git/crystal/.build/crystal
[0x706ea475d1d0] ?? +123620507898320 in /usr/lib/libc.so.6
[0x5c93ea123687] *Crystal::UnreachableVisitor#process_type<Crystal::NamedType+>:Nil +7 in /home/george/dev/git/crystal/.build/crystal
[0x5c93ea1238ca] *Crystal::UnreachableVisitor#process_type<Crystal::NamedType+>:Nil +586 in /home/george/dev/git/crystal/.build/crystal (32729 times)
[0x5c93ea123042] *Crystal::UnreachableVisitor#process_type<Crystal::Program>:Nil +450 in /home/george/dev/git/crystal/.build/crystal
[0x5c93ea12221f] *Crystal::UnreachableVisitor#process<Crystal::Compiler::Result>:Array(Tuple(Crystal::Def+, Int32)) +79 in /home/george/dev/git/crystal/.build/crystal
[0x5c93ea00e8e7] *Crystal::Command#unreachable:Nil +2327 in /home/george/dev/git/crystal/.build/crystal
[0x5c93ea005dcc] *Crystal::Command#tool:(Bool | Nil) +732 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e9ffdc88] *Crystal::Command#run:(Bool | Nil) +1480 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e9ffd559] *Crystal::Command::run<Array(String)>:(Bool | Nil) +25 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e9ffd51d] *Crystal::Command::run:(Bool | Nil) +29 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e8c6e4d9] __crystal_main +2601 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e8e64e46] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +6 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e8e64dba] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +58 in /home/george/dev/git/crystal/.build/crystal
[0x5c93e8c7bb96] main +6 in /home/george/dev/git/crystal/.build/crystal
[0x706ea4745e08] ?? +123620507803144 in /usr/lib/libc.so.6
[0x706ea4745ecc] __libc_start_main +140 in /usr/lib/libc.so.6
[0x5c93e8c6d9d5] _start +37 in /home/george/dev/git/crystal/.build/crystal
[0x0] ???

IDK how to really fix this, but I'd wager a guess you could avoid it by not monkey patching stdlib types like this 🤷.

EDIT: I think what's going on is compiler sees Bool::Lucky has a type Bool::Lucky::ColumnType and goes to process it, Bool::Lucky::ColumnType is an alias to Bool, so the compiler goes to process that type, which then repeats. I'm not sure what impact it would have on the Lucky codebase, but prefixing the types with :: results the issue:

struct Bool
  module ::Lucky
    alias ColumnType = ::Bool
  end
end

So it's definitely related to foreign things being within the stdlib type, tho TBD on if there's a fix, or the fix is just don't do this.

EDIT2: Not specific to stdlib types, seems more so it's just a circular reference that can happen when an alias defined within a type references that type itself:

struct Foo
  alias ColumnType = Foo
end

@straight-shoota
Copy link
Member

@Blacksmoke16 That's a great find! 🕵️
The issue with circular nested types is also present in other type hierarchy visitors and we're using different way to handle that:

  • Keep a list of visited types and only process ones that are not in the list (e.g. RecursiveStructChecker does this)
  • Skip AliasType and TypeDefType which are the only types that can produce circular hierarchies (e.g. Program#define_default_new does this)

The latter is easier and more efficient. We have no use of processing type aliases, so skipping them is completely fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug A bug in the code. Does not apply to documentation, specs, etc. topic:tools:unreachable
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants