SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469

patrickt · 2020-10-05T15:48:09Z

Subject of the issue

I encounter segfaults when attempting to run haskell-language-server-wrapper in the root of my project (or from within my editor). They only go away when I remove this Template Haskell invocation.

I have tried removing the ~/.cache/ghcide folder, ensuring there are no old haskell-language-server binaries on my PATH, and adding a cabal.project to the project. This happens with versions 0.4 and 0.5. I’m using the latest release of macOS.

EDIT: compiling haskell-language-server from source, passing in --enable-executable-dynamic to configure, appears to be a successful workaround.

Your environment

haskell-language-server version: 0.5.0.0 (GHC: 8.10.1) (PATH: /Users/patrickt/.ghcup/bin/haskell-language-server-wrapper-0.5.0) (GIT hash: 14497f2503a2a0d389fabf3b146d674b9af41a34)
Tool versions found on the $PATH
cabal:		3.4.0.0
stack:		2.3.1
ghc:		8.10.2

Which lsp-client do you use
Emacs (though this happens at the command line)
Describe your project (alternative: link to the project)

https://github.com/patrickt/possession

Steps to reproduce

git clone https://github.com/patrickt/possession.git
cd possession
haskell-language-server-wrapper

Expected behaviour

Successful project compilation

Actual behaviour

fish: '/Users/patrickt/.ghcup/bin/hask…' terminated by signal SIGSEGV (Address boundary error)

Include debug information

Debug output:

haskell-language-server version: 0.5.0.0 (GHC: 8.10.2) (PATH: /Users/patrickt/.ghcup/bin/haskell-language-server-8.10.2~0.5.0) (GIT hash: 14497f2503a2a0d389fabf3b146d674b9af41a34)
(haskell-language-server)Ghcide setup tester in /Users/patrickt/src/possession.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:		3.4.0.0
stack:		2.3.3
ghc:		8.10.2


Step 1/4: Finding files to test in /Users/patrickt/src/possession
Found 16 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/Game/Ecs.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
> Resolving dependencies...
> Build profile: -w ghc-8.10.2 -O0
> In order, the following will be built (use -v for more details):
>  - possession-0.1.0.0 (lib) (configuration changed)
> Configuring library for possession-0.1.0.0..
> Preprocessing library for possession-0.1.0.0..
[INFO] Using interface files cache dir: /Users/patrickt/.cache/ghcide/possession-0.1.0.0-inplace-d45d56728bdc453e2db86899d6a423581f0b2ce7
[INFO] Making new HscEnv[possession-0.1.0.0-inplace]
[INFO] Consulting the cradle for "app/Main.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
fish: '/Users/patrickt/.ghcup/bin/hask…' terminated by signal SIGSEGV (Address boundary error)

The text was updated successfully, but these errors were encountered:

patrickt · 2020-10-05T15:55:02Z

For a commit of the project on which hls executes correctly: 2b14bde26a5f8f0e3f1b31fa6d52410ccab99cf1.
For one that fails: 7ad24b119fcf5655d0692530420d00feb0972125

patrickt · 2020-10-05T16:12:41Z

If this is relevant, here’s the code generated by the splice:

instance
  ( k_aaaH ~ An_Iso,
    a_aaaI ~ Apecs.Entity,
    b_aaaJ ~ Apecs.Entity
  ) =>
  LabelOptic "player" k_aaaH State State a_aaaI b_aaaJ
  where
  {-# INLINE labelOptic #-}
  labelOptic = (iso (\(State x_aaaK) -> x_aaaK)) State

patrickt · 2020-10-05T16:28:56Z

Remembered that is probably a duplicate of https://github.com/haskell/ghcide/issues/775

jneira · 2020-10-05T19:13:49Z

It is almost identical so i would close one of both (the older one?)

pepeiborra · 2020-10-08T19:59:16Z

Could be related to https://github.com/haskell/ghcide/issues/854

jneira · 2020-11-09T09:13:05Z

This one may be fixed in master (and the incoming hls 0.5 release) via ghcide-0.5, @patrickt could you check if master fix the issue for you?

patrickt · 2020-11-10T17:43:55Z

@jneira I’m afraid not; compiling with the latest HEAD gives me

haskell-language-server version: 0.5.1.0 (GHC: 8.10.2) (PATH: /Users/patrickt/.local/bin/haskell-language-server) (GIT hash: 8682517e9ff92caa35e727e28445896f97c61e8d)
(haskell-language-server)Ghcide setup tester in /Users/patrickt/src/possession.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:		3.4.0.0
stack:		2.5.1
ghc:		8.10.2


Step 1/4: Finding files to test in /Users/patrickt/src/possession
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/UI/State.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
> Resolving dependencies...
> Build profile: -w ghc-8.10.2 -O0
> In order, the following will be built (use -v for more details):
>  - possession-0.1.0.0 (lib) (configuration changed)
> Configuring library for possession-0.1.0.0..
> Preprocessing library for possession-0.1.0.0..
[INFO] Using interface files cache dir: /Users/patrickt/.cache/ghcide/possession-0.1.0.0-inplace-089966ab63c3b422ee9e2e4bb9ae32c2a6c4a099
[INFO] Making new HscEnv[possession-0.1.0.0-inplace]
File:     /Users/patrickt/src/possession/src/UI/Sidebar.hs
Hidden:   no
Range:    29:1-29:19
Source:   typecheck
Severity: DsWarning
Message: 
  The import of ‘UI.Resource’ is redundant
  except perhaps to import instances from ‘UI.Resource’
  To import instances alone, use: import UI.Resource()
fish: 'haskell-language-server --debug…' terminated by signal SIGSEGV (Address boundary error)

jneira · 2020-11-10T18:26:32Z

Oh, thanks for trying it anyways, i am afraid that th support needs more work to be reliable enough 😟

patrickt · 2020-11-11T18:55:12Z

Some more information: here’s the DWARF backtrace (EDIT: the line numbers here don’t seem to be hugely reliable)

* thread #55, stop reason = EXC_BAD_ACCESS (code=1, address=0x20)
  * frame #0: 0x0000000105bcda8b haskell-language-server`ocResolve_MachO [inlined] relocateSection(oc=<unavailable>, curSection=0) at MachO.c:680:40 [opt]
    frame #1: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be4adb0) at MachO.c:1529 [opt]
    frame #2: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be4adb0) at Linker.c:1735:9 [opt]
    frame #3: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #4: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmcrzm0zi3zi0zi1zm0e46846a_OpticsziInternalziOpticziTypes_zdtcAzuLens_closure") at Linker.c:878 [opt]
    frame #5: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #6: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be61080) at MachO.c:1529 [opt]
    frame #7: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be61080) at Linker.c:1735:9 [opt]
    frame #8: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #9: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_LanguageziHaskellziTHziOpticsziInternal_zdfSubstTypeTypezuzdcsubstType_closure") at Linker.c:878 [opt]
    frame #10: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #11: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5c210) at MachO.c:1529 [opt]
    frame #12: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5c210) at Linker.c:1735:9 [opt]
    frame #13: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #14: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTHziInternalziUtils_zdwquantifyTypezq_info") at Linker.c:878 [opt]
    frame #15: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #16: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5d400) at MachO.c:1529 [opt]
    frame #17: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5d400) at Linker.c:1735:9 [opt]
    frame #18: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #19: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTHziInternalziProduct_LensRules_con_info") at Linker.c:878 [opt]
    frame #20: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #21: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5a230) at MachO.c:1529 [opt]
    frame #22: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5a230) at Linker.c:1735:9 [opt]
    frame #23: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #24: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTH_makeFieldLabels_closure") at Linker.c:878 [opt]
    frame #25: 0x0000000105b9d560 haskell-language-server`lookupSymbol(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTH_makeFieldLabels_closure") at Linker.c:916:21 [opt]
    frame #26: 0x00000001055d6169 haskell-language-server`Lc3pa_info + 49

wz1000 · 2020-11-11T18:58:34Z

@bgamari any idea what is going on here?

patrickt · 2020-11-11T19:02:51Z

A debug print of the ObjectCode* that causes the crash is here. It looks like the symbol it’s crashing on is __text.

(ObjectCode) $1 = {
  status = OBJECT_NEEDED
  fileName = 0x000000010be4ae80 "/Users/patrickt/.cabal/store/ghc-8.10.2/ptcs-cr-0.3.0.1-0e46846a/lib/libHSptcs-cr-0.3.0.1-0e46846a.a"
  fileSize = 8912
  formatName = 0x0000000105d2db42 "Mach-O"
  archiveMemberName = 0x000000010be4aef0 "/Users/patrickt/.cabal/store/ghc-8.10.2/ptcs-cr-0.3.0.1-0e46846a/lib/libHSptcs-cr-0.3.0.1-0e46846a.a(Types.o)"
  symbols = 0x000000010be4b320
  n_symbols = 41
  image = 0x00000001270cc000 "????"
  info = 0x000000010be4b1a0
  imageMapped = 1
  referenced = 0
  misalignment = 0
  n_sections = 3
  sections = 0x000000010be4b200
  n_segments = 1
  segments = 0x000000010be4ad40
  next = 0x000000010be4a1b0
  proddables = 0x000000010be4b300
  symbol_extras = 0x00000001270ce000
  first_symbol_extra = 0
  n_symbol_extras = 45
  bssBegin = 0x00000001270ce000 ""
  bssEnd = 0x00000001270ce000 ""
  stable_ptrs = 0x0000000000000000
  extraInfos = 0x0000000000000000
  rw_m32 = 0x000000010be4af60
  rx_m32 = 0x000000010be4b080
}
(lldb) p ((ObjectCode*) 0x000000010be4adb0)->info
(ObjectCodeFormatInfo *) $2 = 0x000000010be4b1a0
(lldb) p *(((ObjectCode*) 0x000000010be4adb0)->info)
(ObjectCodeFormatInfo) $3 = {
  header = 0x00000001270cc000
  symCmd = 0x00000001270cc170
  segCmd = 0x00000001270cc020
  dsymCmd = 0x00000001270cc188
  nlist = 0x00000001270cc918
  names = 0x00000001270ccbe8 ""
  macho_sections = 0x00000001270cc068
  macho_symbols = 0x000000010f890200
  n_macho_symbols = 45
  got_start = 0x0000000000000000
  got_size = 0
}

wz1000 · 2020-11-11T19:04:43Z

I think it would be useful if you compiled ghcide with -debug and shared the output of running ghcide with +RTS -Dl

wz1000 · 2020-11-11T19:07:27Z

BTW, the original code snippet you linked to doesn't point to any Template Haskell anymore.

patrickt · 2020-11-11T19:10:03Z

@wz1000 Fixed the original code snippet. Building with -debug now.

patrickt · 2020-11-11T19:30:42Z

@wz1000 The result of executing +RTS -Dl and piping stderr to a file was ~600MB, but here’s a gist of the last 500 lines.

wz1000 · 2020-11-11T20:28:00Z

Huh, I can't reproduce this locally on linux with ghcide on GHC 8.10.2.

ProofOfKeags · 2020-11-11T20:37:22Z

I'd like to advance the theory that this might be restricted to MacOS. My colleague is also experiencing something like this on Mac, but if I try on Linux with her exact same project, I cannot repro.

I'm about to dust off my old Mac to see if there's any credibility to this idea.

wz1000 · 2020-11-11T20:40:37Z

It would be helpful if someone could come up with a test case that fails on the newly added MacOS CI on ghcide.

ProofOfKeags · 2020-11-11T21:02:00Z

The test case that is failing for my colleague is actually quite easy to produce:

stack new test-yesod yesodweb/postgres <- this with no alterations is enough to produce it. This seems to be the case for her on both 8.8.4/stackage-16.21 and 8.10.2/nightly-2020-11-11.

pepeiborra · 2020-11-11T21:46:24Z

The backtrace is super useful.

It might be possible to repro without ghcide, if you have a ghc binary built with DYNAMIC_GHC_PROGRAMS=NO. Compiling the project normally might just be enough to reproduce. It would be great if you can try that out and submit a minified GHC bug report.

And if you link ghcide/HLS with -dynamic, it should/will use the system linker instead of the rts linker and avoid this bug.

patrickt · 2020-11-11T22:35:26Z

@ProofOfKeags @wz1000 Good to hear that this doesn’t repro on Linux: given that this this is dying in the guts of relocating sections in Mach-O executables, we can assume it’s something macOS specific.

patrickt · 2020-11-11T22:36:34Z

@pepeiborra Wow, compiling with -dynamic fixed things! Thank you!

ProofOfKeags · 2020-11-11T22:53:19Z

@patrickt @pepeiborra do you just put the -dynamic flag in hls' cabal file? or is this something that goes into the various stack.yaml. Never done dynamic linking before, sorry if this is a dumb question.

patrickt · 2020-11-11T23:38:15Z

@ProofOfKeags Run cabal configure --enable-optimization --enable-executable-dynamic in the haskell-language-server repo root. Then build, and check to see if the resulting executable handles the file in question correctly.

ProofOfKeags · 2020-11-12T19:12:23Z

Dynamic linking fixed this for us too. So am I to understand that this issue is in GHC?

pepeiborra · 2020-11-12T19:23:18Z

Yes, GHC has its own dynamic linker, in the rts, and it gets used in certain cases:

when GHC the binary is linked statically, by GHCI and TH
when ghc the library is linked statically, by TH and the ghc interactive api

Implementing a dynamic linker with support for >=3 platforms is a risky and complex exercise. Fortunately there is an easy opt-out (-dynamic) but unfortunately that opt-out makes distribution much harder, and the precompiled HLS binaries are statically linked.

As I mentioned it would be great if you can minify this bug and report it upstream in GHC.

sordina · 2020-11-30T10:29:01Z

+1 Confirmed the -dynamic option successfully works around the issue for me in haskell-language-server.

JosephLucas · 2020-12-01T09:18:28Z

+1. It solved an issue with Template Haskell.

HLS was reporting an issue with the linux ld object, that it could not find.
Adding -dynamic seemed to have solved the previous issue, but I got another issue

Cannot load -dynamic objects when GHC is built the normal way
    To fix this, either:
      (1) Use -fexternal-interpreter, or
      (2) Build the program twice: once the normal way, and then
          with -dynamic using -osuf to set a different object file suffix.

Adding -fexternal-interpreter solved this second issue.

Now I still have useless warning waves under quasiquotes of template haskell though.

pepeiborra · 2020-12-01T12:16:41Z

-fexternal-interpreter is not supported with ghcide/hls, it will crash.

Where did you local GHC installation come from? Is it a custom build with DYNAMIC_GHC_PROGRAMS=NO?

JosephLucas · 2020-12-01T21:12:24Z

Strange, I think I really use -fexternal-interpreter
-fexternal-interpreter is in package,yaml and PROJECT.cabal.
I confirm that HLS does work. Maybe I don't use the last version of everything...

My ghc comes from /nix/store/lfv2p3l2q8wahr9gx5ks3fdvlvq7rdaf-ghc-8.8.4-with-packages/bin/ghc

patrickt · 2020-12-02T17:04:50Z

I’ve filed a bug upstream to inform about the dynamic linker issue. Anyone who wanted to help minimize the test case would win my infinite gratitude.

isTesting got added to ShakeExtras unnecessarily

jneira · 2022-01-31T08:55:02Z

I am gonna close this issue as all compiler crashes seems to have the same root cause:

We have a general issue about: Improve template haskell support (upstream fix and workaround for compiler crashes) #1431
And other one with more info about the causes: Document why hls binaries are statically linked (and track ghc changes to allow loading dynamic libs within a static executable) #2000
We have a generic workaround (build hls from source dynamically linked) documented in the troubleshooting guide: https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries-and-template-haskell-support
And a possible fix upstream for ghc described in Document why hls binaries are statically linked (and track ghc changes to allow loading dynamic libs within a static executable) #2000 (comment)

If any of you think the issue should not be included generically feel free to reopen it (with a brief explanation if possible)
Thanks all!

patrickt changed the title ~~SIGSEGV (address boundary error) on invocation~~ SIGSEGV (address boundary error) when encountering Template Haskell splice Oct 5, 2020

jneira added the type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc.. label Oct 5, 2020

jneira added component: ghcide status: needs info Not actionable, because there's missing information labels Nov 9, 2020

jneira added the type: template haskell related label Nov 10, 2020

jneira added the os: macos label Nov 12, 2020

jneira added can-workaround status: blocked Not actionable, because blocked by upstream/GHC etc. and removed status: needs info Not actionable, because there's missing information labels Nov 12, 2020

pepeiborra mentioned this issue Nov 30, 2020

Segfault on empty YESOD project #230

Closed

pepeiborra added a commit that referenced this issue Dec 27, 2020

Refactoring: remove isTesting from ShakeExtras (#469)

0bf4e91

isTesting got added to ShakeExtras unnecessarily

jneira mentioned this issue Feb 10, 2021

HLS crashes with "The Haskell (<proj>) server crashed 5 times in the last 3 minutes. The server will not be restarted." #1342

Closed

pepeiborra mentioned this issue Jun 8, 2021

HLS keeps crashing (exit code -11, segfault, bus error, ..) with plutus pioneer program #1847

Closed

pepeiborra changed the title ~~SIGSEGV (address boundary error) when encountering Template Haskell splice~~ SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice Jun 8, 2021

lapplislazuli mentioned this issue Jun 25, 2021

Segmentation fault on OS X Tritlo/PropR#3

Closed

codygman mentioned this issue Jul 1, 2021

Segfault when running hls with TH and libpq #1982

Closed

jneira mentioned this issue Oct 1, 2021

Segmentation fault when encountering a TemplateHaskell splice #800

Closed

dcastro mentioned this issue Oct 21, 2021

"cabal-hls-install ghcs" does not list all ghc versions available. #2285

Closed

jneira mentioned this issue Oct 21, 2021

Improve template haskell support (upstream fix and workaround for compiler crashes) #1431

Closed

jneira mentioned this issue Nov 20, 2021

Document why hls binaries are statically linked (and track ghc changes to allow loading dynamic libs within a static executable) #2000

Open

pavel-demin mentioned this issue Dec 12, 2021

clashi segfaults on macos clash-lang/clash-compiler#2021

Open

jneira closed this as completed Jan 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469

SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469

patrickt commented Oct 5, 2020 •

edited

Loading

patrickt commented Oct 5, 2020 •

edited

Loading

patrickt commented Oct 5, 2020

patrickt commented Oct 5, 2020

jneira commented Oct 5, 2020

pepeiborra commented Oct 8, 2020

jneira commented Nov 9, 2020

patrickt commented Nov 10, 2020

jneira commented Nov 10, 2020 •

edited

Loading

patrickt commented Nov 11, 2020 •

edited

Loading

wz1000 commented Nov 11, 2020

patrickt commented Nov 11, 2020 •

edited

Loading

wz1000 commented Nov 11, 2020

wz1000 commented Nov 11, 2020

patrickt commented Nov 11, 2020

patrickt commented Nov 11, 2020

wz1000 commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020 •

edited

Loading

wz1000 commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020

pepeiborra commented Nov 11, 2020 •

edited

Loading

patrickt commented Nov 11, 2020

patrickt commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020

patrickt commented Nov 11, 2020 •

edited

Loading

ProofOfKeags commented Nov 12, 2020

pepeiborra commented Nov 12, 2020

sordina commented Nov 30, 2020

JosephLucas commented Dec 1, 2020 •

edited

Loading

pepeiborra commented Dec 1, 2020 •

edited

Loading

JosephLucas commented Dec 1, 2020

patrickt commented Dec 2, 2020

jneira commented Jan 31, 2022

SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469

SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469

Comments

patrickt commented Oct 5, 2020 • edited Loading

Subject of the issue

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Include debug information

patrickt commented Oct 5, 2020 • edited Loading

patrickt commented Oct 5, 2020

patrickt commented Oct 5, 2020

jneira commented Oct 5, 2020

pepeiborra commented Oct 8, 2020

jneira commented Nov 9, 2020

patrickt commented Nov 10, 2020

jneira commented Nov 10, 2020 • edited Loading

patrickt commented Nov 11, 2020 • edited Loading

wz1000 commented Nov 11, 2020

patrickt commented Nov 11, 2020 • edited Loading

wz1000 commented Nov 11, 2020

wz1000 commented Nov 11, 2020

patrickt commented Nov 11, 2020

patrickt commented Nov 11, 2020

wz1000 commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020 • edited Loading

wz1000 commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020

pepeiborra commented Nov 11, 2020 • edited Loading

patrickt commented Nov 11, 2020

patrickt commented Nov 11, 2020

ProofOfKeags commented Nov 11, 2020

patrickt commented Nov 11, 2020 • edited Loading

ProofOfKeags commented Nov 12, 2020

pepeiborra commented Nov 12, 2020

sordina commented Nov 30, 2020

JosephLucas commented Dec 1, 2020 • edited Loading

pepeiborra commented Dec 1, 2020 • edited Loading

JosephLucas commented Dec 1, 2020

patrickt commented Dec 2, 2020

jneira commented Jan 31, 2022

patrickt commented Oct 5, 2020 •

edited

Loading

patrickt commented Oct 5, 2020 •

edited

Loading

jneira commented Nov 10, 2020 •

edited

Loading

patrickt commented Nov 11, 2020 •

edited

Loading

patrickt commented Nov 11, 2020 •

edited

Loading

ProofOfKeags commented Nov 11, 2020 •

edited

Loading

pepeiborra commented Nov 11, 2020 •

edited

Loading

patrickt commented Nov 11, 2020 •

edited

Loading

JosephLucas commented Dec 1, 2020 •

edited

Loading

pepeiborra commented Dec 1, 2020 •

edited

Loading