MarGo installation fails for snap packaged go 1.10

[simonklb]

[23:41:29] margo: agent#007: log: run["go" "install" "-v" "-tags=margo" "-i" "disposa.blue/margo/cmd/margo.sublime"]
[23:41:29] margo: agent#007: log: go install runtime/cgo: open /snap/go/2130/pkg/linux_amd64/runtime/cgo.a: read-only file system
[23:41:29] margo: agent#007: log: cannot install margo.sublime: exit status 1

The reason seems to be that the -i flag is included in the go install command which causes a reinstallation of cgo because it's seen as stale. See this comment for more info: golang/go#24674 (comment)

Is there any particular reason that you explicitly add the -i flag for go 1.10 here? This suggests the opposite:

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.

[DisposaBoy]

Did you verify that it's the -i and not the -tags? go install -i simply restores the behavior go install had before they decided it was a good idea to just change it in go1.10.

The code that adds -i is at $PACKAGES/GoSublime/src/margo.sh/sublime/sublime.go:74. You can simply comment out that line then save the file and it should re-compile.

If it's the -tags then I'd argue that that's a Go and/or the snap distro bug and though I'm willing to work around it — I will have to, in order to support go modules in the future anyway — I don't know when I will get round to it.

[simonklb]

I've confirmed that it's the -i flag that is causing the reinstall of cgo. Commented out that line and got rid of that previous error.

However, on the development branch I'm now seeing another error message instead:

[01:53:13] margo: agent#007: ['/home/simon/.config/sublime-text-3/Packages/GoSublime/bin/margo.sh', 'start', 'margo.sublime', '-codec', 'msgpack']
[01:53:13] margo: agent#007: log: ``` go install -v -tags=margo margo.sh/cmd/margo.sublime ```
[01:53:14] margo: agent#007: log: ``` /home/simon/.config/sublime-text-3/Packages/GoSublime/bin/margo.sublime -codec msgpack ```
[01:53:14] margo: agent#007: log: store.go:137: started
[01:53:14] margo: agent#007: log: restart.go:156: Mg/Restart: exit status 1
[01:53:14] margo: agent#007: log: ``` go install -v -tags=margo margo.sh/cmd/margo.sublime ```
[01:53:14] margo: agent#007: log: check console for errors
[01:53:14] margo: agent#007: log: *Not* using margo extension: cannot find package "margo" in any of:
[01:53:14] margo: agent#007: log: 	/snap/go/2512/src/margo (from $GOROOT)
[01:53:14] margo: agent#007: log: 	/home/simon/.config/sublime-text-3/Packages/User/GoSublime/src/margo (from $GOPATH)
[01:53:14] margo: agent#007: log: 	/home/simon/.config/sublime-text-3/Packages/GoSublime/src/margo
[01:53:14] margo: agent#007: log: agent GOPATH is /home/simon/.config/sublime-text-3/Packages/User/GoSublime:/home/simon/.config/sublime-text-3/Packages/GoSublime

[simonklb]

I deleted the binaries and now there are no errors in the console but I'm not sure it finished the installation completely. It ended after:

02:26:06] margo: agent#007: ['/home/simon/.config/sublime-text-3/Packages/GoSublime/bin/margo.sh', 'start', 'margo.sublime', '-codec', 'msgpack']
[02:26:06] margo: agent#007: log: Using margo extension: /home/simon/.config/sublime-text-3/Packages/User/GoSublime/src/margo
[02:26:06] margo: agent#007: log: ``` go install -v -tags="margo margo_extension" margo.sh/cmd/margo.sublime ```
[02:26:07] margo: agent#007: log: ``` /home/simon/.config/sublime-text-3/Packages/GoSublime/bin/margo.sublime -codec msgpack ```
[02:26:07] margo: agent#007: log: store.go:137: started
[02:26:19] margo: agent#007: log: ``` go install margo.sh/vendor/golang.org/x/tools/cmd/guru ```

I don't seem to have any autocompletion for anything other than stdlib packages.

[DisposaBoy]

You'll need to press ctrl+.,ctrl+x to configure margo — which it appears you've already done.

You can also confirm it's working by making a syntax error which should automatically get highlighted unless you removed the &golang.SyntaxCheck{} reducer.

The &DayTimeStatus{} reducer will also place the time in the status bar.

The gocode implementation still relies on binary packages so if you haven't installed them, you won't get any autocompletion. The workflow for this is typically to just use a go install linter that runs on-save but they call go install -i so you'd have to create your own. The code for the builtin ones are here https://github.com/DisposaBoy/GoSublime/blob/development/src/margo.sh/golang/lint.go#L41

You can just copy one into your margo.go file (ctrl+.,ctrl+x) and customize it to your needs e.g.

&golang.Linter{
	Name:    "go",
	Args:    []string{"install"},
	Label:   "Go/Install (no binaries)",
	// set GOBIN to a temp directory to prevent installing binaries into GOPATH/bin
	TempDir: []string{"GOBIN"},
}

[simonklb]

Ah, I wasn't aware that you had to install to get the completions to work. Is that mentioned in the docs? Otherwise perhaps it would be a good idea to add a comment about it under Installation in README or under Code Completion in USAGE.

Also, just out of curiosity, would it be impossible to do code completion simply from source or why does gocode require binaries?

Thanks for the quick response, really great feedback!

[simonklb]

I think I found the answer to my own question: nsf/gocode#233 (comment)

Thanks again!

[DisposaBoy]

It's possible to use source, but it's currently very slow so I didn't enable it.

[simonklb]

Oh, mind sharing how? Also, feel free to re-open this if you are planning on fixing the issue with snap and the -i flag.

[DisposaBoy]

I think you just need to use the source importer instead of the binary ones. There were some problems with it, but I don't remember what. Maybe something to do with GOPATH and other global variables, but I've come up with a clever work-around for that.

In the newImporter() function https://github.com/DisposaBoy/GoSublime/blob/development/src/margo.sh/golang/gocode_suggest.go#L45 just add return importer.For("source", nil).(types.ImporterFrom).

If you have the console open, you will probably start seeing messages like gocode didn't accept the request after... and gocode eventually responded after ... but other than that, it should Just Work™

[simonklb]

It Just Works™! I noticed that there was some work in progress to cache imports in gocode. That would probably speed things up dramatically. Fingers crossed. :)

[simonklb]

Actually, from what I can tell so far, it's only slow the first time suggestions are requested for a package that hasn't been imported yet. After that completions seem smooth. Are you doing something clever in GoSublime or could it be some internal caching going on in go?

[DisposaBoy]

I do some caching in margo:

• For packages in GOPATH, there is no caching — I use a new importer instance every time.

• For stdlib packages, I always use the same importer instance, so it should only ever load a stdlib package once. This was based on the observation that most imports will come from the stdlib which almost never changes.

[simonklb]

I'm curious then why there is only a delay the first time margo imports (source) and not every time it creates suggestions. I've also noticed that when I add new code to a file that has been imported previously I need to restart margo for it to fetch the new symbols.

The only thing I've patched is this:

 func (gsu *gcSuggest) newImporter() types.ImporterFrom {
-   // TODO: switch to source importer only
-   switch {
-   case runtime.Compiler == "gc":
-       return gcexportdata.NewImporter(token.NewFileSet(), map[string]*types.Package{})
-   default:
-       return importer.Default().(types.ImporterFrom)
-   }
+   return importer.For("source", nil).(types.ImporterFrom)
 }

Are you sure there is no internal cache in go (or something similar) causing this behavior even though you create a new importer instance each time?

[DisposaBoy]

It's probably just a case of failing to cache only GOROOT packages here https://github.com/DisposaBoy/GoSublime/blob/development/src/margo.sh/golang/gocode_suggest.go#L139 it should be if err == nil && p.Complete() && bpkg.Goroot {

[DisposaBoy]

dev branch now excludes -i. I also forgot that golang.GoInstall*() doesn't include -i by default, so you don't need to customise it.

latest changes also includes a fix for the gocode caching and a new Source option

[simonklb]

Installation works great now, thanks!

Source does unfortunately not seem to work anymore. Removing the && bpkg.Goroot condition from the if makes suggestions pop up again, so I think the problems is there somewhere. Let me know if you want me to create a new issue for this.

[DisposaBoy]

This is most likely the slowness. Do you see lines like gocode eventually responded after 4.321s in the console?

[simonklb]

Yes, every time I request suggestions and that makes suggestions never pop up.

[11:39:00] margo: agent#018: log: gocode.go:155: gocode didn't respond after 250ms taking 250ms
[11:39:00] margo: agent#018: log: gocode.go:152: gocode eventually responded after 1.039s
[11:39:01] margo: agent#018: log: gocode.go:155: gocode didn't respond after 250ms taking 250ms
[11:39:02] margo: agent#018: log: gocode.go:152: gocode eventually responded after 1.044s
[11:39:03] margo: agent#018: log: gocode.go:155: gocode didn't respond after 250ms taking 250ms
[11:39:03] margo: agent#018: log: gocode.go:152: gocode eventually responded after 1.036s
[11:39:04] margo: agent#018: log: gocode.go:155: gocode didn't respond after 250ms taking 250ms
[11:39:05] margo: agent#018: log: gocode.go:152: gocode eventually responded after 1.034s

[simonklb]

Would it be possible to use the cache for non-GOROOT packages as well and dirty the cache on save? Because now you really feel how slow it is every time you type which isn't really usable.

[DisposaBoy]

I plan to look at better caching soon, but I might have to fork or re-implement the srcimporter.

[simonklb]

Great to hear, thanks! Looking forward to see it in the changelog. :)

[DisposaBoy]

Experimental caching is implemented on the gsu-pkg-cache branch. The main issue I see is that memory use is too high (for my liking), but I don't (yet) see anything that's broken.