Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

website: tip.golang.org is down #29251

Closed
andybons opened this issue Dec 14, 2018 · 10 comments
Closed

website: tip.golang.org is down #29251

andybons opened this issue Dec 14, 2018 · 10 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations)
Milestone

Comments

@andybons
Copy link
Member

Fails with this message on all pages:

builder.Init: build cache is disabled by GOCACHE=off, but required as of Go 1.12

exit status 1

/cc @bradfitz @bcmills

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 14, 2018
@andybons andybons added this to the Unreleased milestone Dec 14, 2018
@bradfitz
Copy link
Contributor

Bryan, can you fix this soon?

/cc @dmitshur

@bradfitz bradfitz added the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Dec 14, 2018
@bradfitz
Copy link
Contributor

I see no mention of GOCACHE in x/tools or x/net. We're not setting this explicitly anywhere that I can see.

@bradfitz
Copy link
Contributor

GOCACHE doesn't appear to be set in that container:

bradfitz@gdev:~$ kubectl exec -ti tipgodoc-deployment-5c777d44fd-6g7b7 /bin/bash
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# ps afx
    PID TTY      STAT   TIME COMMAND
  54205 ?        Ss     0:00 /bin/bash
  54212 ?        R+     0:00  \_ ps afx
      1 ?        Ssl    0:20 /go/bin/tip --autocert=tip.golang.org,beta.golang.org --autocert-bucket=golang-tip-autocert
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# cat /proc/1/environ  | grep GOCACHE
root@tipgodoc-deployment-5c777d44fd-6g7b7:/go/src/tip# 

@kevinburke
Copy link
Contributor

kevinburke commented Dec 14, 2018 via email

@bradfitz bradfitz self-assigned this Dec 14, 2018
@bradfitz
Copy link
Contributor

@kevinburke, ah, great, thanks!

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/154181 mentions this issue: tip: fix, update tip.golang.org

@bradfitz
Copy link
Contributor

I got it kinda back alive, but now the godoc processes are failing after a few minutes with:

process in /build/tip/b exited: signal: killed

I added some more debugging,

                go func() {
                        // TODO(adg,bradfitz): be smarter about dead processes
                        if err := cmd.Wait(); err != nil {
                                log.Printf("process in %v exited: %v (%T)", dir, err, err)
                                if ee, ok := err.(*exec.ExitError); ok {
                                        if ws, ok := ee.ProcessState.Sys().(syscall.WaitStatus); ok {
                                                log.Printf("  ... ws = %v, stopped=%v, exited=%v, exit=%v", ws, ws.Stopped(), ws.Exited(), ws.ExitStatus())
                                                if ws.Signaled() {
                                                        log.Printf("  ... stop signal: %v", ws.StopSignal())
                                                }
                                        }
                                }
                        }
                }()

And now see:

2018/12/14 16:09:55 process in /build/tip/b exited: signal: killed (*exec.ExitError)
2018/12/14 16:09:55   ... ws = 9, stopped=false, exited=false, exit=-1
2018/12/14 16:09:55   ... stop signal: signal -1

What is stop signal -1 !? @ianlancetaylor would be my usual person to ask about that. @bcmills, do you know?

@bradfitz
Copy link
Contributor

Nevermind. I was misusing StopSignal. I thought it was something else. WaitStatus of 9 just means it got SIGKILLed.

@bradfitz
Copy link
Contributor

As expected, OOM:

[55181.043263] Memory cgroup out of memory: Kill process 158615 (godoc) score 1861 or sacrifice child
[55181.052447] Killed process 158615 (godoc) total-vm:5326320kB, anon-rss:4174076kB, file-rss:1072kB, shmem-rss:0kB
[56028.461663] godoc invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=864
[56028.472283] godoc cpuset=0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c mems_allowed=0
[56028.481937] CPU: 0 PID: 167205 Comm: godoc Not tainted 4.14.65+ #1
[56028.488424] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[56028.497766] Call Trace:
[56028.500342]  dump_stack+0x63/0x82
[56028.503776]  dump_header+0x9f/0x22a
[56028.507386]  ? mem_cgroup_scan_tasks+0xaf/0xd0
[56028.511964]  oom_kill_process+0x232/0x430
[56028.516098]  out_of_memory+0x100/0x4c0
[56028.519968]  mem_cgroup_out_of_memory+0x3f/0x60
[56028.524629]  mem_cgroup_oom_synchronize+0x2dd/0x300
[56028.529625]  ? unlock_page_memcg+0x50/0x50
[56028.533846]  pagefault_out_of_memory+0x25/0x56
[56028.538421]  __do_page_fault+0x478/0x4e0
[56028.545848]  ? page_fault+0x2f/0x50
[56028.549549]  page_fault+0x45/0x50
[56028.552989] RIP: fee4c0:          (null)
[56028.557042] RSP: 70000000:000000c001d623a8 EFLAGS: 0a38b3d2
[56028.557132] Task in /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c killed as a result of limit of /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a
[56028.584011] memory: usage 4194304kB, limit 4194304kB, failcnt 38477
[56028.590440] memory+swap: usage 4194304kB, limit 9007199254740988kB, failcnt 0
[56028.597736] kmem: usage 14084kB, limit 9007199254740988kB, failcnt 0
[56028.604292] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[56028.629615] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/f04d13c3e17abb2424036943fd9ad241ca2708e91cdb8a621d48212e23b3a72d: cache:0KB rss:40KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:40KB inactive_file:0KB active_file:0KB unevictable:0KB
[56028.659538] Memory cgroup stats for /kubepods/burstable/podfd6d16ff-ffb9-11e8-a19e-42010a80021a/0a21ec026540ed621ad864464dd7753c8e2b8dbab1b42d20152ef2d7f02da85c: cache:104KB rss:4179924KB rss_huge:1441792KB shmem:0KB mapped_file:48KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:4179924KB inactive_file:100KB active_file:4KB unevictable:0KB
[56028.691290] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[56028.700052] [151592]     0 151592      256        1       4       2        0          -998 pause
[56028.709049] [151679]     0 151679   138583     3513      38       4        0           864 tip
[56028.717989] [167201]     0 167201  1384927  1043295    2249       8        0           864 godoc
[56028.727279] Memory cgroup out of memory: Kill process 167201 (godoc) score 1860 or sacrifice child
[56028.736654] Killed process 167201 (godoc) total-vm:5539708kB, anon-rss:4172504kB, file-rss:676kB, shmem-rss:0kB
[56028.926586] oom_reaper: reaped process 167201 (godoc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Something between Go 1.9 and Go 1.11 uses more memory I guess. Yay.

Will increase it a bit for now.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/163798 mentions this issue: playground: use Go 1.12

gopherbot pushed a commit to golang/playground that referenced this issue Feb 26, 2019
Go 1.12 has been released¹ and should be used, so that it's possible
to play with Go programs that rely on Go 1.12-only features.

In Go 1.12, build cache is required. Either GOCACHE must be set,
or HOME must be set so that GOCACHE can be implicitly deduced.
Set HOME and pass it on to the go build invocation inside the
compileAndRun function.

This fixes the following error, detected by the playground test:

	2019/02/26 06:28:44 compile error: build cache is required, but could not
	be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME
	are defined
	The command '/bin/sh -c /app/playground test' returned a non-zero code: 1

This is related to issues golang/go#29243 and golang/go#29251,
and the fix in CL 154181.

¹ https://groups.google.com/d/msg/golang-announce/Y1RZqnCKUKY/N9yK5c8iBgAJ

Fixes golang/go#30397

Change-Id: I44c5c8928060732f31fd935b114568258c7298c7
Reviewed-on: https://go-review.googlesource.com/c/163798
Reviewed-by: Brad Fitzpatrick <[email protected]>
@golang golang locked and limited conversation to collaborators Feb 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations)
Projects
None yet
Development

No branches or pull requests

4 participants