Use single CPU for building site preview #41142
Conversation
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
size/XS
Based on some experiments, I am suspecting that there are some concurrency issues when building the site for preview. In this PR, I'm proposing we limit the CPUs to 1 (which is actually what Netlify provides us today). The site preview build should normally complete within 4 minutes. The chance of a successful build is much higher than the default setting.
✅ Pull request preview available for checking
To edit notification comments on pull requests, go to your Netlify site settings. |
| @@ -40,7 +40,7 @@ build-preview: module-check ## Build site with drafts and future posts enabled | |||
| hugo --cleanDestinationDir --buildDrafts --buildFuture --environment preview | |||
|
|
|||
| deploy-preview: ## Deploy preview site via netlify | |||
| hugo --cleanDestinationDir --enableGitInfo --buildFuture --environment preview -b $(DEPLOY_PRIME_URL) | |||
| GOMAXPROCS=1 hugo --cleanDestinationDir --enableGitInfo --buildFuture --environment preview -b $(DEPLOY_PRIME_URL) | |||
There was a problem hiding this comment.
Should we apply this limit to main branch builds as well?
There was a problem hiding this comment.
| GOMAXPROCS=1 hugo --cleanDestinationDir --enableGitInfo --buildFuture --environment preview -b $(DEPLOY_PRIME_URL) | |
| GOMAXPROCS=2 hugo --cleanDestinationDir --enableGitInfo --buildFuture --environment preview -b $(DEPLOY_PRIME_URL) |
could this work OK too?
There was a problem hiding this comment.
Not sure. The container we got from Netlify is 1C1G. I don't think relaxing this to 2 processors will buy us anything. I'm more inclined to start with 1 processor so that any concurrency bugs can be ruled out. Later on, if this proved to be working as expected, we can try relax the constraint.
There was a problem hiding this comment.
Should we apply this limit to main branch builds as well?
I'm more concerned about the netlify preview at this stage. Surely we can apply this to main branch if a similar timeout issue is there.
There was a problem hiding this comment.
Ah, OK. I am less worried about the previews and more worried about the main branch builds (which produce the live site).
Anyway, if this change is helpful, I support it. Do we know why it helps?
There was a problem hiding this comment.
golang/go#33803 again 🙃
we should consider PR-ing hugo to have https://github.com/uber-go/automaxprocs in the meantime
There was a problem hiding this comment.
With throttling to 1 CPU, should setting GOMAXPROCS to 32 explicitly lead to build times going from 4 to over 60 minutes? We get timeouts when Hugo uses the automatic setting which we assume is 32.
I can picture how that might happen, but those scenarios feel more like bugs in something (maybe an old kernel?)
There was a problem hiding this comment.
If netlify is hosting on a 32 core host but running in a container throttled to 1 core then Go will attempt to use 32 cores and will quickly exhaust the process's quota leading to aggressive throttling in the kernel. It's hard to say exactly what slowdown will result but it can be large.
If you control the execution environment it's often recommended to only set CPU requests (CFS shares -- proportional time ~guaranteed to schedule) and not limits (CFS quota -- time bound above which process is throttled) unless you're benchmarking. Linux cfs quota throttling is known to be heavy handed.
I took a quick peek through hugo yesterday and AFAICT it's using go's default runtime.GOMAXPROCS() which would match the number of cores on the underlying host (roughly speaking) rather than the cfs quotas (see linked go issue as wel).
I'm not sure where best to insert it into hugo, I'd usually say in main() but they have a very tiny main, I'd guess this affects many other projects and hugo would generally benefit from defaulting GOMAXPROCS in a CFS aware way until go standard runtime solves this (as java did already).
There was a problem hiding this comment.
filing an upstream issue / PR
There was a problem hiding this comment.
|
FWIW it looks like the deploy preview took 63.6s to build using Hugo. |
|
Thanks! /approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mimani68, sftim The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 1a240bc13e0d454eaf6a2d1f8dcab75cfc25d9a9
|
|
This has really helped! |
|
The next hugo release should be CPU quota aware 🤞 gohugoio/hugo#10952 |
|
PR #41301 copies this approach for live deploys. Once we adopt Hugo 0.112 we should be able to drop the workaround. |
Based on some experiments, I am suspecting that there are some concurrency issues when building the site for preview. In this PR, I'm proposing we limit the CPUs to 1 (which is actually what Netlify provides us today). The site preview build should normally complete within 4 minutes. The chance of a successful build is much higher than the default setting.