azurefile-controller does not work on Ubuntu 20.04 #495

emiliodangelo · 2020-12-11T13:05:59Z

What happened:
After installing azurefile-csi-driver in a Kubernetes cluster, csi-attacher and csi-resizer containers (inside *csi-azurefile-controller pod) are crashing every 1 or 2 minutes with the following message:

csi-attacher log:

...
I1210 14:36:58.833263       1 reflector.go:153] Starting reflector *v1beta1.CSINode (10m0s) from k8s.io/client-go/informers/factory.go:135
I1210 14:36:58.833363       1 reflector.go:188] Listing and watching *v1beta1.CSINode from k8s.io/client-go/informers/factory.go:135
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0x15dc7fc, 0xc)
	/usr/lib/go-1.14/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc000502a80)
	/usr/lib/go-1.14/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000500380)
	/usr/lib/go-1.14/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000500380)
	/usr/lib/go-1.14/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc00006b000, 0x167d548, 0x22b8718)
	/usr/lib/go-1.14/src/runtime/proc.go:1390 +0x14e
runtime.newm(0x167d548, 0xc00006b000)
	/usr/lib/go-1.14/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc0004ca401)
	/usr/lib/go-1.14/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
	/usr/lib/go-1.14/src/runtime/proc.go:1953
runtime.resetspinning()
	/usr/lib/go-1.14/src/runtime/proc.go:2415 +0x93
runtime.schedule()
	/usr/lib/go-1.14/src/runtime/proc.go:2527 +0x2de
runtime.mstart1()
	/usr/lib/go-1.14/src/runtime/proc.go:1104 +0x8e
runtime.mstart()
	/usr/lib/go-1.14/src/runtime/proc.go:1062 +0x6e

goroutine 1 [select]:
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1.1(0x1380640, 0x0, 0xc000628180)
...

csi-resizer log:

...
I1211 12:27:26.339777       1 leaderelection.go:283] successfully renewed lease kube-system/external-resizer-file-csi-azure-com
I1211 12:27:31.349381       1 leaderelection.go:283] successfully renewed lease kube-system/external-resizer-file-csi-azure-com
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0x15d27f3, 0xc)
	/usr/lib/go-1.14/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc000682a80)
	/usr/lib/go-1.14/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000079180)
	/usr/lib/go-1.14/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000079180)
	/usr/lib/go-1.14/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc00004f800, 0x1672e98, 0x14f676dd7e26c)
	/usr/lib/go-1.14/src/runtime/proc.go:1390 +0x14e
runtime.newm(0x1672e98, 0xc00004f800)
	/usr/lib/go-1.14/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc000103201)
	/usr/lib/go-1.14/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
	/usr/lib/go-1.14/src/runtime/proc.go:1953
runtime.resetspinning()
	/usr/lib/go-1.14/src/runtime/proc.go:2415 +0x93
runtime.schedule()
	/usr/lib/go-1.14/src/runtime/proc.go:2527 +0x2de
runtime.park_m(0xc000103200)
	/usr/lib/go-1.14/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x0)
	/usr/lib/go-1.14/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [select, 2 minutes]:
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1.1(0x13763e0, 0x0, 0xc0000eee40)
...

The same behavior can be observed in csi-azuredisk-controller pod.

What you expected to happen:
The containers should not fail so frequently.

How to reproduce it:
The failure started right after installing v0.7.0 of azurefile-csi-driver. I upgraded to v0.9.0 (for both, azurefile and azuredisk) with the same results. The Kubernetes cluster is composed of 3 master nodes and 3 workers running on Azure VMs (not AKS).

Anything else we need to know?:
Found a couple issues in golang/go repository that seems to be related:

runtime: mlock of signal stack failed: 12 [1.14 backport] #37807 (runtime: mlock of signal stack failed: 12 [1.14 backport] golang/go#37807)
runtime: mlock of signal stack failed: 12 (runtime: mlock of signal stack failed: 12 golang/go#37436)

Possibly upgrading golang version from 1.14 to 1.15 will solve the problem.

Environment:

CSI Driver version: v0.7.0 and v0.9.0
Kubernetes version (use kubectl version): v1.19.14
OS (e.g. from /etc/os-release): Ubuntu v20.04.1 LTS
Kernel (e.g. uname -a): 5.4.0-1032-azure [feature] Create Helm chart for driver #33-Ubuntu SMP Fri Nov 13 14:23:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools: Helm v3.4.2
Others:
- Master node size: Standard D2ds_v4 (2 vcpus, 8 GiB memory)
- Worker node size: Standard D16ds_v4 (16 vcpus, 64 GiB memory)

Complete log files:
csi-attacher.log
csi-resizer.log

The text was updated successfully, but these errors were encountered:

andyzhangx · 2020-12-11T13:09:17Z

it looks like csi-attacher and csi-resizer container cannot run on Ubuntu v20.04.1

emiliodangelo · 2020-12-11T13:09:27Z

Should I open another issue for azuredisk-csi-driver?

andyzhangx · 2020-12-11T13:11:09Z

I think you could open an issue in https://github.com/kubernetes-csi/external-attacher and https://github.com/kubernetes-csi/external-resizer

andyzhangx · 2020-12-11T13:13:04Z

Ubuntu 18.04 should work now, and this driver is not tested on Ubuntu 20.04 now. Anyway, thanks for reporting

emiliodangelo · 2020-12-11T13:20:27Z

Didn't try yet on Ubuntu 18.04, but created a VM with that version and is using the same kernel: 5.4.0-1032-azure #33~18.04.1-Ubuntu SMP Tue Nov 17 11:40:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

andyzhangx · 2020-12-11T13:24:31Z

our e2e test is on Ubuntu 18.04.5 LTS 5.4.0-1032-azure, it should work otherwise e2e test won't pass:

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_azurefile-csi-driver/492/pull-azurefile-csi-driver-e2e/1336671736728391680/build-log.txt

2020/12/09 14:31:39 ===================azurefile log===================
print out all nodes status ...
NAME                        STATUS   ROLES    AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s-agentpool1-88346041-0   Ready    agent    26m   v1.16.15   10.240.0.4     <none>        Ubuntu 18.04.5 LTS   5.4.0-1032-azure   containerd://1.3.7+azure
k8s-agentpool1-88346041-1   Ready    agent    26m   v1.16.15   10.240.0.35    <none>        Ubuntu 18.04.5 LTS   5.4.0-1032-azure   containerd://1.3.7+azure
k8s-master-88346041-0       Ready    master   26m   v1.16.15   10.255.255.5   <none>        Ubuntu 18.04.5 LTS   5.4.0-1032-azure   containerd://1.3.7+azure

fejta-bot · 2021-03-11T14:19:05Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot · 2021-04-10T14:58:59Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

fejta-bot · 2021-05-10T15:14:57Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

k8s-ci-robot · 2021-05-10T15:15:07Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2021-06-10T02:47:41Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

k8s-ci-robot · 2021-06-10T02:47:45Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2021-07-10T03:47:56Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

k8s-ci-robot · 2021-07-10T03:48:00Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2021-10-08T04:46:50Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2021-11-07T05:25:35Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2021-12-07T05:35:55Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2021-12-07T05:36:08Z

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2022-01-06T05:56:27Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-01-06T05:56:38Z

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

andyzhangx changed the title ~~azurefile-controller pod crashing~~ azurefile-controller does not work on Ubuntu 20.04 Dec 11, 2020

andyzhangx added kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 11, 2020

This was referenced Dec 11, 2020

external-attacher crashing in azuredisk-csi-driver and azurefile-csi-driver kubernetes-csi/external-attacher#284

Closed

external-resizer crashing in azuredisk-csi-driver and azurefile-csi-driver kubernetes-csi/external-resizer#130

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 11, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 10, 2021

k8s-ci-robot closed this as completed May 10, 2021

andyzhangx reopened this May 11, 2021

k8s-ci-robot closed this as completed Jun 10, 2021

andyzhangx reopened this Jun 10, 2021

k8s-ci-robot closed this as completed Jul 10, 2021

andyzhangx reopened this Jul 10, 2021

andyzhangx removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 10, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 8, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 7, 2021

k8s-ci-robot closed this as completed Dec 7, 2021

andyzhangx reopened this Dec 7, 2021

k8s-ci-robot closed this as completed Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

azurefile-controller does not work on Ubuntu 20.04 #495

azurefile-controller does not work on Ubuntu 20.04 #495

emiliodangelo commented Dec 11, 2020 •

edited

Loading

andyzhangx commented Dec 11, 2020

emiliodangelo commented Dec 11, 2020

andyzhangx commented Dec 11, 2020

andyzhangx commented Dec 11, 2020 •

edited

Loading

emiliodangelo commented Dec 11, 2020

andyzhangx commented Dec 11, 2020

fejta-bot commented Mar 11, 2021

fejta-bot commented Apr 10, 2021

fejta-bot commented May 10, 2021

k8s-ci-robot commented May 10, 2021

fejta-bot commented Jun 10, 2021

k8s-ci-robot commented Jun 10, 2021

fejta-bot commented Jul 10, 2021

k8s-ci-robot commented Jul 10, 2021

k8s-triage-robot commented Oct 8, 2021

k8s-triage-robot commented Nov 7, 2021

k8s-triage-robot commented Dec 7, 2021

k8s-ci-robot commented Dec 7, 2021

k8s-triage-robot commented Jan 6, 2022

k8s-ci-robot commented Jan 6, 2022

azurefile-controller does not work on Ubuntu 20.04 #495

azurefile-controller does not work on Ubuntu 20.04 #495

Comments

emiliodangelo commented Dec 11, 2020 • edited Loading

andyzhangx commented Dec 11, 2020

emiliodangelo commented Dec 11, 2020

andyzhangx commented Dec 11, 2020

andyzhangx commented Dec 11, 2020 • edited Loading

emiliodangelo commented Dec 11, 2020

andyzhangx commented Dec 11, 2020

fejta-bot commented Mar 11, 2021

fejta-bot commented Apr 10, 2021

fejta-bot commented May 10, 2021

k8s-ci-robot commented May 10, 2021

fejta-bot commented Jun 10, 2021

k8s-ci-robot commented Jun 10, 2021

fejta-bot commented Jul 10, 2021

k8s-ci-robot commented Jul 10, 2021

k8s-triage-robot commented Oct 8, 2021

k8s-triage-robot commented Nov 7, 2021

k8s-triage-robot commented Dec 7, 2021

k8s-ci-robot commented Dec 7, 2021

k8s-triage-robot commented Jan 6, 2022

k8s-ci-robot commented Jan 6, 2022

emiliodangelo commented Dec 11, 2020 •

edited

Loading

andyzhangx commented Dec 11, 2020 •

edited

Loading