[fnserver] fn invoke myapp hello on Windows: Error invoking function. status: 502 message: Container failed to initialize

[ceefour]

Description

fn invoke myapp hello failed with:

Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk and check the logs

Steps to reproduce the issue:

D:\project_amanah\fn-tutorial>fn init --runtime node hello
Creating function at: ./hello
Function boilerplate generated.
func.yaml created.

D:\project_amanah\fn-tutorial>cd hello

D:\project_amanah\fn-tutorial\hello>fn create app myapp
Successfully created app:  myapp

D:\project_amanah\fn-tutorial\hello>fn deploy --app myapp --local
Deploying hello to app: myapp
Bumped to version 0.0.2
Building image hello:0.0.2 ......................................
Updating function hello using image hello:0.0.2...
Successfully created function: hello with hello:0.0.2

D:\project_amanah\fn-tutorial\hello>fn invoke myapp hello
Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk and check the logs

Describe the results you received:

Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk and check the logs

Logs:

INFO[2020-04-01T04:22:13Z] Fn serving on `:8080`                         type=full version=0.3.749INFO[2020-04-01T04:22:24Z] hot function terminated                       app_id=01E4SZ3B07NG8G00GZJ0000001 container_id=01E4SZKNSRNG8G00GZJ0000002 cpus= error="container exit code 3" fn_id=01E4SZ4R2VNG8G00GZJ0000002 idle_timeout=30 image="hello:0.0.2" memory=128
INFO[2020-04-01T04:22:24Z] hot function failure                          app_id=01E4SZ3B07NG8G00GZJ0000001 container_id=01E4SZKNSRNG8G00GZJ0000002 cpus= error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4SZ4R2VNG8G00GZJ0000002 idle_timeout=30 image="hello:0.0.2" memory=128
INFO[2020-04-01T04:22:24Z] container wait error, sending error to client  action="server.handleFnInvokeCall)-fm" app_id=01E4SZ3B07NG8G00GZJ0000001 call_id=01E4SZKNSQNG8G00GZJ0000001 error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4SZ4R2VNG8G00GZJ0000002
ERRO[2020-04-01T04:22:24Z] api error                                     action="server.handleFnInvokeCall)-fm" code=502 error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4SZ4R2VNG8G00GZJ0000002

Describe the results you expected:

hello function executed.

Additional information you deem important (e.g. issue happens only occasionally):

Always reproducible. Also reproducible with go:

D:\project_amanah\fn-tutorial>fn init --runtime go hello-go
Creating function at: ./hello-go
Function boilerplate generated.
func.yaml created.

D:\project_amanah\fn-tutorial>cd hello-go

D:\project_amanah\fn-tutorial\hello-go>fn deploy --app myapp --local
Deploying hello-go to app: myapp
Bumped to version 0.0.2
Building image hello-go:0.0.2 .............................................................................................................................
Updating function hello-go using image hello-go:0.0.2...
Successfully created function: hello-go with hello-go:0.0.2

D:\project_amanah\fn-tutorial\hello-go>fn invoke myapp hello-go
Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk and check the logs

Logs:

INFO[2020-04-01T04:29:44Z] hot function terminated                       app_id=01E4SZ3B07NG8G00GZJ0000001 container_id=01E4T013ESNG8G00GZJ0000005 cpus= error="container exit code 1" fn_id=01E4T00X9CNG8G00GZJ0000003 idle_timeout=30 image="hello-go:0.0.2" memory=128
INFO[2020-04-01T04:29:44Z] hot function failure                          app_id=01E4SZ3B07NG8G00GZJ0000001 container_id=01E4T013ESNG8G00GZJ0000005 cpus= error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4T00X9CNG8G00GZJ0000003 idle_timeout=30 image="hello-go:0.0.2" memory=128
INFO[2020-04-01T04:29:44Z] container wait error, sending error to client  action="server.handleFnInvokeCall)-fm" app_id=01E4SZ3B07NG8G00GZJ0000001 call_id=01E4T013ESNG8G00GZJ0000004 error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4T00X9CNG8G00GZJ0000003
ERRO[2020-04-01T04:29:44Z] api error                                     action="server.handleFnInvokeCall)-fm" code=502 error="Container failed to initialize, please ensure you are using the latest fdk and check the logs" fn_id=01E4T00X9CNG8G00GZJ0000003

Output of fn version (CLI command):

Client version is latest version: 0.5.96
Server version:  0.3.749

Additional environment details (OSX, Linux, flags, etc.):

Windows 10 Pro.

[geroe1]

Hi, getting the same error, when doing the tutorial https://fnproject.io/tutorials/JavaFDKIntroduction/
Is there an easy explanation, fix or workaround?
Thanks alot,
Georg

[geroe1]

I found this log output in the server:

time="2022-08-12T09:58:19Z" level=debug msg="Exception in thread "main" com.fnproject.fn.runtime.exception.FunctionInitializationException: Unable to bind to unix socket in /tmp/iofs/lsnr.sock\n" app_id=01G7C30SYVNG8G00GZJ0000001 container_id=01GA8QV18RNG8G00GZJ000000T fn_id=01G7C753WANG8G00GZJ0000002 image="fndemo/javafn:0.0.11" tag=stderr
time="2022-08-12T09:58:19Z" level=debug msg="\tat com.fnproject.fn.runtime.HTTPStreamCodec.(HTTPStreamCodec.java:174)\n" app_id=01G7C30SYVNG8G00GZJ0000001 container_id=01GA8QV18RNG8G00GZJ000000T fn_id=01G7C753WANG8G00GZJ0000002 image="fndemo/javafn:0.0.11" tag=stderr
time="2022-08-12T09:58:19Z" level=debug msg="\tat com.fnproject.fn.runtime.EntryPoint.main(EntryPoint.java:69)\n" app_id=01G7C30SYVNG8G00GZJ0000001 container_id=01GA8QV18RNG8G00GZJ000000T fn_id=01G7C753WANG8G00GZJ0000002 image="fndemo/javafn:0.0.11" tag=stderr
time="2022-08-12T09:58:19Z" level=debug msg="Caused by: com.fnproject.fn.runtime.ntv.UnixSocketException: Error in bind:Operation not permitted\n" app_id=01G7C30SYVNG8G00GZJ0000001 container_id=01GA8QV18RNG8G00GZJ000000T fn_id=01G7C753WANG8G00GZJ0000002 image="fndemo/javafn:0.0.11" tag=stderr

Obviously, docker (colima) was not started with root user.

So, the fdk version was not the problem...

[pcross616]

I am getting the same issue on MacOS when using colima or rancher desktop.

the invoked function that is deployed when the fn container spins up doesn't seem to have a valid iofs on the host machine.

I tested with Docker Desktop and this works as expected.

[yufeifly]

您好，您的邮件已收到，谢谢。——熊宇飞

[suneelkandali]

I am also having same issue, any solution to it?

[subprotocol]

Our group recently started experiencing this issue when switching from Docker Desktop to Rancher Desktop. The issue we see applies to all FDK languages. We have included a workaround below.

Specifically we see in our logs:

Unable to connect to unix socket /tmp/iofs/lsnr.sock.tmp Error: listen EPERM: operation not permitted /tmp/iofs/lsnr.sock.tmp
Container failed to initialize, please ensure you are using the latest fdk and check the logs

Analysis if using Rancher Desktop

fn start will run a docker command that binds to volumes inside a users home directory. Binding volumes inside a users home directory works fine in Docker Desktop, but in Rancher Desktop volumes are mounted with the nodev option. This option prevents unix domain sockets from working—something all the FDKs require!

For example, if using the node-fdk with debug logging turned on—you'll see this issue cause an EPERM error here: (https://github.com/fnproject/fdk-node/blob/master/fn-fdk.js#L336).

Workaround for Rancher Desktop:

Instead of using fn start, try the command below. It does the same thing but binds volumes under /tmp instead of your home directory. This got us unblocked.

docker run --rm -i --name fnserver \
-v /tmp/iofs:/iofs \
-e FN_IOFS_DOCKER_PATH=/tmp/iofs \
-e FN_IOFS_PATH=/iofs \
-v /tmp/data:/app/data \
-v /var/run/docker.sock:/var/run/docker.sock \
--privileged \
-p 8080:8080 \
--entrypoint ./fnserver \
-e FN_LOG_LEVEL=DEBUG fnproject/fnserver:latest

@ceefour @geroe1 @pcross616 @suneelkandali - Are any of you using Rancher Desktop by chance? If so, does this workaround resolve your issue?

Thanks

[pcross616]

Yup, can confirm the issue is nodev on the container mount volume for iofs. I was able to change it to anything not in my home dir path. I feel like it's a security policy issue with rancher desktop when mounting in the user home. The above docker command works great.

[suneelkandali]

The issue was resolved for me after applying above Docker command, ty

[ongnxco]

I have the same issue on my Mac M1 with Docker and a fresh installation of FN. I am following the Node.js tutorial, and when I reach the part "fn invoke nodeapp nodefn," I encounter the error mentioned above. Here is the command to start FN with debug log level:

fn start --log-level DEBUG

¡¡¡ 'fn start' should NOT be used for PRODUCTION !!! see https://github.com/fnproject/fn-helm/
docker: Error response from daemon: Conflict. The container name "/fnserver" is already in use by container "d7ac23b0f59ed43134dd4e09415bcb0ff261cb23127af791b068474d0b6dd965". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

[Verun]

I'm seeing this exact issue now on my Mac M1 (Ventura 13.2.1).

Docker Desktop version: 4.22.1
FnProject version: 0.6.26

Any workarounds?

[mariobustosjmz]

Hi All,
About :
Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk and check the logs

I've faced this problem using my Mac, after 2 hours spend and trying to understand all error logs , I found a config in my docker settings...
so , I've updated file sharing implementation of containers from VirtioFs (default) to osxfs legacy

after what then I'm able to invoke my function correctly.

[funtom-io]

Error invoking function. status: 502 message: Container failed to initialize, please ensure you are using the latest fdk an

Thank you, @mariobustosjmz, for this workaround. It solved the issue for me.
I'd like to know whether there will be a fix for that issue soon since I did not find a way to change the sharing implementation only for a group of containers.

[yufeifly]

您好，您的邮件已收到，谢谢。——熊宇飞

[jean-simon-barry1]

@mariobustosjmz You're my hero. I'm really curious how you pinpointed that setting as the problem. In any event thanks that seems to have done the trick (for now)

In my case the tutorial from the docs would throw

time="2024-09-04T15:01:33Z" level=debug msg="  File \"/python/bin/fdk\", line 8, in <module>\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="    sys.exit(main())\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="             ^^^^^^\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="  File \"/python/fdk/scripts/fdk.py\", line 38, in main\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="    fdk.handle(handler)\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="  File \"/python/fdk/__init__.py\", line 125, in handle\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="    start(handle_code, lsnr, loop=loop)\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="  File \"/python/fdk/__init__.py\", line 87, in start\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="    raise ex\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="  File \"/python/fdk/__init__.py\", line 73, in start\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr
time="2024-09-04T15:01:33Z" level=debug msg="    os.chmod(phony_socket_path, 0o666)\n" app_id=01J6YPJJKYNG8G00GZJ0000003 container_id=01J6YRW74YNG8G00GZJ0000002 fn_id=01J6YPNXBBNG8G00GZJ0000004 image="fndemouser/pythonfn:0.0.3" tag=stderr

If this can help anyone in the future.

[yufeifly]

您好，您的邮件已收到，谢谢。——熊宇飞

[boyzoid]

Our group recently started experiencing this issue when switching from Docker Desktop to Rancher Desktop. The issue we see applies to all FDK languages. We have included a workaround below.

Specifically we see in our logs:

Unable to connect to unix socket /tmp/iofs/lsnr.sock.tmp Error: listen EPERM: operation not permitted /tmp/iofs/lsnr.sock.tmp
Container failed to initialize, please ensure you are using the latest fdk and check the logs

Analysis if using Rancher Desktop

fn start will run a docker command that binds to volumes inside a users home directory. Binding volumes inside a users home directory works fine in Docker Desktop, but in Rancher Desktop volumes are mounted with the nodev option. This option prevents unix domain sockets from working—something all the FDKs require!

For example, if using the node-fdk with debug logging turned on—you'll see this issue cause an EPERM error here: (https://github.com/fnproject/fdk-node/blob/master/fn-fdk.js#L336).

Workaround for Rancher Desktop:

Instead of using fn start, try the command below. It does the same thing but binds volumes under /tmp instead of your home directory. This got us unblocked.

docker run --rm -i --name fnserver \
-v /tmp/iofs:/iofs \
-e FN_IOFS_DOCKER_PATH=/tmp/iofs \
-e FN_IOFS_PATH=/iofs \
-v /tmp/data:/app/data \
-v /var/run/docker.sock:/var/run/docker.sock \
--privileged \
-p 8080:8080 \
--entrypoint ./fnserver \
-e FN_LOG_LEVEL=DEBUG fnproject/fnserver:latest

@ceefour @geroe1 @pcross616 @suneelkandali - Are any of you using Rancher Desktop by chance? If so, does this workaround resolve your issue?

Thanks

I'm now getting a different message: Fn not cound.

[yufeifly]

您好，您的邮件已收到，谢谢。——熊宇飞