[v2.2.0.16 Docker ARM] - SHDR in Connector::reader: End of file: End of file

[MRIIOT]

I found something intermittent across installations, but I can reproduce it on my Fanuc simulator (used to work without exception in 2.2.0.3 as that was our last Docker build). I start up agent and then my adapter, I can see the entire SHDR stream on 7878, and it looks like agent sees it as well. But it looks like there is an error. What is the best way to troubleshoot this? I am using https://github.com/TrakHound/MTConnect.NET 5.4.1 as the SHDR adapter. @PatrickRitchie

no eol found, waiting for more characters

2023-12-14T23:11:54.140136Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4475343Z|f_sim_p1_Z_power|0
2023-12-14T23:11:54.140235Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 121 characters in incomming buffer
2023-12-14T23:11:54.140258Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4475547Z|f_sim_p1_Z_feed|0
2023-12-14T23:11:54.140362Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 73 characters in incomming buffer
2023-12-14T23:11:54.140385Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4475774Z|f_sim_p1_Z_state|STOPPED
2023-12-14T23:11:54.140482Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 18 characters in incomming buffer
2023-12-14T23:11:54.140514Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) no eol found, waiting for more characters
2023-12-14T23:11:54.140556Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 1024 characters in incomming buffer
2023-12-14T23:11:54.140576Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4475979Z|f_sim_p1_S_load|2
2023-12-14T23:11:54.140672Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 976 characters in incomming buffer
2023-12-14T23:11:54.140718Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4476168Z|f_sim_p1_S_temp|0
2023-12-14T23:11:54.140809Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 928 characters in incomming buffer
2023-12-14T23:11:54.140832Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4476399Z|f_sim_p1_S_power|0
2023-12-14T23:11:54.140928Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 879 characters in incomming buffer
2023-12-14T23:11:54.140961Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4476600Z|f_sim_p1_S_speed|300
2023-12-14T23:11:54.141057Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 828 characters in incomming buffer
2023-12-14T23:11:54.141081Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4476833Z|f_sim_p1_S_state|TRAVEL
2023-12-14T23:11:54.141197Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 774 characters in incomming buffer
2023-12-14T23:11:54.141221Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4477075Z|f_sim_p1_S_rotary|SPINDLE
2023-12-14T23:11:54.141316Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 718 characters in incomming buffer
2023-12-14T23:11:54.141348Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4477317Z|f_sim_availability|AVAILABLE
2023-12-14T23:11:54.141449Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 659 characters in incomming buffer
2023-12-14T23:11:54.141471Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4477516Z|f_sim_adapter_ip|127.0.0.1;172.18.0.4
2023-12-14T23:11:54.141598Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 591 characters in incomming buffer
2023-12-14T23:11:54.141632Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4477702Z|f_sim_adapter_port|7878
2023-12-14T23:11:54.141731Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 537 characters in incomming buffer
2023-12-14T23:11:54.141754Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4477907Z|f_sim_machine_ip|192.168.111.12
2023-12-14T23:11:54.141851Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 475 characters in incomming buffer
2023-12-14T23:11:54.141874Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:10:40.4478128Z|f_sim_machine_port|8193
2023-12-14T23:11:54.141967Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 421 characters in incomming buffer
2023-12-14T23:11:54.142000Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:19.7572698Z|f_sim_p1_tmr_powered|48550620
2023-12-14T23:11:54.142118Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 362 characters in incomming buffer
2023-12-14T23:11:54.142141Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alarm||UNAVAILABLE
2023-12-14T23:11:54.142232Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 305 characters in incomming buffer
2023-12-14T23:11:54.142254Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_message||UNAVAILABLE
2023-12-14T23:11:54.142338Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 246 characters in incomming buffer
2023-12-14T23:11:54.142370Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_line||O0001(ROUGH)(TOOL=D12FEM)
2023-12-14T23:11:54.142466Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 176 characters in incomming buffer
2023-12-14T23:11:54.142490Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_comms|NORMAL||||
2023-12-14T23:11:54.142609Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 117 characters in incomming buffer
2023-12-14T23:11:54.142632Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_logic|NORMAL||||
2023-12-14T23:11:54.142738Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 58 characters in incomming buffer
2023-12-14T23:11:54.142760Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) no eol found, waiting for more characters
2023-12-14T23:11:54.142803Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 1024 characters in incomming buffer
2023-12-14T23:11:54.142824Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_motion|NORMAL||||
2023-12-14T23:11:54.142935Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 964 characters in incomming buffer
2023-12-14T23:11:54.142961Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_system|NORMAL||||
2023-12-14T23:11:54.143066Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 904 characters in incomming buffer
2023-12-14T23:11:54.143099Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_axes|NORMAL||||
2023-12-14T23:11:54.143207Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 846 characters in incomming buffer
2023-12-14T23:11:54.143242Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_spindles|NORMAL||||
2023-12-14T23:11:54.143363Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 784 characters in incomming buffer
2023-12-14T23:11:54.143398Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_alm_all|NORMAL||||
2023-12-14T23:11:54.143500Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 727 characters in incomming buffer
2023-12-14T23:11:54.143522Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_X_alm_heat|NORMAL||||
2023-12-14T23:11:54.143631Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 667 characters in incomming buffer
2023-12-14T23:11:54.143664Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_X_alm_travel|NORMAL||||
2023-12-14T23:11:54.143764Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 605 characters in incomming buffer
2023-12-14T23:11:54.143787Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_X_alm_servo|NORMAL||||
2023-12-14T23:11:54.143882Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 544 characters in incomming buffer
2023-12-14T23:11:54.143915Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Y_alm_heat|NORMAL||||
2023-12-14T23:11:54.144068Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 484 characters in incomming buffer
2023-12-14T23:11:54.144103Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Y_alm_travel|NORMAL||||
2023-12-14T23:11:54.144228Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 422 characters in incomming buffer
2023-12-14T23:11:54.144252Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Y_alm_servo|NORMAL||||
2023-12-14T23:11:54.144375Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 361 characters in incomming buffer
2023-12-14T23:11:54.144428Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Z_alm_heat|NORMAL||||
2023-12-14T23:11:54.144567Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 301 characters in incomming buffer
2023-12-14T23:11:54.144601Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Z_alm_travel|NORMAL||||
2023-12-14T23:11:54.144725Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 239 characters in incomming buffer
2023-12-14T23:11:54.144749Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_Z_alm_servo|NORMAL||||
2023-12-14T23:11:54.144863Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 178 characters in incomming buffer
2023-12-14T23:11:54.144897Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_S_alm_lnk|NORMAL||||
2023-12-14T23:11:54.144999Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 119 characters in incomming buffer
2023-12-14T23:11:54.145022Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_p1_S_alm_amp|NORMAL||||
2023-12-14T23:11:54.145120Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 60 characters in incomming buffer
2023-12-14T23:11:54.145153Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) no eol found, waiting for more characters
2023-12-14T23:11:54.145241Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 76 characters in incomming buffer
2023-12-14T23:11:54.145264Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: 2023-12-14T23:11:54.1287719Z|f_sim_adapter_health|WARNING||||
2023-12-14T23:11:54.145375Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 14 characters in incomming buffer
2023-12-14T23:11:54.145408Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: * PONG 10000
2023-12-14T23:11:54.145427Z (0x0000007f98734aa8) [debug] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (Port:45192) Received a PONG for fanuc_driver on port 7878
2023-12-14T23:11:54.145452Z (0x0000007f98734aa8) [debug] Connector::reader->Connector::parseSocketBuffer->Connector::processLine->Connector::startHeartbeats: (Port:45192)Received PONG, starting heartbeats every 10000ms
2023-12-14T23:11:54.982404Z (0x0000007f98734aa8) [error] Connector::reader: End of file: End of file
2023-12-14T23:11:54.982541Z (0x0000007f98734aa8) [error] Connector::reader->Connector::reconnect->Connector::close: Closing fanuc_driver:7878 (Local Port:45192)
2023-12-14T23:11:54.982873Z (0x0000007f98734aa8) [debug] Connector::reader->Connector::reconnect->Connector::close: Disconnected from adapter, setting all values to UNAVAILABLE
2023-12-14T23:11:54.986688Z (0x0000007f98734aa8) [info] Connector::reader->Connector::reconnect: reconnect: retry connection in 10000ms
2023-12-14T23:12:04.127434Z (0x0000007f98734aa8) [debug] AgentConfiguration::monitorThread: Monitoring files: "/mtconnect/config/agent.cfg" and /mtconnect/config/devices.xml, will warm start if they change.
2023-12-14T23:12:04.127623Z (0x0000007f98734aa8) [info] pipeline.deliver: _fanuc_driver_7878_asset_update_rate - Average for last 1 minutes: 0
2023-12-14T23:12:04.127662Z (0x0000007f98734aa8) [info] pipeline.deliver: _fanuc_driver_7878_asset_update_rate - Delta for last 10 seconds: 0
2023-12-14T23:12:04.127684Z (0x0000007f98734aa8) [info] pipeline.deliver: _fanuc_driver_7878_observation_update_rate - Average for last 1 minutes: 1.32023
2023-12-14T23:12:04.127699Z (0x0000007f98734aa8) [info] pipeline.deliver: _fanuc_driver_7878_observation_update_rate - Delta for last 10
seconds: 8.5982
2023-12-14T23:12:04.987938Z (0x0000007f98734aa8) [info] : reconnect: retrying connection
2023-12-14T23:12:04.988274Z (0x0000007f98734aa8) [debug] Connector::connect: Connecting to data source: fanuc_driver on port: 7878
2023-12-14T23:12:04.989017Z (0x0000007f98734aa8) [info] Connector::connected: Connected with: 172.18.0.4:7878
2023-12-14T23:12:04.989293Z (0x0000007f98734aa8) [debug] Connector::connected->Connector::sendCommand: (Port:45202) Sending PING
2023-12-14T23:12:05.084268Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer: (fanuc_driver:7878) 14 characters in incomming buffer
2023-12-14T23:12:05.084410Z (0x0000007f98734aa8) [trace] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (fanuc_driver:7878) Received line: * PONG 10000
2023-12-14T23:12:05.084448Z (0x0000007f98734aa8) [debug] Connector::reader->Connector::parseSocketBuffer->Connector::processLine: (Port:45202) Received a PONG for fanuc_driver on port 7878
2023-12-14T23:12:05.084484Z (0x0000007f98734aa8) [debug] Connector::reader->Connector::parseSocketBuffer->Connector::processLine->Connector::startHeartbeats: (Port:45202)Received PONG, starting heartbeats every 10000ms

[MRIIOT]

The other strange observation:

1. Start up adapter.
2. Connect with putty to 7878 right after startup. I see all SHDR in terminal.
3. All data also goes cleanly into agent.

[wsobel]

The agent is getting an end of file that means the other side has closed the connection. The agent would have logged if it had initiated the shutdown.

It occurs right after the first heartbeat, so I wonder if something went wrong on the adapter side.

[MRIIOT]

Hmmm could be something with the ping pong. I will check.

[MRIIOT]

Definitely something on adapter side and very Docker and ARM specific.

TrakHound/MTConnect.NET#53

[MRIIOT]

Actually previous comment is unrelated to this issue, which indeed is a socket close by adapter.

But I can't reproduce it.... yet

[MRIIOT]

This is a non-issue. It was related to the restart sequence of agent and adapter.