ShdrAdapter - sending latest values for all DataItems on new Agent connection

[MRIIOT]

Should either of these methods send the latest values of all DataItems when a new Agent comes along and connects?

MTConnect.NET/src/MTConnect.NET-SHDR/Adapters/Shdr/ShdrAdapter.cs

Lines 319 to 352 in 62709fa

	/// <summary>
	/// Sends all Items that have changed since last sent to the Agent
	/// </summary>
	public void SendChanged()
	{
	WriteChangedDataItems();
	WriteChangedMessages();
	WriteChangedConditions();
	WriteChangedTimeSeries();
	WriteChangedDataSets();
	WriteChangedTables();
	WriteChangedAssets();
	// Call Overridable Method
	OnChangedSent();
	}
	/// <summary>
	/// Sends all of the last sent Items, Assets, and Devices to the Agent. This can be used upon reconnection to the Agent
	/// </summary>
	public void SendLast(long timestamp = 0)
	{
	WriteLastDataItems(timestamp);
	WriteLastMessages(timestamp);
	WriteLastConditions(timestamp);
	WriteLastTimeSeries(timestamp);
	WriteLastDataSets(timestamp);
	WriteLastTables(timestamp);
	WriteAllAssets();
	WriteAllDevices();
	// Call Overridable Method
	OnLastSent();
	}

[PatrickRitchie]

Yes the SendLast() method is called when an Agent is connected. It sends all DataItems.

private void ClientConnected(string clientId, TcpClient client)
{
    AddAgentClient(clientId, client);
    AgentConnected?.Invoke(this, clientId);

    SendLast(UnixDateTime.Now);
}

[MRIIOT]

You're correct. At least from a Telnet session. Works both with ShdrAdapter.FilterDuplicates true or false.

2023-12-16T04:15:26.9643773Z|f_sim_p1_hwver|Machining center Series 0i D4F1 MODEL D 30.0
2023-12-16T04:15:26.9889871Z|f_sim_p1_axes|X Y Z S
2023-12-16T04:15:26.9890389Z|f_sim_p1_tmr_powered|48654840
2023-12-16T04:15:26.9890500Z|f_sim_p1_tmr_operating|5640180
2023-12-16T04:15:26.9890618Z|f_sim_p1_tmr_loaded|1880040
2023-12-16T04:15:26.9890700Z|f_sim_p1_ctl_mode|AUTOMATIC
2023-12-16T04:15:26.9890785Z|f_sim_p1_ctl_exec|READY
2023-12-16T04:15:26.9890864Z|f_sim_p1_tool_num|5
2023-12-16T04:15:26.9890951Z|f_sim_p1_ovr_feed|50
2023-12-16T04:15:26.9891031Z|f_sim_p1_ovr_rapid|0
2023-12-16T04:15:26.9891114Z|f_sim_p1_ovr_spindle|120
2023-12-16T04:15:26.9891194Z|f_sim_estop|ARMED
2023-12-16T04:15:26.9891273Z|f_sim_p1_tmr_cycle|420.944
2023-12-16T04:15:26.9891354Z|f_sim_p1_part_count_life|1081
2023-12-16T04:15:26.9891435Z|f_sim_p1_part_count_complete|231
2023-12-16T04:15:26.9891520Z|f_sim_p1_part_count_remain|0
2023-12-16T04:15:26.9891606Z|f_sim_p1_prg_name_selected|O1
2023-12-16T04:15:26.9891691Z|f_sim_p1_prg_cmt_selected|(ROUGH)
2023-12-16T04:15:26.9891779Z|f_sim_p1_prg_size_selected|26500
2023-12-16T04:15:26.9891859Z|f_sim_p1_prg_mod_selected|2021-05-10T08:07:00.0000000-05:00
2023-12-16T04:15:26.9891945Z|f_sim_p1_prg_name_current|O1
2023-12-16T04:15:26.9892025Z|f_sim_p1_prg_cmt_current|(ROUGH)
2023-12-16T04:15:26.9892106Z|f_sim_p1_prg_size_current|26500
2023-12-16T04:15:26.9892186Z|f_sim_p1_prg_mod_current|2021-05-10T08:07:00.0000000-05:00
2023-12-16T04:15:26.9892266Z|f_sim_availability|AVAILABLE
2023-12-16T04:15:26.9892356Z|f_sim_adapter_ip|192.168.100.4;192.168.56.1;192.168.150.237;127.0.0.1
2023-12-16T04:15:26.9892440Z|f_sim_adapter_port|7878
2023-12-16T04:15:26.9892519Z|f_sim_machine_ip|192.168.111.12
2023-12-16T04:15:26.9892602Z|f_sim_machine_port|8193
                                                    2023-12-16T04:15:43.0223649Z|f_sim_adapter_health|NORMAL||||
                                                                                                                * PONG 10000
                                                                                                                             * PONG 10000

[MRIIOT]

I am testing now with FilterDuplicates=true.

Docker on Linux x86 the behavior is correct and all DataItems are sent over.

But actually in Docker on ARM it behaves differently. Only the last DataItem that was updated is sent over when connecting with Telnet. All other observations are not sent.

2023-12-16T04:50:01.9328677Z|f_sim_p1_tmr_powered|48656940
                                                          * PONG 10000
                                                                       * PONG 10000

[PatrickRitchie]

I ran across the same issue using Docker a few days ago. If it is the same issue, the Apline image doesn't include all of the CultureInfo libraries in order to save space.

This article explains the issue (although the error was slightly different):
https://andrewlock.net/dotnet-core-docker-and-cultures-solving-culture-issues-porting-a-net-core-app-from-windows-to-linux/

The solution for me was to add the following lines to the Docker file:

RUN apk add --no-cache icu-libs
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

The Docker file I used for building an Agent is below with the two lines added:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
WORKDIR /app
COPY . ./
FROM mcr.microsoft.com/dotnet/runtime:8.0.0-alpine3.18-arm64v8
RUN apk add --no-cache icu-libs
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
WORKDIR /app
COPY --from=build-env /app .
EXPOSE 5000/tcp
EXPOSE 1883/tcp
EXPOSE 7878/tcp
ENTRYPOINT ["dotnet", "agent.dll"]
CMD ["debug"]'

I'm looking at a solution to not require this but this worked to resolve the issue for me. Let me know if you still have issues with it.

[MRIIOT]

What issue did it fix for you?

I tried it after the last stage, but no effect. I'm also building against Debian Bullseye Slim ARM32.

RUN apt-get update && apt-get install -y --no-install-recommends libicu67
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

[PatrickRitchie]

Sorry I didn't see you tagged me in the other issue. The issue I ran into was with creating an instance of a CultureInfo class when that culture info wasn't included in the Alpine image. Although I was using Docker for an Agent, the library that was causing the issue should have caused an issue on either Agent or Adapter since it was in the ShdrLine.cs class that both use.

I'm trying to see if I can get a similar issue on my end. So far it works on ARM64 but need to test on the ARM32 image you mentioned.

Are you using the ShdrAdapter class in a library? If so, you should be able to see any errors by subscribing to the SendError event.

private bool WriteLineToClient(AgentClient client, string line)
{
    if (client != null && !string.IsNullOrEmpty(line))
    {
        var lines = SplitLines(line);
        if (!lines.IsNullOrEmpty())
        {
            foreach (var singleLine in lines)
            {
                try
                {
                    // Convert string to ASCII bytes and add line terminator
                    var bytes = Encoding.ASCII.GetBytes(singleLine + "\n");

                    // Get the TcpClient Stream
                    var stream = client.TcpClient.GetStream();
                    stream.ReadTimeout = Timeout;
                    stream.WriteTimeout = Timeout;

                    // Write the line (in bytes) to the Stream
                    stream.Write(bytes, 0, bytes.Length);

                    LineSent?.Invoke(this, new AdapterEventArgs(client.Id, singleLine));
                }
                catch (Exception ex)
                {
                    SendError?.Invoke(this, new AdapterEventArgs(client.Id, ex.Message));
                    return false;
                }
            }

            return true;
        }
    }

    return false;
}

Let me know if you are available for me to login remotely to help as I might not be able to reproduce it on my end.

[MRIIOT]

@PatrickRitchie. I am using an instance of the ShdrAdapter class. SendError is never invoked. What I have observed is that when FilterDuplicates=false, then all the data comes over the wire just fine as it's fed into ShdrAdapter. When FilterDuplicates=true, then I am seeing that SendLast only sends a single observation, the one that has changed last, not all the observations that ShdrAdapter is holding.

[MRIIOT]

So after our discussion, it appears that ShdrQueueAdapter and FilterDuplicates=true might be fighting each other. Which makes sense to turn off FilterDuplicates when using ShdrQueueAdapter.