Intellichlor failing - RS485 message decoding

[CodeSlinger69]

I have an IC20 that has a manufacture date of January 2015. These have an expected lifetime of 5 years and/or 10,000 hours, so it very well might be failing. The symptom I am seeing is the that unit power cycles every few minutes. My first steps were to give the IC20 a vinegar bath to remove scale (its clean) and check my salt level (3600 ppm).

I thought I might use MessageManager to see if there is anything I can learn from the packets that are going back and forth between Nixie and the chlorinator. packetLog(2024-08-19_14-31-54).log

Until message #182 the IC20 is offline. This packet is a "Response to Get Status" (1) message and according to the wiki the first payload byte should be a "1" if "Status = OK". As you can see, my status byte is "0" indicating that things may not be OK.

For the next 2 min 37 sec Nixie and the IC20 exchange messages successfully. Of interest is the "Response to Set Salt %" (18) message. The first payload byte is 78, which is what the IC20 thinks the salt level is at. 78*50 = 3900, close to the value I measured by titration. Interestingly, the njsPC dashpanel shows 3200ppm. The second payload byte is 128, which is an error byte according to the wiki, and this value shouldn't exist. Perhaps this is a previously undocumented error value?

After message #631 the IC20 goes offline due to a power cycle.

So two questions:

1. Does the status byte 0 in the "Response to Get Status" message or the error byte 128 in the "Response to Set Salt %" message give me any additional information?

2. Why would the njsPC dashPanel not show the 3900ppm response from the chlorinator?

Replacement chlorinators cost $1200 these days, so anything I can do to stretch the life of my old unit is still appreciated.

[rstrouse]

I'll have a look at your log. The status byte on the current setting is a bitmask and I believe it is only 7 bits so it it is simply poop at the MSB of that byte. Essentially, this is OK. Double check to ensure you do not have the Ignore Salt Reading setting checked on the configuration because the salt reading should be set to 3900 on the display.

[CodeSlinger69]

Yes that was dumb of me. I was testing whether the salt level was impacting the IC20 and had "Ignore Salt Reading" enabled. The unit is still power cycling even with that option enabled, so obviously that wasn't the problem.

Unticking that option allowed the correct 3900ppm value to be displayed in dashPanel.

[rstrouse]

The most common issue with these are the flow switch which is an inexpensive thing to replace.

[CodeSlinger69]

I did replace the flow switch a few months ago, as the unit was not measuring salt levels effectively, suggesting that the thermistor in the flow switch was broken. It's been measuring flow and salt correctly since that replacement.

I'm guessing there is nothing to be gleaned from the RS485 message exchange then? I'm not surprised as it does not appear that Nixie is controlling the IC20 power state. Since I hacked my Nixie together from cannibalized pieces of an old EasyTouch, the power for the IC20 still comes from the board that was in that EasyTouch. The power cycling must be triggered either by that power board or electronics in the IC20 itself.

My next step will be to get out the multimeter and start probing the power board from the EasyTouch to see if I can learn anything. I'll report back.

[rstrouse]

Yeah the the transformer will draw a load from the surge board and if that isn't right it will not keep the constant current. I haven't had the misfortune to have to troubleshoot one as my last cell went 15 years.

[CodeSlinger69]

Well that was the wrong rabbit hole. I think that my original suspicions of a failing Intellichlor were wrong.

I went out to the OCP with my probe, intending on checking input and output voltages on the power/surge board, and heard relays clicking. Not the little ones on the RPi stack, but the big Omron ones that switch the 120/240 power. Since I only have one big relay in use, the one that controls the pump/chlorinator, and since the "Pool" circuit in njsPC was "on", this relay should not be switching. A probe confirmed that the RPi/njsPC was pulling the relay's coil to 0VDC, thus cutting off power to the Intellichlor.

I then captured logs to determine why njsPC was deciding to turn off that relay. I captured these logs multiple times and saw the same thing each time:

[8/20/2024, 11:26:11 AM] error: Error Sending REM Request: PUT /state/device/i2c:1:2:1 Request timeout after 3600ms: PUT /state/device/i2c:1:2:1
[8/20/2024, 11:26:11 AM] verbose: REM server request returned. PUT /state/device/i2c:1:2:1 {"isOn":true,"latch":10000}
[8/20/2024, 11:26:11 AM] debug: REM server request returned. PUT /state/device/i2c:1:2:1 {"isOn":true,"latch":10000} {"status":{"code":-1,"message":""},"error":{},"data":""}
...
[8/20/2024, 11:26:14 AM] verbose: REM server request initiated. PUT /state/device/i2c:1:2:1 {"isOn":true,"latch":10000}
[8/20/2024, 11:26:14 AM] error: Error sending Request: PUT /state/device/i2c:1:2:1 connect ENETUNREACH 192.168.1.3:8080 - Local (0.0.0.0:0)
[8/20/2024, 11:26:14 AM] error: Error Sending REM Request: PUT /state/device/i2c:1:2:1 Error sending Request: PUT /state/device/i2c:1:2:1 connect ENETUNREACH 192.168.1.3:8080 - Local (0.0.0.0:0)
[8/20/2024, 11:26:14 AM] verbose: REM server request returned. PUT /state/device/i2c:1:2:1 {"isOn":true,"latch":10000}
[8/20/2024, 11:26:14 AM] debug: REM server request returned. PUT /state/device/i2c:1:2:1 {"isOn":true,"latch":10000} {"status":{"code":-1,"message":""},"error":{},"data":""}

OK, so over to REM, enable Silly logging. Nothing unusual in the REM logs that I could see, but here's the log from around the same time as the errors above:

[20/08/2024, 11:26:13] debug: 39 - Sequent 8 Relay Hat Executed read command 0x03 byte read:0x00
[20/08/2024, 11:26:13] debug: 39 - Sequent 8 Relay Hat Executed read command 0x00 byte read:0x01
[20/08/2024, 11:26:13] debug: Persisting Configuration data... /home/paul/relayEquipmentManager/data/controllerConfig.json
[20/08/2024, 11:26:13] debug: Interpolating map thermistor10k input: 6022 {"low":{"input":5960,"value":99},"high":{"input":6238,"value":97}}
[20/08/2024, 11:26:13] verbose: Emitting: /temps : {"airSensor":97.55}
[20/08/2024, 11:26:13] verbose: Feed njspc/poolpi sending airSensor: 97.55 to temps
[20/08/2024, 11:26:13] verbose: Feed internal/Internal Devices sending adcValue: 6.022 to generic:5:1
[20/08/2024, 11:26:13] debug: Interpolating map thermistor10k input: 8102 {"low":{"input":7866,"value":87},"high":{"input":8251,"value":85}}
[20/08/2024, 11:26:13] verbose: Emitting: /temps : {"waterSensor1":85.77}
[20/08/2024, 11:26:13] verbose: Feed njspc/poolpi sending waterSensor1: 85.77 to temps
[20/08/2024, 11:26:13] verbose: Feed internal/Internal Devices sending adcValue: 8.102 to generic:5:2
[20/08/2024, 11:26:13] debug: Interpolating map thermistor10k input: 3949 {"low":{"input":3838,"value":119},"high":{"input":4006,"value":117}}
[20/08/2024, 11:26:13] verbose: Emitting: /temps : {"solarSensor1":117.68}
[20/08/2024, 11:26:13] verbose: Feed njspc/poolpi sending solarSensor1: 117.68 to temps
[20/08/2024, 11:26:13] verbose: Feed internal/Internal Devices sending adcValue: 3.949 to generic:5:3
[20/08/2024, 11:26:13] debug: 39 - Sequent 8 Relay Hat Executed read command 0x03 byte read:0x00
[20/08/2024, 11:26:13] debug: 39 - Sequent 8 Relay Hat Executed read command 0x00 byte read:0x01
[20/08/2024, 11:26:14] debug: 39 - Sequent 8 Relay Hat Executed read command 0x03 byte read:0x00
[20/08/2024, 11:26:14] debug: 39 - Sequent 8 Relay Hat Executed read command 0x00 byte read:0x01
[20/08/2024, 11:26:14] debug: 39 - Sequent 8 Relay Hat Executed read command 0x03 byte read:0x00
[20/08/2024, 11:26:14] debug: 39 - Sequent 8 Relay Hat Executed read command 0x00 byte read:0x01
[20/08/2024, 11:26:15] debug: 39 - Sequent 8 Relay Hat Executed read command 0x03 byte read:0x00
[20/08/2024, 11:26:15] debug: 39 - Sequent 8 Relay Hat Executed read command 0x00 byte read:0x01
[20/08/2024, 11:26:15] debug: Executed send command 0x01 0x00  bytes written:2
[20/08/2024, 11:26:15] warn: Relay Latch timer expired Filter: 10000ms

To me it looks like REM and njsPC are trying to talk, but are having problems for some reason. Would that cause njsPC/REM to switch off the pool/intellichlor relay? Any thoughts on why these communication problems might be happening?

I do have one idea myself. I'm running a RPi 3 as I documented here. At the time I did that write up there were RPi supply chain problems, and you couldn't buy an RPi 4 for less than $200. So I found an old RPi 3 in the bottom of my electronics box and used that instead. It's always been a bit janky. It freezes from time to time, so I configured the onboard watchdog to reboot it when needed. It also may have heat issues as it has no active cooling and I live in Phoenix. I will get around to my separate Nixie enclosure as I documented in that 801 thread, I've just been preoccupied with other high priority things since I wrote that thread; hopefully this winter. Anyway, point is that it may be time to get a RPi 4 and eliminate that as a potential source of my problems. I just checked Amazon and they are only about $62 these days.

Good news is that it does not appear to be the 9 year old Intellichlor that is failing. Might have to drop $1200 on a new one in the future, but not today.

[CodeSlinger69]

I went down a long path of trying to figure out why REM and njsPC were having periodic communication issues. They are both running on the same device so the network or wifi shouldn't be an issue. I'll save you all the details of all the rabbit holes I explored. Bottom line is that I gave up, and just decided to reinstall everything from scratch. After backing up my config files and other data, I reimaged my SD card with the standard Raspbian installer and started getting everything reinstalled. It was pretty quick as I had good notes from my previous work.

As a bonus I was able to upgrade to Raspbian Bookworm, which is a full 64 bit os (both kernel and userland). This version was released in Fall 2023 or so.

So far so good. I'm not seeing any of the issues documented earlier in this thread any more. I'm guessing that something just got messed up with all my hackery and experimentation over the years. Much easier to start clean than trying to trace down obscure issues. I did also order a new RPi 4B from Amazon since they are affordable once again. It should be easy to just move the SD Card over to the new device, move the Sequent stack over, and everything should just work. I'll report back.

[CodeSlinger69]

Received the RPi 4 today. Moved the SD card over and everything just continued to work as expected.

Marking this thread as answered.

[CodeSlinger69]

Resolved.