AL1K4 Pico Motor Readback/Setpoint Desync

[ZLLentz]

Bug Report

• When we move axis 1, axis 2's readback moves
• When we move axis 2, axis 3's readback moves
• When we move axis 3, axis 4's readback moves
• When we move axis 4, axis 1's readback moves

This is perplexing because:

1. This worked correctly last month
2. Each axis is connected to the controller by exactly 1 cable (and I'm not even sure they are encoded).

Expected Behavior

The axis setpoint/readback mapping should be consistent

Context

Makes TMO reference laser alignment confusing
Reported by @aegger13

Steps To Reproduce

1. Move any of the al1k4 pico motors

Your Environment

Standard LCLS EPICS environment
IOC config file is:

RELEASE=/reg/g/pcds/epics/ioc/common/picoMotion/R3.0.0
ENGINEER=Zachary Lentz (zlentz)

IP=mcp-tmo-al1k4
IOCNAME=ioc-mcp-tmo-al1k4
PREFIX=AL1K4:L2SI:MCP
IOC_PREFIX=AL1K4:L2SI:MCP:IOC

MOTOR(NAME=01,DESC="TIP")
MOTOR(NAME=02,DESC="TILT")
MOTOR(NAME=03,DESC="X")
MOTOR(NAME=04,DESC="Y")

Modules used are:

IOCADMIN_MODULE_VERSION = R3.1.15-1.0.1
AUTOSAVE_MODULE_VERSION = R5.6.2-2.2.0
ASYN_MODULE_VERSION = R4.31-0.1.0
MOTOR_MODULE_VERSION = R6.10.1-1.0.1

[ZLLentz]

I observed via telnet that the controller sends the response to the nth command ago, rather than my most recent command.

Theory: the controller falls behind, and if we are out of sync by an even number of steps, then the axes are shifted.
We poll for two values (position and is_done), then receive the value from a previous query.
IOC basically stops working if you are out of sync by an odd number of steps since positions all be come 1 or 0 counts (is done, is not done)

To fix live:

• disable the IOC
• ssh psdev, telnet to the controller on port 23
• *RST to reset
• wait for controller to reboot
• start the IOC

[klauer]

This is sadly a very common issue with motor record drivers and other devices that have a too simple communication protocol. A hacky way to fix this is to find a way to get the device to respond with a unique string, so you can correlate request/response. Looks like there's no reasonable way to avoid this according to the command set for the 8742.

[ZLLentz]

You can probably mitigate this by polling very slowly... I agree that the device protocol for the 8742 where it responds with just a number (4 instead of something like POS1=4) makes it rough to know when a problem has occurred.

You could potentially do a periodic check of *IDN? and do... something? when that doesn't respond with the correct string? (not sure what)