Controller not started on ROS-side although external_control is running.

[felixvd]

Summary

After upgrading to master, changing between ROS control and other scripts causes the robot to stop accepting goals. The controller seems to be stuck in a stopped state, although the external control program is running on the robot.

Versions

• ROS Driver version: 09e0d8
• Affected Robot Software Version(s): 5.10
• Affected Robot Hardware Version(s): UR5e
• Robot Serial Number: 20205500607
• UR+ product(s) installed: -
• URCaps Software version(s): 1.0.4

Impact

The robot stops accepting goals, because it thinks that the control script is not running on the robot, although it is.

Issue details

I recently upgraded our driver from commit 7ac5b4 to master (09e0d8), and the external control URCap from 1.0.2 to 1.0.4. Since upgrading, I have had issues changing between ROS control and other scripts.

During our process, we switch between ROS control and "native" scripts on the UR. To do this, we load and start different scripts on the UR, and then load and restart the external_control.urp on the robot like this. After the update, this process has started to fail in a way that it did not before. It appears that the controller sometimes stops when another program is loaded, and does not recover when external control is restarted.

[ INFO] [1616123298.282498467]: Connection to robot dropped, waiting for new connection.
[ INFO] [1616123298.282778955]: Robot ready to receive control commands.
[ INFO] [1616123298.284849363]: Connection to robot dropped, waiting for new connection.
[ERROR] [1616123298.285252965]: Robot control is currently inactive. Starting controllers that claim resources is currently not possible. Not starting controller 'scaled_pos_joint_traj_controller'
[ERROR] [1616123298.285296765]: Could not switch controllers. The hardware interface combination for the requested controllers is unfeasible.

[ INFO] [1616123339.637201982]: Robot requested program
[ INFO] [1616123339.637251554]: Sent program to robot
[ INFO] [1616123339.819919291]: Robot ready to receive control commands.
[ERROR] [1616123341.664520556]: Can't accept new action goals. Controller is not running.

In this state, /ur_hardware_interface/robot_program_running publishes True, so I am not sure how to detect this and recover from it. /controller_manager/list_controllers does publish the status of the controller, but what is the correct method to start it? Sending commands to /controller_manager/switch_controller seems unwise since the robot driver presumably keeps track of the state internally. Please advise.

Project status at point of discovered

It happened after I upgraded, and occurs about 30% of the time that I ask the robot to execute a program other than ROS external control.

Steps to Reproduce

I haven't found a reliable way to reproduce this yet, but running ROS external control and loading/executing other programs like in this code might do the trick.

Workaround Suggestion

Restarting the controller via /controller_manager/switch_controller might work, but it might confuse the driver, too.

[fmauch]

Before digging deeper into this: We are currently working on

• Improving the ReverseInterface class (This is what basically handles the external_control connects / disconnects). See TCP server implementation Universal_Robots_Client_Library#46 for details.
• Correctly handling multiple instances of the External Control program node inside a UR program. See Split the urscript into a header and a control loop. Universal_Robots_ExternalControl_URCap#14 and Added header and control loop definitions. #333 for details.

The latter should make it possible for your use-case to keep everything in one program instead of switching programs all the time.

If you like, it would be nice if you could test, whether those changes (especially the first one) fix the problems mentioned. If so, I would suggest not digging too deep here, as this will get merged soon.

[felixvd]

Thanks for the response. I tried running with your changes in UniversalRobots/Universal_Robots_Client_Library#46, but when I try to build it with the current master branch of this repo, it fails:

Errors     << ur_robot_driver:make /root/underlay_ws/logs/ur_robot_driver/build.make.007.log
CMakeFiles/ur_robot_driver_node.dir/src/hardware_interface.cpp.o: In function `urcl::UrDriver::~UrDriver()':
/opt/ros/melodic/include/ur_client_library/comm/script_sender.h:43: undefined reference to `urcl::comm::URServer::~URServer()'
CMakeFiles/ur_robot_driver_node.dir/src/hardware_interface.cpp.o: In function `urcl::UrDriver::~UrDriver()':
/opt/ros/melodic/include/ur_client_library/comm/reverse_interface.h:81: undefined reference to `urcl::comm::URServer::disconnectClient()'
/opt/ros/melodic/include/ur_client_library/comm/reverse_interface.h:80: undefined reference to `urcl::comm::URServer::~URServer()'
collect2: error: ld returned 1 exit status
make[2]: *** [/root/underlay_ws/devel/.private/ur_robot_driver/lib/ur_robot_driver/ur_robot_driver_node] Error 1
make[1]: *** [CMakeFiles/ur_robot_driver_node.dir/all] Error 2
make: *** [all] Error 2

If I try with commit 7ac5b4, it builds, but the issue described in the OP still occurs.

I'm not sure that multiple instances of the External Control program node will be needed in my case. We do need to switch between programs and turn ROS control on and off, though.

[felixvd]

I just reverted to the URCap version 1.0.2 and commit 7ac5b4, which I considered to be working, but the issue still occurs. Maybe this is introduced by Polyscope 5.10, but I would very much like to avoid downgrading just to attempt to fix this bug. I also feel like the problem is not on the UR side to start with.

For now I check the controller state after reactivating external control. This seems to work, even though it's ugly:

  def check_for_dead_controller_and_force_start(self):
    service_proxy_list = rospy.ServiceProxy('/b_bot/controller_manager/list_controllers', controller_manager_msgs.srv.ListControllers)
    service_proxy_switch = rospy.ServiceProxy('/b_bot/controller_manager/switch_controller', controller_manager_msgs.srv.SwitchController)
    rospy.sleep(2)

    list_req = controller_manager_msgs.srv.ListControllersRequest()
    switch_req = controller_manager_msgs.srv.SwitchControllerRequest()

    list_res = service_proxy_list.call(list_req)
    for c in list_res.controller:
      if c.name == "scaled_pos_joint_traj_controller":
        if c.state == "stopped":
          # Force restart
          rospy.logwarn("Forcing controller start")
          switch_req.start_controllers = ['scaled_pos_joint_traj_controller']
          switch_req.strictness = 1
          switch_res = service_proxy_switch.call(switch_req)
          return switch_res.ok
        else:
          return True

[felixvd]

As an update, I just received another robot arm on Polyscope 5.9, installed the External Control 1.0.4 URCap, and had the same issues. Reverting to 1.0.2 seemed to help. Looking forward to see upcoming changes.

[fmauch]

I have tried to reproduce this with the following setup:

• Polyscope 5.10
• external_control URCap 1.5.0,
• UniversalRobots/Universal_Robots_Client_Library@2b84427
• This repo on 98e0d87

I created two programs on the robot:

• a program only containing the external_control program node external_control.urp
• test move program with a simple movej command with two waypoints ´move.urp` (The first waypoint starts where my ROS trajectory ends)

On the ROS machine I run this bash script:

#!/bin/bash

while [[ true ]]; do
  rosservice call /ur_hardware_interface/dashboard/load_program "filename: 'external_control.urp'"
  rosservice call /ur_hardware_interface/dashboard/play "{}"
  rostopic pub /scaled_pos_joint_traj_controller/follow_joint_trajectory/goal control_msgs/FollowJointTrajectoryActionGoal -f ~/Downloads/trajectory_pp.txt -1
  sleep 5 # too lazy to write a proper action client
  rosservice call /ur_hardware_interface/dashboard/load_program "filename: 'move.urp'"
  rosservice call /ur_hardware_interface/dashboard/play "{}"
done

With this I could not reproduce the issue after letting it run for a couple of minutes.

[felixvd]

Thanks for the detailed testing. Sadly, my hardware_interface does not even start up with this repo on master (13db13, to which I forwarded to get URCap 1.0.5) and the Universal_Robots_Client_Library on 2b84427, neither for Polyscope 5.9 nor 5.10. It crashes on startup.

For what it's worth, when starting up the hardware interface on master with URCap 1.0.5 (and the binary client library I suppose) while the external control is already running on the robot, the topic robot_program_running publishes False for me.

I don't know how to reproduce it yet, but with the commits mentioned above I once again had my robot arms in a state where robot_program_running showed True, external control was running on the robot, but the controller was not running on the ROS driver side. I will keep watching and hope the upcoming changes in the client library improve things.

[EspenTeigen]

I have the exactly same problem as felixvd
I am using ursim e-series with URSoftware 5.11.0.1092249

Has there been found a solution to this problem?

[stefanscherzinger]

@felixvd @EspenTeigen Does this issue still exist with the latest changes of driver and library?

[EspenTeigen]

@stefanscherzinger It seems to be working fine at the moment :)

[stefanscherzinger]

@felixvd feel free to re-open if this issue persists.

[MiroslavKohut]

Hi,

I would like to ask for a help I have same problem as it is mensioned in this thread.

My settings are:

Affected Robot Software Version(s): 3.15
Affected Robot Hardware Version(s): UR10 CB3
URCaps Software version(s): 1.0.5
Universal robot ros driver version 2.0.0

Could you please propose me some solution ?

When I start the robot and try to control it the controllers are stoped by default and GOAL response with FALSE.

When I try to start the controller I am getting this message

[ERROR] [1649346513.402044300]: Robot control is currently inactive. Starting controllers that claim resources is currently not possible. Not starting controller 'scaled_pos_joint_traj_controller'
[ERROR] [1649346513.402104400]: Could not switch controllers. The hardware interface combination for the requested controllers is unfeasible.

Thank you for your help

[fmauch]

@MiroslavKohut the error you posted seems not necessarily related to the OP. Could you please open a new issue and describe exactly the steps that you took?

[sebas775]

Affected Robot Software Version(s): 3.15
Affected Robot Hardware Version(s): UR10 CB3
URCaps Software version(s): 1.0.5
Universal robot ros driver
[ERROR] [1656132074.104878369]: Robot control is currently inactive. Starting controllers that claim resources is currently not possible. Not starting controller 'forward_cartesian_traj_controller'
[ERROR] [1656132074.105071000]: Could not switch controllers. The hardware interface combination for the requested controllers is unfeasible.