Issue when visual automation is run 10-20 times (user uses Ctrl+C to kill TagUI prematurely)

[kensoh]

user query

---

Hi Ken,

Have been actively playing and trying your new TagUI v5. Liked your new keyboard commands (ability to issue Alt-F, Ctrl-A, etc.) and especially the new low-level mouse down and mouse up events that allow dragging and dropping. Thanks once again for making an already powerful program even more powerful!

While trying, there's one recent problem that bugged me.

Please take a look at the following 2-liner code:

https://google.com
enter input1.png as weather singapore

input1.png is this image (the google input field):

While there are times when it works correctly:

But a lot of the time, you will see garbage characters. Below are 5 of the runs, each one different garbage characters:

Output1:

Output 2:

Output 3:

Output 4:

Output 5:

Looking at the pattern, it seems that 2 keyboard inputs events are triggered at the same time - resulting in overlapping text.

What's interesting is that this behaviour occurs in both windows and mac. I first noticed this in windows 10. And today I tried something in mac, the same bug occurs!

Note:

1. this bug does not occur everytime. Try running it 10 or 20 times, it will start to occur when you run more times.
2. I've tried using every other approach I can think of: e.g.
   using: type input1.png as weather singapore
   using vision type("weather singapore")
   using click input1.png, followed by keyboard weather singapore
   Still the same.

Has anyone reported this to you before?

For your kind assistance and advice, please.

[kensoh]

I tried a few times but unable to replicate. Can you send me the logs in tagui\src\tagui.sikuli folder for some cases where error happen and for 1 case where it works correctly? Won't have time to look at this soon, baby girl has arrived last week, now doing full-time confinement daddy taking care of mum and baby.

A possible cause I can think of is unclosed sessions of SikuliX still running, thus SikuliX doubles the input actions from TagUI. But I haven't come across such a feedback for this to happen and for the effects you observed. Also, let me know how the script is run - manually back to back or through some other master script without delay in between. I also assume you run with chrome option. Also, can try if setting up and running in another folder to see if any change.

[kensoh]

user reply

---

Congrats on your new baby!!! You must be so happy and busy now!

No hurry on this. Baby come first!

Attached please find the sikuli log file (in output.zip).
The script is the 2-liner
https://google.com
enter input1.png as weather singapore
output0 is the ok version
output1, output2 and output3 are those with errors
If you take a look at tagui-output0.log, there should be only one "ACTION - type"

As you can see from tagui-output1.log and tagui-output2.log, there are indeed two "ACTION - type" being triggered as follows:

Even more interesting, tagui-output3.log has three "ACTION - type"!

Please let me know if there are any other log files that you need.

output.zip

[kensoh]

user reply

---

Hi Ken,

A possible cause I can think of is unclosed sessions of SikuliX still running, thus SikuliX doubles the input actions from TagUI. But I haven't come across such a feedback for this to happen and for the effects you observed.

I don't think so. But to be sure, is there any command I can run to make sure that the previous session of sikuli is closed?

Also, let me know how the script is run - manually back to back or through some other master script without delay in between. I also assume you run with chrome option.

The script contains only 2 lines:
https://google.com
enter input1.png as weather singapore

I run from the terminal (in MacOS) using:
tagui google1.gui chrome

It's a standalone script with only 2 lines, manually run from the termial. It's not run from any other scripts.

Also, can try if setting up and running in another folder to see if any change.

Have tried running from another folder. Same problem.

Note: again, this does not occur every time. You have to try running it about 10+ times, then the problem start to occur. Every now and then, it seems to be ok again. But after a while, the problem occurs again.

[kensoh]

I tried 10 times without issue. I notice that your logs at the top seem to have some errors while my log attached is the usual expected log output.

Might be some locked files or processes but not sure why and what. Some ideas to try isolating the cause -

• Download a fresh packaged installation and try again, by running the tagui command directly from the newly downloaded tagui\src\tagui. There should be no .jar files in the tagui.sikuli folder but there is in sikulix folder.

• Try restarting computer, and make sure TagUI is not in some autorun script on system reboot. On Mac there is a Java icon on the dock if Sikuli is running. On Windows there should be a command prompt window for Sikuli if it is running.

• Try typing java -version on command prompt to see if it is something like below, if not, try installing version 8 Java from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

tagui.log

[kensoh]

Root cause is user kill existing script from execution halfway and left SikuliX process still running. When script runs again, commands are received by 2 SikuliX processes.

Will advise user to kill remnant processes if manually killing off TagUI execution.

[kensoh]

Solution is to use scripts in #417 for the use case where users manually terminate TagUI with Ctrl+C.

This will end the integration processes cleanly even though TagUI is terminated by OS with Ctrl+C.

---

TagUI will by default send 'finish' signals to its integrations processes automatically (eg Chrome, Python, R, SikuliX). This happens even if errors are encountered during execution, as the cleaning up is done outside the main execution engine, for robustness and reliability.

However, if a user for some reason use Ctrl+C to kill TagUI prematurely, there is no chance for TagUI to send those 'finish' signals to end its integrations processes, as OS will terminate TagUI right away. In this case, users may use the following scripts to terminate the running integration processes.

Without doing so, example of a known side effect (raised in #412) is a SikuliX process when not exit properly, will interfere with execution for the next visual automation script, as there are 2 SikuliX processes running and thus actions are acted out twice (for eg typing text to screen).

This is an uncommon use case, thus it does not make sense to implement as part of main engine. Because it will require adding some delay for each execution for 'finish' signals to be processed. This will slow down execution of all TagUI scripts for the benefit of such edge use case. Thus the following scripts may be used for this purpose.

Note - to rely on programmatically finding the process IDs and killing them is a possible solution but not reliable (for eg, if there are more than 1 existing lost process). And it adds potential failure arcs during executions due to reliance on more OS-level commands. Thus below scripts are the recommended solutions for the use case where users manually terminate TagUI with Ctrl+C.

macOS / Linux - eg end_processes

#!/usr/bin/env bash
# TO KILL INTEGRATION PROCESSES IF CTRL+C WAS USED TO QUIT TAGUI PREMATURELY ~ TEBEL.ORG #

# checking for existence of files before sending finish signal is important
# otherwise for ongoing tagui iterations, all integrations will be invoked

if [ -f "tagui_r/tagui_r.in" ]; then echo "finish" > tagui_r/tagui_r.in; fi
if [ -f "tagui_py/tagui_py.in" ]; then echo "finish" > tagui_py/tagui_py.in; fi
if [ -f "tagui.sikuli/tagui_sikuli.in" ]; then echo "finish" > tagui.sikuli/tagui_sikuli.in; fi
if [ -f "tagui_chrome.in" ]; then echo "finish" > tagui_chrome.in; fi

Windows - eg end_processes.cmd

@echo off
rem # TO KILL INTEGRATION PROCESSES IF CTRL+C WAS USED TO QUIT TAGUI PREMATURELY ~ TEBEL.ORG #

rem checking for existence of files before sending finish signal is important
rem otherwise for ongoing tagui iterations, all integrations will be invoked

if exist "tagui_r\tagui_r.in" echo finish > tagui_r\tagui_r.in
if exist "tagui_py\tagui_py.in" echo finish > tagui_py\tagui_py.in
if exist "tagui.sikuli\tagui_sikuli.in" echo finish > tagui.sikuli\tagui_sikuli.in
if exist "tagui_chrome.in" echo finish > tagui_chrome.in

[kensoh]

User is using Ctrl+C to terminate live mode as he uses live mode a lot for testing. For time saving, he uses Ctrl+C instead of typing done to terminate live mode. Below is my reply on other alternatives -

Above scripts need to be run manually, just as Ctrl+C was used manually to terminate. Don't think it is possible to run above scripts within TagUI script as it will end up killing its real integration processes. It is probably possible to write your own wrapper batch files to run those scripts, wait a while, then run TagUI, but not straightforward to implement and will add overheads.

Easier to type done to quit the live mode or use a wait or ask step to pause execution before quitting. Alternatively, you can edit tagui_parse.php search for below and change it to some other terminating keyword such as 'q' or something else.

(live_input.indexOf('done')