Android in the Zoo:
Chain-of-Action-Thought for GUI Agents

Jiwen Zhang^1,2 , Jihao Wu² , Yihua Teng² , Minghui Liao² , Nuo Xu² , Xiao Xiao² , Zhongyu Wei¹ , Duyu Tang².

¹Fudan University ²Huawei Inc.

---

This work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. To enable an adaptive learning of CoAT process, we construct a benchmark Android-In-The-Zoo, which contains 18,643 screen-action pairs together with CoAT annotations.

📣 Update

• [2024-10-15] Evaluation code has been released!

• [2024-09-20] Our work has been accepted to EMNLP2024 Findings!

• [2024-07-16] We add the demo code for using CoAT on proprietary models (GPT4V, Gemini-Pro and Qwen-VL-Max)!

• [2024-03-31] We release the first version of our AiTZ dataset!

• [2024-03-05] We have our paper arxived, now you can acess it by clicking here !

Android-in-the-Zoo

The data in AiTZ has 18,643 screens together with 2500+ instructions, all annotated with CoAT-driven semantic labels. The sample format for each time step is

{
  "episode_id": "523638528775825151",
  "episode_length": 4,
  "step_id": 0,
  "coat_screen_desc":   "[observe]",
  "coat_action_think":  "[action think]",
  "coat_action_desc":   "[next action description]",
  "coat_action_result": "[action result]",
  ...
}

You can refer to data-example folder for a more specific example.

Download

Our dataset (GoogleDrive or BaiduNetdisk) contains both the screens (.png) and the annotations (.json), consuming about 2.6G device space.

Statistics

Subset	Train		Test
	#Episodes	#Screens	#Episodes	#Screens
General	323	2405	156	1202
Install	286	2519	134	1108
GoogleApps	166	1268	76	621
Single	844	2594	0	0
WebShopping	379	5133	140	1793
Total	1998	13919	506	4724

Chain-of-Action-Thought

Comparison with other context modeling methods

We validate the effectiveness of CoAT by conducting a preliminary experiment on 50 episodes randomly sampled from AITW dataset.

The compared baselines are Chain-of-Thought (CoT) and Chain-of-Actions (CoA).

Prompt	Metric	QwenVL	Gemini-PV	GPT-4V
CoA	hit	94.5	99.8	99.3
	acc	44.4	47.7	62.8
CoT	hit	95.6	97.5	97.1
	acc	49.4	52.0	64.1
CoAT	hit	96.3	96.4	98.2
	acc	52.4	54.5	73.5

where “hit” means format hit rate, and “acc” means action type prediction accuracy. (One can refer to Table 8 in our paper for more details.)

CoAT demo usage

Here we provide a demo code for anyone who wants to try the CoAT on GPT-4V, Qwen-VL-Max and Gemini-1.0-Pro-Vision.

Firstly, go to coat/config.yaml and add your own api-keys and urls.

Secondly, run the folloiwng code in commad line to generate sematic components of CoAT framework:

python run_coat.py --task "flow" --DEMO_MODE "COAT" --MODEL.NAME "openai/gemini/qwenvl" --num-threads 3

Then, you can obtain the action prediction results by

python run_coat.py --task "predict" --DEMO_MODE "COAT" --MODEL.NAME "openai/gemini/qwenvl" --num-threads 3

Citation

If you find our work helpful, please consider citing our paper.

@misc{zhang2024android,
      title={Android in the Zoo: Chain-of-Action-Thought for GUI Agents}, 
      author={Jiwen Zhang and Jihao Wu and Yihua Teng and Minghui Liao and Nuo Xu and Xiao Xiao and Zhongyu Wei and Duyu Tang},
      year={2024},
      eprint={2403.02713},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}