01-user-guide.adoc

// NOTE: release workflow automatically updates etaoin versions in this file
// NOTE: many of the clojure code blocks in this file are tested via lread/test-doc-block
= User Guide
:toclevels: 5
:toc:
:lib-version: 1.0.40
:project-src-coords: clj-commons/etaoin
:project-mvn-coords: etaoin/etaoin
:url-webdriver: https://www.w3.org/TR/webdriver/
:url-sample-page: /doc/user-guide-sample.html
:url-doc: https://cljdoc.org/d/{project-mvn-coords}
:url-tests: /test/etaoin/api_test.clj
:url-slack: https://clojurians.slack.com/archives/C7KDM0EKW

== Introduction

Etaoin offers the Clojure community a simple way to script web browser interactions from Clojure and Babashka.
It is a thin abstraction atop the link:{url-webdriver}[W3C WebDriver protocol] that also endeavors to resolve real-world nuances and implementation differences.

=== History

Ivan Grishaev (https://github.com/igrishaev[@igrishaev]) created Etaoin and published its first release to Clojars in Feb of 2017.
He and his band of faithful contributors grew Etaoin into a well respected goto-library for browser automation.

In May 2022, finding his time had gravitated more to back-end development, Ivan offered Etaoin for adoption to clj-commons.
It is now currently under the loving care of https://github.com/lread[@lread] and https://github.com/borkdude[@borkdude].

=== Interesting Alternatives

If Etaoin is not your cup of tea, you might also consider:

Clojure based:

* https://github.com/pfeodrippe/wally[Wally] - A Clojure Playwright wrapper
* https://github.com/tatut/clj-chrome-devtools[clj-chrome-devtools] - Clojure API for controlling a Chrome DevTools remote

Other:

* https://www.selenium.dev/[Selenium] - A browser automation framework and ecosystem
* https://playwright.dev/[Playwright] - Reliable end-to-end testing for modern web apps
* https://pptr.dev/[Puppeteer] - A high-level API to control Chrome/Chromium over the DevTools Protocol

[[supported-os-browser]]
=== Supported OSes & Browsers

Etaoin's test suite covers the following OSes and browsers for both Clojure and Babashka:

|===
| OS | Chrome | Firefox | Safari | Edge

| Linux (ubuntu)
| yes
| yes
| -
| -

| macOS
| yes
| yes
| yes
| yes

| Windows
| yes
| yes
| -
| yes

|===

NOTE: We did once test against PhantomJS, but since work has long ago stopped on this project, we have dropped testing

== Installation

There are two steps to installation:

. Add the `etaoin` library as a dependency to your project
. Install the WebDriver for each web browser that you want to control with Etaoin

=== Add the Etaoin Library Dependency

==== For Clojure Users

Etaoin supports Clojure v1.9 and above.

Add the following into the `:dependencies` vector in your `project.clj` file:

[source,clojure,subs="attributes+"]
----
   [etaoin "{lib-version}"]
----

//:test-doc-blocks/skip
Or the following under `:deps` in your `deps.edn` file:
[source,clojure,subs="attributes+"]
----
   etaoin/etaoin {:mvn/version "{lib-version}"}
----

==== For Bababashka Users

We recommend the current release of https://book.babashka.org/#_installation[babashka].

Add the following under `:deps` to your `bb.edn` file:

//:test-doc-blocks/skip
[source,clojure,subs="attributes+"]
----
   etaoin/etaoin {:mvn/version "{lib-version}"}
----

[TIP]
====
Babashka uses https://github.com/ptaoussanis/timbre[timbre] for logging.
Timbre's default logging level is debug.
For a quieter Etaoin experience when using babashka, set the timbre default log level to info:

//:test-doc-blocks/skip
[source,clojure]
----
(require '[taoensso.timbre :as timbre])
(timbre/set-level! :info)
----
====

:url-webdriver: https://www.w3.org/TR/webdriver/
:url-tests: https://github.com/{project-src-coords}/blob/master/test/etaoin/api_test.clj
:url-chromedriver: https://sites.google.com/chromium.org/driver/
:url-chromedriver-dl: https://sites.google.com/chromium.org/driver/downloads
:url-geckodriver-dl: https://github.com/mozilla/geckodriver/releases
:url-phantom-dl: http://phantomjs.org/download.html
:url-webkit: https://webkit.org/blog/6900/webdriver-support-in-safari-10/
:url-edge-dl: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

[[install-webdrivers]]
=== Installing the Browser WebDrivers

Etaoin controls web browsers via their WebDrivers.
Each browser has its own WebDriver implementation that must be installed.

[TIP]
====
If it is not already installed, you will need to install the web browser too (Chrome, Firefox, Edge).
This is usually via a download from its official site.
Safari comes bundled with macOS.
====

[TIP]
====
WebDrivers and browsers are updated regularly to fix bugs.
Use current versions.
====

Some ways to install WebDrivers:

* Google link:{url-chromedriver}[Chrome Driver] +
** macOS: `brew install chromedriver`
** Windows: `scoop install chromedriver`
** Download: link:{url-chromedriver-dl}[Official Chromedriver download]

* Geckodriver for Firefox
** macOS: `brew install geckodriver`
** Windows: `scoop install geckodriver`
** Download: link:{url-geckodriver-dl}[Official geckodriver release page]

* Safari Driver
** macOS only: Set up Safari options as the link:{url-webkit}[Webkit page] instructs (scroll down to "Running the Example in Safari" section).

* Microsoft Edge Driver
** macos: (download manually)
** Windows: `scoop install edgedriver` +
Edge and `msedgedriver` must match so you might need to specify the version:
`scoop install edgedriver@101.0.1210.0`
** Download: link:{url-edge-dl}[Official Microsoft download site]

* Phantom.js browser +
(obsolete, no longer tested)
** macOS: `brew install phantomjs`
** Windows: `scoop install phantomjs`
** Download: link:{url-phantom-dl}[Official PhantomJS download site]

Check your WebDriver installations launching by launching these commands.
Each should start a process that includes its own local HTTP server.
Use Ctrl-C to terminate.

[source,bash]
----
chromedriver
geckodriver
safaridriver -p 0
msedgedriver
phantomjs --wd
----

You can optionally run the Etaoin test suite to verify your installation.

TIP: Some Etaoin API tests rely on ImageMagick.
Install it prior to running test.

From a clone of the https://github.com/clj-commons/etaoin[Etaoin GitHub repo]

* To check tools of interest to Etaoin:
+
[source,bash]
----
bb tools-versions
----
* Run all tests:
+
[source,bash]
----
bb test all
----
* For a smaller sanity test, you might want to run api tests against browsers you are particularly intested in. Example:
+
[source,bash]
----
bb test api --browser chrome
----

During the test run, browser windows will open and close in series.
The tests use a local handcrafted HTML file to validate most interactions.

See <<troubleshooting>> if you have problems - or reach out on link:{url-slack}[Clojurians Slack #etaoin] or https://github.com/clj-commons/etaoin/issues[GitHub issues].

== Getting Started

The great news is that you can automate your browser directly from your Babashka or Clojure REPL.
Let's interact with Wikipedia:

// A little invisible codeblock for some setup
ifdef::env-test-doc-blocks[]
[source,clojure]
----
(require '[babashka.fs :as fs])
;; for better test-doc-block reporting when running generated tests
(require '[etaoin.test-report])
;; for screenshots save dir (dir must currently exist)
(fs/create-dirs "target/etaoin-play")
----
endif::[]

[source,clojure]
----
(require '[etaoin.api :as e]
         '[etaoin.keys :as k])

;; Start WebDriver for Firefox
(def driver (e/firefox)) ;; a Firefox window should appear

;; let's perform a quick Wiki session

;; navigate to wikipedia
(e/go driver "https://en.wikipedia.org/")
;; wait for the search input to load
(e/wait-visible driver [{:id :simpleSearch} {:tag :input :name :search}])

;; search for something interesting
(e/fill driver {:tag :input :name :search} "Clojure programming language")
(e/wait driver 1)
(e/fill driver {:tag :input :name :search} k/enter)
(e/wait-visible driver {:class :mw-search-results})

;; click on first match
(e/click driver [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
(e/wait-visible driver {:id :firstHeading})

;; check our new url location
(e/get-url driver)
;; => "https://en.wikipedia.org/wiki/Clojure"

;; and our new title
(e/get-title driver)
;; => "Clojure - Wikipedia"

;; does page have Clojure in it?
(e/has-text? driver "Clojure")
;; => true

;; navigate through history
(e/back driver)
(e/forward driver)
(e/refresh driver)
(e/get-title driver)
;; => "Clojure - Wikipedia"

;; let's explore the info box
;; What's its caption? Let's select it with a css query:
(e/get-element-text driver {:css "table.infobox caption"})
;; => "Clojure"

;; Ok,now let's try something trickier
;; Maybe we are interested what value the infobox holds for the Family row:
(let [wikitable (e/query driver {:css "table.infobox.vevent tbody"})
      row-els (e/children driver wikitable {:tag :tr})]
  (for [row row-els
        :let [header-col-text (e/with-http-error
                                (e/get-element-text-el driver
                                                       (e/child driver row {:tag :th})))]
        :when (= "Family" header-col-text)]
    (e/get-element-text-el driver (e/child driver row {:tag :td}))))
;; => ("Lisp")

;; Etaoin gives you many options, we can do the same-ish in one swoop in XPath:
(e/get-element-text driver "//table[@class='infobox vevent']/tbody/tr/th[text()='Family']/../td")
;; => "Lisp"

;; When we are done we quit, which stops the Firefox WebDriver
(e/quit driver) ;; the Firefox Window should close
----

Most api functions require the driver as the first argument.
The `doto` macro can give your code a DSL feel.
A portion of the above rewritten with `doto`:

[source,clojure]
----
(require '[etaoin.api :as e]
         '[etaoin.keys :as k])

(def driver (e/firefox))

(doto driver
  (e/go "https://en.wikipedia.org/")
  (e/wait-visible [{:id :simpleSearch} {:tag :input :name :search}])
  (e/fill {:tag :input :name :search} "Clojure programming language")
  (e/wait 1)
  (e/fill {:tag :input :name :search} k/enter)
  (e/wait-visible {:class :mw-search-results})
  (e/click [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
  (e/wait-visible {:id :firstHeading})
  (e/quit))
----

== Playing Along in your REPL
We encourage you to try the examples in from this user guide in your REPL.

The Interwebs is constantly changing.
This makes testing against live sites impractical.
The code in this user guide has instead been tested to work against our link:{url-sample-page}[little sample page].

Until we figure out something more clever, it might be easiest to clone the etaoin GitHub repository and run a REPL from there.

Unless otherwise directed, our examples throughout the rest of this guide will assume you've already executed the equivalent of:

[source,clojure]
----
(require '[etaoin.api :as e]
         '[etaoin.keys :as k]
         '[clojure.java.io :as io])

(def sample-page (-> "doc/user-guide-sample.html" io/file .toURI str))

(def driver (e/chrome)) ;; or replace chrome with your preference
(e/go driver sample-page)
----

== More Getting Started

You can use `fill-multi` to shorten the code like so:

[source,clojure]
----
(e/fill driver :uname "username")
(e/fill driver :pw "pass")
(e/fill driver :text "some text")

;; let's get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username" "pass" "some text"]
----

into:

[source,clojure]]
----
;; issue a browser refresh
(e/refresh driver)
(e/fill-multi driver {:uname "username2"
                      :pw "pass2"
                      :text "some text2"})

;; to get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username2" "pass2" "some text2"]
----

If any exception occurs during a browser session, the WebDriver process might hang until you kill it manually.
To prevent that, we recommend the `with-<browser>` macros:

[source,clojure]
----
(e/with-firefox driver
  (doto driver
    (e/go "https://google.com")
    ;; ... your code here
    ))
----

This will ensure that the WebDriver process is closed regardless of what happens.

== Unit Tests as Docs

The sections that follow describe how to use Etaoin in more depth.

In addition to these docs, the link:{url-tests}[Etaoin api tests] are also a good reference.

== Creating and Quitting the Driver

Etaoin comes with many options to create a WebDriver instance.

TIP: As previously mentioned, we recommend the `with-<browser>` convention when you need proper cleanup.

Let's say we want to create a chrome headless driver:

// let's not pollute our main test-doc-block ns with these driver vars:
//{:test-doc-blocks/test-ns user-guide-driver-creation-test}
[source,clojure]
----
(require '[etaoin.api :as e])

;; at the base we have:
(def driver (e/boot-driver :chrome {:headless true}))
;; do stuff
(e/quit driver)

;; This can also be expressed as:
(def driver (e/chrome {:headless true}))
;; do stuff
(e/quit driver)

;; Or...
(def driver (e/chrome-headless))
;; do stuff
(e/quit driver)
----

The `with-<browser>` functions handle cleanup nicely:

[source,clojure]
----
(e/with-chrome {:headless true} driver
  (e/go driver "https://clojure.org"))

(e/with-chrome-headless driver
  (e/go driver "https://clojure.org"))
----

Replace `chrome` with `firefox`, `edge` or `safari` for other variants.
See link:{url-doc}[API docs] for details.

See <<driver-options>> for all options available when creating a driver.

== Selecting Elements [[querying]]

Queries (aka selectors) are used to select the elements on the page that Etaoin will interact with.

[source,clojure]]
----
;; let's start anew by refreshing the page:
(e/refresh driver)
;; select the element with an html attribute id of 'uname' and fill it with text:
(e/fill driver {:id "uname"} "Etaoin")
;; select the first element with an html button tag and click on it:
(e/click driver {:tag :button})
----

[TIP]
====
* A query returns a unique element identifier typically meaningful only as a selector to other functions it is passed to.
* Many functions accept a query directly. For example:
+
[source,clojure]]
----
;; specifying query directly
(e/get-element-text driver {:tag :button})
;; => "Submit Form"
;; specifying the result of a query (notice the `-el` fn variant here)
(e/get-element-text-el driver (e/query driver {:tag :button}))
;; => "Submit Form"
----
====

[TIP]
====
An exception is thrown if a query does not find an element.
Use exists? to check for element existence:

[source,clojure]
----
(e/exists? driver {:tag :button})
;; => true
(e/exists? driver {:id "wont-find-me"})
;; => false
----
====

=== Simple Queries, XPath, CSS

:xpath-sel: https://www.w3schools.com/xml/xpath_syntax.asp
:css-sel: https://www.w3schools.com/cssref/css_selectors.asp

* `:active` finds the current active element.
The Google page, for example, automatically places the focus on the search input.
So there is no need to click on it first:
+
[source,clojure]
----
(e/go driver "https://google.com")
(e/fill driver :active "Let's search for something" k/enter)
----

* any other keyword is translated to an html id attribute:
+
[source,clojure]
----
(e/go driver sample-page)
(e/fill driver :uname "Etaoin" k/enter)
;; alternatively you can:
(e/fill driver {:id "uname"} "Etaoin Again" k/enter)
----

* a string containing an link:{xpath-sel}[XPath] expression.
(Be careful when writing XPath manually, see <<troubleshooting>>.)
Here we find an `input` tag with an attribute `id` of `uname` and an attribute `name` of `username`:
+
[source,clojure]
----
(e/refresh driver)
(e/fill driver ".//input[@id='uname'][@name='username']" "XPath can be tricky")

;; let's check if that worked as expected:
(e/get-element-value driver :uname)
;; => "XPath can be tricky"
----

* a map with either `:xpath` or `:css` key with a string in corresponding syntax:
+
[source,clojure]
----
(e/refresh driver)
(e/fill driver {:xpath ".//input[@id='uname']"} "XPath selector")
(e/fill driver {:css "input#uname[name='username']"} " CSS selector")

;; And here's what we should see in username input field now:
(e/get-element-value driver :uname)
;; => "XPath selector CSS selector"
----
+
This link:{css-sel}[CSS selector reference] may be of help.

=== Map Syntax Queries

A query can also be a map that represents an XPath expression as data.
The rules are:

* A `:tag` key represents a tag's name.
Defaults to `*`.
* An `:index` key expands into the trailing XPath `[x]` clause.
Useful when you need to select a third row from a table, for example.
* Any non-special key represents an attribute and its value.
* `:fn/` is a prefix followed by a supported query function.

Examples:

* find the first `div` tag
+
[source,clojure]
----
(= (e/query driver {:tag :div})
   ;; equivalent via xpath:
   (e/query driver ".//div"))
;; => true
----

* find the n-th (1-based) `div` tag
+
[source,clojure]
----
(= (e/query driver {:tag :div :index 1})
   ;; equivalent via xpath:
   (e/query driver ".//div[1]"))
;; => true
----

* find the tag `a` where the class attribute equals to `active`
+
[source,clojure]
----
(= (e/query driver {:tag :a :class "active"})
   ;; equivalent xpath:
   (e/query driver ".//a[@class='active']"))
----

* find a form by its attributes:
+
[source,clojure]
----
(= (e/query driver {:tag :form :method :GET :class :formy})
   ;; equivalent in xpath:
   (e/query driver ".//form[@method=\"GET\"][@class='formy']"))
----

* find a button by its text (exact match):
+
[source,clojure]
----
(= (e/query driver {:tag :button :fn/text "Submit Form"})
   ;; equivalent in xpath:
   (e/query driver ".//button[text() = 'Submit Form']"))
----

* find an nth element (`p`, `div`, whatever, it does not matter) with "blarg" text:
+
[source,clojure]
----
(e/get-element-text driver {:fn/has-text "blarg" :index 3})
;; => "blarg in a p"

;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(text(), 'blarg')][3]")
;; => "blarg in a p"
----

* find an element that includes a class:
+
[source,clojure]
----
(e/get-element-text driver {:tag :span :fn/has-class "class1"})
;; => "blarg in a span"

;; equivalent xpath:
(e/get-element-text driver ".//span[contains(@class, 'class1')]")
;; => "blarg in a span"
----

* find an element that has the following domain in a `href`:
+
[source,clojure]
----
(e/get-element-text driver {:tag :a :fn/link "clojure.org"})
;; => "link 3 (clojure.org)"

;; equivalent xpath:
(e/get-element-text driver ".//a[contains(@href, \"clojure.org\")]")
;; => "link 3 (clojure.org)"
----

* find an element that includes all of the specified classes:
+
[source,clojure]
----
(e/get-element-text driver {:fn/has-classes [:class2 :class3 :class5]})
;; => "blarg in a div"

;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(@class, 'class2')][contains(@class, 'class3')][contains(@class, 'class5')]")
;; => "blarg in a div"
----

* find explicitly enabled/disabled input widgets:
+
[source,clojure]
----
;; first enabled input
(= (e/query driver {:tag :input :fn/enabled true})
   ;; equivalent xpath:
   (e/query driver ".//input[@enabled=true()]"))
;; => true

;; first disabled input
(= (e/query driver {:tag :input :fn/disabled true})
   ;; equivalent xpath:
   (e/query driver ".//input[@disabled=true()]"))
;; => true

;; return a vector of all disabled inputs
(= (e/query-all driver {:tag :input :fn/disabled true})
   ;; equivalent xpath:
   (e/query-all driver ".//input[@disabled=true()]"))
;; => true
----

=== Vector Syntax Queries

A query can be a vector of any valid query expressions.
For vector queries, every expression matches the output from the previous expression.

A simple, somewhat contrived, example:

[source,clojure]
----
(e/click driver [{:tag :html} {:tag :body} {:tag :button}])
;; our sample page shows form submits, did it work?
(e/get-element-text driver :submit-count)
;; => "1"
----

You may combine both XPath and CSS expressions

TIP: Reminder: the leading dot in an XPath expression means starting at the current node

[source,clojure]
----
;; under the html tag (using map query syntax),
;;  under a div tag with a class that includes some-links (using css query),
;;   click on a tag that has
;;    a class attribute equal to active (using xpath syntax):
(e/click driver [{:tag :html} {:css "div.some-links"} ".//a[@class='active']"])
;; our sample page shows link clicks, did it work?
(e/get-element-text driver :clicked)
;; => "link 2 (active)"
----

=== Advanced Queries

==== Querying the _nth_ Element Matched

Sometimes you may want to interact with the _nth_ element of a query.
Maybe you want to click on the second link within:

[source,html]
----
<ul>
    <li class="search-result">
        <a href="a">a</a>
    </li>
    <li class="search-result">
        <a href="b">b</a>
    </li>
    <li class="search-result">
        <a href="c">c</a>
    </li>
</ul>
----

You can use the `:index` like so:

[source,clojure]
----
(e/click driver [{:tag :li :class :search-result :index 2} {:tag :a}])
;; check click tracker from our sample page:
(e/get-element-text driver :clicked)
;; => "b"
----
:nth-child: https://www.w3schools.com/CSSref/sel_nth-child.asp

or you can use the link:{nth-child}[nth-child trick] with the CSS expression like this:

[source,clojure]

----
;; start page anew
(e/refresh driver)
(e/click driver {:css "li.search-result:nth-child(2) a"})
(e/get-element-text driver :clicked)
;; => "b"
----

Finally it is also possible to obtain the _nth_ element directly by using `query-all`:

[source,clojure]
----
;; start page anew
(e/refresh driver)
(e/click-el driver (nth (e/query-all driver {:css "li.search-result a"}) 1))
(e/get-element-text driver :clicked)
;; => "b"
----

[NOTE]
====
Notice:

* The use of `click-el` here. The `query-all` function returns an element, not a selector that can be passed to `click` directly
* The nth offset of 1 instead of 2. Clojure's nth is 0-based, and our search indexes are 1-based.
====

==== Querying a Tree [[query-tree]]

`query-tree` pipes selectors.
Every selector queries elements from the previous one.
The first selector finds elements from the root, subsquent selectors find elements downward from each of the previous found elements.

Given the following HTML:
[source,html]
----
<div id="query-tree-example">
  <div id="one">
    <a href="#">a1</a>
    <a href="#">a2</a>
    <a href="#">a3</a>
  </div>
  <div id="two">
    <a href="#">a4</a>
    <a href="#">a5</a>
    <a href="#">a6</a>
  </div>
  <div id="three">
    <a href="#">a7</a>
    <a href="#">a8</a>
    <a href="#">a9</a>
  </div>
</div>
----

The following query will find a vector of `div` tags, then return a set of all `a` tags under those `div` tags:

[source,clojure]
----
(->> (e/query-tree driver :query-tree-example {:tag :div} {:tag :a})
     (map #(e/get-element-text-el driver %))
     sort)
;; => ("a1" "a2" "a3" "a4" "a5" "a6" "a7" "a8" "a9")
----

=== Interacting with Queried Elements

To interact with elements found via a `query` or `query-all` function call you have to pass the query result to either `click-el` or `fill-el` (note the `-el` suffix):

[source,clojure]
----
(e/click-el driver (first (e/query-all driver {:tag :a})))
----

You can collect elements into a vector and arbitrarily interact with them at any time:

[source,clojure]
----
(e/refresh driver)
(def elements (e/query-all driver {:tag :input :type :text :fn/disabled false}))

(e/fill-el driver (first elements) "This is a test")
(e/fill-el driver (rand-nth elements) "I like tests!")
----

== Interactions

Some basic interactions are covered under <<querying>>, here we go into other types of interactions and more detail.

=== Emulate how a Real Person Might Type

Real people type slowly and make mistakes.
To emulate these characteristics, you can use the `fill-human` function.
The following options are enabled by default:

[source,clojure]
----
{:mistake-prob 0.1 ;; a real number from 0.1 to 0.9, the higher the number, the more typos will be made
 :pause-max    0.2} ;; max typing delay in seconds
----

which you can choose to override if you wish:

[source,clojure]
----
(e/refresh driver)
(e/fill-human driver :uname "soslowsobad"
              {:mistake-prob 0.5
               :pause-max 1})

;; or just use default options by omitting them
(e/fill-human driver :uname " typing human defaults")
----

For multiple inputs, use `fill-human-multi`

[source,clojure]
----
(e/refresh driver)
(e/fill-human-multi driver {:uname "login"
                            :pw "password"
                            :text "some text"}
                           {:mistake-prob 0.1
                            :pause-max 0.1})
----

=== Mouse Clicks

The `click` function triggers the left mouse click on an element found by a query term:

[source,clojure]
----
(e/click driver {:tag :button})
----

The `click` function uses only the first element found by the query, which sometimes leads to clicking on the wrong items.
To ensure there is one and only one element found, use the `click-single` function.
It acts the same but raises an exception when querying the page returns multiple elements:

[source,clojure]
----
(e/click-single driver {:tag :button :name "submit"})
----

Although double-clicking is rarely purposefully employed on web sites, some naive users might think it is the correct way to click on a button or link.

A double-click can be simulated with `double-click` function (Chrome, Phantom.js).
It can be used, for example, to check your handling of disallowing multiple form submissions.

[source,clojure]
----
(e/double-click driver {:tag :button :name "submit"})
----

There are also "blind" clicking functions.
They trigger mouse clicks on the current mouse position:

[source,clojure]
----
(e/left-click driver)
(e/middle-click driver)
(e/right-click driver)
----

Another set of functions do the same but move the mouse pointer to a specified element before clicking on them:

[source,clojure]
----
(e/left-click-on driver {:tag :a})
(e/middle-click-on driver {:tag :a})
(e/right-click-on driver {:tag :a})
----

A middle mouse click can open a link in a new background tab.
The right click sometimes is used to imitate a context menu in web applications.

=== Selecting an option from a dropdown [[select-dropdown]]

An `<option>` from a `<select>` can be selected via the `click` function.

Given the following HTML:

[source,html]
----
<select id="dropdown" name="options">
  <option value="o1">foo one</option>
  <option value="o2">bar two</option>
  <option value="o3">bar three</option>
  <option value="o4">bar four</option>
</select>
----

Click on option with value `o4`:
[source,clojure]
----
(e/click driver [{:id :dropdown} {:value "o4"}])
(e/get-element-value driver :dropdown)
;; => "o4"
----

Click on option with text `bar three`:
[source,clojure]
----
(e/click driver [{:id :dropdown} {:fn/text "bar three"}])
(e/get-element-value driver :dropdown)
;; => "o3"
----

TIP: Safari Quirk: You might need to first click on the `select` element, then the option.

[NOTE]
====
Etaoin also includes the https://cljdoc.org/d/etaoin/etaoin/CURRENT/api/etaoin.api#select[select] convenience function. It will select the first option from a dropdown that includes the specified text. It also automatically handles the Safari quirk.

Click first matching option with text `bar`:
[source,clojure]
----
(e/select driver :dropdown "bar")
(e/get-element-value driver :dropdown)
;; => "o2"
----

The same operation expressed with `click`:
[source,clojure]
----
(e/click driver :dropdown) ;; needed for Safari quirk only
(e/click driver [{:id :dropdown} {:fn/has-text "bar"}])
(e/get-element-value driver :dropdown)
;; => "o2"
----
====

=== Keyboard Chords

There is an option to input a series of keys simultaneously.
This useful to imitate holding a system key like Control, Shift or whatever when typing.

The namespace `etaoin.keys` includes key constants as well as a set of functions related to keyboard input.

[source,clojure]
----
(require '[etaoin.keys :as k])
----

A quick example of entering ordinary characters while holding Shift:

[source,clojure]
----
(e/refresh driver)
(e/wait 1) ;; maybe we need a sec for active element to focus
(e/fill-active driver (k/with-shift "caps is great"))
(e/get-element-value driver :active)
;; => "CAPS IS GREAT"
----

The main input gets populated with "CAPS IS GREAT".
Let's duplicate the text via select-all, copy, and paste keyboard shortcuts:

[source,clojure]
----
(if (= "Mac OS X" (System/getProperty "os.name"))
  (e/fill-active driver (k/with-command "a") (k/with-command "c") k/arrow-right " " (k/with-command "v"))
  (e/fill-active driver (k/with-ctrl "a") (k/with-ctrl "c") k/arrow-right " " (k/with-ctrl "v")))
(e/get-element-value driver :active)
;; => "CAPS IS GREAT CAPS IS GREAT"
----

And now let's clear the input by:
1. moving the cursor to the beginning of the input field with the home key
2. moving the cursor to the end field while holding shift to select all text
3. deleting the selected text with the delete key

[source,clojure]
----
(e/fill-active driver k/home (k/with-shift k/end) k/delete)
(e/get-element-value driver :active)
;; => ""
----

NOTE: These functions do not apply to the global browser's shortcuts.
For example, neither "Command + R" nor "Command + T" reload the page or open a new tab.

The `etaoin.keys/with-*` functions are just wrappers for the `etaoin.keys/chord` function that might be used for complex cases.

=== File Uploading

Clicking on a file input button opens an OS-specific dialog.
You technically cannot interact with this dialog using the WebDriver protocol.
Use the `upload-file` function to attach a local file to a file input widget.
An exception will be thrown if the local file is not found.

[source,clojure]
----
;; open a web page that serves uploaded files
(e/go driver "http://nervgh.github.io/pages/angular-file-upload/examples/simple/")

;; bind element selector to variable; you may also specify an id, class, etc
(def file-input {:tag :input :type :file})

;; upload a file form your system to the first file input
(def my-file "env/test/resources/html/drag-n-drop/images/document.png")
(e/upload-file driver file-input my-file)

;; or pass a native Java File object:
(require '[clojure.java.io :as io])
(def my-file (io/file "env/test/resources/html/drag-n-drop/images/document.png"))
(e/upload-file driver file-input my-file)
----

When interacting with a remote WebDriver process, you'll need to avoid the local file existence check by using `remote-file` like so:

//:test-doc-blocks/skip
[source,clojure]
----
(e/upload-file driver file-input (e/remote-file "/yes/i/really/do/exist.png"))
----
The remote file is assumed to exist where the WebDriver is running.
The WebDriver will throw an error if it does not exist.

=== Scrolling

Etaoin includes functions to scroll the web page.

The most important one, `scroll-query` jumps the the first element found with the query term:

[source,clojure]
----
(e/go driver sample-page)
;; scroll to the 5th h2 heading
(e/scroll-query driver {:tag :h2} {:index 5})

;; and back up to first h1
(e/scroll-query driver {:tag :h1})
----

To jump to the absolute pixel positions, use `scroll`:

[source,clojure]
----
(e/scroll driver 100 600)
;; or pass a map with x and y keys
(e/scroll driver {:x 100 :y 600})
----

To scroll relatively by pixels, use `scroll-by` with offset values:

[source,clojure]
----
;; scroll right by 100 and down by 300
(e/scroll-by driver 100 300)
;; use map syntax to scroll left by 50 and up by 200
(e/scroll-by driver {:x -50 :y -200})
----

There are two convenience functions to scroll vertically to the top or bottom of the page:

[source,clojure]
----
(e/scroll-bottom driver) ;; you'll see the footer...
(e/scroll-top driver)    ;; ...and the header again
----

The following functions scroll the page in all directions:

[source,clojure]
----
(e/scroll driver [0 0])     ;; let's start at top left

(e/scroll-down driver 200)  ;; scrolls down by 200 pixels
(e/scroll-down driver)      ;; scrolls down by the default (100) number of pixels

(e/scroll-up driver 200)    ;; the same, but scrolls up...
(e/scroll-up driver)

(e/scroll-right driver 200) ;; ... and right
(e/scroll-right driver)

(e/scroll-left driver 200)  ;; ...left
(e/scroll-left driver)

----

NOTE: All scroll actions are carried out via Javascript.
Ensure your browser has it enabled.

=== Working with frames and iframes

You can only interact with items within an individual frame or iframe by first swithing to them.

Say you have an HTML layout like this:

[source,html]
----
<iframe id="frame1" src="...">
  <p id="in-frame1">In frame2 paragraph</p>
  <iframe id="frame2" src="...">
    <p id="in-frame2">In frame2 paragraph</p>
  </iframe>
</iframe>
----

Let's explore switching to `:frame1`.

[source,clojure]
----
(e/go driver sample-page)
;; we start in the main page, we can't see inside frame1:
(e/exists? driver :in-frame1)
;; => false

;; switch context to frame with id of frame1:
(e/switch-frame driver :frame1)

;; now we can interact with elements in frame1:
(e/exists? driver :in-frame1)
;; => true
(e/get-element-text driver :in-frame1)
;; => "In frame1 paragraph"

;; switch back to top frame (the main page)
(e/switch-frame-top driver)
----

To reach nested frames, you can dig down like so:

[source,clojure]
----
;; switch to the first top-level iframe with the main page: frame1
(e/switch-frame-first driver)
;; downward to the first iframe with frame1: frame2
(e/switch-frame-first driver)
(e/get-element-text driver :in-frame2)
;; => "In frame2 paragraph"
;; back up to frame1
(e/switch-frame-parent driver)
;; back up to main page
(e/switch-frame-parent driver)
----

Use the `with-frame` macro to temporarily switch to a target frame, do some work, returning its last expression, while preserving your original frame context.

[source,clojure]
----
(e/with-frame driver {:id :frame1}
  (e/with-frame driver {:id :frame2}
    (e/get-element-text driver :in-frame2)))
;; => "In frame2 paragraph"
----

=== Executing Javascript

Use `js-execute` to evaluate a Javascript code in the browser:

[source,clojure]
----
(e/js-execute driver "alert('Hello from Etaoin!')")
(e/dismiss-alert driver)
----

Pass any additional parameters to the script with the `arguments` array-like object.
[source,clojure]
----
(e/js-execute driver "alert(arguments[2].foo)" 1 false {:foo "hello again!"})
(e/dismiss-alert driver)
----

We have passed 3 arguments:

. `1`
. `false`
. `{:foo "hello again!}` which is automatically converted to JSON `{"foo": "hello again!"}`

The alert then presents the `foo` field of the 3rd (index 2) argument, which is `"hello again!"`.

To return any data to Clojure, add `return` into your script:

[source,clojure]
----
(e/js-execute driver "return {foo: arguments[2].foo, bar: [1, 2, 3]}"
                     ;; same args as previous example:
                     1 false {:foo "hello again!"})
;; => {:bar [1 2 3], :foo "hello again!"}
----

Notice that the JSON has been automatically converted to edn.

==== Asynchronous Scripts

Use `js-async` to deal with scripts that rely on async strategies such as `setTimeout`.
The WebDriver creates and passes a callback as the last argument to your script.
To indicate that work is complete, you must call this callback.

Example:

[source,clojure]
----
(e/js-async
  driver
  "var args = arguments; // preserve the global args
  // WebDriver added the callback as the last arg, we grab it here
  var callback = args[args.length-1];
  setTimeout(function() {
    // We call the WebDriver callback passing with what we want it to return
    // In this case we pass we chose to return 42 from the arg we passed in
    callback(args[0].foo.bar.baz);
  },
  1000);"
  {:foo {:bar {:baz 42}}})
;; => 42
----

If you'd like to override the default script timeout, you can do so for the WebDriver session:

[source,clojure]
----
;; optionally save the current value for later restoration
(def orig-script-timeout (e/get-script-timeout driver))
(e/set-script-timeout driver 5) ;; in seconds
;; do some stuff
(e/set-script-timeout driver orig-script-timeout)
----

or for a block of code via `with-script-timeout`:

//:test-doc-blocks/skip
[source,clojure]
----
(e/with-script-timeout driver 30
  (e/js-async driver "var callback = arguments[arguments.length-1];
                      //some long operation here
                      callback('phew,done!');"))
;; => "phew,done!"
----

=== Wait Functions

The main difference between a program and a human being is that the first one operates very fast.
A computer operates so fast, that sometimes a browser cannot render new HTML in time.
After each action, you might consider including a `wait-<something>` function that polls a browser until the predicate evaluates to true.
Or just `(wait <seconds>)` if you don't care about optimization.

The `with-wait` macro might be helpful when you need to prepend each action with `(wait n)`.
For example, the following form:

[source,clojure]
----
(e/with-wait 1
  (e/refresh driver)
  (e/fill driver :uname "my username")
  (e/fill driver :text "some text"))
----

is executed something along the lines of:

[source,clojure]
----
(e/wait 1)
(e/refresh driver)
(e/wait 1)
(e/fill driver :uname "my username")
(e/wait 1)
(e/fill driver :text "some text")
----

and thus returns the result of the last form of the original body.

The `(doto-wait n driver & body)` acts like the standard `doto` but prepends each form with `(wait n)`.
The above example re-expressed with `doto-wait`:

[source,clojure]
----
(e/doto-wait 1 driver
  (e/refresh)
  (e/fill :uname "my username")
  (e/fill :text "some text"))
----

This is effectively the same as:

[source,clojure]
----
(doto driver
  (e/wait 1)
  (e/refresh)
  (e/wait 1)
  (e/fill :uname "my username")
  (e/wait 1)
  (e/fill :text "some text"))
----

In addition to `with-wait` and `do-wait` there are a number of waiting functions: `wait-visible`, `wait-has-alert`, `wait-predicate`, etc (see the full list in the link:{url-doc}/CURRENT/api/etaoin.api#wait[API docs].
They accept default timeout/interval values that can be redefined using the `with-wait-timeout` and `with-wait-interval` macros, respectively.
They all throw if the wait timeout is exceeded.

[source,clojure]
----
(e/with-wait-timeout 15 ;; time in seconds
  (doto driver
    (e/refresh)
    (e/wait-visible {:id :last-section})
    (e/click {:tag :a})
    (e/wait-has-text :clicked "link 1")))
----

Wait text:

* `wait-has-text` waits until an element has text anywhere inside it (including inner HTML).
+
[source,clojure]
----
(e/click driver {:tag :a})
(e/wait-has-text driver :clicked "link 1")
----

* `wait-has-text-everywhere` like `wait-has-text` but searches for text across the entire page
+
[source,clojure]
----
(e/wait-has-text-everywhere driver "ipsum")
----

=== Load Strategy [[load-strategy]]

When you navigate to a page, the driver waits until the whole page has been completely loaded.
That's fine in most cases but doesn't reflect the way human beings interact with the Internet.

Change this default behavior with the `:load-strategy` option:

* `:normal` (the default) wait for full page load (everything, include images, etc)
* `:none` don't wait at all
* `:eager` wait for only DOM content to load

For example, the default `:normal` strategy:

[source,clojure]
----
(e/with-chrome driver
  (e/go driver sample-page)
  ;; by default you'll hang on this line until the page loads
  ;; (do-something)
)
----

Load strategy option of `:none`:

[source,clojure]
----
(e/with-chrome {:load-strategy :none} driver
  (e/go driver sample-page)
  ;; no pause, no waiting, acts immediately
  ;; (do-something)
)
----

The `:eager` option only works with Firefox at the moment.

=== Actions [[actions]]

Etaoin supports link:{actions}[Webdriver Actions].
They are described as "virtual input devices".
They act as little device input scripts that run simultaneously.

Here, in raw form, we have an example of two actions.
One controls the keyboard, the other the pointer (mouse).

[source,clojure]
----
;; a keyboard input
{:type    "key"
 :id      "some name"
 :actions [{:type "keyDown" :value "a"}
           {:type "keyUp" :value "a"}
           {:type "pause" :duration 100}]}
;; some pointer input
{:type       "pointer"
 :id         "UUID or some name"
 :parameters {:pointerType "mouse"}
 :actions    [{:type "pointerMove" :origin "pointer" :x 396 :y 323}
              ;; double click
              {:type "pointerDown" :duration 0 :button 0}
              {:type "pointerUp" :duration 0 :button 0}
              {:type "pointerDown" :duration 0 :button 0}
              {:type "pointerUp" :duration 0 :button 0}]}
----

You can create a map manually and send it to the `perform-actions` method:

[source,clojure]
----
(def keyboard-input {:type    "key"
                     :id      "some name"
                     :actions [{:type "keyDown" :value "e"}
                               {:type "keyUp" :value "e"}
                               {:type "keyDown" :value "t"}
                               {:type "keyUp" :value "t"}
                               ;; duration is in ms
                               {:type "pause" :duration 100}]})
;; refresh so that we'll be at the active input field
(e/refresh driver)
;; perform our keyboard input action
(e/perform-actions driver keyboard-input)
----

Or you might choose to use Etaoin's action helpers.
First you create the virtual input device:

[source,clojure]
----
(def keyboard (e/make-key-input))
----

and then fill it with the actions:

[source,clojure]
----
(-> keyboard
    (e/add-key-down k/shift-left)
    (e/add-key-down "a")
    (e/add-key-up "a")
    (e/add-key-up k/shift-left))
----

Here's a slightly larger working annotated example:

[source,clojure]
----
;; virtual inputs run simultaneously so we'll create a little helper to generate n pauses
(defn add-pauses [input n]
  (->> (iterate e/add-pause input)
       (take (inc n))
       last))

(let [username (e/query driver :uname)
      submit-button (e/query driver {:tag :button})
      mouse (-> (e/make-mouse-input)
                ;; click on username
                (e/add-pointer-click-el
                  username k/mouse-left)
                ;; pause 10 clicks to allow keyboard action to enter username
                ;; (key up and down for each of keypress for etaoin)
                (add-pauses 10)
                ;; click on submit button
                (e/add-pointer-click-el
                  submit-button k/mouse-left))
      keyboard (-> (e/make-key-input)
                   ;; pause 2 ticks to allow mouse action to first click on username
                   ;; (move to username element + click on it)
                   (add-pauses 2)
                   (e/with-key-down k/shift-left
                     (e/add-key-press "e"))
                   (e/add-key-press "t")
                   (e/add-key-press "a")
                   (e/add-key-press "o")
                   (e/add-key-press "i")
                   (e/add-key-press "n")) ]
  (e/perform-actions driver keyboard mouse))
----

To clear the state of virtual input devices, release all currently pressed keys etc, use the `release-actions` method:

[source,clojure]
----
(e/release-actions driver)
----

== Capturing Screenshots

Calling the `screenshot` function dumps the current visible page into a PNG image file on your disk.
Specify any absolute or relative path.
Specify a string:

[source,clojure]
----
(e/screenshot driver "target/etaoin-play/screens1/page.png")
----

or a `File` object:

[source,clojure]
----
(require '[clojure.java.io :as io])
(e/screenshot driver (io/file "target/etaoin-play/screens2/test.png"))
----

=== Screenshots for Specific Elements

With Firefox and Chrome, you can also capture a single element within a page, say a div, an input widget, or whatever.
It doesn't work with other browsers at this time.

[source,clojure]
----
(e/screenshot-element driver {:tag :form :class :formy} "target/etaoin-play/screens3/form-element.png")
----

=== Screenshots after each form

Use `with-screenshots` to take a screenshot to the specified directory after each form is executed in the code block.
The file naming convention is `<webdriver-name>-<milliseconds-since-1970>.png`

[source,clojure]
----
(e/refresh driver)
(e/with-screenshots driver "target/etaoin-play/saved-screenshots"
  (e/fill driver :uname "et")
  (e/fill driver :uname "ao")
  (e/fill driver :uname "in"))
----

this is equivalent to something along the lines of:

[source,clojure]
----
(e/refresh driver)
(e/fill driver :uname "et")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-1.png")
(e/fill driver :uname "ao")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-2.png")
(e/fill driver :uname "in")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-3.png")
----

== Print Page to PDF

Use `print-page` to print the current page to a PDF file:

[source,clojure]
----
(e/with-firefox-headless driver
  (e/go driver sample-page)
  (e/print-page driver "target/etaoin-play/printed.pdf"))
----

See link:{url-doc}/CURRENT/api/etaoin.api#print-page[API docs] for details.

== Peeking deeper

Sometimes it is useful to go a little deeper.

=== Invoking WebDriver Implementation Specific Features

The Etaoin API exposes an abstraction of the W3C WebDriver protocol.
This is normally all you need, but sometimes you'll want to invoke a WebDriver implementation feature that is not part of the WebDriver protocol.

Etaoin talks to the WebDriver process via its `execute` function.
You can use this lower level function to send whatever you like to the WebDriver process.

As a real-world example, Chrome supports taking screenshots with transparent backgrounds.

Here we use Etaoin's `execute` function to ask Chrome to do this:

[source,clojure]
----
(e/with-chrome driver
  ;; navigate to our sample page
  (e/go driver sample-page)
  ;; send the Chrome-specific request for a transparent background
  (e/execute {:driver driver
              :method :post
              :path [:session (:session driver) "chromium" "send_command_and_get_result"]
              :data {:cmd "Emulation.setDefaultBackgroundColorOverride"
                     :params {:color {:r 0 :g 0 :b 0 :a 0}}}})
  ;; and here we take an element screenshot as per normal
  (e/screenshot-element driver
                        {:tag :form}
                        (str "target/etaoin-play/saved-screenshots/form.png")))
----

=== Reading a Browser's Console Logs [[console-logs]]

Function `get-logs` returns the browser's console logs as a vector of maps.
Each map has the following structure:

// note that we do not verify get-logs output with test-doc-blocks by omitting =>
[source,clojure]
----
(e/js-execute driver "console.log('foo')")
(e/get-logs driver)
;; [{:level :info,
;;   :message "console-api 2:32 \"foo\"",
;;   :source :console-api,
;;   :timestamp 1654358994253,
;;   :datetime #inst "2022-06-04T16:09:54.253-00:00"}]

;; on the 2nd call, for chrome, we'll find the logs empty
(e/get-logs driver)
;; => []
----

Currently, logs are available in Chrome and Phantom.js only.
The message text and the source type will vary by browser vendor.
Chrome wipes the logs once they have been read.
Phantom.js wipes the logs when the page location changes.

=== DevTools: Tracking HTTP Requests, XHR (Ajax) [[devtools]]

You can trace events that come from the DevTools panel.
This means that everything you see in the developer console now is available through the Etaoin API.
This currently only works for Google Chrome.

To start a driver with devtools support enabled specify a `:dev` map.

//let's put this driver in its own namespace
//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
(require '[etaoin.api :as e])

(e/with-chrome driver {:dev {}}
  ;; do some stuff
)
----

The value must not be a map (not `nil`).
When `:dev` an empty map, the following defaults are used.

[source,clojure]
----
{:perf
 {:level :all
  :network? true
  :page? false
  :categories [:devtools.network]
  :interval 1000}}
----

We'll work with a driver that enables everything:

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
(require '[etaoin.api :as e])

(def driver (e/chrome {:dev
                       {:perf
                        {:level :all
                         :network? true
                         :page? true
                         :interval 1000
                         :categories [:devtools
                                      :devtools.network
                                      :devtools.timeline]}}}))
----

Under the hood, Etaoin sets up a special `perfLoggingPrefs` dictionary inside the `chromeOptions` object.

Now that your browser is accumulating these events, you can read them using a special `dev` namespace.

The results will be different when you try this, but here's what I experienced:

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
(require '[etaoin.dev :as dev])

(e/go driver "https://google.com")

(def reqs (dev/get-requests driver))

;; reqs is a vector of maps
(count reqs)
;; 23

;; what were the request types?
(frequencies (map :type reqs))
;; {:script 6,
;;  :other 2,
;;  :xhr 4,
;;  :image 5,
;;  :stylesheet 1,
;;  :ping 3,
;;  :document 1,
;;  :manifest 1}

;; Interesting, we've got Js requests, images, AJAX and other stuff
----

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
;; let's take a peek at the last image:
(last (filter #(= :image (:type %)) reqs))
;;    {:state 4,
;;     :id "14535.6",
;;     :type :image,
;;     :xhr? false,
;;     :url
;;     "https://www.google.com/images/searchbox/desktop_searchbox_sprites318_hr.webp",
;;     :with-data? nil,
;;     :request
;;     {:method :get,
;;      :headers
;;      {:Referer "https://www.google.com/?gws_rd=ssl",
;;       :sec-ch-ua-full-version-list
;;       "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;       :sec-ch-viewport-width "1200",
;;       :sec-ch-ua-platform-version "\"10.15.7\"",
;;       :sec-ch-ua
;;       "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;       :sec-ch-ua-platform "\"macOS\"",
;;       :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;       :sec-ch-ua-wow64 "?0",
;;       :sec-ch-ua-model "",
;;       :sec-ch-ua-bitness "\"64\"",
;;       :sec-ch-ua-mobile "?0",
;;       :sec-ch-dpr "1",
;;       :sec-ch-ua-arch "\"x86\"",
;;       :User-Agent
;;       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;     :response
;;     {:status nil,
;;      :headers
;;      {:date "Sat, 04 Jun 2022 00:11:36 GMT",
;;       :x-xss-protection "0",
;;       :x-content-type-options "nosniff",
;;       :server "sffe",
;;       :cross-origin-opener-policy-report-only
;;       "same-origin; report-to=\"static-on-bigtable\"",
;;       :last-modified "Wed, 22 Apr 2020 22:00:00 GMT",
;;       :expires "Sat, 04 Jun 2022 00:11:36 GMT",
;;       :cache-control "private, max-age=31536000",
;;       :content-length "660",
;;       :report-to
;;       "{\"group\":\"static-on-bigtable\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/static-on-bigtable\"}]}",
;;       :alt-svc
;;       "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;       :cross-origin-resource-policy "cross-origin",
;;       :content-type "image/webp",
;;       :accept-ranges "bytes"},
;;      :mime "image/webp",
;;      :remote-ip "142.251.41.68"},
;;     :done? true}
----

TIP: The details of these responses come from Chrome and are subject to changes to Chrome.

Since we're mostly interested in AJAX requests, there is a function `get-ajax` that does the same but filters XHR requests:

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
;; refresh to fill the logs again
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish

(last (dev/get-ajax driver))
;; {:state 4,
;;  :id "14535.59",
;;  :type :xhr,
;;  :xhr? true,
;;  :url
;;    "https://www.google.com/complete/search?q&cp=0&client=gws-wiz&xssi=t&hl=en-CA&authuser=0&psi=OtuaYq-xHNeMtQbkjo6gBg.1654315834852&nolsbt=1&dpr=1",
;;  :with-data? nil,
;;  :request
;;  {:method :get,
;;   :headers
;;   {:Referer "https://www.google.com/",
;;    :sec-ch-ua-full-version-list
;;    "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;    :sec-ch-viewport-width "1200",
;;    :sec-ch-ua-platform-version "\"10.15.7\"",
;;    :sec-ch-ua
;;    "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;    :sec-ch-ua-platform "\"macOS\"",
;;    :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;    :sec-ch-ua-wow64 "?0",
;;    :sec-ch-ua-model "",
;;    :sec-ch-ua-bitness "\"64\"",
;;    :sec-ch-ua-mobile "?0",
;;    :sec-ch-dpr "1",
;;    :sec-ch-ua-arch "\"x86\"",
;;    :User-Agent
;;    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;  :response
;;  {:status nil,
;;   :headers
;;   {:bfcache-opt-in "unload",
;;    :date "Sat, 04 Jun 2022 04:10:35 GMT",
;;    :content-disposition "attachment; filename=\"f.txt\"",
;;    :x-xss-protection "0",
;;    :server "gws",
;;    :expires "Sat, 04 Jun 2022 04:10:35 GMT",
;;    :accept-ch
;;    "Sec-CH-Viewport-Width, Sec-CH-Viewport-Height, Sec-CH-DPR, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version, Sec-CH-UA-Arch, Sec-CH-UA-Model, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version-List, Sec-CH-UA-WoW64",
;;    :cache-control "private, max-age=3600",
;;    :report-to
;;    "{\"group\":\"gws\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/gws/cdt1\"}]}",
;;    :x-frame-options "SAMEORIGIN",
;;    :strict-transport-security "max-age=31536000",
;;    :content-security-policy
;;    "object-src 'none';base-uri 'self';script-src 'nonce-xM7BqmSpeu5Zd6usKOP4JA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/cdt1",
;;    :alt-svc
;;    "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;    :content-type "application/json; charset=UTF-8",
;;    :cross-origin-opener-policy "same-origin-allow-popups; report-to=\"gws\"",
;;    :content-encoding "br"},
;;   :mime "application/json",
;;   :remote-ip "142.251.41.36"},
;;  :done? true};; => nil
----

A typical pattern of `get-ajax` usage is the following.
You'd like to check if a certain request has been fired to the server.
So you press a button, wait for a while, and then read the requests made by your browser.

Having a list of requests, you search for the one you need (e.g. by its URL) and then check its state.
The `:state` field has got the same semantics of the `XMLHttpRequest.readyState`.
It's an integer from 1 to 4 with the same behavior.

To check if a request has been finished, done or failed, use these predicates:

// Having too many failures on CI with this one, skip for now
//:test-doc-blocks/skip
[source,clojure]
----
;; fill the logs
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish

(def reqs (dev/get-ajax driver))
;; you'd search for what you are interested in here
(def req (last reqs))

(dev/request-done? req)
;; => true

(dev/request-failed? req)
;; => nil

(dev/request-success? req)
;; => true
----

Note that `request-done?` doesn't mean the request has succeeded.
It only means its pipeline has reached a final step.

TIP: when you read dev logs, you consume them from an internal buffer that gets flushed.
The second call to `get-requests` or `get-ajax` will return an empty list.

Perhaps you want to collect these logs.
A function `dev/get-performance-logs` return a list of logs so you accumulate them in an atom or whatever:

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
;; setup a collector
(def logs (atom []))

;; make requests
(e/refresh driver)

;; collect as needed
(do (swap! logs concat (dev/get-performance-logs driver))
    true)

(count @logs)
;; 136
----

The `+logs->requests+` and `+logs->ajax+` functions convert already fetched logs into requests.
Unlike `get-requests` and `get-ajax`, they are pure functions and won't flush anything.

//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
;; convert our fetched requests from our collector atom
(dev/logs->requests @logs)
(last (dev/logs->requests @logs))
;;    {:state 4,
;;     :id "14535.162",
;;     :type :ping,
;;     :xhr? false,
;;     :url
;;     "https://www.google.com/gen_204?atyp=i&r=1&ei=Zd2aYsrzLozStQbzgbqIBQ&ct=slh&v=t1&m=HV&pv=0.48715273690818806&me=1:1654316389931,V,0,0,1200,1053:0,B,1053:0,N,1,Zd2aYsrzLozStQbzgbqIBQ:0,R,1,1,0,0,1200,1053:93,x:42832,e,U&zx=1654316432856",
;;     :with-data? true,
;;     :request
;;     {:method :post,
;;      :headers
;;      {:Referer "https://www.google.com/",
;;       :sec-ch-ua-full-version-list
;;       "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;       :sec-ch-viewport-width "1200",
;;       :sec-ch-ua-platform-version "\"10.15.7\"",
;;       :sec-ch-ua
;;       "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;       :sec-ch-ua-platform "\"macOS\"",
;;       :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;       :sec-ch-ua-wow64 "?0",
;;       :sec-ch-ua-model "",
;;       :sec-ch-ua-bitness "\"64\"",
;;       :sec-ch-ua-mobile "?0",
;;       :sec-ch-dpr "1",
;;       :sec-ch-ua-arch "\"x86\"",
;;       :User-Agent
;;       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;     :response
;;     {:status nil,
;;      :headers
;;      {:alt-svc
;;       "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;       :bfcache-opt-in "unload",
;;       :content-length "0",
;;       :content-type "text/html; charset=UTF-8",
;;       :date "Sat, 04 Jun 2022 04:20:32 GMT",
;;       :server "gws",
;;       :x-frame-options "SAMEORIGIN",
;;       :x-xss-protection "0"},
;;      :mime "text/html",
;;      :remote-ip "142.251.41.36"},
;;     :done? true}
----

When working with logs and requests, pay attention to their count and size.
The maps have plenty of keys and the number of items in collections can become very large.
Printing a slew of events might freeze your editor.
Consider using `clojure.pprint/pprint` as it relies on max level and length limits.

// hidden cleanup of our devtools driver
ifdef::env-test-doc-blocks[]
//{:test-doc-blocks/test-ns user-guide-devtools-test}
[source,clojure]
----
(e/quit driver)
----
endif::[]

=== Postmortem: Auto-save Artifacts in Case of Exception [[postmortem]]

Sometimes, it can be difficult to diagnose what went wrong during a failed UI test run.
Use the `with-postmortem` to save useful data to disk before the exception was triggered:

* a screenshot of the visible browser page
* HTML code of the current browser page
* JS console logs, <<console-logs,if available for your browser>>

Example:

[source,clojure]
----
(try
  (e/with-postmortem driver {:dir "target/etaoin-play/postmortem"}
    (e/click driver :non-existing-element))
  (catch Exception _e
    "yup, we threw!"))
;; => "yup, we threw!"
----

An exception will occur. Under `target/etaoin-postmortem` you will find three postmortem files named like so: `<browser>-<host>-<port>-<datetime>.<ext>`, for example:

[source,shell]
----
$ tree target
target
└── etaoin-postmortem
    ├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.html
    ├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.json
    └── chrome-127.0.0.1-49766-2022-06-04-12-26-31.png
----

The available `with-postmortem` options are:

[source,clojure]
----
{;; directory to save artifacts
 ;; will be created if it does not already exist, defaults to current working directory
 :dir "/home/ivan/UI-tests"

 ;; directory to save screenshots; defaults to :dir
 :dir-img "/home/ivan/UI-tests/screenshots"

 ;; the same but for HTML sources
 :dir-src "/home/ivan/UI-tests/HTML"

 ;; the same but for console logs
 :dir-log "/home/ivan/UI-tests/console"

 ;; a string template to format a timestamp; See SimpleDateFormat Java class
 :date-format "yyyy-MM-dd-HH-mm-ss"}
----

== Driver Options [[driver-options]]

When creating a driver instance, a map of additional parameters can optionally be passed to tweak the WebDriver and web browser behaviour.

Here, for example, we set an explicit path to the chrome WebDriver binary:

//:test-doc-blocks/skip
[source,clojure]
----
(def driver (e/chrome {:path-driver "/Users/ivan/downloads/chromedriver"}))
----

[cols="70,30"]
|===
| Option | Defaults

a|`:host` for *WebDriver* process. When:

* omitted, creates a new local WebDriver process (unless `:webdriver-url` was specified).
* specified, attempts to connect to an existing running WebDriver process.
See <<connecting-existing>>.

Alternative: see `:webdriver-url` below.

Example: `:host "192.68.1.12"`
| <not set>

|`:port` for *WebDriver* process.
If `:port` is found to  already in use when creating a new local WebDriver process (see `:host`), a random port will be automatically selected. +
See also <<connecting-existing>>.

Example: `:port 9997`
a| Random port when lanching local WebDriver process, else varies by vendor:

* chrome `9515`
* firefox `4444`
* safari `4445`
* edge `17556`
* phantom `8910`

a| `:webdriver-url` for *WebDriver* process. When:

* omitted, creates a new local WebDriver process (unless `:host` was specified).
* specified, attempts to connect to an existing running WebDriver process. 

See <<connecting-existing>>.

Alternative: see `:host` above.

Example: `"https://chrome.browserless.io/webdriver"`

| <not set>

| `:path-driver` to *WebDriver* binary. +
Typically used if your WebDriver is not on the PATH.

Example:
`:path-driver "/Users/ivan/Downloads/geckodriver"`
a| As you would expect, varies by vendor:

* chrome `"chromedriver"`
* firefox `"geckodriver"`
* safari `"safaridriver"`
* edge `"msedgedriver"`
* phantom `"phantomjs"`

| `:args-driver` specifies extra command line arguments to *WebDriver*.

Example: `:args-driver ["-b" "/path/to/firefox/binary"]`
| <not set>

| `:webdriver-failed-launch-retries`

Introduced to compensate for mysterious but recoverable failed launches of safari driver.
a| - safari `4` (for a total of 5 tries)
- all other drivers `0`

| `:path-browser` to *web browser* binary. +
Typically used if your browser is not on the PATH.

Example: `:path-browser "/Users/ivan/Downloads/firefox/firefox"`
| By default, the WebDriver process automatically finds the web browser.

| `:args` specifies extra command line arguments to *web browser*, see your web browser docs for what is available.

Example: `:args ["--incognito" "--app" "http://example.com"]`
| <not set>

a| `:log-level` *web browser* minimal console log level.
Only messages with this level and above will be collected.
From least to most verbose:

* `nil`, `:off` or `:none` for no messages
* `:err`, `:error`, `:severe`, `:crit` or `:critical`
* `:warn` or `:warning`
* `:debug`
* `:all` for all messages.

See <<console-logs>>

Example: `:log-level :err`

| `:all`

a| `driver-log-level` *WebDriver* minimal log level.
values vary by browser driver vendor:

* chrome `"OFF"` `"SEVERE"` `"WARNING"` `"INFO"` or `"DEBUG"`
* firefox `"fatal"` `"error"` `"warn"` `"info"` `"config"` `"debug"` or `"trace"`
* phantomjs `"ERROR"` `"WARN"` `"INFO"` `"DEBUG"`

Example: `:driver-log-level "INFO"`

a| * phantomjs `"INFO"`

a| `:log-stdout` and `:log-stderr` *WebDriver* stdout and stderr log files

Specify `:inherit` to have WebDriver process output destination inherit from calling process (for example, the console).

Example:
[source,clojure]
----
  :log-stdout "target/chromedriver-out.log"
  :log-stderr "target/chrmoedriver-err.log"
----
| `/dev/null`, on Windows `NUL`

| `:profile` path to custom *web browser* profile, see <<browser-profile>>

Example: +
`:profile "/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test"`

| <not set>

| `:env` map of environment variables for *WebDriver* process.

Example: `:env {:MOZ_CRASHREPORTER_URL "http://test.com"}`
| <not set>

| `:size` initial *web browser* window width and height in pixels

Example: `size: [640 480]`
| [1024 680]

| `:url` default URL to open in *web browser*.+
Only works in Firefox at this time.

Example: `:url "https://clojure.org"`
| <not set>

| `:user-agent` overrides the *web browser* `User-Agent`.
Useful for headless mode.
See <<user-agent>>.

Example: `:user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"`
| Default is governed by WebDriver vendor.

| `:download-dir` directory for *web browser* downloads files.
See <<download-dir>>

Example: `:download-dir "target/chrome-downloads"`
| Default is governed by browser vendor.

| `:headless` run the *web browser* without a UI.
See <<headless>>.

Example `:headless true`
| Normally `false`, but automatically set for driver creation functions like `chrome-headless`, `with-chrome-headless` etc.

| `:prefs` map of *web browser* specific preferences.

Example: see one usage in <<download-dir>>.
| <not set>

| `:proxy` to set *web browser* proxy.

Example: see <<http-proxy>>.
| <not set>

| `:load-strategy` controls how long the *WebDriver* should wait before interacting with a page.
See <<load-strategy>>.

Example: `:load-strategy :none`
| `:normal`

| `:capabilities` *WebDriver*-specific options.
Read vendor docs for WebDriver before setting anything here.
You'll find an example usage under <<http-proxy>>.
| <none>

|===

=== Using Headless Drivers [[headless]]

Google Chrome, Firefox, and Microsoft Edge can be run in headless mode.
When headless, none of the UI windows appear on the screen.
Running without a UI is helpful when:

* running integration tests on servers that do not have a graphical output device
* running local tests without having them take over your local UI

Ensure your browser supports headless mode by checking if it accepts `--headless` command-line argument when running it from the terminal.
The Phantom.js driver is headless by its nature (it was never been developed for rendering UI).

When starting a driver, pass the `:headless` boolean flag to switch into headless mode.
This flag is ignored for Safari which, as of June 2022, still does not support headless mode.

//{:test-doc-blocks/test-ns user-guide-headless-test}
[source,clojure]
----
(require '[etaoin.api :as e])

(def driver (e/chrome {:headless true})) ;; runs headless Chrome
;; do some stuff
(e/quit driver)
----

or

//{:test-doc-blocks/test-ns user-guide-headless-test}
[source,clojure]
----
(def driver (e/firefox {:headless true})) ;; runs headless Firefox
;; you can also check if a driver is in headless mode:
(e/headless? driver)
;; => true
(e/wait 1) ;; seems to appease Firefox on Linux
(e/quit driver)
----

NOTE: PhantomJS will always be in headless mode.

There are several shortcuts to run Chrome or Firefox in headless mode:

//{:test-doc-blocks/test-ns user-guide-headless-test}
[source,clojure]
----
(def driver (e/chrome-headless))
;; do some stuff
(e/quit driver)

;; or

(def driver (e/firefox-headless {:log-level :all})) ;; with extra settings
;; do some stuff
(e/quit driver)

;; or

(require '[etaoin.api :as e])

(e/with-chrome-headless driver
  (e/go driver "https://clojure.org"))

(e/with-firefox-headless {:log-level :all} driver ;; notice extra settings
  (e/go driver "https://clojure.org"))
----

There are also the `when-headless` and `when-not-headless` macros that conditonally execute a block of commands:

//{:test-doc-blocks/test-ns user-guide-headless-test}
[source,clojure]
----
(e/with-chrome driver
  (e/when-not-headless driver
    ;;... some actions that might be not available in headless mode
    )
  ;;... common actions for both versions
  )
----

=== File Download Directory [[download-dir]]

To specify a directory where the browser should download files, use the `:download-dir` option:

//:test-doc-blocks/skip
[source,clojure]
----
(def driver (e/chrome {:download-dir "target/etaoin-play/chrome-downloads"}))
;; do some downloading
(e/driver quit)
----

Now, when you click on a download link, the file will be saved to that folder.
Currently, only Chrome and Firefox are supported.

Firefox requires specifying MIME-types of the files that should be downloaded without showing a system dialog.
By default, when the `:download-dir` parameter is passed, the library adds the most common MIME-types: archives, media files, office documents, etc.
If you need to add your own one, override that Firefox preference manually via the `:prefs` option:

//:test-doc-blocks/skip
[source,clojure]
----
(def driver (e/firefox {:download-dir "target/etaoin-play/firefox-downloads"
                        :prefs {:browser.helperApps.neverAsk.saveToDisk
                                "some-mime/type-1;other-mime/type-2"}}))
;; do some downloading
(e/driver quit)
----

To check whether a file was downloaded during UI tests, see <<test-file-downloads>>.

=== Managing User-Agent [[user-agent]]

Set a custom `User-Agent` header with the `:user-agent` option when creating a driver, for example:

[source,clojure]
----
(e/with-firefox {:user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"}
                driver
  (e/wait 1) ;; seems to appease Firefox on Linux
  (e/get-user-agent driver))
;; => "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
----

Setting this header is important when using <<headless,headless browsers>> as many websites implement some sort of blocking when the User-Agent includes the "headless" string.
This can lead to 403 response or some weird behavior of the site.

=== HTTP Proxy [[http-proxy]]

To set proxy settings use environment variables `HTTP_PROXY`/`HTTPS_PROXY` or pass a map of the following type:

//:test-doc-blocks/skip
[source,clojure]
----
{:proxy {:http "some.proxy.com:8080"
         :ftp "some.proxy.com:8080"
         :ssl "some.proxy.com:8080"
         :socks {:host "myproxy:1080" :version 5}
         :bypass ["http://this.url" "http://that.url"]
         :pac-url "localhost:8888"}}

;; example
(e/chrome {:proxy {:http "some.proxy.com:8080"
                   :ssl "some.proxy.com:8080"}})
----

NOTE: A `:pac-url` is for a https://en.wikipedia.org/wiki/Proxy_auto-config#The_PAC_File[proxy autoconfiguration file].
Used with Safari as other proxy options do not work in Safari.

To fine tune the proxy you use the original https://www.w3.org/TR/webdriver/#proxy[object] and pass it to capabilities:

//:test-doc-blocks/skip
[source,clojure]
----
(e/chrome {:capabilities
           {:proxy
            {:proxyType "manual"
             :proxyAutoconfigUrl "some.proxy.com:8080"
             :ftpProxy "some.proxy.com:8080"
             :httpProxy "some.proxy.com:8080"
             :noProxy ["http://this.url" "http://that.url"]
             :sslProxy "some.proxy.com:8080"
             :socksProxy "some.proxy.com:1080"
             :socksVersion 5}}})
----
=== Connecting to an Existing Running WebDriver [[connecting-existing]]

To connect to an existing WebDriver, specify the `:host` parameter.

TIP: When neither the `:host` nor the `:webdriver-url` parameter is specified Etaoin will launch a new WebDriver process.

The `:host` can be a hostname (localhost, some.remote.host.net) or an IP address (127.0.0.1, 183.102.156.31).
If the port is not specified, the <<driver-options,default>> `:port` is assumed.

Both `:host` and `:port` are ignored if `:webdriver-url` is specified.

Example:

//:test-doc-blocks/skip
[source,clojure]
----
;; Connect to an existing chromedriver process on localhost on port 9515
(def driver (e/chrome {:host "127.0.0.1" :port 9515})) ;; for connection to driver on localhost on port 9515

;; Connect to an existing geckodriver process on remote most on default port
(def driver (e/firefox {:host "192.168.1.11"})) ;; the default port for firefox is 4444

;; Connect to a chrome instance on browserless.io via :webdriver-url
;; (replace YOUR-API-TOKEN with a valid browserless.io api token if you want to try this out)
(e/with-chrome {:webdriver-url "https://chrome.browserless.io/webdriver"
                :capabilities {"browserless:token" "YOUR-API-TOKEN"
                               "chromeOptions" {"args" ["--no-sandbox"]}}}
               driver
  (e/go driver "https://en.wikipedia.org/")
  (e/wait-visible driver [{:id :simpleSearch} {:tag :input :name :search}])
  (e/fill driver {:tag :input :name :search} "Clojure programming language")
  (e/fill driver {:tag :input :name :search} k/enter)
  (e/get-title driver))
;; => "Clojure programming language - Search results - Wikipedia"
----

=== Setting the Browser Profile [[browser-profile]]

When running Chrome or Firefox, you may specify a special web browser profile made for test purposes.
A profile is a folder that keeps browser settings, history, bookmarks, and other user-specific data.

Imagine, for example, that you'd like to run your integration tests against a user that turned off Javascript execution or image rendering.

TIP: This is a hypothetical example.
Turning off JavaScript will affect/break certain WebDriver features.
And it can affect certain WebDriver implementations, https://github.com/clj-commons/etaoin/issues/363[for example].

==== Create and Find a Profile in Chrome

. In the right top corner of the main window, click on a user button.
. In the dropdown, select "Manage People".
. Click "Add person", submit a name and press "Save".
. The new browser window should appear.
Now, setup the new profile as you want.
. Open `chrome://version/` page.
Copy the file path that is beneath the `Profile Path` caption.

==== Create and Find a Profile in Firefox

. Run Firefox with `-P`, `-p` or `-ProfileManager` key as the https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles[official page] describes.
. Create a new profile and run the browser.
. Setup the profile as you need.
. Open `about:support` page.
Near the `Profile Folder` caption, press the `Show in Finder` button.
A new folder window should appear.
Copy its path from there.

==== Running a Driver with a Profile

Once you've got a profile path, launch a driver with the `:profile` key as follows:

//:test-doc-blocks/skip
[source,clojure]
----
;; Chrome
(def chrome-profile
  "/Users/ivan/Library/Application Support/Google/Chrome/Profile 2/Default")

(def chrome-driver (e/chrome {:profile chrome-profile}))

;; Firefox
(def ff-profile
  "/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test")

(def firefox-driver (e/firefox {:profile ff-profile}))
----

== Writing and Running Integration Tests For Your Application

=== Headless Testing [[headless-testing]]
Is is not unusual for Continuous Integration services to have no display.
This seems to be especially true for Linux runners.

When running your tests on Linux with no display, you have 2 choices:

* run the WebDriver in <<headless,headless>> mode
* use a virtual display

NOTE: Things being what they are, WebDrivers can behave differently when run headless.

The technologies we use for Etaoin's CI testing on GitHub Actions for Linux are:

* Xvfb - acts as an X virtual display
* fluxbox - a lightweight windows manager (needed by geckodriver/Firefox to support window positioning operations)

You can see how we make use of these tools in the Etaoin link:/script/test.clj[test script], but in a nutshell:

To install:
[source,shell]
----
sudo apt get install -y xvfb fluxbox
----

TIP: As of this writing Xvfb is pre-installed on the linux runner on GitHub Actions, but fluxbox is not.

Ensure `DISPLAY` env var is set:
[source,shell]
----
export DISPLAY=:99.0
----

Launch the virtual display and fluxbox:
[source,shell]
----
Xvfb :99 -screen 0 1024x768x24 &
fluxbox -display :99 &
----

=== Basic Fixture

It is desirable to have your tests be independent of one another.
One way to achieve this is through the use of a test fixture.
The fixture's job is to, for each test:

1. create a new driver
2. run the test with the driver
3. shutdown the driver

A dynamic `+*driver*+` var might be used to hold the driver.

//:test-doc-blocks/skip
[source,clojure]
----
(ns project.test.integration
  "A module for integration tests"
  (:require [clojure.test :refer [deftest is use-fixtures]]
            [etaoin.api :as e]))

(def ^:dynamic *driver*)

(defn fixture-driver
  "Executes a test running a driver. Bounds a driver
   with the global *driver* variable."
  [f]
  (e/with-chrome [driver]
    (binding [*driver* driver]
      (f))))

(use-fixtures
  :each ;; start and stop driver for each test
  fixture-driver)

;; now declare your tests

(deftest ^:integration
  test-some-case
  (doto *driver*
    (e/go url-project)
    (e/click :some-button)
    (e/refresh)
    ...
    ))
----

If for some reason you want to reuse a single driver instance for all tests:

//:test-doc-blocks/skip
[source,clojure]
----
(ns project.test.integration
  "A module for integration tests"
  (:require [clojure.test :refer [deftest is use-fixtures]]
            [etaoin.api :as e]))

(def ^:dynamic *driver*)

(defn fixture-browser [f]
  (e/with-chrome-headless {:args ["--no-sandbox"]} driver
    (e/disconnect-driver driver)
    (binding [*driver* driver]
      (f))
    (e/connect-driver driver)))

;; creating a session every time that automatically erases resources
(defn fixture-clear-browser [f]
  (e/connect-driver *driver*)
  (e/go *driver* "http://google.com")
  (f)
  (e/disconnect-driver *driver*))

;; this is run `once` before running the tests
(use-fixtures
  :once
  fixture-browser)

;; this is run `every` time before each test
(use-fixtures
  :each
  fixture-clear-browser)

...some tests
----

For faster testing you can use this example:

//:test-doc-blocks/skip
[source,clojure]
----
.....

(defn fixture-browser [f]
  (e/with-chrome-headless {:args ["--no-sandbox"]} driver
    (binding [*driver* driver]
      (f))))

;; note that resources, such as cookies, are deleted manually,
;; so this does not guarantee that the tests are clean
(defn fixture-clear-browser [f]
  (e/delete-cookies *driver*)
  (e/go *driver* "http://google.com")
  (f))

......
----

=== Multi-Driver Fixtures

In the example above, we examined a case when you run tests against a single type of driver.
However, you may want to test your site on multiple drivers, say, Chrome and Firefox.
In that case, your fixture may become a bit more complex:

//:test-doc-blocks/skip
[source,clojure]
----

(def driver-type [:firefox :chrome])

(defn fixture-drivers [f]
  (doseq [type driver-types]
    (e/with-driver type {} driver
      (binding [*driver* driver]
        (testing (format "Testing in %s browser" (name type))
          (f))))))
----

Now, each test will be run twice.
Once for Firefox and then once Chrome.
Please note the test call is prepended with the `testing` macro that puts the driver name into the report.
Once you've got an error, you'll easily find what driver failed the tests exactly.

TIP: See also link:{url-tests}[Etaoin's API tests] for an example of this strategy.

=== Postmortem Handler To Collect Artifacts

To save some artifacts in case of an exception, wrap the body of your test into the `with-postmortem` handler as follows:

//:test-doc-blocks/skip
[source,clojure]
----
(deftest test-user-login
  (e/with-postmortem *driver* {:dir "/path/to/folder"}
    (doto *driver*
      (e/go "http://127.0.0.1:8080")
      (e/click-visible :login)
      ;; any other actions...
      )))
----

If any exception occurs in that test, artifacts will be saved.

To not copy and paste the options map, declare it at the top of the module.
If you use Circle CI, it would be great to save the data into a special artifacts directory that might be downloaded as a zip file once the build has been finished:

//:test-doc-blocks/skip
[source,clojure]
----
(def pm-dir
  (or (System/getenv "CIRCLE_ARTIFACTS") ;; you are on CI
      "/some/local/path"))               ;; local machine

(def pm-opt
  {:dir pm-dir})
----

Now pass that map everywhere into PM handler:

//:test-doc-blocks/skip
[source,clojure]
----
  ;; test declaration
  (e/with-postmortem *driver* pm-opt
    ;; test body goes here
    )
----

Once an error occurs, you will find a PNG image that represents your browser page at the moment of exception and HTML dump.

See <<postmortem>>.

=== Running Tests By Tag

Since UI tests may take lots of time to pass, it's definitely a good practice to pass both server and UI tests independently from each other.

If you are using leiningen, here are a few tips.

First, add `+^:integration+` tag to all the tests that are run under the browser like follows:

//:test-doc-blocks/skip
[source,clojure]
----
(deftest ^:integration
  test-password-reset-pipeline
  (doto *driver*
    (go url-password-reset)
    (click :reset-btn)
    ;; and so on...
  ))
----

Then, open your `project.clj` file and add test selectors:

[source,clojure]
----
:test-selectors {:default (complement :integration)
                 :integration :integration}
----

Now, when you launch `lein test` you will run all the tests except browser integration tests.
To run integration tests, launch `lein test :integration`.

=== Check Whether a File has been Downloaded [[test-file-downloads]]

Sometimes, a file starts to download automatically when you click on a link or just visit some page.
In tests, you might need to ensure a file really has been downloaded successfully.
A common scenario would be:

* provide a custom empty download folder when running a browser (see <<download-dir>>).
* Click on a link or perform any action needed to start file downloading.
* Wait for some time;
for small files, 5-10 seconds would be enough.
* Using files API, scan that directory and try to find a new file.
Check if it matches a proper extension, name, creation date, etc.

Example:

//:test-doc-blocks/skip
[source,clojure]
----
(require '[clojure.java.io :as io]
         '[clojure.string :as str])

;; Local helper that checks whether it is really an Excel file.
(defn xlsx? [file]
  (-> file
      .getAbsolutePath
      (str/ends-with? ".xlsx")))

;; Top-level declarations
(def DL-DIR "/Users/ivan/Desktop")
(def driver (e/chrome {:download-dir DL-DIR}))

;; Later, in tests...
(e/click-visible driver :download-that-application)
(e/wait driver 7) ;; wait for a file has been downloaded

;; Now, scan the directory and try to find a file:
(let [files (file-seq (io/file DL-DIR))
      found (some xlsx? files)]
  (is found (format "No *.xlsx file found in %s directory." DL-DIR)))
----

== Running Selenium IDE files [[selenium-ide]]

Etaoin can play the files produced by link:{ide}[Selenium IDE].
Selenium IDE allows you to record web interactions for later playback.
It is installed as an optional extension in your web browser.

Once installed, and activated, it records your actions into a JSON file with the `.side` extension.
You can save that file and run it with Etaoin.

Let's imagine you've installed the IDE and recorded some actions as per Selenium IDE documentation.
Now that you have a `test.side` file, you could do something like this:

//:test-doc-blocks/skip
[source,clojure]
----
(require '[clojure.java.io :as io]
         '[etaoin.api :as e]
         '[etaoin.ide.flow :as flow])

(def driver (e/chrome))

(def ide-file (io/resource "ide/test.side"))

(def opt
    {;; The base URL redefines the one from the file.
     ;; For example, the file was written on the local machine
     ;; (http://localhost:8080), and we want to perform the scenario
     ;; on staging (https://preprod-001.company.com)
     :base-url "https://preprod-001.company.com"

     ;; keywords :test-.. and :suite-.. (id, ids, name, names)
     ;; are used to select specific tests. When not passed,
     ;; all tests get run. For example:

     :test-id "xxxx-xxxx..."         ;; a single test by its UUID
     :test-name "some-test"          ;; a single test by its name
     :test-ids ["xxxx-xxxx...", ...] ;; multiple tests by their ids
     :test-names ["some-test1", ...] ;; multiple tests by their names

     ;; the same for suites:

     :suite-id    ...
     :suite-name  ...
     :suite-ids   [...]
     :suite-names [...]})

(flow/run-ide-script driver ide-file opt)
----

Everything related to the IDE feature can be found under the link:{url-doc}/CURRENT/api/etaoin.ide[etaoin.ide] namespace.

=== CLI Arguments [[selenium-ide-cli]]

You may also run a `.side` script from the command line.
Here is a `clojure` example:

[source,shell]
----
clojure -M -m etaoin.ide.main -d firefox -p '{:port 8888 :args ["--no-sandbox"]}' -r ide/test.side
----

As well as from an uberjar.
In this case, Etaoin must be in the primary dependencies, not the `:dev` or `:test` related.

[source,shell]
----
java -cp .../poject.jar -m etaoin.ide.main -d firefox -p '{:port 8888}' -f ide/test.side
----

We support the following arguments (check them out using the `clojure -M -m etaoin.ide.main -h` command):

----
  -d, --driver-name name   :chrome  The name of driver. The default is `:chrome`
  -p, --params params      {}       Parameters for the driver represented as an
                                    EDN string, e.g '{:port 8080}'
  -f, --file path                   Path to an IDE file on disk
  -r, --resource path               Path to an IDE resource
      --test-ids ids                Comma-separeted test ID(s)
      --suite-ids ids               Comma-separeted suite ID(s)
      --test-names names            Comma-separeted test name(s)
      --suite-names names           Comma-separeted suite name(s)
      --base-url url                Base URL for tests
  -h, --help
----

Pay attention to `--params`.
This must be an EDN string representing a Clojure map.
That's the same map that you pass into a driver at creation time.

Please note the IDE support is still experimental.
If you encounter unexpected behavior feel free to open an issue.
At the moment, we only support Chrome and Firefox for IDE files.

== Webdriver in Docker

To work with the driver in Docker, you can take ready-made images:

Example for https://hub.docker.com/r/robcherry/docker-chromedriver/[Chrome]:

[source,shell]
----
docker run --name chromedriver -p 9515:4444 -d -e CHROMEDRIVER_WHITELISTED_IPS='' robcherry/docker-chromedriver:latest
----

for https://hub.docker.com/r/instrumentisto/geckodriver[Firefox]:

[source,shell]
----
docker run --name geckodriver -p 4444:4444 -d instrumentisto/geckodriver
----

To connect to an existing running WebDriver process you need to specify the `:host`.
In this example `:host` would be `localhost` or `127.0.0.1`.
The `:port` would be the appropirate port for the running WebDriver process as exposed by docker.
If the port is not specified, the <<driver-options,default>> port is set.

//:test-doc-blocks/skip
[source,clojure]
----
(def driver (e/chrome-headless {:host "localhost" :port 9515 :args ["--no-sandbox"]}))
(def driver (e/firefox-headless {:host "localhost"})) ;; will try to connect to port 4444
----

== Troubleshooting [[troubleshooting]]

=== Old Versions of WebDrivers can have Limitations

[horizontal]
Reproduction:: For example, `chromedriver` used to throw an error when calling `maximize`:
+
[source,clojure]
----
(e/with-chrome driver
  (e/maximize driver))
;; an exception with "cannot get automation extension" was thrown
----
Cause:: This was a bug in `chromedriver` that was fixed in chromdriver v2.28.
Solution:: Updating to the current WebDriver resolved the issue.

=== New Versions of WebDrivers can Introduce Bugs

[horizontal]
Reproduction:: For example, `chromedriver` v103 started sporadically failing with `unknown error: cannot determine loading status\nfrom unknown error: unexpected command response`.

Cause:: Likely a https://bugs.chromium.org/p/chromedriver/issues/detail?id=4121[bug in chromedriver]

Solution:: Upgrade to newer version that fixes bug.
For this particular bug I simply suffered retrying failed tests while waiting for newer version.

=== Clicking on a Link Does not Work with Safari

[horizontal]
Reproduction:: An attempt to click on a link does nothing.
+
Given the following HTML
+
[source,html]
----
<div class="faculty_course_title" title="Course Title">
 <a href="https://clojure.org">Course Title</a>
</div>
----
+
An attempt to click on the link does nothing:
//:test-doc-blocks/skip
+
[source,clojure]
----
(e/click-single driver [{:tag :div :class "faculty_course_title"} {:tag :a}])
----

Cause:: Odd as it may seem, this is a long-standing (maybe on-again-off-again?) bug in `safaridriver`.

Work-around:: You can, if you wish, employ JavaScript to click the link:
//:test-doc-blocks/skip
+
[source,clojure]
----
(let [elem (e/query driver [{:tag :div :class "faculty_course_title"} {:tag :a}])]
  (e/js-execute driver "arguments[0].click()" (e/el->ref elem)))
----

=== XPath and Searching from Root vs Current node

Reproduction::
+
[source,clojure]
----
;; we intend to find an element with the text 'some' under an element with id 'mishmash'
(e/get-element-text driver [{:id :mishmash} "//*[contains(text(),'some')]"])
;; => "A little sample page to illustrate some concepts described in the Etaoin user guide."
;; but we've found the first element with text 'some'
----
Cause:: In a vector, every expression searches from the previous one in a loop.
Without a leading dot, the XPath `+"//..."+` clause means to find an element from the root of the whole page.
With a dot, it means to find from the current node, which is one from the previous query, and so forth.
Solution:: Add the XPath dot.
+
[source,clojure]
----
(e/get-element-text driver [{:id :mishmash} ".//*[contains(text(),'some')]"])
;; => "some other paragraph"
;; that's what we were looking for!
----

=== Clicking On Non-Visible Element

Reproduction::
//:test-doc-blocks/skip
+
[source,clojure]
----
(e/click driver :cantseeme)
;; as of this writing, on chrome throws an exception with message containing 'not interactable'
----

Cause:: You cannot interact with an element that is not visible or is so small that a human could not click on it.

=== Selectors not Working

Symptom:: Selectors for locating elements are not working, even though the elements are clearly available.

Possible cause:: Your script may have clicked a link that opened a new tab or window.
Even though the new window is in the foreground, the driver instance is still connected to the original window.

Solution:: Call `switch-window-next` when a new tab or window is opened to point the driver to the new tab/window.

=== Unpredictable errors in Chrome when the window is not active

Reproduction:: when you focus on another window, a WebDriver session that is run under Google Chrome fails.

Solution:: Google Chrome may suspend a tab when it has been inactive for some time.
When the page is suspended, no operation can be done on it.
No clicks, Js execution, etc.
So try to keep Chrome window active during test session.

=== Invalid argument: can't kill an exited process for Firefox

Reproduction:: When you try to start the driver you get an error:
//:test-doc-blocks/skip
+
[source,clojure]
----
(def driver (e/firefox {:headless true}))
;; throws an exception containing message with 'invalid argument: can't kill an exited process'
----

Possible Cause:: Running Firefox as root in a regular user's session is not supported

To Diagnose:: Run the driver with the path to the log files and the "trace" log level and explore the output.
//:test-doc-blocks/skip
+
[source,clojure]
----
(def driver (firefox {:log-stdout "ffout.log" :log-stderr "fferr.log" :driver-log-level "trace"}))
----
Similar Problem:: https://github.com/mozilla/geckodriver/issues/1655

=== DevToolsActivePort file doesn't exist error on Chrome

Reproduction:: When you try to start the chromedriver you get an error:

//:test-doc-blocks/skip
[source,clojure]
----
(def driver (e/chrome))
;; throws an exception with message containing 'DevToolsActivePort file doesn't exist'
----

Possible Cause::
A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux.
While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and discouraged.
Ideally you would configure your environment to run Chrome as a regular user instead.
+
NOTE: We have noticed that the Selenium docker images always invoke chrome with `--no-sandbox`, so the caveat is a little confusing.
We've naively replicated this in our own dev docker images.
Maybe `--no-sandbox` is acceptable, or even needed, when running from a docker container?
If you have intel here, let us know!

Potential Solution:: Run driver with argument `--no-sandbox`.
Caution!
This bypasses the OS security model.
+
[source,clojure]
----
(e/with-chrome {:args ["--no-sandbox"]} driver
  (e/go driver "https://clojure.org"))
----

Similiar Problem:: A similar problem is described https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t[here]

// A little invisible cheat to cleanup our driver
ifdef::env-test-doc-blocks[]
[source,clojure]
----
(e/quit driver)
----
endif::[]