Add support for shadow DOM

[dgr]

Currently, Etaoin doesn't have any convenience functions that help a user deal with the browser's shadow DOM capabilities. After some pointers from @lread and a lot of playing with execute in the REPL, I was able to get this working.

I would propose the following additions to the Etaoin API:

(defn shadow-root? [driver q] ...): Returns true if the element matching the query q has a shadow root. This is similar to enabled? and visible?.

(defn shadow-root-el? [driver el] ...): Returns true if element el has a shadow root. The is similar to enabled-el?, etc.

(defn get-shadow-root [driver q] ...): If the element matching the query q has a shadow root, return the shadow root element, otherwise nil.

(defn get-shadow-root-el [driver el] ...): Same as get-shadow-root but with element instead of query.

(defn query-shadow [driver shadow-q q] ...): This is basically the same as query but queries the shadow DOM starting at the shadow root specified by q. Returns an error/exception if q doesn't specify a shadow root. Note that we can't use query here (without a bunch of changes) because the Web Driver spec requires different request URLs for searching the shadow DOM. See the Web Driver spec, sections 12.3.6 and 12.3.7

(defn query-shadow-el [driver shadow-el q] ...): Same as query-shadow but using an element to identify the root rather than an element.

:fn/shadow-root true/false: Create another pseudo-function to be used with query to find an element with a shadow root.

(defmulti ^:private find-element-from-shadow-root* ...): Private implementation function dispatching on driver type, similar to find-element-from*. Maps to Web Driver spec section 12.3.6.

(defmulti ^:private find-elements-from-shadow-root* ...): Private implementation function dispatching on driver type, similar to find-elements-from*. Maps to Web Driver spec section 12.3.7.

Discuss. If we can reach a consensus, I'll try coding it up and will submit a PR.

[dgr]

Ah, I should also note that Web Driver support for shadow DOM is a bit splotchy. All browsers currently support it to a certain extent, with Firefox being the best but still not complete. Notably, xpath is not supported by any browser currently, so users would have to manually specify CSS selectors for now. See https://wpt.fyi/results/webdriver/tests/classic/find_element_from_shadow_root/find.py?label=experimental&label=master&aligned for more info.

[lread]

Nice digging @dgr! Thanks for doing all that investigation.

I like your API naming; it fits in nicely with current conventions.

Since Etaoin is a light wrapper over WebDrivers, I think you are right to expose the limitations and nuances as is (i.e., query-shadow-* is separate from query-*) and not invent any big compensations.

It's too bad about splotchy support, but that seems to be the way things go.
If you find it is too splotchy to be testable on some WebDrivers, you could add tests only for WebDrivers where support is more solid and document known limitations in the user guide and docstrings with the expectation that maybe someday WebDriver implementations will be more complete.

[dgr]

The API changed slightly for this as I started to look at the rest of the Etaoin API. The final implementation tries to fit in with the naming conventions and other API functions better than the original proposal.

Private:

(defn find-element-from-shadow-root* ...)
(defn find-elements-from-shadow-root* ...)

Public:

(defn get-element-shadow-root-el [driver el] ...)
(defn get-element-shadow-root [driver q] ...)
(defn has-shadow-root? [driver q] ...)
(defn has-shadow-root-el? [driver el] ...)
(defn query-shadow-root-el [driver shadow-root-el shadow-q] ...)
(defn query-all-shadow-root-el [driver shadow-root-el shadow-q] ...)
(defn query-shadow-root [driver q shadow-q] ...)
(defn query-all-shadow-root [driver q shadow-q] ...)

Note that :fn/shadow-root was dropped because a shadow root is only identified by a DOM property, not an HTML/DOM attribute, and both XPath and CSS can only query for attributes, not properties.

[lread]

Closed by #606

[dgr]

So, in researching #559, I started looking through the shadow root functions I added. I’d like to make a series of name tweaks that would bring them more in-line with what I proposed for new functions in 559. Now, I have, for example, query-shadow-root. I think that should be renamed to query-from-shadow-root. That’s more verbose, but it’s more descriptive and similar to the underlying Web Driver API, Find Element From Shadow Root. Any issues with that? Also, in looking through the query-shadow-… functions, I’ve made the same mistake as 559 (child/children), where I’m not actually handling the shadow-q parameter in the same way that query handles the q parameter, in spite of docstrings saying that’s the case. I think it’s OK breaking compatibility for the naming as Etaoin hasn’t had an official release with those in there, and the docs on cljdocs.org are still the 1.0.40 docs. I'll submit a PR shortly.