Help need regarding "HTTP method not allowed" exception on certain elements

[abhi18av]

Hi @igrishaev,

First off, thanks for this wonderful package - I've come to really enjoy using clojure for browser automation!

During a recent project, I noticed some strange behavior in etaoin.
I have a collection of elements called page3, defined like so

(def page3 (rest
             (children driver
                       (query driver {:css "html body table tbody tr td.main table tbody tr td.maincontent table tbody"})
                       {:css "tr"})))

I've defined a wrapper function get-problem-url like so

  (defn get-problem-url [el]
    (get-element-attr-el driver
                         (second (children driver
                                           el
                                           {:tag "a"}))
                         :href))

I noticed that at the following element, which works fine for get-element-text-el as show below

(get-element-text-el driver (nth page3 2))
=> "2002 - Crosswords 72\n0.00%\n39\n0.00%"

... something unexpected happens when I try to use the get-problem-url function on this specific element like so

(get-problem-url (nth page3 2))
Execution error (ExceptionInfo) at slingshot.support/stack-trace (support.clj:201).
throw+: {:response "HTTP method not allowed", :path "session/b99f7cb0-227c-0145-8f65-7e0f51cc626d/element//attribute/href", :payload nil, :method :get, :type :etaoin/http-error, :port 4444, :host "127.0.0.1", :status 405, :driver {:args ("/usr/local/bin/geckodriver" "--port" 4444), :capabilities {:loggingPrefs {:browser "ALL"}, :moz:firefoxOptions {:binary "/Applications/Firefox Developer Edition.app/Contents/MacOS/firefox"}}, :process #object[java.lang.ProcessImpl 0x4294a11b "Process[pid=41358, exitValue=\"not exited\"]"], :locator "xpath", :type :firefox, :env nil, :port 4444, :host "127.0.0.1", :url "http://127.0.0.1:4444", :session "b99f7cb0-227c-0145-8f65-7e0f51cc626d"}}

After trying to debug and check using other functions in the library for an element, I still can't figure out why exactly this exception is raised at this specific element.

Could you please help me out a bit?

[igrishaev]

@abhi18av thanks for your report, looks like the issue is the following. In your exception, there is a URL like this:

"session/b99f7cb0-227c-0145-8f65-7e0f51cc626d/element//attribute/href"

Pay attention to the element//attribute part. It should have been like this instead: element/xxxxxxxxxxx/attribute, where xxx... is the element ID. The double slash in your URL means that the ID was empty, or nil. That's because the following expression:

(second (children driver
                      el
                      {:tag "a"}))

also returns nil. I may guess that there is something wrong with the data structure you're expecting.

[igrishaev]

What I can propose here is, you'd better to check every element you receive for nil before passing them into a query function. Meanwhile, we'll improve the query functions so that they check elements for nil.

[igrishaev]

@Uunnamed I think we can add some sort of :pre checks to all query functions (query, children, etc) to ensure the element parameter is not nil. Like this:

(defn children
  "Finds multiple elements under given root element."
  [driver ancestor-el q]
  {:pre [(some? ancestor-el)]} ;; this
  (let [[loc term] (query/expand driver q)]
    (find-elements-from* driver ancestor-el loc term)))

[abhi18av]

What I can propose here is, you'd better to check every element you receive for nil before passing them into a query function. Meanwhile, we'll improve the query functions so that they check elements for nil.

Yeah, I ended up doing the same actually and the problem was resolved :)

I think, the :pre check warning would be nice.

[igrishaev]

In the latest master, we added some :pre statements to ensure the element is not nil. It will be available in the upcoming 0.3.9 release.

[abhi18av]

That sounds great thanks!