quickstrom should expose console.log ...

[jleonard-r7]

... such that specs can check for presence of and specific contents of the javascript console log (i.e., info, debug, error, warning).

This would be useful, for example, if an application wants to ensure that under no circumstance do console.errors occur (among other use cases).

[owickstrom]

I can see how it would be a useful property, like "there are no errors in the log", but other than that I'm not sure what it would be useful for.

The fact that you can only assert on things in the DOM is a design choice for Quickstrom, with the aim to reduce coupling between specs and implementations with their technical complexities. The idea is that you should assert on things that the user can see. If you have nothing to assert on in the DOM, then you're users probably don't have much to go on either. I'd say that coupling specs to the JS log is one such technical coupling that I've been trying to avoid.

[jleonard-r7]

I don’t see it quite that dogmatically. Certainly one could get too tied to implementation details that way. On the other hand, it is a common practice for applications to provide some level of instrumentation intended for testing purposes rather than for user consumption.

Regardless, if I really want this, I can just embed the log into hidden DOM elements. But I do think that potential for misuse should not be a consideration— one should assume a competent user (especially when that user is in the first place, himself). 😀

[jleonard-r7]

And yes "there are no errors in the log" would be quite a useful property and in fact was the original inspiration for this feature request. However, I can see other "instrumentation"-related uses and I generally prefer to leave future possibilities (i.e., use cases) open rather than restricting them unnecessarily, thus the request being generalized to "expose console.log" unconditionally.

[owickstrom]

Gotcha. How would you represent the log for specs? Would it always be the entire log, or just what was added since the last state?

Another problem is that there seems to be no supported way of reading console entries in JS. The suggested approach (https://stackoverflow.com/a/19846113, https://stackoverflow.com/a/24059643) is to overwrite console.log and store it manually. This is very problematic for Quickstrom as webapps may navigate between different pages, so any JS state on window can be lost at any point. All JS interaction in Quickstrom is therefore stateless, except for the DOM mutation watchers (but it's kind of a special case).

[jleonard-r7]

Yea, I would see the log as a stream of discrete state changes just like in the DOM. I think it would fit into the existing Quickstrom model like that.

As for reading it, I don’t think we’d have to do it from JavaScript. Could we utilize WebDriver/ChromeDriver support for it?

https://stackoverflow.com/questions/18261338/get-chromes-console-log

[jleonard-r7]

To elaborate on the first point, for my purposes, it would be fine to have a query type “selector” of sorts which would be a two-tuple of (log_type, regex) where log_type would be one of: info, error, debug, warning and regex may default to “.*” or be specified to match particular log entry content.

[jleonard-r7]

Sorry, I guess I didn’t really answer the question wrt history being returned or not from the query. I hadn’t thought of that aspect of it until now. Even just returning the “last” for each particular log type would allow some use cases. But of course a comprehensive set of matches (at least back until the epoch of the current Quickstrom “state”) would be more powerful/flexible. I think this may actually be best since the logic in specs is based on properties that hold true for the current (and perhaps next) state regardless of what happened prior to that.

[jleonard-r7]

Does that sound ok to you? I am thinking to implement this once I return from holiday since it is blocking progress on our application/use case. Please let me know if you have any concerns/questions/reservations or if that design sounds ok.

[owickstrom]

OK. I think in that case that this new query should return the logged messages since the last state:

logEntries :: Array LogEntry

So, in the initial state, logEntries == []. And values in this array would be something like:

type LogEntry = { level :: LogLevel, message :: String }

You could then do filtering on your own with regular PureScript functions.

Regarding WebDriver, it doesn't seem like webdriver-w3c currently has support for retrieving logs. I guess we could either make a PR for it, or implement the endpoints in Quickstrom manually.

[jleonard-r7]

Ok, sounds good. I will see what I can do.

[jleonard-r7]

I've been doing some more research on this and yes I think I've confirmed that webdriver-w3c does not expose those logs. I have added the capabilities and desiredCapabilities to our WebDriver config but that only makes them available to be retrieved. They still need to be explicitly retrieved with the equivalent of: driver.get_log(browser); which webdriver-w3c doesn't do because it is using the HTTP API rather than a language SDK (which is what all examples of doing this seem to use). I've also looked through the webdriver endpoints documented here:

https://www.w3.org/TR/webdriver/#endpoints

but don't see anything pertaining to "logs" or a "get_log" equivalent.

So, for now, the only workaround I see is to have a Haskell filesystem watcher stream in lines as written to chrome_debug.log under the configured profile (typically "Default") [for Chrome obviously]. I think I will pursue this option at this point and we can figure out something similar for FIrefox. Note that it requires passing a --user-data-dir via the previously exposed --browser-options.

If this all seems too hacky for you, perhaps it would be best addressed by having a "logic extension/plugin" feature in the purescript specfication language which would let users do arbitrary evaluation to be integrated with the logic evaluator (wherein I could hide this ugly chrome_debug.log reading & parsing etc etc).

[owickstrom]

Ah, that's unfortunate. Yeah let me think a bit about this one. As you say, it seems hacky and brittle, especially with browser special cases. Let's try to figure out a good way for users to extend Quickstrom instead.

[owickstrom]

So, I have given this some thought. I'm again back to this being out of scope for Quickstrom, both exposing console logs (even if we could) and some kind of plugin or arbitrary external execution that would get fed into the spec checking. It just gets too messy and complicated, and it'll be a heavy maintenance thing, I think.

[jleonard-r7]

That's fine. I already have a workaround. However, from my point of view every user of quickstrom will want to do the same thing. e.g., react often lets users do ill-advised things and merely warns them in the console. Quickstrom should be able to detect that sort of thing, effectively turning a "runtime" error into an "integration time" error (assuming that quickstrom is part of one's continuous integration [and it should be]).

[owickstrom]

Yeah, I can see the usefulness of it. If and when it can be done with regular WebDriver calls, I'd consider adding it again.

[jleonard-r7]

fyi... I've opened this here: nbloomf/webdriver-w3c#26

Hopefully webdriver-w3c can support this.

[nbloomf]

I agree this is useful and probably belongs in the webdriver spec. Until that happens, as mentioned above, we can get at these logs for chrome with the appropriate command line arguments. Here is a small example of how to do that with webdriver-w3c: nbloomf/webdriver-w3c#26 (comment)