v12.0.0

Breaking changes in logging during UI tests

One of the implicit assertions that the UI Testing Toolbox does automatically is checking the Orchard Core application logs for any errors. By default, any error fails the test.

Previously, this relied on the tested web app using NLog for logging, and it parsed NLog log files. This not only required the app to use NLog over Serilog (that Orchard also supports out of the box) or any other logging framework, but it also depended on the structure of an otherwise freely configurable textual log format.

Now, the UI Testing Toolbox configures the app to use FakeLogger in addition to any logging framework it has set up, and uses it for assertions. FakeLogger not only doesn't need anything specific in the app, but it also lets you access and assert on log entries in a structured, statically typed way.

Using NLog is still support, and log files will be included in the test dump. Any other logging framework is also supported, your app doesn't need to do anything special.

What changed

• All log management methods are now async.
• Log assertions now don't happen on a string representing the whole log, but rather on IApplicationLog and IApplicationLogEntry instances.
• The LogsShouldBeEmptyAsync and GetLogOutputAsync methods have changed, see the diff.
• CreateAppLogAssertionForSecurityScan and CreateAppLogAssertionForSecurityScan work the same but were moved into the Lombiq.Tests.UI.SecurityScanning namespace.
• A new OrchardCoreConfiguration.AfterFakeLoggingConfiguration event handler is exposed to change the default configuration of FakeLogger.
• OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAsync and OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAndCacheFolderErrorsAsync are no longer available.

How to adapt

• Use OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainCacheFolderErrorsAsync instead of OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAndCacheFolderErrorsAsync .
• OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAsync is not needed, since the new behavior is to not log warnings in the first place.
• The examples are updated in Lombiq.Tests.UI.Samples.UITestBase, see the diff. Check them out if you have custom app log assertion code.

What's Changed

• OFFI-132: Shift time by @sarahelsaig in #418
• OFFI-153: Making JsonSerializerOptions editable by @wAsnk in #420
• OSOE-402: Custom logging instead of NLog-based one by @Piedone in #419
• OSOE-402: Branch references by @dministro in #423
• OSOE-904: Create an example of using FrontendServer and context.ExecuteJavascriptTestAsync by @sarahelsaig in #421
• OSOE-916: Change "shift time" terminology to "time shift" in Lombiq.UITestingToolbox by @sarahelsaig in #425
• OSOE-921: Update Atata.WebDriverSetup package to v3.1.0 and enable Edge tests again by @Piedone in #427
• LMBQ-418: Not dismissing alerts by default by @MZole in #424
• OSOE-917: Update vulnerable NuGet packages by @Piedone in #428
• OSOE-903: Update to Orchard Core 2.1.0 by @Piedone in #429

Full Changelog: v11.1.0...v12.0.0