v12.0.0

Breaking changes in logging during UI tests

One of the implicit assertions that the UI Testing Toolbox does automatically is checking the Orchard Core application logs for any errors. By default, any error fails the test.

Previously, this relied on the tested web app using NLog for logging, and it parsed NLog log files. This not only required the app to use NLog over Serilog (that Orchard also supports out of the box) or any other logging framework, but it also depended on the structure of an otherwise freely configurable textual log format.

Now, the UI Testing Toolbox configures the app to use FakeLogger in addition to any logging framework it has set up, and uses it for assertions. FakeLogger not only doesn't need anything specific in the app, but it also lets you access and assert on log entries in a structured, statically typed way.

Using NLog is still support, and log files will be included in the test dump. Any other logging framework is also supported, your app doesn't need to do anything special.

What changed

• All log management methods are now async.
• Log assertions now don't happen on a string representing the whole log, but rather on IApplicationLog and IApplicationLogEntry instances.
• The LogsShouldBeEmptyAsync and GetLogOutputAsync methods have changed, see the diff.
• CreateAppLogAssertionForSecurityScan and CreateAppLogAssertionForSecurityScan work the same but were moved into the Lombiq.Tests.UI.SecurityScanning namespace.
• A new OrchardCoreConfiguration.AfterFakeLoggingConfiguration event handler is exposed to change the default configuration of FakeLogger.
• OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAsync and OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAndCacheFolderErrorsAsync are no longer available.

How to adapt

• Use OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainCacheFolderErrorsAsync instead of OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAndCacheFolderErrorsAsync .
• OrchardCoreUITestExecutorConfiguration.AssertAppLogsCanContainWarningsAsync is not needed, since the new behavior is to not log warnings in the first place.
• The examples are updated in Lombiq.Tests.UI.Samples.UITestBase, see the diff. Check them out if you have custom app log assertion code.

What's Changed

• OFFI-132: Shift time by @sarahelsaig in #418
• OFFI-153: Making JsonSerializerOptions editable by @wAsnk in #420
• OSOE-402: Custom logging instead of NLog-based one by @Piedone in #419
• OSOE-402: Branch references by @dministro in #423
• OSOE-904: Create an example of using FrontendServer and context.ExecuteJavascriptTestAsync by @sarahelsaig in #421
• OSOE-916: Change "shift time" terminology to "time shift" in Lombiq.UITestingToolbox by @sarahelsaig in #425
• OSOE-921: Update Atata.WebDriverSetup package to v3.1.0 and enable Edge tests again by @Piedone in #427
• LMBQ-418: Not dismissing alerts by default by @MZole in #424
• OSOE-917: Update vulnerable NuGet packages by @Piedone in #428
• OSOE-903: Update to Orchard Core 2.1.0 by @Piedone in #429

Full Changelog: v11.1.0...v12.0.0

v11.1.0

What's Changed

• OSOE-818: Upgrade to Orchard Core 2.0 by @sarahelsaig in #409
• OSOE-906: Use WCAG 2.2 as the default accessibility standard by @Piedone in #411
• LMBQ-401: Adapting security scanning URL excludes to OC 2.0 upgrade by @DemeSzabolcs in #412
• OFFI-135: DRYing and using system json by @wAsnk in #413
• OSOE-909: Updating Atata.WebDriverSetup and re-enabling Edge tests by @Piedone in #414
• OFFI-142: Handle dynamic setup operation by @sarahelsaig in #415
• OFFI-85: Allow viewport screenshot for cases where element selection won't work. by @sarahelsaig in #417
• INFRA-164: Compatibility with Ubuntu 2024-based (v24.04) runners by @BenedekFarkas in #416

Full Changelog: v11.0.0...v11.1.0

v11.0.0

What's Changed

• OSOE-795: Upgrade to latest OC preview to test System.Text.Json by @sarahelsaig in #348
• OSOE-859: Update HL NuGet version by @sarahelsaig in #374
• NEST-538: Adjusting workflow testing in case there are already one page of workflows by @DemeSzabolcs in #371
• OFFI-33: Add GetBoolAttribute and IsDisabled extension methods by @sarahelsaig in #376
• OCC-245: Add publish-cloudsmith workflow by @sarahelsaig in #377
• OSOE-852: Change NuGet metadata to use license expression by @AydinE in #379
• OSOE-867: Addressing analyzer warnings by @Psichorex in #378
• OSOE-537: Migrate from Selenium.Axe to TWP.Selenium.Axe.Html in Lombiq.UITestingToolbox by @Psichorex in #380
• OSOE-873: Disable "allow pasting" warning in Chrome's dev tools by @sarahelsaig in #383
• OSOE-759: Fixing YAML lint warnings by @DemeSzabolcs in #385
• TDEAL-38: Save suggested image if it's the final try regardless of retry count by @sarahelsaig in #389
• ADHOC-463: Optimizing basic Media operations testing by @Piedone in #384
• OCORE-191: Updating Refit to 7.1.2 by @Piedone in #391
• OFFI-78: Support different base path for OC app by @wAsnk in #386
• NEST-501: Updating ZAP to 2.15.0, regex patterns for LogsShouldBeEmptyAsync(), giving access to the ZAP log, Swagger auto-discovery for OpenAPI scan by @Piedone in #388
• LMBQ-373: Fix visual verification tests instead of using Firefox and skipping them by @DemeSzabolcs in #392
• OCC-245: Update OC preview version by @sarahelsaig in #375
• OSOE-889: Update Atata packages to latest by @Piedone in #394
• OSOE-890: Updating Atata.WebDriverSetup to v2.13.0 to fix Edge UI tests by @Piedone in #398
• OSOE-365: Support for running tests without a browser by @Piedone in #396
• OFFI-92: Better default log assertion configuration, disabling UseCdn by default by @Piedone in #395
• OSOE-893: Fixing SMTP settings management by @Piedone in #400
• LMBQ-336: Adding sample for custom false positive alert filter by @barthamark in #401
• LMBQ-178: Enhancing support for remote tests working with apps behind Cloudflare by @Piedone in #402
• NEST-536: Add necessary security exceptions by @sarahelsaig in #403
• LMBQ-377: More logging for CloudflareHelper by @DemeSzabolcs in #404
• OFFI-87: BE: Add endpoint testing by @porgabi in #405
• OSOE-732: Generalize Failure Dump to Test Dump by @Piedone in #406
• OFFI-101: Frontend server support and interactive mode improvements by @sarahelsaig in #407

Full Changelog: v10.0.1...v11.0.0

v10.0.1

What's Changed

• OSOE-835: OSOE-835: Add GetTenantOptionsAsync() by @AydinE in #366
• OSOE-859: Change the projects to target OC 1.8 by @sarahelsaig in #373

New Contributors

• @AydinE made their first contribution in #364

Full Changelog: v9.1.1...v10.0.1

v10.0.0

Breaking changes

While the impact should be low for most projects, this version does introduce some breaking changes:

• Atata was updated to v3.1.0 from v2.9.0. Atata v3 brought breaking changes as well. So, if you integrated with some Atata features directly, you may need to adapt your code, see the migration guide. If you only used the higher-level APIs provided by the UI Testing Toolbox, then nothing to do (any problems will be apparent as build errors).
• The TestBasicOrchardFeatures* extension methods are now in their own namespace. You'll need to thus add using Lombiq.Tests.UI.BasicOrchardFeaturesTesting; to files using them. Everything works otherwise the same.

What's Changed

• OSOE-844: Don't include a file called .htmlvalidate.json by default in Lombiq.UITestingToolbox by @sarahelsaig in #361
• OSOE-770: Add support for structured html-validate output by @AydinE in #364
• OSOE-847: Basic Orchard features testing should include Workflows and Audit Trail by @DemeSzabolcs in #365
• LMBQ-323: Adding OS based visual verification option by @DemeSzabolcs in #368
• LMBQ-323: Adding AssertVisualVerificationApprovedOnAllResolutionsWithPlatformSuffix by @DemeSzabolcs in #369
• OSOE-770: Fix uncovered edge cases by @AydinE in #370
• OFFI-37: Adding SignInDirectlyAndGoToAdminRelativeUrlAsync extension method by @porgabi in #367
• OSOE-838: Updating Atata v3.1.0, and other dependencies to latest too, reliability improvements by @Piedone in #362

New Contributors

• @AydinE made their first contribution in #364

Full Changelog: v9.1.1...v10.0.0

v9.1.1

This release is only to fix a NuGet publishing issue that you can read about here. There are no other changes, but the packages now don't mistakenly define the NuGetBuild MSBuild property.

Full Changelog: v9.1.0...v9.1.1

v9.1.0

What's Changed

• OCC-218: Exclude "The filename, directory name, or volume label syntax is incorrect" error during security testing. by @sarahelsaig in #350
• LMBQ-326: Adding AxeResultItemExtensions by @DemeSzabolcs in #353
• LMBQ-267: Fixing that only a single security scan could run at the same time, extending configurability, adjusting full scan config by @Piedone in #351
• WALMA-144: Fixing PermissionExistsAsync parallelization problem by @DemeSzabolcs in #357
• OSOE-840: Fix Chrome processes remaining after test runs by @Piedone in #360

Full Changelog: v9.0.0...v9.1.0

v9.0.0

Default browser configuration

The UI Testing Toolbox now offers a default for the browser being used. The endless [Theory, Chrome] attributes and passing the browser down the call chain are now while still supported, unnecessary, unless you want to use a different or multiple browsers.

Everything should keep on working, but it's still nicer to remove the now unnecessary code. So, in all projects, please do this:

1. Update to the latest UI Testing Toolbox. Then code cleanup time! Warning: UITestBase classes should usually remain so check and revert them if they change after the below steps!
2. Do a Find and Replace in the UI test projects with the following will cover most cases: [Theory, Chrome] → [Fact], (Browser browser) => → () =>, , browser); → );.
3. Run the following with "Use regular expressions" checked: ,\r\n.*browser\); → );, ,\r\n.*browser, → ,.
4. Do a Find and Replace of (Browser browser) → ().
5. Do a reference lookup for Browser.Chrome (Shift+F12) to find any missed cases.
6. Finally, you'll need to check the formatting of all tests, because e.g. you may be able to make methods a lot more compact with fewer line breaks. All tests will have now unused references too, but if you have cleanup on save configured in VS, then just saving them will get rid of that.

All other changes

• LMBQ-249: Updating Lombiq.HelpfulLibraries NuGet reference by @MZole in #318
• OSOE-430: Improve stability directly after a page load by @sarahelsaig in #316
• NEST-462: Comply with new OC preview version as well by @wAsnk in #317
• OSOE-677: New baseline images by @wAsnk in #319
• TDEAL-2: Add EnableAutoSetupIfNotUITesting extension method by @sarahelsaig in #320
• OSOE-675: Set stopOnFail flag to true for Lombiq.Tests.UI.Samples by @tteguayco in #325
• OSOE-733: Ability to configure default browser by @Piedone in #326
• OSOE-351: Offer security checks by @Piedone in #322
• OSOE-755: Adding windows headless shell visual testing baseline image by @wAsnk in #330
• OSOE-755: Fix the BasicVisualVerificationTests.VerifyNavbar sample for Unix also by @wAsnk in #332
• TDEAL-8: Use the new AddAsyncResultFilter extension method by @sarahelsaig in #331
• TDEAL-23: Updating visual verification documentation by @porgabi in #335
• TDEAL-16: ZAP improvements by @sarahelsaig in #334
• NEST-113: Updating Configuration.md with info about changing HTML validation rules for specific tests by @DemeSzabolcs in #333
• OSOE-767: Fixing that the sarif reports used different risk and confidence levels than the HTML reports by @Piedone in #339
• OSOE-799: Renaming SelectThemeAsync() to SetThemeDirectlyAsync() by @Piedone in #340
• GOV-29: Adding the ability to do UI testing on a remote environment by @Piedone in #341
• OSOE-807: Removing unneeded MultipleActiveResultSets=True config from the default SQL Server connection string by @Piedone in #344
• OSOE-771: Adding more navigation methods by @Piedone in #342
• LMBQ-299: Adding SetTaxonomyFieldByTextAsync function by @DemeSzabolcs in #345
• OSOE-817: "Creating the web driver failed..." exception when trying to run tests locally by @Piedone in #347
• OSOE-751: Upgrade to Orchard Core 1.8 by @Psichorex in #338

New Contributors

• @tteguayco made their first contribution in #325

Full Changelog: v8.2.0...v9.0.0

v8.2.0

What's Changed

• OSOE-638: Adding monaco editor extensions by @wAsnk in #315

Full Changelog: v8.0.2...v8.2.0

v8.0.2

What's Changed

• OSOE-683: Update all NuGet dependencies by @Psichorex in #314

Full Changelog: v8.0.1...v8.0.2