Optimizations for Synchronous Context

[jongpie]

Discussed in #303

^{Originally posted by LawrenceLoz April 28, 2022}
Hi there,

Firstly big kudos to author/authors and contributors on this project - a really rich and intuitive set of features and some great configuration flexibility.

Running a few experiments and looking at debug logs, I've noticed that when the Logger object class is called several queries are run within the transaction which contribute to the limit of 100 SOQL statements - in my tests this is 4 queries and I think would be 5 for logs being made for community users. Whilst this isn't an especially high, in an ideal world given how widely a logging framework should be used it would be great to reduce the utilisation and improve synchronous performance where possible.

There are a couple of areas resulting in queries being run, and I wonder whether it's viable to refactor to avoid these:

1. Line 2556 in Logger initialises LogEntryEventHandler: LogEntryEventHandler().executeBeforeInsert(logEntryEvents); As part of this, TAG_ASSIGNMENT_RULES variable is set from the results of a CMDT query, which includes flitering on SObjectField__r.QualifiedApiName (the bit which makes the CMDT query contribute to limits like a normal SOQL object query). This variable isn't used until after PEs are created so feels like this should be called only at that stage
2. There are methods queryOrganization(), queryUser(), queryAuthSession() and queryNetwork() in the LogEntryEventBuilder, which all query for details to populate fields on the event. Could it make sense instead to populate just the user id on the event and avoid checking for this extra information until LogEntryEventHandler is processing the events? The downside of this approach is that events would be missing the extra info so an external system consuming these wouldn't have the detail, so maybe the choice of whether these queries are made before or after events should be configurable in settings

I'm happy to create issues and help out with PRs suggesting changes for some or all of these areas if you agree there's a better approach? Or let me know if I'm missing something about how any of these can be avoided today?

Also a heads up I'm hoping to touch on Nebula Logger as part of a talk I'm co-presenting at London's Calling soon (https://www.londonscalling.net/sessions/session-apex-frameworks-for-architects/). Let me know if you'd like more insight into how this would be included (or if for any reason you'd prefer Nebula Logger not to be included!)

Again, thanks for the great work - I've developed a similarly architected logging framework myself and familiar with many of the challenges you've addressed really well here!

Acceptance Criteria:

• Lazy-load query for LogEntryEventHandler.TAG_ASSIGNMENT_RULES - Included in PR Bugfix: logging no longer fails when parent transaction ID is not found #307
• Cache the call to UserInfo.getSessionId()
• Add a configuration option to asynchronously run queryAuthSession()
• Add a configuration option to asynchronously run queryNetwork()
• Add a configuration option to asynchronously run queryOrganization()
• Add a configuration option to asynchronously run queryUser()
• Add a configuration option to skip running LogEntryEventBuilder.parseStackTrace()
• Consider using platform cache to cache queries. Some queries would make sense to cache within the org partition; other queries (such as queries about the current user) should be cached within the session partition.
  • Since not all orgs have platform cache storage, this would need to be optional (probably controlled via a new LoggerParameter__mdt record?). If org/session cache isn't available (or disabled), then Nebula Logger should continue to use the existing transaction cache

Next Steps:

• In order to make the queries reusable (in both the synchronous and asynchronous contexts), all of the queries will need to be centralized into new class(es). This was completed in v4.7.3 (PR Added query selector classes, fixed some recurring bugs #314)
• Need to consider the availability of platform cache in Professional and Essentials editions (and confirm availability for Enterprise & Unlimited)

[jongpie]

Scope creep time! After chatting with @rmccu about some other optimization needs for orgs with large data volume, I'm going to also include 2 more areas in the scope of this issue to partially address some related areas of the codebase:

• Based on Redundant conditions? #384 - Leverage platform cache, with a LoggerParameter__mdt record to enable/disable it - when disabled, Nebula Logger will only use the transaction cache (the current functionality)
  • I've done 2-3 prototypes of platform cache over the last few years, and I've always been hesitant to implement it in Nebula Logger due to concerns with availability of platform cache. But after looking into it further, I think it's feasible & useful to finally leverage it in Nebula Logger. In order to use Nebula Logger, you have to have an edition of Salesforce that supports platform events (an integral part of nebula's architecture). Platform events are currently only supported for Performance, Unlimited, Enterprise, and Developer Editions (source). Similarly, the same editions are required to use platform cache, and each edition has some cache space included (source), so it should be safe to at least reference the Cache namespace in the core nebula package.
  • For orgs that either don't have enough free cache space, or simply don't want Nebula Logger to use platfom cache, a new LoggerParameter__mdt record will be used to disable it in the org.
• Based on Performance issue in LogEntryEventHandler #386 - In addition to the original scope of this issue (specifically, being able to run some queries async), there are some orgs that may want to completely disable some/all queries used in LogEntryEventHandler, LogEventHandler, and LogHandler. This means that some fields on Log__c or LogEntry__c that rely on queried data would simply not be populated. For orgs that don't need some of the fields to be populated & would rather streamline the efficiency of Nebula's performance, several new LoggerParameter__mdt records will be used to disable various queries in the org. I may split some of this work up into smaller issues - there are several queries throughout the codebase, and disabling some of them could cause issues unless I refactor some parts of the Apex. But I'll at least start the process of adding new LoggerParameter__mdt to fully enable/disable queries.