Plugin framework v2 + new Big Object Archive plugin + new Log Retention Rules plugin #298
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stalled plugins, added some prototype changes to LoggerPlugin__mdt
…me placeholder CMDT files for planned plugins
…rt dashboards Not sure if I'll keep this long term
…ignment Rules' plugin
… seems like the better option
…on</excludeButtons> and <excludedStandardButtons>OpenListInQuip</excludedStandardButtons> because dealing with standard buttons is the worst In orgs that don't have these features enabled, the package installation fails
The Spring '22 release includes a beta feature for filtering platform events, so this plugin doesn't seem necessary anymore - https://help.salesforce.com/s/articleView?id=release-notes.rn_platform_events_filtering_beta.htm&type=5&release=236
…eature/plugin-framework-overhaul
…ectHandler to use both old LoggerSObjectHandler class & new LoggerPlugin class, bumped package versionNumber, added a null check for Slack's endpoint, created beta package for core
… on LoggerPlugin__mdt (instead of the old SObjectType__c field)
…ObjectHandlerPlugin
… new LoggerParameter__mdt records to control via configuration
…SObjectHandlerInput, and updated LoggerSObjectHandler to internally use input class
…t configuration, LoggerSObjectHandler.SObjectHandlerInput input)
…h handler's related records in LoggerParameter__mdt & LoggerPlugin__mdt
…erSettings LWC via LoggerParameter__mdt record AvailableSaveMethods
…t add to the list (instead of replacing the list)
…ts for 3 condition logic types (AND, OR, and Custom)
…r__c - sobject-specific fields are now dynamically used instead
… started new test class LoggerSObjectHandler_Tests
* Picking #117 back up with @jongpie to finally add BigObject support through Logger plugin package - includes a new BigObject LogEntryArchive__b and new plugin class * Cleaned up some picklist values for LoggerPlugin__mdt.PluginType__c with an eye towards future BigObject changes * Added an permission set LoggerLogEntryArchiveAdmin within BigObject plugin to grant access to the BigObject LogEntryArchive__b * Switched to lazy-loading the instance of SObjectHandlerInput - tests were previously failing because of the input being generated too early by the trigger framework * Added apex script with sample query for LogEntryArchive__b that includes filters on the indexed fields * Added plugins folder to the script for generating apex doc files, generated updated docs for Apex * Bumped package version number for the managed package * Added 5 minute wait between unlocked package version creation & package installation, added deprecated LoggerPlugin__mdt validation rule & field back into the managed-package folder, cleaned up some docs markdown files Co-authored-by: Jonathan Gillespie <[email protected]>
…BigObject plugin (#288) * Replaced LoggerSObjectHandlerPlugin abstract class with new class LoggerPlugin that contains 2 interfaces + helper methods, switched to using multiple fields on LoggerPlugin__mdt to indicate Apex classes & Flows to run for a plugin, removed SObject-specific fields on LoggerPlugin__mdt, added new fields Log__c.LogPurgeAction__c and LoggerSettings__c.DefaultLogPurgeAction__c * Added tests in LoggerSettingsController_Tests & LogHandler_Tests for new custom setting LoggerSettings__c.DefaultLogPurgeAction__c * Updated LoggerSObjectHandler & plugin classes to use the NEW new plugin overhaul changes via LoggerPlugin class, removed old class LoggerSObjectHanderPlugin * Expanded the plugin framework to support some aspects of #128 by adding the ability to create plugins for LogBatchPurger, using the interface LoggerPlugin.Batchable - The BigObject plugin (and other plugins) can then leverage this to run additional logic before Log__c, LogEntry__c & LogEntryTag__c records are hard-deleted - For the BigObject plugin, it will be able to archive data into LogEntryArchive__b before LogBatchPurger deletes the data within the custom objects - Also finished some test improvements for triggerable plugins within LoggerSObjectHandler * Renamed BigObject plugin's CMDT file for save method to reflect the naming convention change ('CustomSaveMethodBigObject' instead of 'AdditionalSaveMethodsBigObject'), and renamed plugin CMDT file from 'BigObjectArchiving' to 'LogEntryArchiving', updated labels on some deprecated fields * Added 'deprecated' to the label of several deprecated fields on LoggerPlugin__mdt & removed the related handler methods * WIP Made progress on #128 - Split part of LogEntryArchiveBuilder into a new class, LogEntryArchivePlugin, and implemented interface LoggerPlugin.Batchable so that Log__c/LogEntry__c/LogEntryTag__c records can be archived via LogBatchPurger before they're deleted - The plugin class handles talking with Logger, and builder class handles converting LogEntryEvent__e or LogEntry__c records to LogEntryArchive__b records * Added tests for LogBatchPurger integration in LogEntryArchivePlugin_Tests * WIP Stubbed out a new LWC + controller class + custom tab for viewing LogEntryArchive__b as part of #117 * Standardized test-visible method naming conventions to start with `setMock` & updated approaches to use a Map instead of List for CMDT records * Finished some TODOs & standardized mocking approach for CMDT in LogHandler * Added Log__c list view to show logs that will be purged in the next 10 days, fixed LogEntryArchiveBuilder using the wrong value for LoggedById__c, updated index on LogEntryArchive__b again (still a WIP), retrieved & formatted metadata for LogEntryArchive__b, updated Admin.profile * Fixed some FLS issues * More improvements for CMDT records used in the classes LoggerParameter & LoggerPlugin - Added LoggerParameter.matchOnPrefix() to return CMDT records with a specified prefix in the DeveloperName field - Added inner Comparable class in LoggerPlugin to handle custom sorting, since there are some limitations with SOQL queries for CMDT - Classes like LogBatchPurger, LoggerSObjectHandler, and LoggerSettingsController no longer need to track their own mock CMDT records, the LoggerPlugin & LoggerParameter classes now fully handle mocks for their corresponding CMDT objects - LoggerParameter & LoggerPlugin now require mock CMDT records to have DeveloperName populated - they'll throw errors if the field is null - Also added System namespace to some calls for Test.isRunningTest() to handle crazy orgs that have a custom Test class deployed * Added the ability to disable all triggers during tests via new @testvisible method LoggerSObjectHandler.shouldExecute(Boolean) * Removed old integration test class + Flow for testing Flow plugins in LoggerSObjectHandler - I'll reimplement new tests when I also implement Flow plugins for LogBatchPurger * Moved LoggerTestUtils to the configuration folder, added public methods for mocking all CMDT records * Consolidated plugin-framework folders back into the configuration folders
jamessimone
reviewed
Apr 26, 2022
| } | ||
|
|
||
| private void truncateFieldValues() { | ||
| // TODO |
jamessimone
reviewed
Apr 26, 2022
| public void start(LoggerPlugin__mdt configuration, LoggerBatchableContext input) { | ||
| // Skip directly deleting LogEntryTag__c records so that the tags can be included when LogEntry__c records | ||
| // are archived into LogEntryArchive__b | ||
| if (input.sobjectType == Schema.LogEntryTag__c.SObjectType) { |
There was a problem hiding this comment.
I feel like it makes more sense to do != Schema.LogEntry__c.SObjectType here - comparing to LogEntryTag__c feels like this class knows a bit too much
jamessimone
reviewed
Apr 26, 2022
| } | ||
| String closedByUsernameField = | ||
| logLookupRelationshipName + | ||
| '.' + |
There was a problem hiding this comment.
this looks like it could be cleaned up a bit, or at least have a TODO for drying it up
There was a problem hiding this comment.
Good call, there's a lot of duplicated code here - I'll try to move some of the duplicate logic here into a private method for reuse + cleanup.
jamessimone
reviewed
Apr 26, 2022
| } | ||
|
|
||
| public Boolean matchesFilter() { | ||
| if (this.filterCondition.ValueType__c == 'RegEx') { |
There was a problem hiding this comment.
long term, I feel like these strings could be represented by a private enum that you always fetch when referencing ValueType__c, and maybe short term have constants for them? This is really a nice to have, but just something I noticed while scrolling through this file
There was a problem hiding this comment.
Yeah, I love that idea - I'll add a TODO to make that change in a future release.
jamessimone
reviewed
Apr 26, 2022
| // Object yesterday = System.today().addDays(-1); | ||
| // System.assert(today > yesterday); // This line cannot execute since it's comparing Object | ||
| private Boolean compareDatetime(Datetime recordFieldValue, Datetime comparisonValue) { | ||
| switch on this.filterCondition.Operator__c { |
There was a problem hiding this comment.
same thing here, with Operator__c. Again, no expectations as far as making these changes as part of this PR, but good food for thought!
There was a problem hiding this comment.
Same, I'll add a TODO for now, but I love the idea of using a private enum
jamessimone
reviewed
Apr 26, 2022
nebula-logger/plugins/log-retention-rules/plugin/classes/LogRetentionFilter.cls
Show resolved
Hide resolved
jamessimone
reviewed
Apr 26, 2022
|
|
||
| for (Schema.SObjectField field : this.fieldChain) { | ||
| Schema.DescribeFieldResult fieldDescribe = field.getDescribe(); | ||
| // TODO delete? String relationshipName = fieldDescribe.getRelationshipName(); |
jamessimone
reviewed
Apr 26, 2022
nebula-logger/plugins/log-retention-rules/plugin/classes/LogRetentionFilter.cls
Show resolved
Hide resolved
jamessimone
reviewed
Apr 26, 2022
| } | ||
| } | ||
|
|
||
| // TODO Keep in this class |
jamessimone
reviewed
Apr 26, 2022
| return (List<LogEntry__c>) Database.query(logEntryQuery); | ||
| } | ||
|
|
||
| // TODO Keep in this class |
There was a problem hiding this comment.
same for these other TODOs?
jamessimone
reviewed
Apr 26, 2022
| AND SendSlackNotification__c = TRUE | ||
| AND SlackNotificationDate__c = NULL | ||
| ]; | ||
| } | ||
|
|
||
| @SuppressWarnings('PMD.AvoidDebugStatements') | ||
| private HttpRequest createSlackHttpRequest() { | ||
| System.debug(notificationLoggingLevel, 'endpoint==' + endpoint); | ||
| System.debug(NOTIFICATION_LOGGING_LEVEL, 'ENDPOINT==' + ENDPOINT); |
There was a problem hiding this comment.
going back to the "does this make sense to call Logger.debug?" question
- Eliminated a double-hardcoded string in LoggerParameter - Removed @TestVisbile annotation on some properties in LogBatchPurger, added TODO on some @testvisible methods to potentially raise the methods' visibility in a future release - Cleaned up error message creation in ComponentLogger - Added data masking for new fields HttpRequestBody__c and HttpResponseBody__c, and added new boolean fields to indicate if the fields have been masked (HttpRequestBodyMasked__c and HttpResponseBodyMasked__c) - Eliminated some hardcoded strings in Logger where enum values could be used instead - Updated some internal debug messages in various places to use Logger's logging methods, instead of System.debug() - Removed some old/completed TODOs, cleaned up some code formatting - Deprecated the fields LogEntryDataMaskRule__mdt.ApplyToMessage__c and LogEntryDataMaskRule__mdt.ApplyToRecordJson__c - Added new methods LoggerMockDataCreator.createHttpRequest() and LoggerMockDataCreator.createHttpResponse()
…retention rules plugin) to provide better examples of how to configure rules
jongpie
force-pushed
the
feature/plugin-framework-overhaul
branch
from
a8ba81e to
5a2dd4d
Compare
…fixed test failures that occur when email deliverability is disabled in an org
…ject Archiving, Log Retention Rules, and Slack
jongpie
changed the title
Codecov Report
@@ Coverage Diff @@
## main #298 +/- ##
==========================================
- Coverage 95.80% 95.57% -0.24%
==========================================
Files 30 43 +13
Lines 3173 4583 +1410
Branches 63 92 +29
==========================================
+ Hits 3040 4380 +1340
- Misses 132 200 +68
- Partials 1 3 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
jongpie
added
the
Plugin: Big Object Archiving
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Core Package Changes
This release of the core unlocked package is primarily focused on improving how plugins can interact with the core package - while working towards that goal, several enhancements & bugfixes have been included this release
Replaced the checkbox field
LoggerSettings__c.IsPlatformEventStorageEnabled__cwith a new text fieldLoggerSettings__c.DefaultPlatformEventStorageLocation__c, providing a way for plugins (big object plugin in particular - see below) to provide additional storage location.LoggerSettings__c.DefaultPlatformEventStorageLocation__c- if you previously hadLoggerSettings__c.IsPlatformEventStorageEnabled__cset totrue, then you should updateLoggerSettings__c.DefaultPlatformEventStorageLocation__cto be set toCUSTOM_OBJECTS. This is the default value going forward, and can be updated using the "Logger Settings" tab available in the Logger Console app (see screenshot below)Added new picklist field
Log__c.LogPurgeAction__c- out of the box, only the value 'Delete' is included/used, but plugins (like the Big Object plugin below) can add new picklist values to support new actions.Fixed an issue in
LogEntryEventBuilderwhere some stack trace lines would be duplicatedRenamed class
LoggerEmailUtilstoLoggerEmailSenderAdded additional fixes for LoggerEmailUtils_Tests failed #276 that was partially fixed in
v4.7.0- some of the unit tests had not been updated to check if deliverability was enabled, resulting in tests still failing in orgs with deliverability disabled. Thanks to @gjslagle12 for reporting these test failures!Added new public method
LogBatchPurger.setChainedBatchSize(Integer)that's used internally to ensure any chained batch jobs use the same batch size as the original job. Previously, only the first job would use the specified batch size, and any chained jobs then used the default of 200.Started adding data classifications to custom fields throughout the data model to start progress on Start tracking the data classification info on each custom field #292
New fields
DatabaseResultCollectionSize__candRecordCollectionSize__c(originally planned as part of Add support for logging Map<Id, SObject> #222)Partially implemented Add support for logging HttpRequest and HttpResponse objects #240 by adding new methods
LogEntryEventBuilder.setHttpRequestDetails(request)andLogEntryEventBuilder.setHttpResponseDetails(response), which populates new fields onLogEntryEvent__eandLogEntry__c. In a future release, I am going to consider also adding overloads to the logging methods inLogger. The new fields onLogEntryEvent__eandLogEntry__care:HttpRequestBody__cHttpRequestBodyMasked__cHttpRequestCompressed__cHttpRequestEndpoint__cHttpRequestMethod__cHttpResponseBody__cHttpResponseBodyMasked__cHttpResponseHeaderKeys__cHttpResponseStatus__cHttpResponseStatusCode__cVersion 2 of Plugin Framework
This release includes a new & improved approach for building plugins for Nebula Logger. The first plugin framework beta (referred to as
plugin-v1-beta) was originally released last year in thev4.5.0release of the unlocked package. Since then, it's remained largely unchanged, but there has been a lot of feedback in the last ~9 months. The new beta of the plugin framework (plugin-v2-beta) is a complete overhaul of how plugins are built for Nebula Logger, allowing much more control and functionality for plugins.Redesigned plugins for Nebula Logger's trigger handler framework, and added the ability to create plugins for the batch class
LogBatchPurger. The old Apex classLoggerSObjectHandlerPluginhas been removed - Apex plugins can now be created by implementing one (or both) of the new interfaces:LoggerPlugin.Batchable- this interface is used to define & run plugins within the batch jobLogBatchPurgerLoggerPlugin.Triggerable- this interface is used to define & run plugins within Nebula Logger's trigger framework,LoggerSObjectHandlerReintroduced
LoggerSObjectHandler__mdtcustom metadata type. This can be used to enable/disable some of Nebula Logger's trigger handler classes, as well as a way to override the default trigger handlers with a custom oneNew Plugin: Log Entry Archive Plugin
This new plugins provides archiving of logging data in a Big Object, allowing you to clear up data storage used by the custom objects (
Log__c,LogEntry__c, andLogEntryTag__c) while still housing your logging data within your Salesforce org. A huge 'thank you' to @jamessimone for all of his work on this - he and I originally started work on this over a year ago, and it unfortunately was put on hold for several months while other technical debt & enhancements were first prioritized. It's incredible to finally see this being released!The Big Object
LogEntryArchive__bcontains all of the same fields (or comparable fields) asLogEntryEvent__e,Log__c, andLogEntry__ccombined.Closes Add BigObject support as the logging location #117 - a huge thanks to @jamessimone for implementing this via PR Big Object Plugin #287 (and for creating Big Object prototypes last year!). The plugin provides 2 custom save methods that can be used to bypass platform events (
LogEntryEvent__e) and custom objects (Log__c,LogEntry__c, andLogEntryTag__c) and instead use the Big ObjectLogEntryArchive__bas the primary storage location. This also closes Add support for archiving Log__c and LogEntry__c records into a BigObject #128 - implemented via PR Added support for plugins within LogBatchPurger + added archiving in BigObject plugin #288, the plugin can also archiveLog__c,LogEntry__candLogEntryTag__cdata before the batch job deletes any records whereLog__c.LogPurgeAction__c == 'Archive'. This means that the plugin can be configured in 4 ways:LoggerSettings__c.DefaultSaveMethod__c=EVENT_BUS,LoggerSettings__c.DefaultPlatformEventStorageLocation__c=BIG_OBJECT- with these options, Nebula Logger will still leverage the Event Bus, which ensures that log entries are saved, even if an exception is thrown. This may not be ideal for all orgs/users due to org limits for platform events, but this would provide the most reliable way of logging directly toLogEntryArchive__b& circumvent the custom objectsLog__c,LogEntry__candLogEntryTag__cLoggerSettings__c.DefaultSaveMethod__c=BIG_OBJECT_IMMEDIATE- with this option, Nebula Logger will skip the Event Bus, and instead try to write directly to the Big ObjectLogEntryArchive__b. Any Big Object records that are saved will not be rolled back if there are any exceptions in the transaction - however, this option only works if you save the Big Objects before performing DML on any "normal" SObjects. If you perform DML on another SObject first, and then attempt to save directly to the Big Object, the platform will throw a mixed DML exception, and no Big Object records will be saved.LoggerSettings__c.DefaultSaveMethod__c=BIG_OBJECT_QUEUEABLE- with this option, Nebula Logger will asynchronously save Big Object records using a queueable job. This is helpful in avoiding hitting limits in the original transaction, and also avoids the mixed DML exception that can occur when usingBIG_OBJECT_IMMEDIATE(above). However, if an exception occurs in the current transaction, then the queueable job will not be enqueued.LoggerSettings__c.DefaultSaveMethod__c=EVENT_BUS,LoggerSettings__c.DefaultPlatformEventStorageLocation__c=CUSTOM_OBJECTS,LoggerSettings__c.DefaultLogPurgeAction__c= 'Archive' - with these options configured, Nebula Logger will utilize the Event Bus to ensure any log entries are published (even if an exception occurs), and the data is then initially stored in the custom objectsLog__c,LogEntry__candLogEntryTag__c. Once the log's retention date has passed (Log__c.LogRetentionDate__c <= System.today(), then the plugin will archive the custom object data intoLogEntryArchive__bbefore the custom object data is deleted.The included permission set
LoggerLogEntryArchiveAdminprovides all of the permissions needed forLogEntryArchive__band the includedLog Entry ArchivestabIncludes a custom tab 'Log Entry Archives' to display the LWC
logEntryArchives. This LWC provides a datatable view ofLogEntryArchive__bdata, with the ability to filter onTimestamp__c,LoggingLevel__c, andMessage__cfields.New Plugin: Log Retention Rules
This new plugin closes #226 by adding the ability to create & deploy advanced, configurable rules for setting the retention date of
Log__crecords, using custom metadata typesLogRetentionRule__mdtandLogRetentionRuleCondition__mdt. This provides a way to create more advanced log retention policies for logging data stored inLog__c,LogEntry__c, andLogEntryTag__c, using functionality that's similar to adding custom logic to list views. This plugin's code is based on another open source project of mine, ApexValidationRulesAnd for anyone wondering "can I use the Log Retention Rules plugin with the Log Entry Archive plugin?" - the answer is yes! If you install & configure both plugins in your org, then you can create log retention rules to better control when logs are purged - the Log Entry Archive plugin will then run within the
LogBatchPurgerjob, and anyLog__crecords withLogPurgeAction__c == 'Archive'will be archived into the Big ObjectLogEntryArchive__bon the log's retention end date.Updated Plugin: Logger Slack Plugin
Only a few small changes have been included in this release of the Slack plugin, but upgrading is required in order to work with the latest changes to Nebula Logger's plugin framework
LogEntry__c.StackTrace__candLogEntry__c.ExceptionStackTrace__cto the Slack notification messageTest Improvements
LoggerMockDataCreator: Utility class used to help with generating mock data when writing Apex tests for Nebula Logger. These methods are generic, and should work in any Salesforce org. They can be used when writing Apex tests for plugins.LoggerMockDataStore: Utility class used to mock any data-related operations, including DML statements, Event Bus publishing, and enqueuing async queueable jobs. These methods are generic, and should work in any Salesforce org. They can be used when writing Apex tests for plugins.LoggerTestConfigurator: Utility class used to help with setting up Nebula Logger's configurations within a test context. These methods are specific to metadata implemented within Nebula Logger. They can be used when writing Apex tests for plugins.LoggerMockDataStore. More improvements to come in future releases that will further reduce the average test speed, and improve the overall quality of the tests.@IsTest(IsParallel=true)wherever possible to try to further improve test speeds. Some classes that perform DML onUserrecords cannot leverage parallel test runs, so a few test classes use@IsTest(IsParallel=false)