Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for plugins within LogBatchPurger + added archiving in BigObject plugin #288

Merged

Conversation

jongpie
Copy link
Owner

@jongpie jongpie commented Mar 22, 2022

As a follow-up to #287 (and part of a larger overhaul of the plugin framework for the upcoming release v4.7.1), I've reworked even more of the plugin framework to add support for plugins within the LogBatchPurger class

Plugin Framework Changes

  • Eliminated the old plugin approach by deleting the abstract class LoggerSObjectHandlerPlugin. It has been replaced with 2 new interfaces:
    1. LoggerPlugin.Batchable - used to create plugins for LogBatchPurger (batch deletion job)
    2. LoggerPlugin.Triggerable - used to create plugins for LoggerSObjectHandler (trigger framework used by Logger)
  • Cleaned up the fields on LoggerPlugin__mdt + added new fields to control the sort order of plugins used by LogBatchPurger
  • Added a new settings field LoggerSettings__c.DefaultLogPurgeAction__c - this then sets the default value of the field Log__c.LogPurgeAction__c. This field value isn't used by LogBatchPurger directly, but it provides a place for plugins to set/read additional actions that can be used within plugin logic (see BigObject changes below for an example)
    • I've also updated LoggerSettingsController to query for LoggerParameter__mdt records that start with 'CustomLogPurgeAction' so that plugins can define & package new actions. Any matching values will be displayed in the LWC loggerSettings as a picklist option for LoggerSettings__c.DefaultLogPurgeAction__c
  • LogBatchPurger now uses a new inner class BatchPurgerInput to provide contextual info to plugins
    • I realized earlier today that the class LogBatchPurger.BatchPurgerInput and LoggerSObjectHandler.SObjectHandlerInput will both need to be changed to top-level classes so that Flow plugins can use them. From this page, Apex-defined types can't be used in Flow unless they are top-level classes. I've already scope-creeped in this branch a lot, so I'll make this change after merging this PR into the plugin-framework-overhaul branch.
  • Scope creep: standardized some of the @TestVisible methods used throughout the codebase to mock custom metadata records

BigObject Plugin Changes

The BigObject plugin has been updated to also work as a batch purger plugin (in addition to being an SObject handler plugin that @jamessimone implemented in #287)

  • Split the class LogEntryArchiveBuilder into 2 classes:
    1. LogEntryArchiveBuilder - this now only handles converting a LogEntryEvent__e into a LogEntryArchive__b record (implemented via Big Object Plugin #287), as well as converting a LogEntry__c record into a LogEntryArchive__b record (to cover most of the functionality in Add support for archiving Log__c and LogEntry__c records into a BigObject #128)
    2. LogEntryArchivePlugin - this new class implements the interfaces LoggerPlugin.Batchable and LoggerPlugin.Triggerable so archiving can be used as the initial storage place (when a user's save method == BIG_OBJECT) as well as a secondary storage place (when a Log__c has LogPurgeAction__c == 'Archive')
  • Added more fields on LogEntryArchive__b to (more or less) match the list of fields on LogEntryEvent__e
  • The BigObject plugin now includes a custom value 'Archive' for the field LoggerSettings__c.DefaultLogPurgeAction__c. When LogBatchPurger executes, any matching Log__c/LogEntry__c/LogEntryTag__c data will be archived into LogEntryArchive__b before being deleted by LogBatchPurger
  • I've also made a few tweaks to the index for LogEntryArchive__b, but I'm guessing we'll probably try a few other variations on what to use within the index
  • I also included some placeholder metadata that will eventually be used to display LogEntryArchive__b records via a new LWC + custom tab

Slack Plugin Changes & Log Retention Rules Plugin Changes

  • The metadata for both plugins have been updated to use the latest changes to the plugin framework - there shouldn't be any functional changes

jongpie added 14 commits March 20, 2022 00:34
…gerPlugin that contains 2 interfaces + helper methods, switched to using multiple fields on LoggerPlugin__mdt to indicate Apex classes & Flows to run for a plugin, removed SObject-specific fields on LoggerPlugin__mdt, added new fields Log__c.LogPurgeAction__c and LoggerSettings__c.DefaultLogPurgeAction__c
…new custom setting LoggerSettings__c.DefaultLogPurgeAction__c
…in overhaul changes via LoggerPlugin class, removed old class LoggerSObjectHanderPlugin
…ng the ability to create plugins for LogBatchPurger, using the interface LoggerPlugin.Batchable

- The BigObject plugin (and other plugins) can then leverage this to run additional logic before Log__c, LogEntry__c & LogEntryTag__c records are hard-deleted
- For the BigObject plugin, it will be able to archive data into LogEntryArchive__b before LogBatchPurger deletes the data within the custom objects
- Also finished some test improvements for triggerable plugins within LoggerSObjectHandler
…aming convention change ('CustomSaveMethodBigObject' instead of 'AdditionalSaveMethodsBigObject'), and renamed plugin CMDT file from 'BigObjectArchiving' to 'LogEntryArchiving', updated labels on some deprecated fields
… a new class, LogEntryArchivePlugin, and implemented interface LoggerPlugin.Batchable so that Log__c/LogEntry__c/LogEntryTag__c records can be archived via LogBatchPurger before they're deleted

- The plugin class handles talking with Logger, and builder class handles converting LogEntryEvent__e or LogEntry__c records to LogEntryArchive__b records
…gin_Tests, fixed tests in LogBatchPurger_Tests
…tMock` & updated approaches to use a Map instead of List for CMDT records
…0 days, fixed LogEntryArchiveBuilder using the wrong value for LoggedById__c, updated index on LogEntryArchive__b again (still a WIP), retrieved & formatted metadata for LogEntryArchive__b, updated Admin.profile
@jongpie jongpie added Type: Enhancement New feature or request Plugin: Big Object Archiving Items related to archiving logging data into the LogEntryArchive__b big object Layer: Log Management Items related to the custom objects & Logger Console app Package Type: Unlocked Package Issues and enhancements that specifically apply to the unlocked package Feature: Log Retention Items related to LogBatchPurger or LogBatchPurgeScheduler labels Mar 22, 2022
@jongpie jongpie requested a review from jamessimone March 22, 2022 19:15
@jongpie
Copy link
Owner Author

jongpie commented Mar 22, 2022

@jamessimone there are a few failing tests in this PR that I'm not too worried about for this PR - 1 is an issue for Flows that I'll fix in the plugin-framework-overhaul branch, and a few test failures are because I've only created the skeleton for a new LWC/tab to show LogEntryArchive__b (and I included some System.assert(false) calls as placeholders/reminders)

@jamessimone
Copy link
Collaborator

Awesome. Incredible write-up — looking forward to reviewing!

jongpie added 3 commits March 23, 2022 15:05
…r & LoggerPlugin

- Added LoggerParameter.matchOnPrefix() to return CMDT records with a specified prefix in the DeveloperName field
- Added inner Comparable class in LoggerPlugin to handle custom sorting, since there are some limitations with SOQL queries for CMDT
- Classes like LogBatchPurger, LoggerSObjectHandler, and LoggerSettingsController no longer need to track their own mock CMDT records, the LoggerPlugin & LoggerParameter classes now fully handle mocks for their corresponding CMDT objects
- LoggerParameter & LoggerPlugin now require mock CMDT records to have DeveloperName populated - they'll throw errors if the field is null
- Also added System namespace to some calls for Test.isRunningTest() to handle crazy orgs that have a custom Test class deployed
…stantSettings to both scratch org definition files
@jongpie jongpie had a problem deploying to Experience Cloud Scratch Org March 25, 2022 21:10 Failure
… LoggerSObjectHandler - I'll reimplement new tests when I also implement Flow plugins for LogBatchPurger
@jongpie jongpie force-pushed the feature/batch-purger-plugins branch from 9ed34d6 to b3f1907 Compare March 25, 2022 21:34
@jongpie jongpie temporarily deployed to Base Scratch Org March 25, 2022 21:36 Inactive
@jongpie jongpie temporarily deployed to Experience Cloud Scratch Org March 25, 2022 21:36 Inactive
@jongpie jongpie temporarily deployed to Demo Org March 25, 2022 21:46 Inactive
@jongpie jongpie force-pushed the feature/batch-purger-plugins branch from 232a85a to 90fa34b Compare March 26, 2022 02:47
…lse to reduce issues in orgs upgrading, re-addedthe deprecated class LoggerSObjectHandlerPlugin to the managed package's deprecated folder
@jongpie jongpie force-pushed the feature/batch-purger-plugins branch from 90fa34b to 54b1c0f Compare March 26, 2022 02:50
@jongpie jongpie temporarily deployed to Experience Cloud Scratch Org March 26, 2022 02:53 Inactive
@jongpie jongpie temporarily deployed to Base Scratch Org March 26, 2022 02:53 Inactive
@jongpie jongpie temporarily deployed to Demo Org March 26, 2022 03:06 Inactive
@jongpie jongpie temporarily deployed to Demo Org March 26, 2022 03:06 Inactive
@jongpie jongpie force-pushed the feature/batch-purger-plugins branch from b109fcf to 8345106 Compare March 27, 2022 01:50
@jongpie jongpie force-pushed the feature/batch-purger-plugins branch from 8345106 to 5bc2e2e Compare March 27, 2022 02:08
@jongpie jongpie had a problem deploying to Experience Cloud Scratch Org March 27, 2022 02:11 Error
@jongpie jongpie temporarily deployed to Experience Cloud Scratch Org March 27, 2022 02:21 Inactive
@jongpie jongpie temporarily deployed to Base Scratch Org March 27, 2022 02:21 Inactive
@jongpie jongpie temporarily deployed to Demo Org March 27, 2022 02:34 Inactive
@jongpie jongpie temporarily deployed to Demo Org March 27, 2022 02:34 Inactive
@codecov
Copy link

codecov bot commented Mar 27, 2022

Codecov Report

❗ No coverage uploaded for pull request base (feature/plugin-framework-overhaul@b3ebafc). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 872b052 differs from pull request most recent head f19e999. Consider uploading reports for the commit f19e999 to get more accurate results

@@                         Coverage Diff                          @@
##             feature/plugin-framework-overhaul     #288   +/-   ##
====================================================================
  Coverage                                     ?   93.67%           
====================================================================
  Files                                        ?       40           
  Lines                                        ?     4176           
  Branches                                     ?       86           
====================================================================
  Hits                                         ?     3912           
  Misses                                       ?      259           
  Partials                                     ?        5           
Flag Coverage Δ
Apex 94.52% <0.00%> (?)
LWC 88.60% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3ebafc...f19e999. Read the comment docs.

@jongpie jongpie merged commit d25e766 into feature/plugin-framework-overhaul Mar 27, 2022
@jongpie jongpie deleted the feature/batch-purger-plugins branch March 27, 2022 03:25
jongpie added a commit that referenced this pull request May 2, 2022
…on Rules plugin (#298)

- Replaced `LoggerSObjectHandlerPlugin` abstract class with new class `LoggerPlugin` that contains 2 interfaces + helper methods, switched to using multiple fields on `LoggerPlugin__mdt` to indicate Apex classes & Flows to run for a plugin, removed SObject-specific fields on `LoggerPlugin__mdt`, added new fields `Log__c.LogPurgeAction__c` and `LoggerSettings__c.DefaultLogPurgeAction__c`

- Added support for plugins within LogBatchPurger + added archiving in BigObject plugin (#288)

- Added SObject-specific 'execution order' fields on `LoggerPlugin__mdt`

- Added picklist values for API versions in calendar year 2022

- Reintroduced the ability to disable trigger handler classes by adding new `LoggerParameter__mdt` records to control via configuration

- Renamed CMDT records that control each trigger handler

- Added new abstract methods in LoggerSObjectHandler for retrieving each handler's related records in `LoggerParameter__mdt` & `LoggerPlugin__mdt`

- Added the ability to configure the list of save methods shown in `loggerSettings` LWC via `LoggerParameter__mdt`, using the prefix 'CustomSaveMethod'

- Standardized test-visible method naming conventions to start with `setMock` & updated approaches to use a Map instead of List for CMDT records

- Fixed issues in LoggerSObjectHandler for invalid Apex & Flow plugins, started new test class LoggerSObjectHandler_Tests

- Added `Log__c` list view to show logs that will be purged in the next 10 days, fixed LogEntryArchiveBuilder using the wrong value for LoggedById**c, updated index on LogEntryArchive**b again (still a WIP), retrieved & formatted metadata for LogEntryArchive\*\*b, updated Admin.profile

- Closed #226 by adding new "Log Retentions Rule" plugin, based on my other project, ApexValidationRules

- @jamessimone closed #117, and I closed #228 by adding new "Big Object Archiving" plugin

- Lowercased the plugin directories

- Added plugins folder to the script for generating apex doc files, generated updated docs for Apex

- Bumped package version number for the managed package

- Added 5 minute wait between unlocked package version creation & package installation, added deprecated `LoggerPlugin__mdt` validation rule & field back into the managed-package folder, cleaned up some docs markdown files

- Added tests in LoggerSettingsController_Tests & LogHandler_Tests for new custom setting LoggerSettings**c.DefaultLogPurgeAction**c

- Added 'deprecated' to the label of several deprecated fields on `LoggerPlugin__mdt` & removed the related handler methods

- More improvements for CMDT records used in the classes LoggerParameter & LoggerPlugin

  - Added LoggerParameter.matchOnPrefix() to return CMDT records with a specified prefix in the DeveloperName field
  - Added inner Comparable class in LoggerPlugin to handle custom sorting, since there are some limitations with SOQL queries for CMDT
  - Classes like LogBatchPurger, LoggerSObjectHandler, and LoggerSettingsController no longer need to track their own mock CMDT records, the LoggerPlugin & LoggerParameter classes now fully handle mocks for their corresponding CMDT objects
  - LoggerParameter & LoggerPlugin now require mock CMDT records to have DeveloperName populated - they'll throw errors if the field is null
  - Also added System namespace to some calls for Test.isRunningTest() to handle crazy orgs that have a custom Test class deployed

- Added the ability to disable all triggers during tests via new @testvisible method LoggerSObjectHandler.shouldExecute(Boolean)

- Moved LoggerTestUtils to the configuration folder, added public methods for mocking all CMDT records

- Consolidated plugin-framework folders back into the configuration folders

- Moved demo experience cloud metadata under config, instead of under nebula-logger, updated build.yml to deploy instead of push metadata

- Updated test classes to use new overloads for LoggerTestUtils.setMock()

- Added new class LoggerDataStore to centralize all database, event bus & queueable operations

- Closed #193 by adding classes `LoggerMockDataCreator` for generic mock & data creation (part of configuration layer), `LoggerTestConfigurator` forNebula Logger-specific mocking & setup (part of the log-management layer), and `LoggerMockDataStore` for mocking database & event bus operations throughout the codebase

- Started work on #292 by adding businessStatus, complianceGroup & securityClassification nodes to several custom fields (still a WIP)

- Rewrote tests in `LogEntryEventBuilder_Tests` to use new methods in `LoggerMockDataCreator` for creating mock instances of database result classes

- Scope creep: added fallback fields for setting `RecordName__c` in LogEntryHandler on objects that don't have a display name field, also added some caching

- Moved permissionSets and customPermissions folders from configuration to log-management

  - This should make the configuration folder fully standalone (i.e., it could be deployed by itself)
  - The perm sets were already referencing all of the objects within log-management, so it seems like it's better to keep the perm sets organized with

- Changed labels to just "Entry #" on both `LogEntryEvent__e` and `LogEntryArchive__b`

- Fixed an issue in `LogEntryEventBuilder` where some stack trace lines would be duplicated

- Scope creeped a bit by adding new fields `DatabaseResultCollectionSize__c` and `RecordCollectionSize__c` - I had planned to add these as part of #222, but decided to add them now so I could duplicate the fields to the `LogEntryArchive__b` big object as well. This also helped to identify (and fix) some issues with missing asserts in LogEntryEventHandler_Tests.

- Replaced settings field `IsPlatformEventStorageEnabled__c` with new field `DefaultPlatformEventStorageLocation__**__c`, providing a way for plugins (big object plugin in particular) to provide additional storage location. The big object plugin provides a custom option BIG_OBJECT, in addition to the default value CUSTOM_OBJECTS that will now be used out of the box

- Added stack trace fields to the Slack message created via SlackLoggerPlugin

- Updated most test classes to use `@IsTest(isParallel=true)`. A few tests (especially integration tests) that do DML on User can't use parallel testing, but most test classes are now using it

- Partially implemented #240 by adding HTTP Request & HTTP Response fields to `LogEntryEvent__e`, `LogEntry__c` and `LogEntryArchive__b`. More scope creep yet again, but this will save some headaches long term - since Big Objects can't (easily) have new fields added, I wanted to go ahead and get these new fields added to the data model before the first release of the Big Object plugin. I'll probably add some logging overloads in Logger for these items in a future release, but for now, only LogEntryEventBuilder has new methods for HttpRequest & HttpResponse

- Updated some internal debug messages in various places to use Logger's logging methods, instead of `System.debug()`

- Deprecated the fields `LogEntryDataMaskRule__mdt.ApplyToMessage__c` and `LogEntryDataMaskRule__mdt.ApplyToRecordJson__c`

- Fixed test failures that occur when email deliverability is disabled in an org

Co-authored-by: James Simone <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature: Log Retention Items related to LogBatchPurger or LogBatchPurgeScheduler Layer: Log Management Items related to the custom objects & Logger Console app Package Type: Unlocked Package Issues and enhancements that specifically apply to the unlocked package Plugin: Big Object Archiving Items related to archiving logging data into the LogEntryArchive__b big object Type: Enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants