Add BigObject support as the logging location

[jamessimone]

Larger orgs with significant logging volume may be leery of using a custom object to store the Log__c and LogEntry__c items, due to storage space alone. With a high enough throughput, multiple LogBatchPurger jobs would be necessary per hour purely to keep storage space at a sane level. For orgs like this, using BigObjects to store the Log and Log Entry items would be a happier path to success. If this were configurable within Nebula Logger itself, people could choose whether or not their log items were appended to BigObjects or the traditional custom object locations.

[jongpie]

I love this idea - I think the feature makes a lot of sense to add as a configurable option, probably with 4 possible configurations:

1. Enable only log & log entry creation - create Log__c and LogEntry__c records (existing functionality) / don't create BigObject records
2. Enable only big object creation - don't create Log__c and LogEntry__c records/ only create BigObject records
3. Enable everything - create Log__c and LogEntry__c AND BigObject records
4. Disable everything - don't create Log__c, LogEntry__c or BigObject records

I think that would cover all scenarios (the LogEntryEvent__e platform events would always be created since they're the starting pointing for logging)

[jongpie]

@jamessimone one other thought on this - if we have LoggerSettings__c hierarchy setting control the behavior, then that opens up a lot of possibilities with the 4 scenarios that I mentioned above. For example, you could have big objects enabled for the entire org (org defaults), and for certain profiles/users, you could also configure the settings to have have Log__c/LogEntry__c records still created, letting you could mix & match which records get created.

[arvindnarsimhan]

Thanks, @jamessimone for considering this feature. Also, would like to look into the option of writing to BigObjects directly using Queable or Future Apex instead of using Platform Events. Large organizations might want to preserve the limits for Platform Events for Business use cases like CDC etc instead of logging.

[jamessimone]

@arvindnarsimhan as an aside, Nebula Logger already supports the usage of Queueables as the log saving implementation - the saveLog method exposed via the Logger class allows for you to stipulate Logger.SaveMethod.QUEUEABLE to achieve this. Once Big Object support is implemented, you’ll have all the tools you need!

[arvindnarsimhan]

@jamessimone The QueableSaver.execute method again publishes the log to the EventBus. The publishing of events is what happens in the Queable Apex. Was looking to write the log to the BigObject directly instead of using PlatformEvents pub/sub model.

[jongpie]

@arvindnarsimhan that should be fine - I think 2 parts of incorporating BigObject support are:

• Add field(s) on the custom setting to control if it writes to Platform Event or BigObject (or both)
• Update the current queueable job to insert BigObject records directly when writing directly to the BigObject is enabled (if Platform Events are enabled, it'll still use the current EventBus approach)

[arvindnarsimhan]

@arvindnarsimhan that should be fine - I think 2 parts of incorporating BigObject support are:

• Add field(s) on the custom setting to control if it writes to Platform Event or BigObject (or both)
• Update the current queueable job to insert BigObject records directly when writing directly to the BigObject is enabled (if Platform Events are enabled, it'll still use the current EventBus approach)

@jongpie Here is my suggestion. The strategy to move the logs(Platform Events or Future Apex ) and medium to persist the logs (Big Object vs Custom Object ) should be 2 different separate options.

This should give the flexibility to use

1. Platform Events + Custom Objects ( Current approach )
2. Platform Events + Big Object
3. Future APEX method + Custom Object
4. Future APEX method + Big Object

Also, if we discussed an option to write to both Custom Object + Big Object

Here is my approach

1. Add the 4th option to the SaveMethod enum FUTURE_METHOD
2. Config option called LogStorage for NONE, BIG_OBJECTS_ONLY, CUSTOM_OBJECT_ONLY, BIG_AND_CUSTOM_OBJECTS

[jongpie]

@arvindnarsimhan conceptually I like the approach, but a couple of follow up thoughts

• Future method: what's your thought process behind wanting to include a future method? I don't see any benefit to it - the queueable job serves the same purpose of saving asynchronously (we'll update the existing queueable class to support saving to BigObject), and future methods are much more limited overall compared to queueable classes. But let me know if there's a particular scenario where you think a future method would be more advantageous.

• Config options: your list of LogStorage config options doesn't cover one scenario that I think should be handled for Platform Events

  • Publish platform events, but do not store them in custom objects: This scenario isn't covered with your approach. Some orgs will still want to publish the platform events, but not store them in the custom objects. This can happen in orgs that want to continue using the benefits of the pub/sub model without using up data storage. I think using 3 checkbox fields (examples shown in image below) will give more control over the different combinations/scenarios. And since custom settings don't support picklist fields, using checkboxes is less error prone than a text field.

  

[arvindnarsimhan]

@jongpie -

1. Yes you are right. We can augment the current QueableSaver to persist the events in BigObjects or Custom Object. No particular advantage on using Future methods.

2. Your design is very neat. Does it make sense to call LogEntryEventHandler.insertLogEntries directly in QueableSaver.execute to bypass Platform Events?

3. Also, if Platform Events are enabled, we can use the current pub/sub model. The documentation says limits for event delivery only apply for CometD and empApi. We should be good.

4. Think we should have a LogWriter interface with 2 implementations CustomObjectLogWriter and BigObjectLogWriter. We can have an array of LogWriter in LogEntryEventHandler. The array will have CustomObjectLogWriter or BigObjectLogWriter or both depending on user preference. The write method LogWriter will abstract the logic for writing to Big Objects or Custom Object

[jongpie]

@arvindnarsimhan sorry for the late reply!

I've talked with @jamessimone some about how to best approach this - right now, we're planning to have the Logger class internally handle creating LogEntryEvent__e platform event records and/or LogEntryArchive__b BigObject records (new object). The problem with putting the logic into LogEntryEventHandler is that that class is really intended to handle the platform events - so if orgs want to bypass platform events altogether and only use BigObjects, then LogEntryEventHandler won't execute/shouldn't execute (it only executes after insert on LogEntryEvent__e). I think having Logger handle determining where to store the logs is will be a cleaner approach. We'll keep you updated on our progress!

[jongpie]

I think we're going to finally revisit this item - we'll be releasing this functionality as a plugin initially. Long-term, I'd like to make this part of the core package, but starting with it as a plugin will give us the flexibility to make changes to things like the big object's index, which is hard/impossible to change later on.

A few MVP goals for the first release of the plugin:

• It should support logging to a new big object called LogEntryArchive__b - it will look nearly identical to the platform event object LogEntryEvent__e
• The plugin should be able to provide its own save method (BIG_OBJECT or something similar) that can be selected in the LWC loggerSettings
  • When the user's save method is BIG_OBJECT, then log entries should only be saved in the big object - it should not use platform events, and it should not store any data in Log__c or LogEntry__c
• The plugin should run on BEFORE_INSERT for LogEntryEvent__e
• The plugin should include an LWC + custom tab to display & filter on LogEntryArchive__b records. Long term, this custom tab will become part of the Logger Console app, once the plugin is merged into core
• The plugin should include an LWC that can be embedded on a record page (using the target lightning__RecordPage), and only display LogEntryArchive__b records for the current record (based on the field RecordId__c
  • Due to how SOQL & indexes work for big objects, this might involve having a second big object that only contains log entries with a RecordId__c value. If that ends up being necessary, then this big object will be called RecordLogEntryArchive__b or something similar

[jongpie]

@arvindnarsimhan, it took over a year to get this completed, but the v4.7.1 release introduced a new Big Object Archiving plugin that can be used for archiving logging data into a Big Object (closing this issue, as well as the related issue #128).

Let me know if you/your team try out the new plugin, I'd love to hear any feedback that you have!