FATAL ERROR: Database.executeBatch cannot be called from a batch start, batch execute, or future method.

[VigneshRS7101]

Hi @jamessimone ,
I initiated "Start Rollup" from UI which is FULL RECALCULATION CMDT , batch apex 'RollupFullBatchRecalculator' is initiated but there are few failures occurs frequently. And, there was this error 'Database.executeBatch cannot be called from a batch start, batch execute, or future method.' Additionally, 'RollupAsyncProcessor' is queued multiple times. I see in one of your previous discussions Rollup "#139" it is not a expected behaviour. Even that is fixed, do you have solutions on why it is showing this error ?

I am experiencing much data mismatch issues because of this.

Thanks in advance

[jamessimone]

@VigneshRS7101 do you have any additional information? I'm assuming you're using the most up-to-date version?

With this sort of thing, it's going to be hard for me to assist without quite a bit more info - logs, with rollup logging enabled via the Org Default rollup control record, for instance. A good place to start is adding the parent recalc button to a parent record's flexipage and ensuring that for a singular record there are no issues recalculating. From there, we can work backwards to figuring out what the issue is.

[VigneshRS7101]

I am using v1.6.6

Example #1: There are 27 fields we need to run Rollup. If, we run a batch individually for every fields batch is successful, while for all fields it is failing.

My assumption is, there is two execute methods in 'RollupAsyncProcessor' class. One is Database.batchable and other one is Queueable. In that second execute method (queueable) it is calling finish method.

[jamessimone]

You're quite a few versions behind and I would recommend upgrading. From what you're saying, if singular recalcs are working, the issue is much more likely to be a CPU timing issue - which can occur when there is too much automation on the parent object(s) being updated. You might want to try lowering your Batch Chunk Size on the rollup control record to see if that helps. But again, I'd highly recommend upgrading to the latest version, as I am continually refining the performance of batch full recalculations and every little bit helps.

Something else to note - enabling rollup logging consumes additional CPU time, and it should be disabled particularly for batch full recalcs when not debugging

[VigneshRS7101]

@jamessimone I've updated Package today, set rollup logging to false, and reduced Batch Chunk Size to 100. When I initiate Full Recalculation, it shows error "Batchable instance is too big". And, we don't have much automations on Parent object. Still hitting heap size limit.

[jamessimone]

@VigneshRS7101 any chance you're able to reproduce this issue in a sandbox environment? It sounds like you'll need to increase the batch chunk size back up. Without being able to see logs, or getting a look at what's going in a lower environment, I can only guess at what the issue is. I will try to reproduce this issue, but I personally know of a few orgs running more rollups than what you've described without a problem. Hopefully we can work together to understand what's going on here

[VigneshRS7101]

@jamessimone
Hope this helps,
I replicated this in sandbox with Batch Chunk sizes as '50', '100', '200' & '2000' . I am running rollup update batch for 20+ fields on Single Parent record. Still throws same error.
Just a add-on, there should be 3700 batches to be processed, when Batch chunk size is set to '100'

[VigneshRS7101]

@jamessimone I did run rollup batch for 350,000+ Parent records of 30, 100 & 1000 batch chunk sizes. It still throws error 'Batchable instance is too big'. Eventhough, for every batch heap size is around 5-6 MB. For single Parent, Maximum of 5000 child records are present. Can you help resolve me this.

[jamessimone]

I don't think I'll be able to take a look at this issue this week. Try checking the box for Should Skip Resetting Parent Fields and see if that helps. You are not working with large data volume, and I know of (and have worked with) much larger rollup runs than what you currently have set up. So far, you've included a screenshot of the rollup control record - which helps - but no information at all about the sort of rollups you're running, or whether or not you've been able to get an exception stacktrace. I will do what I can to help, but knowing more about your setup might help to gain insight into why you're experiencing an issue.

[VigneshRS7101]

@jamessimone Thanks for your help, I am attaching the logs from batch run. Also adding summary of our rollup process, which may help to find root cause.

1. We have enabled Should Skip Resetting Parent Fields checkbox and ran the batch, still we had got same error. [We don't have
   Parent to Grandparent rollups]
2. The Parent Record which Account has 344000+ and child record has 376000 records. We have Around 24 fields ( 3 Datetime and rest number/currency) to rollup from child to parent, it includes Min, Max, Sum and Avg type of rollups.
3. Around 5 Accounts has more than 1000 child records and around 20 Accounts has more than 100 child records.
4. As we mentioned earlier, I also ran batch with size of 30,50 and 100 and got error for all.
5. The error Batchable instance is too big: RollupFullBatchRecalculator is coming after updgrading the package to 1.6.32 in Fullcopy sandbox, earlier we had 1.6.6.
6. We are getting error, Database.executeBatch cannot be called from a batch start, batch execute, or future method after package upgrade in Fullcopy sandbox.
7. We have very less automations (flow/triggers) for Account. And most of the logic we have bypassed for batch running user.

Stack trace is not printed, you can take look in the logs. Logs are generated at Apex Finest level.

apex-07LU7000006kyVZMAY (1).log
apex-07LU7000006kyVaMAI (1).log

[jamessimone]

Still looking into this. I've been running some full recalcs with 40+ rollups without issue so it's not at all the number of operations per batch chunk that's causing the issue. Yesterday I made some progress in investigating and at least now I have some hypotheses to test. I will keep you updated

[jamessimone]

@VigneshRS7101 I've made progress on this issue, and I'm hopeful to soon wrap up that work and release it. Apologies that it's taken a while, but solving this particular problem in a scalable way has been my primary concern.

I should have more details to share out on this soon.

[jamessimone]

this is fixed in #636