clarify microbatch per feedback #6544
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
github-actions
bot
added
content
|
|
||
| When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` and `batch_size` you configure. | ||
|
|
||
| Each "batch" corresponds to a single bounded time period (by default, a single day of data). Where other incremental strategies operate only on "old" and "new" data, microbatch models treat every batch as an atomic unit that can be built or replaced on its own. Each batch is independent and <Term id="idempotent" />. This is a powerful abstraction that makes it possible for dbt to run batches [separately](#backfills) — in the future, concurrently — and to [retry](#retry) them independently. |
There was a problem hiding this comment.
We just added support for concurrency - see #6550
There was a problem hiding this comment.
great stuff, thanks Grace. i've tweaked this and will link out to the 'concurrently' line once we add a section about it and resolve 6550 🙏 that way, i don't block this pr.
Co-authored-by: Grace Goheen <[email protected]>
Co-authored-by: Grace Goheen <[email protected]>
| Microbatch incremental models make it possible to process transformations on very large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the [`event_time`](/reference/resource-configs/event-time) and `batch_size` you configure. | ||
| Microbatch is an incremental strategy designed for large time-series datasets: | ||
| - It relies solely on a time column ([`event_time`](/reference/resource-configs/event-time)) to define time-based ranges for filtering. Set the `event_time` column for your microbatch model and its direct parents (upstream models). Note, this is different to `partition_by`, which groups rows into partitions. | ||
| - It complements, rather than replaces, existing incremental strategies by focusing on efficiency and simplicity in batch processing. |
There was a problem hiding this comment.
Hiya @mirnawong1
Just to ask, should "complements" be "complements" or "compliments" I wasn't sure if complements is a US spelling.
Kind Regards
Natalie
There was a problem hiding this comment.
hey @nataliefiann , good question! this is right IMHO as complements means something that enhancements it. compliment generally means praise so in this context, microbatch enhances existing strategies.
There was a problem hiding this comment.
Hiya @mirnawong1
Thanks for creating this PR. I've approved this for you with a non-blocking QQ.
Kind Regards
Natalie
this pr makes updates to the microbatch doc based on internal slack feedback:
🚀 Deployment available! Here are the direct links to the updated files: