Skip to content
This repository has been archived by the owner on Jul 19, 2024. It is now read-only.

User configurable initial offset support #79

Merged
merged 7 commits into from
Nov 19, 2020
Merged

Conversation

sidkri
Copy link
Member

@sidkri sidkri commented Oct 27, 2020

This change allows users to specify the point in an event stream that processing should start at if no checkpoint exists. Checkpoints per Event Hubs partition is maintained in storage and existing values if any should be deleted for this configuration to apply on the first run of the function(s).

Event Hubs extension configuration options added to host.json:

"eventHubs": {
    ....
    "initialOffsetOptions": {
        "type": "<fromStart OR fromEnd OR fromEnqueuedTime>",
        "enqueuedTime": "<time in any format supported DateTime.Parse() e.g. 2020-10-26T20:31Z>"
    }
}

Initial Offset options supported:

  • fromStart: The default option if not specified. Will start processing from the start of the stream.
  • fromEnd: Will start processing events from the end of the stream. Use this option if you only want to process events that are added after the function starts.
  • fromEnqueuedTime: Will process events that were enqueued by Event Hubs on or after the specified time. Note that this applies to all Event Hubs partitions and there is no support for specifying a per-partition value.

Design considerations:

  • Of the five supported offset options supported by the Event Hubs processor, only three are supported by this change as the other two make sense in the context of a partition and allowing for partition IDs to be specified complicates the host.json configuration and could lead to user errors. We will consider supporting "fromSequenceNumber" and "fromOffset" if there is a need by customers.

Pull request checklist

  • My changes do not require documentation changes
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • I have added all required tests (Unit tests, E2E tests)

@sidkri sidkri requested a review from alrod October 27, 2020 00:01
@sidkri sidkri self-assigned this Oct 27, 2020
@sidkri sidkri added this to the Functions Sprint 87 milestone Oct 27, 2020
@sidkri sidkri marked this pull request as ready for review November 9, 2020 13:03
@sidkri
Copy link
Member Author

sidkri commented Nov 11, 2020

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@sidkri sidkri merged commit 5eb8543 into dev Nov 19, 2020
@sidkri
Copy link
Member Author

sidkri commented Nov 19, 2020

Will not publish this version until the Azure docs PR is ready to merge as well

sidkri added a commit that referenced this pull request Dec 8, 2020
* Add EventHub output logging. Fixes #60.
* Unifying logging of exceptions. Fixes #59.
* Fix nuget errors on pack (#73)
* Increasing max message size and batch size. Fixes #29
* User configurable initial offset support (#79)
* Instructions via Readme.md for setting up local environment to run integration tests

Co-authored-by: Alexey Rodionov <[email protected]>
Co-authored-by: Pragna Gopa <[email protected]>
Co-authored-by: Sid Krishna <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow users to specify offset / timestamp to configure starting point for Event Hub
2 participants