Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 refactor timer coros awspr multi worker #30

Open
wants to merge 36 commits into
base: 2.31.12-all-cherrypicks
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
5562a2b
output: add support for scheduled timer jobs with coroutines
PettitWesley Sep 26, 2023
fdda9d5
output_thread: add support for scheduled timer jobs with coroutines
PettitWesley Sep 26, 2023
a4407b6
engine: print and track scheduled timer coroutines on shutdown
PettitWesley Sep 26, 2023
c73fef0
out_s3: stability refactor
PettitWesley Sep 26, 2023
5202199
output: fix variable name in output_pre_timer_cb
PettitWesley Sep 26, 2023
814c8c7
out_s3: remove unneeded info message
PettitWesley Sep 26, 2023
c710a19
WIP: renaming
PettitWesley Oct 6, 2023
3d8d326
wip
PettitWesley Oct 11, 2023
d152dd6
output: init timer coroutine lists
PettitWesley Oct 11, 2023
b087a0d
out_s3: add to all index error messages to be clear
PettitWesley Oct 16, 2023
382068f
WIP: flb_sched_out_async_timer_cb_create and cb_flush_mutex
PettitWesley Oct 16, 2023
5fccd08
refactor async timers into their own code file
PettitWesley Oct 17, 2023
19dff4c
Add missing imports of #include <fluent-bit/flb_async_timer.h
PettitWesley Oct 17, 2023
3adf204
wip
PettitWesley Oct 17, 2023
de4aa1b
wip
PettitWesley Oct 17, 2023
c74dc5f
wip
PettitWesley Oct 17, 2023
9a5fc2a
wip
PettitWesley Oct 17, 2023
3c27f4f
Add flb_async_timer.c to CMakelists.txt
PettitWesley Oct 18, 2023
26de4ea
wip
PettitWesley Oct 18, 2023
93b2f8e
wip
PettitWesley Oct 18, 2023
5504b1e
wip
PettitWesley Oct 18, 2023
6605425
wip
PettitWesley Oct 18, 2023
6d3ab25
Proposed change to associate sched with the output thread and then pu…
PettitWesley Oct 20, 2023
8576725
Switch async timers to flb_sched struct
PettitWesley Oct 20, 2023
e5d6818
Prototype for creating one timer coro per output thread
PettitWesley Oct 23, 2023
0e0116d
out_s3: create an async timer on each worker thread
PettitWesley Oct 26, 2023
09d72bf
wip
PettitWesley Oct 26, 2023
c8dbf2a
wip
PettitWesley Oct 26, 2023
415b593
wip
PettitWesley Oct 31, 2023
8de7ac3
??? build failing why ???
PettitWesley Nov 17, 2023
2289649
wip
PettitWesley Nov 28, 2023
85ab1a3
wip
PettitWesley Nov 28, 2023
4e22809
wip
PettitWesley Nov 28, 2023
0ba44ad
Use lecaros suggestion to fix build issue
PettitWesley Nov 30, 2023
3c5f046
wip
PettitWesley Nov 30, 2023
e0c34ac
wip
PettitWesley Dec 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
wip
  • Loading branch information
PettitWesley committed Nov 28, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 4e22809a3f06fab9135b10227e4c59ee0caf1de5
38 changes: 0 additions & 38 deletions include/fluent-bit/flb_output.h
Original file line number Diff line number Diff line change
@@ -76,44 +76,6 @@

struct flb_output_flush;

struct flb_out_thread_instance {
struct mk_event event; /* event context to associate events */
struct mk_event_loop *evl; /* thread event loop context */
struct flb_bucket_queue *evl_bktq; /* bucket queue for evl track event priority */
flb_pipefd_t ch_parent_events[2]; /* channel to receive parent notifications */
flb_pipefd_t ch_thread_events[2]; /* channel to send messages local event loop */
struct flb_output_instance *ins; /* output plugin instance */
struct flb_config *config;
struct flb_tp_thread *th;
struct mk_list _head;

/*
* In multithread mode, we move some contexts to independent references per thread
* so we can avoid to have shared resources and mutexes.
*
* The following 'coro' fields maintains a state of co-routines inside the thread
* event loop.
*
* note: in single-thread mode, the same fields are in 'struct flb_output_instance'.
*/
int flush_id; /* coroutine id counter */
struct mk_list flush_list; /* flush context list */
struct mk_list flush_list_destroy; /* flust context destroy list */

/*
* If the main engine (parent thread) needs to query the number of active
* 'flushes' running by a threaded instance, then the access to the 'flush_list'
* must be protected: we use 'flush_mutex for that purpose.
*/
pthread_mutex_t flush_mutex; /* mutex for 'flush_list' */

/* List of mapped 'upstream' contexts */
struct mk_list upstreams;

/* Each event loop has a scheduler instance */
struct flb_sched *sched;
};

struct flb_out_thread_instance;
int flb_output_thread_pool_coros_size(struct flb_output_instance *ins);
struct flb_out_thread_instance *flb_output_thread_instance_get();
Loading