Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 refactor timer coros awspr multi worker #30

Open
wants to merge 36 commits into
base: 2.31.12-all-cherrypicks
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
5562a2b
output: add support for scheduled timer jobs with coroutines
PettitWesley Sep 26, 2023
fdda9d5
output_thread: add support for scheduled timer jobs with coroutines
PettitWesley Sep 26, 2023
a4407b6
engine: print and track scheduled timer coroutines on shutdown
PettitWesley Sep 26, 2023
c73fef0
out_s3: stability refactor
PettitWesley Sep 26, 2023
5202199
output: fix variable name in output_pre_timer_cb
PettitWesley Sep 26, 2023
814c8c7
out_s3: remove unneeded info message
PettitWesley Sep 26, 2023
c710a19
WIP: renaming
PettitWesley Oct 6, 2023
3d8d326
wip
PettitWesley Oct 11, 2023
d152dd6
output: init timer coroutine lists
PettitWesley Oct 11, 2023
b087a0d
out_s3: add to all index error messages to be clear
PettitWesley Oct 16, 2023
382068f
WIP: flb_sched_out_async_timer_cb_create and cb_flush_mutex
PettitWesley Oct 16, 2023
5fccd08
refactor async timers into their own code file
PettitWesley Oct 17, 2023
19dff4c
Add missing imports of #include <fluent-bit/flb_async_timer.h
PettitWesley Oct 17, 2023
3adf204
wip
PettitWesley Oct 17, 2023
de4aa1b
wip
PettitWesley Oct 17, 2023
c74dc5f
wip
PettitWesley Oct 17, 2023
9a5fc2a
wip
PettitWesley Oct 17, 2023
3c27f4f
Add flb_async_timer.c to CMakelists.txt
PettitWesley Oct 18, 2023
26de4ea
wip
PettitWesley Oct 18, 2023
93b2f8e
wip
PettitWesley Oct 18, 2023
5504b1e
wip
PettitWesley Oct 18, 2023
6605425
wip
PettitWesley Oct 18, 2023
6d3ab25
Proposed change to associate sched with the output thread and then pu…
PettitWesley Oct 20, 2023
8576725
Switch async timers to flb_sched struct
PettitWesley Oct 20, 2023
e5d6818
Prototype for creating one timer coro per output thread
PettitWesley Oct 23, 2023
0e0116d
out_s3: create an async timer on each worker thread
PettitWesley Oct 26, 2023
09d72bf
wip
PettitWesley Oct 26, 2023
c8dbf2a
wip
PettitWesley Oct 26, 2023
415b593
wip
PettitWesley Oct 31, 2023
8de7ac3
??? build failing why ???
PettitWesley Nov 17, 2023
2289649
wip
PettitWesley Nov 28, 2023
85ab1a3
wip
PettitWesley Nov 28, 2023
4e22809
wip
PettitWesley Nov 28, 2023
0ba44ad
Use lecaros suggestion to fix build issue
PettitWesley Nov 30, 2023
3c5f046
wip
PettitWesley Nov 30, 2023
e0c34ac
wip
PettitWesley Dec 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
out_s3: add to all index error messages to be clear
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
PettitWesley committed Oct 16, 2023

Verified

This commit was signed with the committer’s verified signature.
PettitWesley Wesley Pettit
commit b087a0d87480fb2a98891e12598335ac99cf4f0d
14 changes: 7 additions & 7 deletions plugins/out_s3/s3.c
Original file line number Diff line number Diff line change
@@ -416,22 +416,22 @@ static int init_seq_index(void *context) {

ctx->seq_index_file = flb_sds_create(ctx->metadata_dir);
if (ctx->seq_index_file == NULL) {
flb_plg_error(ctx->ins, "Failed to create sequential index file path");
flb_plg_error(ctx->ins, "Failed to create $INDEX file path");
flb_errno();
return -1;
}
tmp = "/seq_index_";
ret = flb_sds_cat_safe(&ctx->seq_index_file, tmp, strlen(tmp));
if (ret < 0) {
flb_plg_error(ctx->ins, "Failed to create sequential index file path");
flb_plg_error(ctx->ins, "Failed to concat $INDEX file path");
flb_errno();
return -1;
}

sprintf(tmp_buf, "%d", ctx->ins->id);
ret = flb_sds_cat_safe(&ctx->seq_index_file, tmp_buf, strlen(tmp_buf));
if (ret < 0) {
flb_plg_error(ctx->ins, "Failed to create sequential index file path");
flb_plg_error(ctx->ins, "Failed to concat output ID to $INDEX file path");
flb_errno();
return -1;
}
@@ -455,11 +455,11 @@ static int init_seq_index(void *context) {
else {
ret = read_seq_index(ctx->seq_index_file, &ctx->seq_index);
if (ret < 0) {
flb_plg_error(ctx->ins, "Failed to read from sequential index "
flb_plg_error(ctx->ins, "Failed to read from $INDEX "
"metadata file");
return -1;
}
flb_plg_info(ctx->ins, "Successfully recovered index. "
flb_plg_info(ctx->ins, "Successfully recovered existing $INDEX in store_dir. "
"Continuing at index=%d", ctx->seq_index);
}
return 0;
@@ -1469,7 +1469,7 @@ static int s3_put_object(struct flb_s3 *ctx, const char *tag, time_t file_first_
if (ret < 0 && access(ctx->seq_index_file, F_OK) == 0) {
ctx->seq_index--;
flb_sds_destroy(s3_key);
flb_plg_error(ctx->ins, "Failed to update sequential index metadata file");
flb_plg_error(ctx->ins, "Failed to update $INDEX tracking file");
return -1;
}
}
@@ -1612,7 +1612,7 @@ static struct multipart_upload *create_upload(struct flb_s3 *ctx, const char *ta
if (ret < 0) {
ctx->seq_index--;
flb_sds_destroy(s3_key);
flb_plg_error(ctx->ins, "Failed to write to sequential index metadata file");
flb_plg_error(ctx->ins, "Failed to write to $INDEX tracking file");
return NULL;
}
}

Unchanged files with check annotations Beta

steps:
# Set up the list of target to build so we can pass the JSON to the reusable job
- id: set-matrix
run: |

Check failure on line 26 in .github/workflows/build-master-packages.yaml

GitHub Actions / PR - Actionlint

workflow command "set-output" was deprecated. use `echo "{name}={value}" >> $GITHUB_OUTPUT` instead: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions
matrix=$((
echo '{ "distro" : [ "debian/bullseye", "ubuntu/16.04", "ubuntu/18.04", "ubuntu/20.04" ]}'
) | jq -c .)