Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network message monitoring and collection #712

Open
Kubuxu opened this issue Oct 9, 2024 · 0 comments
Open

Network message monitoring and collection #712

Kubuxu opened this issue Oct 9, 2024 · 0 comments
Assignees

Comments

@Kubuxu
Copy link
Contributor

Kubuxu commented Oct 9, 2024

Set up network message monitoring and collection for the purpose of post-factum analysis and metric collection.
Current idea is based on the parquet data format, f3 observe command and duckdb for parquet processing.

  • f3 observe to collect messages
  • maybe improve f3 observe to include message validation
  • periodically upload the parquet files to S3
  • build a script to download parquet files for time range, compact and optimize them to ease analysis
@Kubuxu Kubuxu self-assigned this Oct 9, 2024
@masih masih self-assigned this Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

2 participants