Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qualification tool: Add option to filter event logs for a maximum file system size #1275

Merged
merged 5 commits into from
Aug 12, 2024

Conversation

tgravescs
Copy link
Collaborator

@tgravescs tgravescs commented Aug 12, 2024

We have customers with TBs of event logs and some of them can be in the 100's of GB. Right now the qual tool needs as much memory as the uncompressed size. To allow for us to skip these add an option that skips event logs over a user specified size. Note this doesn't handle rolling event logs as it says in the option description below. I will file a followup to add that and add it to profiling tool.

The logic is first to filter by eventlog size and then by the file system time and number.
I also add a log info message to print the logs we filtered out.

Add a new option:

-m, --max-event-log-size  <arg>    Process only application event logs whose
                                    size is less than or equal to the size
                                    specified. Valid units of size are
                                    b(bytes),k(kilobytes),m(megabytes),g(gigabytes).
                                    If no units are specified, the size is
                                    assumed to be m. Note, this does not
                                    support event log rolling which puts
                                    multiple event logs for the same
                                    application into a single directory.

Testing was manual on some event logs and verified it only picked up ones <= to the size specified.

@tgravescs tgravescs added the feature request New feature or request label Aug 12, 2024
@tgravescs tgravescs self-assigned this Aug 12, 2024
@tgravescs
Copy link
Collaborator Author

#1276 is followup

Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tgravescs ! LGTM.

Copy link
Collaborator

@parthosa parthosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tgravescs. LGTM

@tgravescs tgravescs merged commit 46f7cab into NVIDIA:dev Aug 12, 2024
14 checks passed
@amahussein amahussein added the core_tools Scope the core module (scala) label Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants