Add support for automatic removal of old logs #432

aesophor · 2019-02-17T16:19:21Z

This PR is in regard to #423. The code has been added and tested , but not yet integrated into glog.

Example usage:

vector<string> overdue_log_names = GetOverdueLogNames("/tmp", 1);

for (const auto& name : overdue_log_names) {
  unlink(name.c_str());
}

GetOverdueLogNames() will check all filenames under log_directory, and return a list of files whose last modified time is over the given days ( calculated with difftime() ), so that we can simply call unlink() on the filenames in the returned vector.

However, I'm not sure when is the right time to call it.

GetOverdueLogNames(string log_directory, int days) will check all filenames under log_directory, and return a list of files whose last modified time is over the given days (calculated using difftime()). So that we can easily for unlink all files stored in the returned vector.

googlebot · 2019-02-17T16:19:24Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

Googlers can find more info about SignCLA and this PR by following this link.

aesophor · 2019-02-17T16:20:36Z

I signed it!

googlebot · 2019-02-17T16:20:38Z

CLAs look good, thanks!

Googlers can find more info about SignCLA and this PR by following this link.

Steven0917 · 2019-03-05T06:51:38Z

Guess the patch was implemented based on Linux. For Windows user, https://github.com/tronkko/dirent will be needed.

In this commit, at the end of LogFileObject::Write, it will perform clean up for old logs. It uses GetLoggingDirectories() and for each file in each directory, it will check if a file is a log file produced by glog. If it is, and it is last modified 3 days ago, then it will unlink() this file. (It will only remove the project's own log files, it won't remove the logs from other projects.) Currently it is hardcoded to 3 days, I'll see if this can be implemented in a more flexible manner.

The log cleaner can be enabled and disabled at any given time. By default, the log cleaner is disabled. For example, this will enable the log cleaner and delete the log files whose last modified time is >= x days google::EnableLogCleaner(x days); To disable it, simply call google::DisableLogCleaner(); Please note that it will only clean up the logs produced for its own project, the log files from other project will be untouched.

Also replaced the hardcoded overdue days with the correct variable.

src/logging.cc

aesophor · 2019-08-21T22:55:33Z

Guess the patch was implemented based on Linux. For Windows user, https://github.com/tronkko/dirent will be needed.

Thank you very much for your help! CI has passed on windows!
This is what I've done currently:

To enable log cleaner, in your project (log cleaner is disabled by default).

google::EnableLogCleaner(3);

And then your project will check if there are overdue logs whenever a flush is performed. In this example, the logs whose last modified times are >= 3 will be unlink(). It calls GetLoggingDirectories() to look for overdue logs.

You may also disable it at anytime.

google::DisableLogCleaner();

I try to check if a file is a glog file produced by the project it is linked in carefully. It will only remove a log if its filename matches: <program name>.<hostname>.<user name>.log.<severity level>, so logs from other projects should be untouched.

glog/src/logging.cc

Line 871 in b8a61d7

    
           return log_name_tokens[0] == glog_internal_namespace_::ProgramInvocationShortName()

Edit:
This has been tested on Linux and it works well on my machine.
I'll see if I can test it on Windows, or if anyone is willing to help me I'll be very grateful.

src/logging.cc

Previously the full path to a file is passed into IsGlogLog(), and then std::string::erase() is used to get the filename part. If a directory name contains '.', then this function will be unreliable. Now only the filename it self is passed into IsGlogLog(), so this problem will be eradicated.

src/logging.cc

Splitting a filename into tokens by '.' causes problems if the executable's filename contains a dot. Filename should be matched keyword by keyword in the following order: 1. program name 2. hostname 3. username 4. "log"

aesophor · 2019-08-22T14:52:40Z

@sergiud Sorry for asking but any chance this patch could be merged?

I've tested them on linux (4.19.66-gentoo) and windows 10, old log files can be removed automatically. This feature is disabled by default.

cswuyg · 2019-12-01T04:18:49Z

"It will only remove a log if its filename matches: ...log., so logs from other projects should be untouched."

@aesophor If I use SetLogDestination API to change base_filename_, old logs will not delete automatically. is a bug?

aesophor · 2019-12-01T10:23:21Z

"It will only remove a log if its filename matches: ...log., so logs from other projects should be untouched."

@aesophor If I use SetLogDestination API to change base_filename_, old logs will not delete automatically. is a bug?

@cswuyg Yes, this is a bug. I'm working on a patch for this.

aesophor · 2019-12-01T12:56:17Z

@cswuyg Fixed in #502

googlebot added the cla: no label Feb 17, 2019

googlebot added cla: yes and removed cla: no labels Feb 17, 2019

Replaced the lines that require C++11

4dd1386

aesophor added 2 commits August 21, 2019 20:24

Merge branch 'master' into RemoveOldLogs

d8d0c14

embed dirent.h in project

55cfc38

aesophor force-pushed the RemoveOldLogs branch from e42db36 to 55cfc38 Compare August 21, 2019 12:44

aesophor added 4 commits August 21, 2019 23:19

logging: log_cleaner: Use blackslash for windows dir delim

5eef96f

logging: log_cleaner: remove the range-based loops

b8a61d7

Also replaced the hardcoded overdue days with the correct variable.

aesophor commented Aug 21, 2019

View reviewed changes

src/logging.cc Outdated Show resolved Hide resolved

aesophor commented Aug 21, 2019

View reviewed changes

src/logging.cc Show resolved Hide resolved

aesophor mentioned this pull request Aug 21, 2019

Glog should delete old log files automaticly #423

Closed

Add Marco Wang to AUTHORS and CONTRIBUTORS

6dc3e0f

aesophor commented Aug 22, 2019

View reviewed changes

src/logging.cc Outdated Show resolved Hide resolved

aesophor added 4 commits August 22, 2019 15:15

logging: log_cleaner: improve readability

3d715d7

Add google::EnableLogCleaner() to windows logging.h

4136403

logging: log_cleaner: Remove perror message

a1a9dc8

aesophor commented Aug 22, 2019

View reviewed changes

src/logging.cc Outdated Show resolved Hide resolved

logging: IsGlogLog: match filename keyword by keyword

77a1f39

Splitting a filename into tokens by '.' causes problems if the executable's filename contains a dot. Filename should be matched keyword by keyword in the following order: 1. program name 2. hostname 3. username 4. "log"

sergiud merged commit a6f7be1 into google:master Nov 1, 2019

aesophor deleted the RemoveOldLogs branch November 4, 2019 10:32

aesophor mentioned this pull request Dec 1, 2019

src/logging.cc: encapsulate log cleaner, match logfile with <base_filename_><date>-<time>.<pid> #502

Merged

sergiud added this to the 0.5 milestone Mar 30, 2021

sergiud mentioned this pull request May 6, 2021

release 0.5 #646

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for automatic removal of old logs #432

Add support for automatic removal of old logs #432

aesophor commented Feb 17, 2019 •

edited

Loading

googlebot commented Feb 17, 2019

aesophor commented Feb 17, 2019

googlebot commented Feb 17, 2019

Steven0917 commented Mar 5, 2019

aesophor commented Aug 21, 2019 •

edited

Loading

aesophor commented Aug 22, 2019

cswuyg commented Dec 1, 2019 •

edited

Loading

aesophor commented Dec 1, 2019

aesophor commented Dec 1, 2019

Add support for automatic removal of old logs #432

Add support for automatic removal of old logs #432

Conversation

aesophor commented Feb 17, 2019 • edited Loading

googlebot commented Feb 17, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

aesophor commented Feb 17, 2019

googlebot commented Feb 17, 2019

Steven0917 commented Mar 5, 2019

aesophor commented Aug 21, 2019 • edited Loading

aesophor commented Aug 22, 2019

cswuyg commented Dec 1, 2019 • edited Loading

aesophor commented Dec 1, 2019

aesophor commented Dec 1, 2019

aesophor commented Feb 17, 2019 •

edited

Loading

aesophor commented Aug 21, 2019 •

edited

Loading

cswuyg commented Dec 1, 2019 •

edited

Loading