Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing the naming convention of the event log files #4

Closed
jeffbass opened this issue Jan 9, 2021 · 1 comment
Closed

Changing the naming convention of the event log files #4

jeffbass opened this issue Jan 9, 2021 · 1 comment

Comments

@jeffbass
Copy link
Owner

jeffbass commented Jan 9, 2021

For an overview of the yin-yang-ranch distributed computer vision pipeline design, see this repo. The essence of the design is that multiple imagenodes send images and event messages to each imagehub. The imagehub uses Python logging to save the imagenode event messages since it is easy to use and allows easy rotation of log files. In the current version of imagehub, the RotatingFileHandler file handler is used, which means that the log files are named like this at each rotation:

        current (today)  imagehub.log
        previous log     imagehub.log.1
        previous log     imagehub.log.2
        previous log     imagehub.log.3

As my development of the librarian code progresses, the current naming convention for the event log files presents 2 problems:

  1. It is not easy to find the event text lines for a specific date; the log files have to be searched.
  2. The librarian uses rsync for backups of imagehub data, since it is very good at efficiently backing up only what changes. But with the current RotatingFileHandler, all the log files are backed up every time because every existing log file is renumbered at each rotation.

To address these problems, I am changing the imagehub logging to use the TimedRotatingFileHandler. This is what the log files will look like after the change:

        current (today)  imagehub.log
        previous log     imagehub.log.2020-10-22
        previous log     imagehub.log.2020-10-21
        previous log     imagehub.log.2020-10-20

It solves both of the above problems, and, frankly, it should have been done that way from the start. For anyone currently using imagehub and using a program that reads the event log files, this change will require that those programs be modified.

The actual code change is simple and only involves changing 2 lines in the logging section of imagehub.py. Here is the existing code:

    handler = logging.handlers.RotatingFileHandler(hub.logfile,
        maxBytes=99000, backupCount=995)

which becomes:

    handler = logging.handlers.TimedRotatingFileHandler(hub.logfile,
        when='midnight', backupCount=995)

My existing librarian code has been changed and I'll be testing it for a few more weeks. The new imagehub code will be merged into the master branch when the librarian testing is done. This issue is a heads up to anyone currently using imagehub that the change is coming. Feel free to comment.

@jeffbass
Copy link
Owner Author

I have pushed the above mentioned change to the master branch. The imagehub event message log files now rotate at midnight and look like the TimedRoatingFileHandler example above.

Closing #4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant