Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛[BUG]: MLFlow Logging Typically fails for Multi-process #66

Closed
NickGeneva opened this issue Jul 28, 2023 · 2 comments · Fixed by #121
Closed

🐛[BUG]: MLFlow Logging Typically fails for Multi-process #66

NickGeneva opened this issue Jul 28, 2023 · 2 comments · Fixed by #121
Assignees
Labels
0 - Backlog In queue waiting for assignment bug Something isn't working

Comments

@NickGeneva
Copy link
Collaborator

Version

0.1.0

On which installation method(s) does this occur?

Source

Describe the issue

Probably need to have it just log on root process.

Minimum reproducible example

No response

Relevant log output

No response

Environment details

No response

@NickGeneva NickGeneva added the bug Something isn't working label Jul 28, 2023
@NickGeneva NickGeneva self-assigned this Jul 28, 2023
@NickGeneva NickGeneva added the 0 - Backlog In queue waiting for assignment label Jul 28, 2023
@mariusaurus
Copy link
Contributor

mine works when I tried using the two GPUs on my desktop machine, but each process was logged individually. Could we have this fixed as I have a customer who wants to use the MLFlow logger in multi-GPU and multi-node settings

@mariusaurus
Copy link
Contributor

mariusaurus commented Oct 4, 2023

since this is related to logging and only requires small changes: file logging at https://github.com/NVIDIA/modulus-launch/blob/main/modulus/launch/logging/console.py#L40C5-L47C48 should have only the first line after the if statement indented:

    def file_logging(self, file_name: str = "launch.log"):
        """Log to file"""
        if os.path.exists(file_name):
            os.remove(file_name)
        filehandler = logging.FileHandler(file_name)
        filehandler.setFormatter(self.formatter)
        filehandler.setLevel(logging.DEBUG)
        self.logger.addHandler(filehandler)

Also, formatter has to be passed from the init routine, ideally by making it self.formatter in line 28. And in the current form, this only logs process one, if this could include the whole logging would be great! LMK if I should open a seperate issue for all that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 - Backlog In queue waiting for assignment bug Something isn't working
Projects
None yet
2 participants