Train different tasks at same time

[acai66]

🚀 Feature

Train different tasks at same time.

Motivation

there always are multi gpu in a machine, We should have been able to train different models at same time, but outputs and results are stored in same directory now, it may be conflict.

Pitch

split outputs and results include weights in separate directories.

Alternatives

Additional context

I made a temporary change to train.py so i can train different tasks, but i really hope this funiction will be official support.
tkanks.

    wdir = 'weights' + os.sep + opt.name + os.sep  # weights dir
    if not os.path.exists(wdir):
        os.mkdir(wdir) 
    last = wdir + 'last.pt'
    best = wdir + 'best.pt'
    results_dir = 'logs' + os.sep + opt.name + os.sep
    results_file = results_dir + 'results.txt' 

[github-actions]

Hello @acai66, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Google Colab Notebook, Docker Image, and GCP Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI surveillance systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@acai66 yes you make a good point. We use multiple docker containers on a single machine to exploit multiple single-gpu trainings simultaneously.

Without docker containers you might simply copy the directory, one per gpu.

For a more comprehensive solution, we might be better off depositing all run-related items (jpgs, results.txt, checkpoints etc.) into the unique ./runs directory already created automatically by tensorboard when a training run starts. What do you think?

[acai66]

@acai66 yes you make a good point. We use multiple docker containers on a single machine to exploit multiple single-gpu trainings simultaneously.

Without docker containers you might simply copy the directory, one per gpu.

For a more comprehensive solution, we might be better off depositing all run-related items (jpgs, results.txt, checkpoints etc.) into the unique ./runs directory already created automatically by tensorboard when a training run starts. What do you think?

good idea, thanks

[glenn-jocher]

The unique directory is defined in

yolov5/train.py

Lines 394 to 399 in b810b21

	# Train
	if not opt.evolve:
	tb_writer = SummaryWriter(comment=opt.name)
	print('Start Tensorboard with "tensorboard --logdir=runs", view at http://localhost:6006/')
	train(hyp)

tb_writer.log_dir
Out[3]: 'runs/Jun07_09-10-55_Glenns-MBP.attlocal.net'

[glenn-jocher]

@acai66 see #104, this PR seems to address many of your concerns. Perhaps you could look it over and give feedback to the PR author.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.