Enhance feature: better SAM / adetailer integration (mashb1t fork v2.5.0)

[mashb1t]

Now part of v2.5.0-rc1 / #3282

WIP, but works (had to fully restructure the async_worker to allow for multi-step processing).
You can still see the boxes, currently no feathering, just raw yolov8 and ultralytics.

See mashb1t/Fooocus@main...mashb1t:Fooocus:feature/multi-step-rendering

This also allows potentially automated generation + upscaling + adetailer in the future.

generated image	merged mask	output

[mashb1t]

Eyes model (https://civitai.com/models/178518/eyeful-or-robust-eye-detection-for-adetailer):

Before	After

[PeterJared]

Looks quite good, I find the current scaler a bit basic and this could improve a lot the results.

[mashb1t]

optimized with GroundingDINO + SAM, so masks can be prompted + are (most of the time) very good

Yesterday (ultralytics/adetailer)	Today (GroundingDINO + SAM, no ultralytics/adetailer)

[xhoxye]

Probably try to fix a broken human face

[mashb1t]

works, but a limitation is that in order to use an existing image (and you don't have the prompt anymore, as me in this example) you have to use inpaint and paint a small area not affecting the composition.
But this isn't the use case anyways (for now), the main intention is to auto-fix generated images.

UI

Original	Variant 1 (0-shot)	Variant 2 (0-shot)

Both variants are from the same batch, only one enhancement iteration.

[mashb1t]

remaining open ToDo's for UI:

• max num boxes in developer debug mode (currently hardcoded to 2)
• box_erode_or_dilate support
• steps (maybe same as upscaler?)
• detection + replacement prompt
• control for dino mask or sam mask (dino_debug) => use boxes and skip sam or use sam instead
• (optional) inpaint mode select
• save intermediate stage2 images to developer debug mode
• save stage2 masks to developer debug mode
• display stage2 masks to developer debug mode
• (optional) enabled tab => green

[mashb1t]

Quick update: still a few things on the ToDo list, but the technical base is done. The new and significantly better structured generation process also allows for positive and negative prompt adjustments as well as future expansion to upscale directly after generation (if needed) + to use different steps / dilate and erode values from the global setting per iteration.

Demo screenshots of branch multi-step-rendering (diff):

without mask debugging	with mask debugging

[PeterJared]

What are the plans for merging it?

[mashb1t]

@PeterJared code is 90%+ done, RC is dropping either tomorrow or on Friday

[PeterJared]

Cool, we have in YouTube already two guides for that feature, I guess outdated in some areas in this moment, specially the older one.

[mashb1t]

@PeterJared may i ask which videos you're referring to?

[PeterJared]

https://www.youtube.com/watch?v=RheMMNpWuzE&t=398s&ab_channel=KleebzTechAI

https://www.youtube.com/watch?v=9qnl8SnjRGA&t=191s&ab_channel=JumpIntoAI

[ILikeToasters]

I had a feeling my video was one of the ones you were talking about. Those cover just the auto masking feature in his fork and not this new implementation. I will probably have a new video once the feature is added and I figure it all out.

[mashb1t]

v2.5.0-rc1 is now available on my fork and I'm looking forward to your feedback.
You can simply pull the tag for a fixed version or branch feature/multi-step-rendering for the latest version.

Open ToDo's: mashb1t#40 (comment)

[ILikeToasters]

Will be checking it out very soon!

[mashb1t]

HINT: use with performance Quality or Speed for best results when using an inpaint engine (inpaint mode default or add objects, not improve details), as the inpaint engine models are LoRAs and are not compatible with performance LoRAs on larger inpainting areas. This is also true for normal inpainting, not only with the new enhance feature.

[ILikeToasters]

I love that you used "enhance".... "Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there." - Rick Deckard

[DavidDragonsage]

Oooh Bladerunner, yes! 😎

@mashb1t This is going to be great! Thank you for all the hard work! 🧙👍

[Alexdnk]

@mashb1t Why does the input field for "Detection prompt" only appear when model "sam" is selected?
Are the models in some way specific and narrowly focused on something in particular? If so, could it be possible to show in a few words somewhere near the model selection what is the specialty of the currently selected model?

[mashb1t]

The other models do not support the usage of detection with prompts, only general subject masking. The model descriptions can be found here:
https://github.com/danielgatis/rembg?tab=readme-ov-file#models
Would you propose to add a link to Fooocus discussion with an explanatory page in a discussion, a separate readme file or some different solution?
(fyi i've implemented SAM from scratch for better performance and higher level of control, not using rembg)

[Alexdnk]

Thanks, it became clearer. The description in the future should be made in the style of the interface of Fooocus with such a link "📔 Document". Thank you for your hard work.

[mashb1t]

Upscale or Vary has been added as automated 1st option in enhance, closes #3113

[ILikeToasters]

Getting an error with Upscale (Fast 2x).

Fooocus] Processing enhance ...
[Fooocus] Downloading upscale models ...
[Fooocus] Upscaling image from (1024, 1024) ...
Upscaling image with shape (1024, 1024, 3) ...
Image upscaled.
[Fooocus] Saving image 1/2 to system ...
Traceback (most recent call last):
File "F:\Mashb1t testing\Fooocus\modules\async_worker.py", line 1374, in worker
handler(task)
File "F:\Mashb1t testing\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\Mashb1t testing\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\Mashb1t testing\Fooocus\modules\async_worker.py", line 1274, in handler
current_progress, img = process_enhance(
File "F:\Mashb1t testing\Fooocus\modules\async_worker.py", line 976, in process_enhance
uov_input_image_path = log(async_task.uov_input_image, d, output_format=async_task.output_format)
File "F:\Mashb1t testing\Fooocus\modules\private_logger.py", line 31, in log
image = Image.fromarray(img)
File "F:\Mashb1t testing\python_embeded\lib\site-packages\PIL\Image.py", line 2943, in fromarray
arr = obj.array_interface
AttributeError: 'NoneType' object has no attribute 'array_interface'

[mashb1t]

@ILikeToasters i assume you're on the branch not on the tag, and you're using the step 1 upscaling and not the normal one, correct? I can check this issue tomorrow.

[ILikeToasters]

Yes on the feature/multi-step-rendering branch and using the enhance to auto upscale. The other upscales work fine just not the fast one it seems.

[mashb1t]

fixed, please pull again

[bobsponge76]

Hello, how can I try this fork? Should I just download it and overwrite the regular Fooocus installation?

[ILikeToasters]

I would suggest using a different folder so you can still use the original if you want. You can use the config to configure them to share the same model folders.

[mashb1t]

@bobsponge76 you can find the instructions in https://github.com/mashb1t/Fooocus?tab=readme-ov-file#switching-from-fooocus-to-this-fork.
To be 100% safe you can alternatively also follow @ILikeToasters variant of installing my fork in a new folder (using the zip file) and then adjust the config + switch the branch there.

[mashb1t]

works, but a limitation is that in order to use an existing image (and you don't have the prompt anymore, as me in this example) you have to use inpaint and paint a small area not affecting the composition.
But this isn't the use case anyways (for now), the main intention is to auto-fix generated images.

#3122 (reply in thread)

Another update: it's now possible to start with an image without having to do inpaint/upscale before by using the new tab Input Image > Enhance.
@xhoxye fyi

--enable-describe-uov-image has also been renamed to --enable-auto-describe-image and the enhance image now also uses auto descriptions if the arg is set.

[mashb1t]

Draft Release Notes: mashb1t#42

Now with support for upscaling either before or after enhancement

[mashb1t]

v2.5.0 has been released in my fork, see https://github.com/mashb1t/Fooocus/releases/tag/v2.5.0. Happy to hear your opinion!

[Alexdnk]

I think mask generation model u2net_cloth_seg is not working correctly. A tri-level mask is drawn. And it's superimposed exactly as it is, which is wrong. Along with this such an error in the console. How to repeat - on the screenshot.

Traceback (most recent call last):
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\async_worker.py", line 1435, in worker
    handler(task)
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\async_worker.py", line 1382, in handler
    current_progress, img, enhance_prompt_processed, enhance_negative_prompt_processed = process_enhance(
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\async_worker.py", line 1013, in process_enhance
    imgs, img_paths, current_progress = process_task(all_steps, async_task, callback, controlnet_canny_path,
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\async_worker.py", line 312, in process_task
    imgs = [inpaint_worker.current_task.post_process(x) for x in imgs]
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\async_worker.py", line 312, in <listcomp>
    imgs = [inpaint_worker.current_task.post_process(x) for x in imgs]
  File "C:\Users\Alex\AppData\Local\Programs\Fooocus\Fooocus-mashb1t\Fooocus\modules\inpaint_worker.py", line 258, in post_process
    result[a:b, c:d] = content
ValueError: could not broadcast input array from shape (3588,832,3) into shape (1156,832,3)

[mashb1t]

@Alexdnk thanks for the hint, i'll check this in the evening

[mashb1t]

@Alexdnk fixed & pushed in main, see mashb1t@b3a4b4e. Seems like i missed this field when refactoring the code. Thank you for testing & reporting, much appreciated!

[ILikeToasters]

@ILikeToasters Hmm it does both, enable mask upload and show the mask generation functionality. What would you propose to change it to?

Maybe "Enable Advanced Masking Features" or "Enable Mask Upload/Auto-Gen"?

[mashb1t]

@ILikeToasters renamed to "Enable Advanced Masking Features" and pushed to main, thanks! Was thinking about "Manual / Automatic Masking" but one can already do manual masking without enabling the checkbox.

[Kaneki010]

i'm really hyped up for when this gets merged with the ability to automate it simply because of how time-consuming it is to inpaint everytime i get a bad face in a picture

[ZeroCool22]

Can someone do a tutorial video showcasing these new features¡

[mashb1t]

see mashb1t#42 (reply in thread), @ILikeToasters is https://www.youtube.com/@KLEEBZTECH

[ILikeToasters]

@ZeroCool22 This should get you started. https://youtu.be/NiGbksd9B_Y

[mashb1t]

Now part of v2.5.0-rc1, docs in #3282