-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added an API layer #194
added an API layer #194
Conversation
Can anyone confirm that Text-To-Image works? I'm getting bad outputs that seem like normal Text-To-Image. Everything loads and there are no errors though. And could you also return the generated control image at either the first or last image index if the pre-processor was used? |
"controlnet_input_image": [img], This is how json objects are formated and those that are necessary. My setup works fine this way with depth, pose and scribble. Img is a base64 string for the image that goes into the ControlNet UI on the SD WebUI. EDIT: control img can be added but procrastinating on it. If interested in doing so, just need to encode the img in the second index of the output that is in numpy array format. |
Good. Small advice:
|
I see the issue now. Doesn't work with Euler apparently? Switched to DDIM and I'm getting proper results now. Edit: Just tried with DPM++ 2M and the output was bad as well. Using canny btw Edit: Just tested with depth and the same thing happens. It works with DDIM and PLMS but with Euler, LMS, Heun or DPM++ 2M the results become random. I've tried most of the samplers now and ControlNet appears to only be applying to DDIM and PLMS. They all work in the webui though. It could just be an issue with my installation. |
I've dropped this change in, but seems to break when visiting the docs; http://127.0.0.1:7860/docs#/ |
For those this doesn't work, forgot the highlight it only works only when ControlNet is installed as a script in img2img and txt2img. Try it without other scripts installed in the extension tab. |
Will do when I have time. Quick Q: I see in your script, that you arrange script args based on script index. Any tips in populating script args for multiple alwayson_scripts properly? Thanks! |
@kft334 this might be the issue |
Creating a new API was a wonderful idea, even if it's 'makeshift'. It makes things so much easier to edit. One thing though, I pass in the controlnet_input_image as an array of base64 strings, and it throws an error ("Incorrect Padding"). Replacing cn_image = Image.open(io.BytesIO(base64.b64decode(controlnet_input_image[0]))) with cn_image = decode_base64_to_image(controlnet_input_image[0]) Fixed that error. |
Hello! I'm the author of the post on Reddit. I have a couple of suggestions to make your api layer better.
I still don't know what would be the best solution from the architectural standpoint. You cloned the whole A1111 api for txt2img and img2img. I don't think it's a good idea, as it's hard to maintain. You'll have to track changes both in A1111 api and in the ControlNet extension. My solution is not great either. I made the api only to retrieve the models, and created an A1111 script to control the ControlNet extension, so a user only have to add the Script name and arguments to the existing API call. It's hacky too (and adds an extra script with UI), but at least, you don't have to worry about maintaining the cloned API. I don't know if there is another way to do this apart from adding the ControlNet to the core of A1111. Any suggestions would be appreciated. |
Hey, I really loved your work! And great suggestions : ) Yeah definitely this is a super makeshift effort : ) I just needed quick and dirty for the moment, but as you said maintaining will be a headache potentially. I think adding the 1) GET methods, 2) move to separate script (done on my system locally), 3) returning control results as you and others suggested would be good for the moment. But also a fan of exploring the right implementation model (which would be why I wanted to start this thread) that would be needed going forward. |
@sangww - having problems running this, the fallback case line 350 always is True. therefore control net is not used? Am I missing something? |
Shouldn't be true. I have updated the commit which also includes an example api call from a python notebook. This could be helpful in comparing with your setup There are a few things to check given how makeshift is this:
|
No error message, no extra modules, installed your branch on a fresh install - I use the same encoding for the input for ControlNet as the initImage (without the "data:image/png;base64," substring) and still the flag is checked True. Could you double check if this is occurring for you too? -> There is no exception clause in the chain of execution afterward - whence there is no error displayed. |
That is strange indeed. I am suspecting it could be the extension's own seeting. |
I have the same issue still, with the settings changes.
Processed is always none here |
The new version of ControlNet adds Guidance strength, but there is no such parameter in the api. The default value of Guidance strength is zero when calling the api to generate, resulting in serious error in the result. Please add this parameter to the api or tell me how to add it? Please refer to the above three pictures for the difference in results. |
add more entries to script_args should fix it. (PR welcome sd-webui-controlnet/scripts/api.py Lines 170 to 180 in 5fed282
|
@Mikubill Confused, when I add the following code like other codes it doesn't work.
|
Working on it in #384 |
Just started getting this error;
I've installed the LoRA network today? |
Hey, so I had to disable "additional networks" extension and it started working again. Another issue, GET request on "controlnet/model_list" returns an empty array? |
Is there any documentation of how to use the controlnet api other than what is in the A1111 api documentation? It would be great to see an example of successfully running a txt2img and an img2img using controlnet with POST requests. I've used POST for regular img2img from a 3d software environement and would like to add the capability to use CN as well. Thanks! |
We definitely need to describe what each field of In the meantime, tl;dr: Each field of the You can start by specifying only the Here's an example img2img object that I used when testing the multi-controlnet api pr: (ellipsis json object
{
"init_images": [...],
"resize_mode": 0,
"denoising_strength": 0.5,
"image_cfg_scale": 0,
"mask_blur": 4,
"inpainting_fill": 0,
"inpaint_full_res": true,
"inpaint_full_res_padding": 0,
"inpainting_mask_invert": 0,
"initial_noise_multiplier": 0,
"prompt": "1girl",
"styles": [],
"seed": -1,
"subseed": -1,
"subseed_strength": 0,
"seed_resize_from_h": -1,
"seed_resize_from_w": -1,
"sampler_name": "Euler",
"batch_size": 1,
"n_iter": 1,
"steps": 50,
"cfg_scale": 7,
"width": 512,
"height": 512,
"restore_faces": false,
"tiling": false,
"negative_prompt": "",
"eta": 0,
"s_churn": 0,
"s_tmax": 0,
"s_tmin": 0,
"s_noise": 1,
"override_settings": {},
"override_settings_restore_afterwards": true,
"sampler_index": "Euler",
"include_init_images": false,
"controlnet_units": [
{
"model": "diff_control_sd15_depth_fp16 [978ef0a1]"
}
]
} In this case, the controlnet unit specified inherits the image specified in the You can probably omit most of the root values. In my case I used a diff model. |
This is great info, I think I understand better already -- only thing is that I'm not seeing the ControlNetUnitRequest documented in the localhost docs, so I can't see the fields available. I only see 3 things documented related to controlnet:
And in Schemas: Body_detect_controlnet_detect_post{
} I may be missing something about how the docs are used. |
I don't, which is weird -- I don't see any txt2img or img2img routes at all.... in the past I did see those routes, that's how I learned how to do my first successful POSTs on txt2img and img2img... |
This is what I'm seeing @ljleb |
Yeah I'm not sure why it happens to be honest. One fix could be to hijack the webui |
Note that even with a fix, soon the webui will start supporting always on scripts in api calls: AUTOMATIC1111/stable-diffusion-webui#8253 When/if this PR is eventually merged, we will probably deprepcate/remove our own |
so is this a bug in CN or in A1111? How do I get back to seeing all the routes in the api docs, even just for basic a1111? wait and keep trying? :) |
If you try uninstalling the controlnet extension you should still not see any |
Guys, I am experiencing the same issue as @kft334:
I am using the route Click to see the payload{
"enable_hr": false,
"denoising_strength": 0,
"prompt": "...",
"styles": [
"string"
],
"seed": -1,
"subseed": -1,
"subseed_strength": 0,
"seed_resize_from_h": -1,
"seed_resize_from_w": -1,
"sampler_name": "DDIM",
"batch_size": 1,
"n_iter": 1,
"steps": 20,
"cfg_scale": 7,
"width": 512,
"height": 512,
"restore_faces": false,
"tiling": false,
"negative_prompt": "...",
"eta": 0,
"s_churn": 0,
"s_tmax": 0,
"s_tmin": 0,
"s_noise": 1,
"override_settings": {},
"override_settings_restore_afterwards": true,
"script_args": [],
"sampler_index": "DDIM",
"controlnet_units": [
{
"module": "canny",
"model": "control_canny-fp16 [e3fe7712]",
"weight": 1,
"resize_mode": "Envelope (Outer Fit)",
"lowvram": false,
"guessmode": false,
"input_image": "..."
}
]
} Reading the thread, I tried the following options (without success):
No error message is shown. Could you guys please help me? @sangww @ljleb @Mikubill @chrisbward |
@paulo-coronado |
Hi all, thanks for the development and discussion in this thread, it's very helpful. |
@alexcode0 you can pass preprocessed images as a base64 string to the You will also find an example of the json structure via |
Hm. I tried exactly that before posting my Q. Seemed like the logical way to implement it, but no luck.
|
Can you elaborate on what you expect to see, and then what you see instead happening? I don't have enough info at the moment to help. |
Essentially when I use the webui, I can upload an image to the control net, and get a preprocessed image, which is then ported to the txt2img workflow and is generated using the prompt and the preprocessed image as a guide. When I try to do this via the API, I get the canny preprocess successfully. When I pass that preprocessed image to the control net again and try to apply the model (control_sd15_canny [fef5e48e]), the output image is unchanged from the preprocessed input. See image below: |
Thanks for the info. The api should return 2 images in your case, 1 for the preprocessor (unchanged when As you save both images under the same name, I suspect the txt2img output is just overwritten by the preprocessor image. Consider adding |
You can check "don't save detectmap to output" in webui settings if you dont want the controlnet maps as output (should probably be an api parameter) |
Bingo! I thought that may have been the case and checked the response json but didn't find 2. Went back and checked and yup, it's in there alright. Thank you for taking the time to help |
`import requests Open the control image file in binary modewith open("poses\poses_base.png", "rb") as f:
Define the payload with the prompt and other parameterspayload = { Send the request to the API endpointresponse = requests.post(url=f'http://127.0.0.1:7860/controlnet/txt2img', json=payload) Print the responseprint(response) The response is giving 500 server error, can any one help me what am I doing wrong? |
With latest version of webui it is possible that the controlnet API does not work anymore. There are efforts currently to put it back on its feet in #527 |
Have you tried decoding the response? decoded = response.read().decode() I also had a 500 Server Error but when decoding it gave me more information, telling me the input image wasn't encoded correctly. But is there any update on whether controlnet API is working in the latest release? I tried both the /controlnet/txt2img and /sdapi/v1/txt2img endpoints and they both aren't going through controlnet |
Waiting on approval for #587. The API will work again once it is merged. |
hi, i also meet this problem, and you solve it? |
This API layer replicates SD-WebUI's txt2img and img2img pipeline, with added script parameters for ControlNet.
It has two caveats that needs addressed, but currently in a working state with full support of SD-WebUI + ControlNet features.