replicating quantisation from training during inference

[michaelcyshield]

hello, I realized that the validation images are consistently of higher quality than the ones I use during inference. Is there a way I can replicate exactly what is going on during validation image generation to my use case?
here is my config
simple_tuner_config.txt

and this is my inference script
'
pipeline = DiffusionPipeline.from_pretrained(args.model_id,torch_dtype=torch.float16,)
# pipeline.vae.enable_slicing()
# pipeline.vae.enable_tiling()

# Apply LoRA weights
# Load LyCORIS configuration
with open(args.lycoris_config, "r") as f:
    lycoris_config = json.load(f)
    


# Apply LoRA weights with LyCORIS configuration
# lycoris_network, weights_sd = create_lycoris_from_weights(
#     file=args.adapter_id,
#     module=pipeline.transformer,
#     device=args.device,
#     **lycoris_config  # Pass the configuration parameters
# )
assert (
    "multiplier" in lycoris_config
), "lycoris_config JSON must contain multiplier key"
multiplier = int(lycoris_config["multiplier"])
assert (
    "linear_dim" in lycoris_config
), "lycoris_config JSON must contain linear_dim key"
linear_dim = int(lycoris_config["linear_dim"])
assert (
    "linear_alpha" in lycoris_config
), "lycoris_config JSON must contain linear_alpha key"
linear_alpha = int(lycoris_config["linear_alpha"])

apply_preset = lycoris_config.get("apply_preset", None)
if apply_preset is not None and apply_preset != {}:
    LycorisNetwork.apply_preset(apply_preset)

# This is a kwarg, but mandatory.
assert "algo" in lycoris_config, "lycoris_config JSON must contain algo key"

# Remove the positional arguments we extracted.
del lycoris_config["multiplier"]
del lycoris_config["linear_dim"]
del lycoris_config["linear_alpha"]

lycoris_network = create_lycoris(
    pipeline.transformer,
    multiplier,
    linear_dim,
    linear_alpha,
    **lycoris_config,
)

lycoris_network.load_weights(args.adapter_id)

    # Move weights to the correct device manually (since the function loads them to CPU)
# for key in weights_sd:
#     weights_sd[key] = weights_sd[key].to(args.device)


#lycoris_network.to(args.device)
#pipeline.to(args.device)
lycoris_network.apply_to()  # Merge the LoRA weights into the transformer
lycoris_network.merge_to() # Merge the LoRA weights into the transformer
# Quantize the pipeline using the same method as training
#Hard-code precision settings to 'int8-quanto'
base_model_precision ='fp8-quanto'
text_encoder_1_precision = 'quanto'
text_encoder_2_precision ='quanto'
text_encoder_3_precision = 'quanto'

quantoise(
    unet=getattr(pipeline, 'unet', None),
    transformer=getattr(pipeline, 'transformer', None),
    text_encoder_1=getattr(pipeline, 'text_encoder_1', None),
    text_encoder_2=getattr(pipeline, 'text_encoder_2', None),
    text_encoder_3=getattr(pipeline, 'text_encoder_3', None),
    controlnet=None,
     args=argparse.Namespace(
        base_model_precision=base_model_precision,
        text_encoder_1_precision=text_encoder_1_precision,
        text_encoder_2_precision=text_encoder_2_precision,
        text_encoder_3_precision=text_encoder_3_precision
    )
)   


print("Model loaded, LoRA applied, and quantized successfully. Ready to generate images.")
styles = [ 'hand drawn water color image']

if args.prompts_path is not None:
    prompt_files = [file for file in os.listdir(args.prompts_path.strip()) if file.endswith(".txt")]
    for prompt_file in prompt_files:
        
        with open(os.path.join(args.prompts_path.strip(), prompt_file), "r") as f:
            prompt = f.read() 
        
        for style in styles:
            for i in range(3):
                negative_prompt = ''

                prompt =  style + ' ' + prompt
                prompt = prompt  + negative_prompt
                style_output = style.replace(' ','_')
                output_file_name = prompt_file.replace('_prompt.txt','').replace('_summary.txt', '')+ '_' + style_output +f"_{i+1}_" + '.png'
                output_file = os.path.join(args.output_folder.strip(),output_file_name)
                
                # Use default or previously set resolution
                width = locals().get('width', 1024)
                height = locals().get('height', 1024)
                
                # Generate image with the specified parameters
                generate_image(
                    pipeline=pipeline,
                    prompt=prompt,
                    output_file=output_file,
                    num_inference_steps=args.num_inference_steps,
                    width=width,
                    height=height,
                    guidance_scale=args.guidance_scale,
                    seed=args.seed,
                    device=args.device
                )

'

sorry for the messy code in advance but I am really stuck here, currently the output of this inference script is just poor for some odd reason

[bghira]

don't quantise the text encoders

[michaelcyshield]

I run out of memory trying to run inference with 'no_change' on the text encoders

[mhirki]

I run out of memory trying to run inference with 'no_change' on the text encoders

You can just offload the text encoders. Then you don't have to quantize them.

pipeline.enable_model_cpu_offload()

[michaelcyshield]

I run out of memory trying to run inference with 'no_change' on the text encoders

You can just offload the text encoders. Then you don't have to quantize them.

pipeline.enable_model_cpu_offload()

I've tried this and its leading to weird behavior, if I do it before the lycoris merge, it gets mad because of tensors being on both gpu and cpu, if I do it after the lycornis merge it works but I need to move the pipeline to gpu after calling the model cpu offload, I get a warning that thats bad but the model actually fits in vram

this is the error I get when I use cpu offload without moving the pipeline to gpu after the offload
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[mhirki]

What's the value of args.guidance_scale? If you're using the config parser from SimpleTuner, you should probably be using args.validation_guidance.

[mhirki]

Ok, guidance should be fine then.

Upon further thought, I think you're supposed to call either apply_to() or merge_to() for lycoris. Try removing one of them from your script.

[michaelcyshield]

2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=3072 and factor=16, using full matrix mode.
2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=12288 and factor=8, using full matrix mode.

not sure if this is something to be weary about, but I do get this, i trained the lycoris on the default config from simple tuner

[mhirki]

2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=3072 and factor=16, using full matrix mode. 2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=12288 and factor=8, using full matrix mode.

not sure if this is something to be weary about, but I do get this, i trained the lycoris on the default config from simple tuner

No need to be concerned about those warnings. The default configuration triggers "full matrix mode" which is generally though to be the best way to train a LoKr.

[michaelcyshield]

removing the .apply_to and only working with the merge_to seems to be working just fine and possibly improved performance (either than or the random seeds were in my favor)

[michaelcyshield]

2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=3072 and factor=16, using full matrix mode. 2024-10-03 11:41:40|[LyCORIS]-WARNING: lora_dim 10000 is too large for dim=12288 and factor=8, using full matrix mode.
not sure if this is something to be weary about, but I do get this, i trained the lycoris on the default config from simple tuner

No need to be concerned about those warnings. The default configuration triggers "full matrix mode" which is generally though to be the best way to train a LoKr.

oh thats a good insight to have, thank you, I did not know that