[tests] Refactor TorchAO serialization fast tests (#10271) #4194

Job	Run time
Setup Torch Pipelines CUDA Slow Tests Matrix	1m 59s
Flax TPU Tests	2m 53s
ONNX CUDA Tests	5m 28s
Torch CUDA Tests (models)	6m 18s
Torch CUDA Tests (schedulers)	4m 40s
Torch CUDA Tests (lora)	4m 56s
Torch CUDA Tests (others)	45s
Torch CUDA Tests (single_file)	1m 36s
Examples PyTorch CUDA tests on Ubuntu	29m 10s
PyTorch Compile CUDA tests	3m 56s
PyTorch xformers CUDA tests	6m 2s
Torch Pipelines CUDA Tests (cogvideo)	3m 37s
Torch Pipelines CUDA Tests (controlnet)	12m 2s
Torch Pipelines CUDA Tests (ddpm)	2m 18s
Torch Pipelines CUDA Tests (deepfloyd_if)	4m 6s
Torch Pipelines CUDA Tests (flux)	3m 49s
Torch Pipelines CUDA Tests (ip_adapters)	2m 17s
Torch Pipelines CUDA Tests (kandinsky)	4m 55s
Torch Pipelines CUDA Tests (kandinsky2_2)	5m 13s
Torch Pipelines CUDA Tests (latent_consistency_models)	3m 0s
Torch Pipelines CUDA Tests (ltx)	1m 14s
Torch Pipelines CUDA Tests (pixart_alpha)	2m 35s
Torch Pipelines CUDA Tests (stable_diffusion)	3m 56s
Torch Pipelines CUDA Tests (stable_diffusion_2)	4m 37s
Torch Pipelines CUDA Tests (stable_diffusion_3)	1m 22s
Torch Pipelines CUDA Tests (stable_diffusion_adapter)	1m 51s
Torch Pipelines CUDA Tests (stable_diffusion_xl)	4m 38s
Torch Pipelines CUDA Tests (stable_video_diffusion)	2m 33s
	2h 11m 46s

Provide feedback