Controllable Human Image Generation with Personalized Multi-Garments

1KAIST, 2OMNIOUS.AI

TL;DR: BootComp generates human images with multi-garments and various conditions.

teaser

Abstract

We present BootComp, a novel framework based on text-to-image diffusion models for controllable human image generation with multiple reference garments. Here, the main bottleneck is data acquisition for training: collecting a large-scale dataset of high-quality reference garment images per human subject is quite challenging, i.e., ideally, one needs to manually gather every single garment photograph worn by each human. To address this, we propose a data generation pipeline to construct a large synthetic dataset, consisting of human and multiple-garment pairs, by introducing a model to extract any reference garment images from each human image. To ensure data quality, we also propose a filtering strategy to remove undesirable generated data based on measuring perceptual similarities between the garment presented in human image and extracted garment. Finally, by utilizing the constructed synthetic dataset, we train a diffusion model having two parallel denoising paths that use multiple garment images as conditions to generate human images while preserving their fine-grained details. We further show the wide-applicability of our framework by adapting it to different types of reference-based generation in the fashion domain, including virtual try-on, and controllable human image generation with other conditions, e.g., pose, face, etc.

Decomposition module

We train a decomposition network mapping from a segmented garment to a product garment, and bootstrap synthetic paired data of human and multiple garment.

decomp

Composition module

We train a composition module with the synthetic paired dataset enabling BootComp to generate human images with multiple reference garment images.

comp

Multi-garments based human image generation

BootComp can generate human images wearing multiple reference garments given by images.

applications

Controllable human image generation

BootComp enables controllable generation, handling various conditions like pose and style, as well as personalized generation like virtual try-on.

applications

Decomposition module in various applications

Our decomposition module can be applied to general domains that include common objects and can also be utilized for multi-view generation.

applications

BibTeX

@article{choi2024controllable,
      title={Controllable Human Image Generation with Personalized Multi-Garments},
      author={Choi, Yisol and Kwak, Sangkyung and Yu, Sihyun and Choi, Hyungwon and Shin, Jinwoo},
      journal={arXiv preprint arXiv:2411.16801},
      year={2024}
    }