Abstract

Diffusion models have demonstrated significant potential in image generation. However, their ability to replicate training data presents a privacy risk, particularly when training data includes confidential information. While existing mitigation strategies primarily focus on augmenting the training dataset, we provide new perspectives to address the replication issue by considering the diffusion model architecture and the training procedure. We observe that although skip connections in U-Net-based diffusion model improve image quality, they also increase data memorization and replication risk. To address this, we propose a replication-aware U-Net (RAU-Net) architecture that incorporates information transfer blocks into the skip connections that are the most sensitive to replication. Recognizing the potential impact of RAU-Net on generated image quality, we further investigate and identify specific timesteps that have less impact on generated image quality. By applying RAU-Net selectively at these timesteps, we couple our novel diffusion model with specialized training and inference strategies, forming a framework we refer to as LoyalDiffusion. Additionally, we reassess the replication score, which is a prevalent metric that quantifies the severity of data replication, and establish an empirical target benchmark. Our extensive experiments demonstrate that LoyalDiffusion outperforms the state-of-the-art replication mitigation method by achieving a 48.63% reduction in replication while maintaining comparable generated image quality.

Date

2026

Authors

Chenghao Li, Yuke Zhang, Dake Chen, Jingqi Xu, Peter A Beerel

Conference

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Pages

4230-4239