HE-Diffusion: Privacy-Preserving Diffusion Model using Homomorphic Encryption

We introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model’s output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement the HE-based private preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500X speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Read the Paper, Code available, Cite

                            @misc{chen2024privacypreserving,
                                title={Privacy-Preserving Diffusion Model Using Homomorphic Encryption},
                                author={Yaojian Chen and Qiben Yan},
                                year={2024},
                                eprint={2403.05794},
                                archivePrefix={arXiv},
                                primaryClass={cs.CR}
                            }

Team

HE-diffusion was discovered by the following team of academic researchers:

Yaojian Chen at SEIT Lab, Michigan State University
Qiben Yan at SEIT Lab, Michigan State University

Contact us at chenyaojian17@gmail.com, qyan@msu.edu

Features

Why is privacy-preserving stable diffusion inference important?

In the scenario of AI as a service (AIaaS), features such as text-to-image conversion have gained widespread popularity among users. Yet, the high computational demands of these tasks often preclude local processing, compelling users to rely on external servers. This reliance raises substantial privacy concerns, as both the input prompts and the resulting images can contain sensitive personal information. Exposing this data on potentially unsecured servers introduces significant risks. To address these concerns, privacy-preserving stable diffusion inference has been developed. This approach seeks to safeguard private data while still leveraging the computational power of external servers, offering a balanced solution to maintain privacy without sacrificing functionality.

How does HE-diffusion work?

HE-diffusion utilizes homomorphic encryption to secure sensitive data throughout the computational process. To initiate this, users are required to encode their prompts, after which the model performs several forward layers of processing. The unique aspect of homomorphic encryption is its application during the denoising phase to ensure the protection of outputs. By employing HE-diffusion, users are empowered to conduct privacy-preserving computations on external servers, effectively safeguarding their data while leveraging the computational resources of these servers.

Will the generation capability be degraded as a result of privacy protection?

HE-diffusion is capable of generating images with negligible loss in quality compared to the original stable diffusion model. The only instance of approximation occurs during the encryption step, attributable to the characteristics of the CKKS encryption scheme. Throughout the entire computational process, optimizations are carefully implemented in a manner that preserves both the results and their precision, ensuring that the integrity of the output remains uncompromised.

Results from stable diffusion (top) and HE-diffusion (bottom)

Can HE-diffusion inference be as fast as stable diffusion?

HE-diffusion delivers performance comparable to the original stable diffusion. Homomorphic encryption works during the denoising phase where computational demands are relatively moderate. Additionally, the implementation of strategic optimizations, such as partial encryption, further enhances efficiency. As a result of these measures, the inference times for HE-diffusion and the original stable diffusion model are similar, falling within the same order of magnitude.

Case	Plain	Non-Optimized Encryption	Enc_opt	HE-Diffusion (Sparse)
Denoise once	0.12s	354.34s	1.67s	0.70s
Inference (CPU)	0.24h	10.06h	0.41h	0.32h
Inference (GPU)	35s	9.84h	203s	106s