Diffusion Plugin Upgrade Improves AI Image Generation

Want to improve your AI image generations using older models? Researchers have come up with a way to upgrade older diffusion plugins to make them compatible with current-gen models like Stable Diffusion XL without retraining.

Dubbed X-Adapter, the tool is designed to control the upgraded model with a new text-image data pair – it universally upgrades plugins, making them directly compatible with the upgraded model.

It effectively makes a copy of the old model, preserving connectors for different plugins and adds trainable mapping layers that bridge decoders from models of different versions, allowing for feature remapping. The remapped features are then used as guidance for the upgraded model.

Examples of results on SDXL and SD 2.1 using X-Adapter and pre-rained SD 1.5 plugins and the related prompts.

It's like playing older video games on a new console – enhancing graphics (or in this case images) while retaining the core aspects of the original system. For example, the foundational qualities of Stable Diffusion 1.5 can be retained while still leveraging the power of Stable Diffusion XL.

The team behind X-Adapter is from the AI lab of Chinese tech giant Tencent, Show Lab at the National University of Singapore and Fudan University in China. They experimented with plugins like ControlNet and LoRA on Stable Diffusion 1.5, and then, using X-Adapter, upgraded them so they could work with Stable Diffusion XL.

The creators of X-Adapter said the concept could “facilitate wider application in the upgraded foundational diffusion model.”

Retaining original models would allow developers and engineers to retain investments in older systems. It could also allow researchers to compare and contrast older and newer models directly. In terms of use cases, marketers could employ X-Adapter to blend the unique characteristics of various models to create customized visuals.

There were some issues with X-Adapter. The paper outlining it suggested that some plugins failed to maintain the identity of personalized concepts. The researcher said this was because the custom plugins “work on the text encoder rather than the feature space concepts that are not directly injected into the upgraded model other than fused as guidance because of the custom plugins.”

Access X-Adapter

At the time of writing the code for X-Adapter is not available, however, it’s due soon. It will be accessible on the X-Adapter GitHub page.