One of the most frustrating challenges in generative image creation is character drift: you create a perfect protagonist in one image, but in the next scene, their facial structure, nose shape, or eye spacing changes completely. This guide explains why facial drift occurs and gives you four proven, hands-on methods to lock in character identity across diverse poses, outfits, and backgrounds.
Why AI Models Change Character Faces Between Generations
Diffusion models do not “remember” previous generations unless you explicitly provide conditioning constraints. Every standard generation starts with random latent noise determined by a numerical seed. When you alter descriptive tokens—such as changing an outfit from a “navy trench coat” to a “casual linen shirt”—the model distributes semantic attention differently across the entire image matrix, inadvertently recalculating the facial features along with the clothing.
Method 1: Using Dedicated Character Reference Tags (–cref in Midjourney)
If you use Midjourney v6 or later, the Character Reference parameter (--cref) is the fastest way to anchor facial features. It directs the model to isolate and transfer facial morphology from a source URL while allowing the prompt to dictate clothing and setting.
Step-by-Step Workflow:
- Generate and upscale an anchor portrait: Create a clean, front-facing or three-quarter portrait with neutral studio lighting and minimal accessories. Copy the image URL.
- Append the reference flag to your new prompt: Paste the URL after
--cref [IMAGE_URL]at the end of your prompt. - Tune character reference weight (–cw): By default,
--cw 100copies face, hair, and clothing. When you want to change outfits or hairstyles, set--cw 20to focus strictly on the facial features.
/imagine prompt: A 30-year-old architect with short curly brown hair and hazel eyes, wearing a yellow raincoat under city drizzle, cinematic street photography, 35mm lens --cref https://your-image-url.jpg --cw 25 --v 6.1
Why this works: Setting --cw 25 tells Midjourney to pull only facial landmarks from the reference URL, allowing the yellow raincoat and drizzle to render without clothing bleed from the original portrait.
Method 2: Stable Diffusion & IP-Adapter FaceID
For local and cloud-hosted Stable Diffusion workflows (such as ComfyUI or Forge), text prompts alone are rarely sufficient for face permanence. The industry benchmark is IP-Adapter FaceID (InsightFace model), which extracts identity vectors directly from pixel space.
- Load IP-Adapter FaceID Plus: Connect the FaceID model alongside your base SDXL checkpoint.
- Set Adapter Weight between 0.65 and 0.80: Weights above 0.85 frequently cause plastic skin texture and pose rigidity, while weights below 0.50 allow facial drifting.
- Pair with Inpainting for Close-ups: If rendering a wide-angle shot, generate the full body first, then inpaint the face area with IP-Adapter active to ensure high-resolution facial feature transfer.
Method 3: Prompt Anchoring Without Reference Images (DALL-E 3 & Free Web Generators)
When using consumer chatbots like DALL-E 3 or free generators that do not support image references, you must rely on hyper-specific descriptor anchoring:
- Invent a Unique Name or Signature Descriptor: Instead of “a young woman”, define: “A 28-year-old Norwegian botanist named Freja, sharp angular jawline, slight bridge bump on nose, dark green asymmetrical eyes, copper-red pixie cut with shaved left temple.”
- Keep the Descriptor Block Identical: Copy and paste this exact physical description string at the very beginning of every subsequent prompt, only altering the action, setting, and wardrobe.
Sources & Further Reading
- Midjourney Documentation: Character Reference & Weight Parameters (docs.midjourney.com)
- InsightFace Project: Deep Face Analysis and Landmark Extraction
- Stable Diffusion IP-Adapter: Text-Compatible Image Prompt Adapter Research