Need help generating flat image from button photo (AI or OpenCV approach)

1

Hi everyone :waving_hand:

I’m working on a project where I need to generate a flat image from a button photo — basically an image with the same color and pattern as the button, but without any holes, lighting, or shadows.

The goal is simple:
Take a photo of a button (which has holes and curved reflections), and create a flat version of it where the area inside the holes is intelligently filled with matching pattern and color.

images (1)
images (1)225×225 4.7 KB

:brain: What I’ve tried so far

  • OpenCV Inpainting:
    It fills the holes, but produces blurry or uneven color results.

  • Hugging Face Stable Diffusion Inpainting:
    Tried using the runwayml/stable-diffusion-inpainting model, but it often generates unrelated textures or random background.

  • Prompt-based approach:
    Even with prompts like “fill the button hole intelligently and make it flat,” the output doesn’t match the original pattern.

:light_bulb: What I’m looking for

  1. A model or technique that can fill small areas with the same texture or pattern as the surrounding region.

  2. Whether this can be done using Diffusers, ControlNet, or texture synthesis models.

  3. Any working examples, prompts, or notebooks that can help me achieve this flat output image.

:puzzle_piece: My Setup

  • Language: Python

  • Libraries: Diffusers, OpenCV

  • System: macOS M1

  • Torch Backend: MPS (Metal)

from diffusers import StableDiffusionInpaintPipeline
from PIL import Image
import torch

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16
).to("mps")

result = pipe(
    prompt="create a flat version of this button image, fill the holes with same color and pattern, realistic surface",
    image=Image.open("button.png"),
    mask_image=Image.open("mask.png")
).images[0]

result.save("flat_button.png")

:framed_picture: Example

I’ll attach a few button images for reference — each has small holes and patterns that need to be filled seamlessly.

:heart: Any help appreciated

If you know of a specific model, technique, or prompt strategy that could handle this better, please let me know.
Your suggestions will help me move this project forward.

Thanks in advance! :folded_hands: