Index Color Separation for Screen Printing: The Definitive Technical Guide

Index color separation is one of the most powerful — and most misunderstood — techniques in screen printing. It solves problems that CMYK halftone process printing cannot: it eliminates moiré, it works beautifully on textured garments, it handles opaque inks on dark substrates, and it produces vivid, photographic-quality results with as few as six colors. Used correctly, index color separation can outperform CMYK process printing in many real-world screen printing scenarios.

This guide covers what index color separation is, why it works, how to execute it in Photoshop, how to choose the right color count, and a complete troubleshooting section covering the most common problems printers encounter with indexed separations.

What Is Index Color Separation?

Index color separation is the process of reducing a full-color image (which may contain millions of colors) down to a small, defined set of colors — the “index” — and then generating a separate film/screen for each of those colors.

Unlike CMYK process printing, where four transparent inks are overlaid at different halftone angles to reproduce color through optical mixing, index color printing places each ink color side by side, pixel by pixel, with no intentional overlap. Each ink covers its area completely with solid dots.

The core mechanism that makes this work is diffusion dither — a noise-based dithering algorithm that distributes the error of replacing one color with another in a visually random, spatially distributed pattern. The result is a pseudo-random pixel array that, when viewed from a normal distance, blends into smooth gradients and nuanced color transitions.

Index Color vs. CMYK Process: A Direct Comparison

Understanding when to use index color separation versus CMYK requires understanding the fundamental difference in ink behavior.

CMYK Process Printing

CMYK uses four transparent inks (Cyan, Magenta, Yellow, Black) applied in overlapping halftone dots at offset angles. Color is produced by the eye mixing the overlapping transparent layers.

Strengths of CMYK:

• Theoretical infinite color gamut within the CMYK space
• Well-understood, industrially standardized
• Smooth gradients at high resolution

Weaknesses of CMYK for screen printing:

• Moiré risk from halftone angle interactions
• Transparent inks cannot print opaque on dark substrates without a white underbase
• Sensitive to dot gain (especially in shadows)
• Requires precise press registration (typically ±0.1 mm or better)
• Color shifts when print order changes

Index Color Printing

Index uses a set of opaque spot colors placed in a diffusion dither pattern. No two colors intentionally overlap; each ink sits cleanly on the substrate (or underbase).

Strengths of index color:

• No moiré (random dot placement eliminates angle interference)
• Works with opaque inks — ideal for dark garments
• Forgiving of minor registration variation
• Excellent color vibrancy — opaque colors don’t mix and muddy
• Predictable dot gain behavior (all dots same size)
• Lower mesh count requirements than fine AM halftone

Weaknesses of index color:

• Limited color gamut compared to full-process CMYK
• More screens required to expand color range (8–12 colors for photo-quality)
• Not ideal for very smooth gradient work in small color counts
• Visible pixel texture at close viewing distances (by design)

How Many Colors Does Index Color Need?

This is the most common question and the most important decision in index color separation. The answer depends on the complexity of the artwork and the required print quality.

A useful rule of thumb: 12-color index looks approximately twice as smooth as a 6-color index separation of the same image. Each additional color adds detail and tonal nuance, particularly in skin tones, skies, and shadow areas.

How to Create Index Color Separation in Photoshop: Step by Step

Photoshop’s Indexed Color mode is the standard tool for index color separation. Here is the complete process.

Step 1: Prepare the Image

Start with an RGB image at the correct print resolution (typically 200–300 DPI at print size for index color work). The image should be:

• Color-corrected and color-balanced for accurate reproduction
• Sharpened appropriately (unsharp mask: amount 120–150%, radius 0.8–1.2px, threshold 2–4)
• Sized to final print dimensions

Do not convert to CMYK before performing index separation — work from RGB.

Step 2: Flatten and Merge All Layers

Image → Flatten Image. All layers must be merged before conversion.

Step 3: Convert to Indexed Color

Image → Mode → Indexed Color

In the Indexed Color dialog:

• Palette: Select “Local (Perceptual)” or “Local (Selective)” — these give the most visually accurate color selection
• Colors: Start with your target count (e.g., 8)
• Forced: None (unless you need to guarantee white or black is included)
• Transparency: Leave unchecked unless working with a transparent background
• Matte: None
• Dither: Diffusion — this is critical. Always use Diffusion dither for index color separation. Never use Pattern or Noise.
• Dither Amount: 75–100%. Higher values = more dithering, smoother gradients but more visual texture. Lower values = less texture but more banding.
• Preserve Exact Colors: Check this to prevent colors that are already exact palette matches from being dithered.

Click OK.

Step 4: Review and Edit the Color Table

Image → Mode → Color Table

This shows the exact index colors Photoshop has selected. Here is where expert judgment comes in:

Review each color for printability:

• Are any colors too similar to be worth a separate screen? If two colors differ by only 5–8% in one channel, consider reducing the color count by 1 and re-separating.
• Are any critical colors (flesh tones, brand colors, sky blues) accurately represented? If not, manually edit those entries in the Color Table.
• Does the palette include a useful near-white for highlight areas? Does it include a good near-black for shadows?

Manual color table editing: Double-click any color chip to edit it. You can shift a color to be more printable — for example, converting a slightly greenish neutral gray to a true neutral. Remember: every color in the index is a real ink that will be mixed and printed, so it must be a color your print shop can actually produce.

Step 5: Check Image Quality

Zoom to 100% and inspect:

• Gradient areas (are they smooth or does banding show?)
• Shadow areas (is there sufficient detail or are they flat black?)
• Highlight areas (are pale tones preserved or washed out?)
• Skin tones (the most critical area — do they look natural?)

If quality is insufficient, go back and increase the color count by 2 and repeat. The difference between 6 and 8 colors is dramatic; between 10 and 12 is subtle.

Step 6: Generate Individual Channel Files

Once the indexed image is approved, each color in the index becomes a separate film. To extract each color:

Method A: Convert back to RGB and use channels

1. Convert Indexed → RGB
2. Use Select → Color Range to select each index color
3. Create a layer mask from each selection
4. Output each layer as a grayscale film

Method B: Use separation software Professional separation tools (AccuRIP, Separation Studio, FastFilms) can automate indexed color to film, with automatic halftone dot output and registration marks.

Step 7: Film Output

Each color outputs as a grayscale film — black areas indicate where that color’s ink will print. Film should be output at 1200–2400 DPI. Unlike CMYK halftone films, index color films do not require specific halftone angles because the diffusion dither pattern is non-directional.

Choosing the Ink Colors for Index Separation

The quality of an index color print depends not just on how many colors are used, but which colors are chosen. The index palette represents the physical inks that will be mixed and put on press — every color must be achievable in ink.

Working with the Printer’s Ink Set

The ideal workflow is to provide the index color table to your printer and have them match each color to their available mixing system (Pantone, custom lab). For a Dragonfly Colors separation, we provide exact Lab color values for each index color so printers can match accurately.

Common Index Color Palettes

For photographic work on white or light garments, a typical 8-color index palette might include:

• Near-white highlight
• Light skin / warm highlight
• Vivid yellow
• Orange / warm mid
• Red or magenta
• Dark blue or teal
• Dark brown / near-black shadow
• True black or very dark shadow

For dark garment work, the palette shifts to account for the dark background:

• The substrate (dark shirt) serves as the darkest shadow
• A pure white provides highlights (often applied as underbase)
• Mid-range colors fill in between

Index Color on Dark Garments: Working with an Underbase

Printing on dark garments with index color typically requires a white underbase. The underbase is a solid or partial coverage white layer that creates a neutral foundation for the index colors to print on.

Two approaches to underbase + index color:

Option 1: Full White Underbase, Print All Colors on Top

A full coverage white underbase is flashed, then all index colors print wet-on-wet. This gives maximum color vibrancy but can feel heavy on the garment.

Option 2: Selective Underbase (Choke and Spread)

The underbase is sized slightly smaller (“choked”) than the printed colors, allowing dark areas to show through as part of the shadow tones. This uses the dark garment as a shadow color, reducing ink build-up and giving a softer hand.

For index separation specifically, option 2 is preferable. The dark garment contributes shadow values; the index palette only needs to represent the mid-tones and highlights. This typically allows satisfactory results with 6–8 colors instead of 10–12.

Mesh Selection for Index Color Printing

Index color diffusion dither pixels are all the same size — unlike AM halftone dots which vary. The pixel size depends on the image resolution (200 DPI image = 200 pixels per inch, each pixel approximately 127 microns).

Match mesh count to ensure clean, complete pixel transfer:

Higher mesh counts reproduce finer pixel detail but require thinner, lower-viscosity inks.

Troubleshooting Index Color Separation

Problem 1: Visible Pixel Grid / Texture Too Obvious

Symptoms: Print looks “blocky” or “pixel-y” at normal viewing distance; texture is more visible than expected.

Causes and fixes:

• Image resolution too low: Increase source image resolution to 200–300 DPI at print size. Low-res images produce large, visible pixels.
• Dither amount too low: In Photoshop Indexed Color dialog, increase Diffusion dither amount to 90–100%. Lower dither amounts produce more regular patterns.
• Color count too low: Increase color count. More colors allow smoother representation of tonal transitions, reducing reliance on dither texture.
• Viewing distance consideration: Index color always shows pixel structure up close. At 50–60 cm viewing distance, the texture should dissolve. If it doesn’t, increase resolution or color count.

Problem 2: Flat, Washed-Out Shadows

Symptoms: Shadow areas lack depth; image looks like it has no dark values.

Causes and fixes:

• Dark colors not included in palette: Ensure the color table includes at least one very dark brown or black. Add one manually if Photoshop’s automatic selection omitted it.
• Index separation working on dark garment without accounting for substrate: If printing on a dark shirt, the shirt provides shadow values. Adjust palette to include more mid-tones.
• Underbase too transparent: White underbase must be opaque enough to support color vibrancy. Check ink opacity and flash settings.

Problem 3: Skin Tones Look Wrong / Unnatural

Symptoms: Flesh tones appear too orange, too pink, or posterized.

Causes and fixes:

• Insufficient skin tone colors in palette: Skin requires at least 2–3 distinct tones for natural appearance. Use “Forced” option in Indexed Color to lock key skin tone values.
• Color table shifted by automatic selection: Manually edit the warm flesh tones in the Color Table after conversion. Shift them toward neutral — remove excess red or yellow saturation.
• Increase color count: The most reliable fix. Going from 6 to 8 colors specifically improves the smooth rendering of flesh tones.

Problem 4: Banding in Gradients

Symptoms: Smooth gradients in the original image appear as distinct bands of color in the print.

Causes and fixes:

• Dither amount too low: Increase to 90–100% in Indexed Color dialog
• Gradient too compressed: Pre-process the gradient in the original image using Curves to expand the tonal range before separation
• Insufficient colors: Add 1–2 more intermediate tones to the color count
• Try “Selective” palette instead of “Perceptual”: Selective palette weights color selection toward colors that appear frequently — gradients often benefit from more evenly spaced color distribution

Problem 5: Colors in Print Don’t Match Screen Proof

Symptoms: Printed result looks different from the digital proof on screen, even though the color separation appears correct.

Causes and fixes:

• Ink colors not matched to index table: Each index color must be physically matched to an ink. Provide Lab values or Pantone equivalents of each index color to the printer.
• Underbase affecting final colors: White underbase reduces apparent saturation of colors printed on top. Compensate by slightly increasing saturation in the separation.
• Print order: On multi-color presses, print order affects final appearance when inks slightly overlap at edges. Standardize print order.
• Screen display not color-calibrated: Use a calibrated monitor (D50 or D65 white point) for proofing. Uncalibrated displays show significant color inaccuracies.

Problem 6: Moiré in Index Color Print

Symptoms: Regular repeating pattern visible in the print — unusual since index color should be moiré-free.

Causes and fixes:

• Dither type set to Pattern instead of Diffusion: In Indexed Color dialog, always select Diffusion dither. Pattern dither produces regular grid structures that can create interference patterns.
• Image was pre-processed with a halftone filter: Some image preparation workflows accidentally apply AM halftone patterns before index separation. Verify source image has no halftone structure.
• Very low color count with large pixel size: At very low color counts (4–5) and low image resolution, the dither pattern can approach regularity. Increase resolution or color count.

Common Mistakes in Index Color Separation

1. Using Pattern dither instead of Diffusion. This is the single most common mistake. Pattern dither creates regular dot arrays that are functionally identical to halftone screens — and just as vulnerable to moiré. Always use Diffusion.

2. Choosing too few colors and expecting photo quality. A 4-color index can look excellent for bold, graphic artwork. For photographic work, plan for at least 8–10 colors. Briefing a client on a 6-color “photo” print on a dark garment will often produce a disappointing result.

3. Not editing the auto-generated color table. Photoshop’s automatic palette selection is statistically optimal but not perceptually optimal. Always review the color table and manually adjust colors for skin tones, critical brand colors, and problematic grays.

4. Working at the wrong image resolution. Index color separation at 72 DPI produces large, blocky pixels — visible from across a room. Work at 200–300 DPI at print size.

5. Using the same palette for dark and light garments. A palette optimized for printing on white needs to include dark shadow values. A palette for dark garments should lean toward mid-tones and highlights, using the substrate as the shadow. Using one palette for both will produce either washed-out light garment prints or dark, heavy dark garment prints.

6. Forgetting to check the minimum dot printability. After separation, inspect the films for 1-pixel isolated dots. Single isolated pixels in fine highlight areas may not print reliably if the mesh opening is too small. Set minimum dot size to 2–3 pixels in the RIP if available.

7. Printing index color at the wrong mesh count. Using a 110 mesh for a 200 DPI index image means pixels are larger than the mesh openings in some orientations — producing irregular, merged ink coverage. Always match mesh count to pixel size.

Summary: When to Choose Index Color Separation

Index color separation is the right choice when:

• Printing on dark garments with a limited number of screens
• Moiré is a problem with your press setup
• Working with textured substrates where AM halftone performs poorly
• You need opaque, vibrant colors rather than transparent process inks
• Your press has limited precision registration capability
• You want predictable, consistent results run-to-run

It is not ideal when:

• Printing on smooth white paper where fine CMYK halftone is possible
• The design requires very specific Pantone color matching throughout
• Budget limits you to 3–4 colors (the visual difference between 3 colors and 4 is less valuable than investing in a better 8-color index setup)

Used correctly, index color separation allows screen printers to achieve genuinely photographic print quality with standard equipment, standard inks, and a fraction of the moiré risk of traditional CMYK process printing.

Dragonfly Colors provides professional index color separations optimized for your specific press, substrate, and ink system. We include complete ink matching data (Lab values) with every separation. Get in touch to discuss your project.