Index color separation is one of the most powerful — and most misunderstood — techniques in screen printing. It solves problems that CMYK halftone process printing cannot: it does away with the angled halftone screens that cause moiré in process work, it works beautifully on textured garments, it handles opaque inks on dark substrates, and it produces vivid, photographic-quality results with as few as six colors. Used correctly, index color separation can outperform CMYK process printing in many real-world screen printing scenarios. This guide covers what index color separation is, why it works, how to execute it in Photoshop, how to choose the right color count, and a complete troubleshooting section covering the most common problems printers encounter with indexed separations.


What Is Index Color Separation?

Index color separation is the process of reducing a full-color image (which may contain millions of colors) down to a small, defined set of colors — the “index” — and then generating a separate film/screen for each of those colors. Unlike CMYK process printing, where four transparent inks are overlaid at different halftone angles to reproduce color through optical mixing, index color printing places each ink color side by side, pixel by pixel, with no intentional overlap. Each ink covers its area completely with solid dots. The core mechanism that makes this work is diffusion dither — a noise-based dithering algorithm that distributes the error of replacing one color with another in a visually random, spatially distributed pattern. The result is a pseudo-random pixel array that, when viewed from a normal distance, blends into smooth gradients and nuanced color transitions.


Index Color vs. CMYK Process: A Direct Comparison

Understanding when to use index color separation versus CMYK requires understanding the fundamental difference in ink behavior.

CMYK Process Printing

CMYK uses four transparent inks (Cyan, Magenta, Yellow, Black) applied in overlapping halftone dots at offset angles. Color is produced by the eye mixing the overlapping transparent layers. Strengths of CMYK:

Weaknesses of CMYK for screen printing:

Index Color Printing

Index uses a set of opaque spot colors placed in a diffusion dither pattern. No two colors intentionally overlap; each ink sits cleanly on the substrate (or underbase). Strengths of index color:

Weaknesses of index color:


Simulated Process — the Method This Comparison Usually Leaves Out

CMYK is not the only alternative to index, and for garment work it is rarely the real one. Simulated process separates the image into a set of opaque spot inks, exactly as index does, but prints each of them as a conventional halftone whose dots grow and shrink. It therefore has the same gamut advantage as index — real inks, so real Pantones, fluorescents and metallics — while keeping the tonal behaviour of a halftone.

The difference that matters is where the tone comes from. In simulated process each screen carries a full range from 0 to 100 percent by itself, so a single ink can render a smooth gradient on its own. In index an ink has one value only: it is either on that pixel or it is not, and tone exists solely because neighbouring pixels of different inks sit side by side. Everything else follows from that:

Neither method replaces the other. Simulated process is the better tool for soft, photographic, gradient-led artwork when the press is accurate and the fabric is smooth. Index is the better tool when the image is busy, the fabric is textured, the press has some play in it, or moiré has already cost a run.

There is one more practical difference: a good simulated process separation is mostly judgement — how the channels are built, where they are held back, how much they are allowed to overlap. That is why it is still done by hand, one job at a time. An index separation is a different kind of problem, with far more of it decided by arithmetic, which is exactly why it can be automated.

How Many Colors Does Index Color Need?

This is the most common question and the most important decision in index color separation. The answer depends on the complexity of the artwork and the required print quality.

Color Count Result Quality Best For
4–5 colors Graphic, stylized look Bold designs, limited ink setups
6–7 colors Good for simpler photography Event shirts, merchandise
8–10 colors Photo-quality on most images Professional commercial work
12–14 colors Excellent photographic reproduction Art prints, premium garments
16+ colors Near-photographic High-end decorative printing

A useful rule of thumb: 12-color index looks approximately twice as smooth as a 6-color index separation of the same image. Each additional color adds detail and tonal nuance, particularly in skin tones, skies, and shadow areas.


How to Create Index Color Separation in Photoshop: Step by Step

Photoshop’s Indexed Color mode is the standard tool for index color separation. Here is the complete process.

Step 0: Check the File Before You Touch Anything

Almost every separation that has to be redone from scratch was decided here, before a single button was pressed.

Check that black is black and white is white wherever they are meant to be, and that any area meant to be one flat color really is one value. A yellow headline on a poster looked perfectly uniform and was stored in the file as #e4af37, #e4b037, #e3b037 and #e3af37 — four values one point apart on a single channel. Clustering saw four groups and split one ink across three palette entries, which is three screens for one color.

Zoom in on edges and on flat fields, sample several points of the same field with the eyedropper, and be especially suspicious of files that have been resized or saved as JPEG. Those are what produce this.

Check the color profile by hand: Edit → Assign Profile. That dialog tells the truth about whether the document is untagged or carries a profile. An untagged file should be assigned sRGB, which describes the pixels without changing them.

Step 1: Prepare the Image

Start with an RGB image at whatever resolution it came in. The 180–220 DPI from the mesh table above is the resolution the separation ends at, not the one it starts at (see Step 3b: resampling the file down before separating throws away the smoother result). The image should be:

Do not convert to CMYK before performing index separation — work from RGB.

Step 2: Flatten and Merge All Layers

Image → Flatten Image. All layers must be merged before conversion.

Step 3: Convert to Indexed Color

Image → Mode → Indexed Color In the Indexed Color dialog:

Click OK.

Step 3b: Two Passes — Index High, Then Down

Do not resample the file down to the print resolution before separating. Index it at the resolution it arrives in — 300 DPI, or whatever the file actually is. Then convert back to RGB, resize to the print resolution from the mesh table above, and index a second time on the same palette: Image → Mode → Indexed Color, with Palette set to Previous.

The reason is what the downsampling does in between. Reducing an image that is already indexed squeezes the blocks of flat color together and softens the hard edges between them, and the second pass then works on that softened material. The result is visibly smoother than indexing straight to the low resolution, where every edge is decided once, at full contrast, on a coarse grid.

Reusing the palette in the second pass is not optional. Let Photoshop choose freely and it will pick a different set of colors from the downsampled image — and the palette you are about to review in the next step is then not the palette that ends up on the films.

If the file already arrives at the print resolution there is nothing to do: the second pass has nothing to reduce, and one pass is the whole job.

Step 4: Review and Edit the Color Table

Image → Mode → Color Table This shows the exact index colors Photoshop has selected. Here is where expert judgment comes in: Review each color for printability:

What a palette color actually controls. An entry in the index table decides which pixels go to which screen, and nothing else. Which ink gets mixed in the bucket is the printer's decision, and the film knows nothing about it. That distinction reverses the usual advice: do not "improve" a palette entry to make it prettier or more printable.

The measured case: pale blue lettering printed as one clean solid because the palette held exactly the value that was in the file. Changing that entry to a stronger blue broke the lettering into dots — no ink now matched the file, so the dither started building the letters out of a mixture. The behaviour was correct. The result was not wanted.

The rule for flat areas assigned to a single entry: leave the value from the file in the palette and take the stronger shade at the bucket. The substitution is one to one and the separation stays solid. The rule stops applying where an area is shaded and built from several inks mixing, because there, changing one of them really does change the result.

A color that is not in the file cannot be added in the palette. It goes like this. An area reads warm — brick red, say — against a large cold field around it, and the printer asks for that warmth on the shirt. Sample it and the pixels turn out to be cold as well: darker than their surroundings and shifted perhaps 15°, but still on the cold side. The eye read them as warm only by contrast. Such a pixel can sit 6 ΔE from a color already in the palette and 29 from the warm one added for it, so the dither keeps choosing the first and the area stays cold. The stronger that warm ink, the further away it is and the fewer pixels it wins. There is no saturation that is both visible and close enough — it is a dead end, not a tuning problem.

The fix is to warm the area in the original with an adjustment layer and extract the colors again. Then the color is in the pixels: extraction finds it, the dither assigns it, and refreshing the separation does not undo it. Retouch before separation, not in the palette.

One trap in between: swapping a color in the palette repaints the document immediately but does not recompute the dither. It therefore looks right, because a strong ink is sitting in dots that were laid out for a pale one. The next refresh wipes that out. Do not build production work on it.

Step 5: Check Image Quality

Zoom to 100% and inspect:

If quality is insufficient, go back and increase the color count by 2 and repeat. The difference between 6 and 8 colors is dramatic; between 10 and 12 is subtle.

Step 6: Generate Individual Channel Files

Once the indexed image is approved, each color in the index becomes a separate film. To extract each color: Method A: Convert back to RGB and use channels

  1. Convert Indexed → RGB
  2. Use Select → Color Range to select each index color
  3. Create a layer mask from each selection
  4. Output each layer as a grayscale film

Method B: Use separation software Professional separation tools (AccuRIP, Separation Studio, FastFilms) can automate indexed color to film, with automatic halftone dot output and registration marks.

Those are halftone RIPs that also accept indexed output. IndexColorExtractor is a Photoshop plugin written for this method alone: it extracts the palette, lets you set the opacity and underbase of every ink, and writes the spot channels with registration marks and choke.

Step 7: Film Output

Each color outputs as a grayscale film — black areas indicate where that color’s ink will print. Film should be output at 1200–2400 DPI. Unlike CMYK halftone films, index color films do not require specific halftone angles because the diffusion dither pattern is non-directional.


Choosing the Ink Colors for Index Separation

The quality of an index color print depends not just on how many colors are used, but which colors are chosen. The index palette represents the physical inks that will be mixed and put on press — every color must be achievable in ink.

Working with the Printer’s Ink Set

The ideal workflow is to provide the index color table to your printer and have them match each color to their available mixing system (Pantone, custom lab). For a Dragonfly Colors separation you get the full print specification: every screen in print order, the Pantone number for each one, and the mesh count. Not numbers to convert — the sheet the press works from.


Index Color on Dark Garments: Working with an Underbase

Printing on dark garments with index color typically requires a white underbase. The underbase is a solid or partial coverage white layer that creates a neutral foundation for the index colors to print on. Two approaches to underbase + index color:

Option 1: Full White Underbase, Print All Colors on Top

A full coverage white underbase is flashed, then all index colors print wet-on-wet. This gives maximum color vibrancy but can feel heavy on the garment.

Option 2: Partial Underbase — Let the Garment Be the Shadow

The white does not have to go under the whole design. Leave it out of the dark parts of the image and the garment itself becomes the darkest value in the separation. Less ink on the shirt, a softer hand, and the palette no longer has to carry the shadows — it only has to cover mid-tones and highlights, which is often the difference between needing 10–12 colors and getting a good result with 6–8.

This is the option worth reaching for on dark garments, and it has nothing to do with choke.

Choke Is Not Part of This — and Index Work Usually Does Not Want It

Choke means printing the underbase a pixel or two smaller than the colors above it, so a misregistered press does not show a white halo around the edge of the print. At 180 DPI one pixel is about 0.14 mm.

It is an allowance at the outline of a shape, and it belongs to flat graphics — a design with a contour and solid fields, the kind with hard, clean edges. That is where a white fringe would be visible and where taking the underbase in by a pixel costs nothing.

It is not something applied to every dot of a dither. Where a design fades out into diffusion dither there is no edge to protect, and choking simply thins the underbase in the mid-tones — it eats pixels that were carrying the image. On gradients and soft transitions, leave it off.

And it is not what lets the garment show through between colors. That comes from the partial underbase above, from ink opacity, and from the dither pattern itself — not from shrinking the white.

When the detail is too thin to choke. Taking one pixel off can eat a thin line down to nothing — there is nothing left in the file to shrink. Instead of choking it, take the same film you use for the color, expose it on a white mesh and give it a longer exposure: the scattered light undercuts the edge and the opening narrows by itself, smoothly, below one pixel. Here the white thread is the tool, not the flaw.


Mesh Selection for Index Color Printing

An index dot is the size of one pixel and never changes — unlike an AM halftone dot, which grows and shrinks. So the resolution you separate at fixes the dot size, and the mesh has to be matched to it.

How fine that dot can go is not decided by whether the screen will hold it. The emulsion will hold a dot smaller than a mesh opening perfectly well. The limit is at exposure and washout: a very small dot on the film is too weak a stencil, light undercuts it, the emulsion hardens partly, and it will not come away when the screen is washed out.

Image resolutionPixel sizeMesh for the colorsMesh for the underbase
180 DPI0.141 mm (141 microns)120/cm (305/inch)90/cm (230/inch)
220 DPI0.115 mm (115 microns)130–140/cm (330–355/inch)120/cm (305/inch)

The underbase has its own column for a reason. It carries no raster — it is a solid layer — so it goes on a coarse mesh, where what counts is ink deposit. Only the screens carrying the indexed dots have to resolve a single pixel, and they want about 1.7 mesh openings across it. A 180 DPI dot on a 90/cm mesh gets 1.28, and what is missing there comes back as dot gain. A base with holes in it — when some colors print straight onto the fabric — carries the same pixel as the colors, and then it goes on the same mesh as they do.

Yellow mesh or white. A white thread scatters light under the film and the edge of the dot can get exposed — the opening in the emulsion then comes out smaller and the dot prints finer than it is on the film. With a well-judged exposure time there is practically none of this. A yellow thread does not scatter at all, so for raster work it is simply the safer one. For a dot gain measurement it means the measuring has to be done on the mesh you print on: white and yellow give different numbers.

Mind the two mesh scales. Europe counts threads per centimetre, the United States per inch, and the same number means the opposite thing in each: 90/cm is a fine mesh, 90/inch a very coarse one. Both counts are given above so there is nothing to misread.

180 DPI is the floor. Below it the pixels are large enough to read as coarse texture on the shirt, which is why almost nobody separates index work any lower.

220 DPI is the practical ceiling — and the reason sits in the washout, not on the press. Going to 250 or 300 DPI asks for dots you may not get out of the screen. It calls for a very good exposure unit, a strong light source and well-judged exposure times, and in practice for a CTS machine printing wax straight onto the emulsion: a film laid against a coated screen is simply too weak a medium at that dot size. Without that equipment the extra resolution buys nothing, and the file swells for no return.

There is a second reason this table is not a formality: interference. The colors in an index separation are scattered irregularly, but the dots themselves sit on a perfectly regular grid of pixels. When the pitch of that grid approaches the pitch of the mesh, the two patterns overlap and moiré appears on the print — indexed separation or not. 220 DPI laid on a 90/cm mesh is the classic case.

Finer meshes resolve the pixel more cleanly, but they want thinner, lower-viscosity inks to clear the opening.


Troubleshooting Index Color Separation

Problem 1: Visible Pixel Grid / Texture Too Obvious

Symptoms: Print looks “blocky” or “pixel-y” at normal viewing distance; texture is more visible than expected. Causes and fixes:

Problem 2: Flat Shadows, Flat Patches, or a Color That Does Not Belong

Symptoms: shadows with no depth; or a large area that came out as one solid patch of ink where the original had texture; or a color turning up somewhere it has no business being — green in an animal's fur, for instance.

These look like three separate faults and are usually one: a gap in the tonal ramp of a hue family. It is worth knowing, because the obvious diagnosis — "the palette picked the wrong colors" — sends you in exactly the wrong direction.

The measured case. Green kept appearing in a large warm brown area where no green belonged. Color matching had never chosen green there: the nearest green was more than twice as far away as the warm color that was chosen. The real cause was 36 units of L* of empty space between #edba15 (L* 78) and #885817 (L* 42). Pixels falling into that gap were reproduced with an error of 13–16 ΔE, diffusion dither spread that error onto their neighbours, and the pushed value eventually crossed a boundary where green was the nearest entry. Adding one color at L* 64 removed the green completely. Five earlier attempts, all of which changed which colors were in the palette, did nothing — the problem was the spacing between them.

The same illness, the opposite symptom. Elsewhere, skin tones and a sky came out as flat patches of ink with no dither at all, although the original had plenty of detail there. Nothing foreign came in; nothing mixed. The cure was different too: not adding a color, but moving the lightness of an entry that was already there. An entry sitting in the middle of a wide range captures that whole range and has no partner to share pixels with. Moved toward the edge, it leaves part of the range closer to its neighbour, and the dither comes back.

How to work through it:

Analyse the index table for this — the ink appearances, meaning each ink on the underbase and the same ink straight on the fabric, plus the underbase and the garment itself — and not the list of screens. A darkened variant of an ink is a different color, and arithmetic on raw pigment values will mislead you.

On opacity. How much an ink covers cannot be read from its color and cannot be computed. It depends on the pigment, the filler and the titanium white in the mix, which is why two inks that look almost the same can cover quite differently — yellows are usually the worst. The only way to know is to print the ink on the fabric you are going to use and hold that print against the screen.

The ordinary causes are still worth ruling out first:

Problem 3: Skin Tones Look Wrong / Unnatural

Symptoms: Flesh tones appear too orange, too pink, or posterized. Causes and fixes:

Problem 4: Banding in Gradients

Symptoms: Smooth gradients in the original image appear as distinct bands of color in the print. Causes and fixes:

Problem 5: Colors in Print Don’t Match Screen Proof

Symptoms: Printed result looks different from the digital proof on screen, even though the color separation appears correct. Causes and fixes:

Problem 6: Moiré in Index Color Print

Symptoms: Regular repeating pattern visible in the print — unusual since index color should be moiré-free. Causes and fixes:


Print Order for Index Color Separations

Print order determines how each screen is sequenced on press — and unlike CMYK process printing, where four transparent inks reliably overlay in a fixed order, index color separation places opaque inks side-by-side with only minor edge overlap. Getting the sequence wrong doesn’t just shift color slightly — it can let one ink visually dominate its neighbors in the dither pattern, or cause dot-edge buildup that looks muddy on press.

The General Rule: Dark to Light

As a starting point, sequence screens from darkest to lightest:

  1. White underbase (if printing on a dark or colored garment) always prints first.
  2. Dark, low-opacity colors print next, establishing the shadow structure of the image.
  3. Mid-tones follow.
  4. Light, highly opaque colors (near-whites, pastels, bright yellows) print last, since opaque light inks sit cleanly on top and read as crisp highlights.

This mirrors how a painter builds an image — blocking in darks first, adding highlights last — and it matches how opaque inks physically behave in a wet-on-wet stack: each screen prints on top of the previous one, so a light ink applied late stays vivid instead of being buried under darker ink printed afterward.

Why Dark-to-Light Isn’t the Whole Story

For index separations specifically, two other factors often matter more than raw lightness:

Hue proximity between neighboring colors. Because index printing places colors side-by-side in a diffusion dither pattern rather than overlaying them, colors that sit close together on the color wheel blend smoothly at their shared edges. Colors far apart in hue (e.g. a saturated red next to a saturated green) create a hard visual seam regardless of print order. When sequencing screens, check which colors actually sit adjacent to each other in the dither pattern — not just their position on a light-to-dark scale.

Matched opacity between adjacent colors. If one ink in a dither pattern has noticeably higher opacity than its neighbor, it will visually dominate that neighbor no matter where it falls in the print sequence. Before finalizing screen order, confirm that colors sharing dither boundaries have similar opacity — this matters more than strict light/dark ordering.

Some bright, saturated colors need white underneath regardless of “lightness.” A pure yellow or vivid red is technically “light” in tonal value, but both are heavily desaturated by a dark garment unless they sit on white. In practice, these colors often need to print over a dedicated patch of underbase even though the overall sequence still runs roughly dark to light.

Treat the Photoshop-derived print order as a starting plan, not a final answer. Run a test pass and check the printed sheet with the screens sitting side by side — this is the only reliable way to see which color is actually dominating its neighbor. It’s common to swap two adjacent screens after seeing the first proof, even when the digital separation looked correct on screen.


Common Mistakes in Index Color Separation

1. Using Pattern dither instead of Diffusion. This is the single most common mistake. Pattern dither creates regular dot arrays that are functionally identical to halftone screens — and just as vulnerable to moiré. Always use Diffusion. 2. Choosing too few colors and expecting photo quality. A 4-color index can look excellent for bold, graphic artwork. For photographic work, plan for at least 8–10 colors. Briefing a client on a 6-color “photo” print on a dark garment will often produce a disappointing result. 3. Not reviewing the auto-generated color table. The automatic palette selection in Photoshop is statistically optimal but not perceptually optimal. Always review it: check that skin tones, brand colors and problematic grays are represented at all, and that no two entries sit so close together that a screen is being wasted. Reviewing does not mean repainting entries to taste — what a palette color actually controls is in Step 4. 4. Working at the wrong image resolution. Index color separation at 72 DPI produces large, blocky pixels — visible from across a room. Work at 180–220 DPI at print size. 5. Using the same palette for dark and light garments. A palette optimized for printing on white needs to include dark shadow values. A palette for dark garments should lean toward mid-tones and highlights, using the substrate as the shadow. Using one palette for both will produce either washed-out light garment prints or dark, heavy dark garment prints. 6. Forgetting to check the minimum dot printability. After separation, inspect the films for 1-pixel isolated dots. Single isolated pixels in fine highlight areas may not survive exposure and washout if the dot is too small for your equipment — which is a reason to check the resolution and mesh table above, not to set a minimum dot size in the RIP. With diffusion dither, isolated single pixels are everywhere by design, and forcing a two- or three-pixel minimum would destroy the highlights it is meant to protect. 7. Printing index color at the wrong mesh count. Using a 110 mesh for a 200 DPI index image means pixels are larger than the mesh openings in some orientations — producing irregular, merged ink coverage. Always match mesh count to pixel size.


Summary: When to Choose Index Color Separation

Index color separation is the right choice when:

It is not ideal when:

Used correctly, index color separation allows screen printers to achieve genuinely photographic print quality with standard equipment, standard inks, and a fraction of the moiré risk of traditional CMYK process printing.


Dragonfly Colors provides professional index color separations optimized for your specific press, substrate, and ink system. Every separation comes with the full print specification: print order, Pantone numbers and mesh counts. Get in touch to discuss your project.

Photoshop plugin

Doing all of this by hand, on every job?

Every step above - extracting the palette, editing the color table, the underbase, the channel output - is what IndexColorExtractor does inside Photoshop. It is built for index separation and nothing else: it works from the measured opacity of your own inks instead of assumptions, keeps one ink on one screen when that ink needs two appearances, and maps where the separation lost the original before a single screen is burned.

See IndexColorExtractor