While individual differences have been found in many aspects of color perception, it is unclear whether mental representations of color vary between individuals in a structured way. To test for structured variability, we collected perceptual similarity judgments for 58 colors in a triadic-comparisons procedure, and from each participant's judgments, embedded the colors into a personalized three-dimensional space. The personalized embedding predicted a participant's similarity judgments on held-out items significantly better than did (a) embeddings from other participants, suggesting reliable individual differences in perceived color similarity (p < 0.0001), and (b) color coordinates in a standardized perceptual color space (CIELAB; p < 0.0005). Across individuals, embedding structure did not vary randomly but fell into two clusters, one encompassing more distinct color categories and the other a more continuous perceptual space. The results suggest the existence of previously unrecognized “motifs” in how people represent colors.