When learning to categorize stimuli, do we assume similar things should have similar labels? Are people more likely to respond with closer labels (e.g. 2-1 vs 2-4) when stimuli are more similar to each other? Across five experiments, we report evidence of such a bias and demonstrate that it can surface across a wide range of stimulus modalities and features, and persists regardless of participants’ prior knowledge of the dimensions relevant for categorization. We also characterize some of the limits of this effect: it appears sensitive to the specific configuration of label-stimulus mappings, and may depend on overt similarity relations in label space. At minimum, our findings indicate the need to consider label-stimulus configurations when designing categorization experiments. They also hint more broadly at how label-to-stimulus mappings may affect how we structure novel categories.