A core challenge in cognitive science is understanding the barriers to intelligence and the circumstances that favor cognitive specialization. General intelligence requires a cognitive architecture that is successful across tasks. However, improving an architecture for a given task is often observed to hinder performance on others. Although trade-offs between tasks are a recurring element of explanations in cognitive science, they have received little direct theoretical attention. We present a formal framework that provides a principled understanding of when trade-offs emerge. In particular, we build on recent advances in applying rate-distortion theory to reinforcement learning. This allows us to formalize the representational capacity an agent can call upon in approaching tasks in terms of information. We find trade-offs occur when components of a task conflict in ways that cannot be easily coarse-grained by the agent’s encoding scheme. Further, cognition may be general, specialized, or implement a coverall strategy, depending on conditions.