Pragmatic atypicality is widely considered to be a central characteristic of autism. This is often explained as a consequence of Theory of Mind deficits. However, this account is flawed and biased. In this paper, we revisit the Double Empathy Problem and provide an experience-first approach to autistic pragmatics. We start with proposing a mechanistic explanation of a link between experiential differences and intentionality understanding in linguistic contexts using the Interpretive Sensory Access theory. Then, we explain how theories of common ground in communication involve factors beyond intention recognition and even beyond cooperation, highlighting how the egocentric nature of communication is relevant to one’s attention and experiences. Taken together, we put forward an experience-based approach to understand autistic pragmatic atypicalities. This view is compatible with many other non-linguistic characteristics well-documented in autism, and prioritizes the experience of autistic people, instead of framing it as a communication disorder with a “mind-reading failure”.