Humans recognize social interactions effortlessly, even when presented with minimal visual information in unfamiliar displays. While force dynamics has been proposed as latent representations for perceiving social interactions, most research has approached this topic from a linguistic perspective based on conceptual knowledge, leaving open the central question of how latent force representations arise from visual inputs. The present study developed a force model that represents social interactions through two types of compositional forces: interactive forces, driven by interactions between agents; and self-propelled forces, driven by intentions of individual agents. Each force was formulated using a physics function to capture the dynamics of repulsive and attractive forces. We conducted two human experiments to measure human similarity judgments across a range of interaction animations and to evaluate recognition performance using generated animations in which the forces applied to individual agents were systematically manipulated. We found that the force model provides a parsimonious account for human judgments in both experiments. These findings suggest that mid-level representations based on compositional forces driven by different goals play an important role in social perception. We conjecture that the development of social perception may be grounded in perceptual mechanisms that support intuitive physics.