Purpose The purpose of this study was to investigate the contribution of sport to the association between physical activity (PA) and quality of life (QoL). Methods Cross-sectional data were gathered on 4,909 subjects (age 15–69) from the French National Barometer 2005 survey. The International Physical Activity Questionnaire (IPAQ) and the abbreviated version of the World Health Organization Quality of Life Questionnaire (WHOQOL-BREF) were administered. An additional question was used to assess sporting activity. All analyses used linear regression models and were adjusted on variables associated with QoL in a general population. Results The mean age of the participants (both men and women) decreased with increasing PA level. Sport was positively associated with QoL among men (β range from 4.2 [95 % CI 3.1–5.4] for physical health to 2.4 [95 % CI 1.1–3.8] for social relationship domains) and women (β range from 3.6 [95 % CI 2.6–4.5] for physical health to 1.6 [95 % CI 0.6–2.8] for social relationship domains). The association between sport and QoL was greater for low or high PA levels rather than moderate PA for men (physical and psychological health) and women (physical health only). For women, there was a dose–response association with psychological health and social relationships (contribution of sport to QoL increased with PA level). Conclusions These results showed that sport was nearly always associated with better QoL, even more so for people who had low or high PA levels (physical and psychological health for men and physical health for women). Prospective studies are necessary to confirm these findings.