<p>Regular physical activity is an important factor in healthy aging. This cross-sectional study analyzed the association between lifelong physical activity and current quality of life and physical fitness in 220 people aged 60 years and over from 2010 to 2016. The subjects were divided into two groups based on World Health Organization (WHO) guidelines on physical activity: consistently active from a young age—active group (143 people) and inactive in their youth and currently—inactive group (77 people). The first two questions from the WHOQoL-BREF questionnaire (quality of life and self-assessment of health) assessed quality of life, and the Senior Fitness Test evaluated physical fitness. Additional data included handgrip strength, age, and gender. Physical activity differentiated lower (<i>p</i> = 0.003, η² = 0.10) and upper limb strength (<i>p</i> = 0.004, η² = 0.11), lower body flexibility (<i>p</i> = 0.024, η² = 0.06), and aerobic capacity (<i>p</i> = 0.009, η²= 0.17) in women, indicating more favourable fitness outcomes in active women. In both gender groups, those who were active performed the 8-foot up-and-go test faster than those who were inactive (<i>p</i> = 0.002 in women and <i>p</i> = 0.033 in men, η² = 0.32). No significant differences in quality of life were found between the active and inactive groups. Lifelong physical activity is associated with the physical fitness of older adults, especially older women. Our research confirms the need for further observation and analysis of quality of life, physical activity, and physical fitness among older adults.</p>