Introduction: Cardiorespiratory fitness (CRF) is associated with cardiovascular co-morbidities and is a strong predictor of adverse cardiovascular outcomes. However, data on the natural history of CRF among healthy subjects is limited. This study investigated what are the predictors of deterioration in CRF over time. Methods: We investigated 36,239 men and women who were annually screened in a tertiary medical center and completed an exercise stress test in all visits, with a total of 175,596 annual visits. Subjects with less than five visits to the center or those who developed ischemic heart disease during follow-up were excluded. Change in CRF between the first baseline visit and the fifth visit was used to calculate fitness deterioration. The primary study endpoint was defined as the lower sex-specific quintile of change in metabolic equivalents (METS) between visits 1 and 5. Logistic regression models were applied. Results: Final study population included 10,841 subjects. The mean age of the study population was 49 ±10 years, the mean BMI was 26±4, and 8107 (75%) were men. Median METS at baseline were 10.8 (IQR 9-12.6) and 11.1 (IQR 9.4-13) at the first and fifth visit, respectively (p<0.001 for METS between visits). Overall, 2189 (20%) subjects met the study endpoint. CFR deterioration was higher among women as compared to men (p=0.023). Out of all baseline clinical and laboratory parameters, only obesity was independently associated with fitness deterioration in the multivariate model (OR=1.4 95% CI 1.2-1.5, p<0.001). The association of obesity with fitness deterioration was modified by sex such that the risk of CRF deterioration was more pronounced in women (OR=1.6 95% CI 1.3-2, p<0.001) than in men (OR=1.3 95% CI 1.1-1.4, p<0.001). Conclusion: Obesity is an independent predictor of future CRF deterioration. The effect of obesity on future CRF deterioration is more pronounced among women as compared to men.