Breast tumors are characterized by an extensive desmoplastic stroma, abundantly populated by fibroblasts. Nevertheless, there is no detailed analysis of the dynamic changes in Cancer-Associated Fibroblasts (CAFs) during tumor progression. We therefore characterized the transcriptome of mammary CAFs isolated from distinct stages of mammary carcinogenesis in a transgenic mouse model of human breast cancer. We found unique CAF gene signatures that correspond to different tumor stages, with only partial overlap between the stages, suggesting co-evolution of the microenvironment with tumorigenic progression. Interestingly, the gene signature of fibroblasts isolated from hyperplastic lesions is inverse to that in CAFs from neoplastic stages, and has a growth inhibitory phenotype, while CAFs from mammary tumors express pro-inflammatory and tumor promoting gene signatures. A proteomic screen of breast cancer cells secretome identified Osteopontin (OPN) in reprogramming of normal mammary fibroblasts to pro-inflammatory CAFs. Knockdown of OPN in tumor cells in vivo resulted in attenuated stromal activation, reduced immune cell recruitment and inhibition of tumor growth. Furthermore, we show that immune cell recruitment and pro-inflammatory signaling by mammary CAFs is mediated by the pro-inflammatory secreted glycoprotein CHI3L1, highly upregulated in CAFs isolated from invasive mammary tumors. Knockdown of Chi3L1 in fibroblasts during orthotopic breast cancer transplantations resulted in attenuated tumor growth, functionally implicating CAF-derived Chi3L1 in breast cancer progression. Moreover, CHI3L1 enhanced tumor vascularization and increased macrophage migration and upregulation of an M2-associated gene signature, implicating fibroblast-derived CHI3L1 as a key player in the crosstalk between cancer-associated fibroblasts and macrophages in the microenvironment of breast tumors. Taken together, our findings reveal a reciprocal crosstalk between fibroblasts, cancer cells and macrophages, which mediates the co-evolution of fibroblasts during breast cancer progression.