A challenge in cancer biology is to understand the interactions between malignant cells and the tumor microenvironment. This requires dissecting the gene expression programs of the diverse stromal and immune cell types and uncovering their specific interactions with the tumor. Here, we use single-cell transcriptomics to assemble a comprehensive cell atlas of the tumor microenvironment of the liver, a major site for metastases and local tumors. We sequenced the transcriptome of more than 6,000 individual cells from 6 patients with colorectal liver metastases or cholangiocarcinomas, including tumor cells and cells from surrounding normal liver. We find that while individual patients significantly differ in their tumor cell expression programs, the liver tumor microenvironment exhibits robust recurring patterns in terms of cellular composition and expression. We characterize the expression program of activated human stellate cells, the diverse liver monocyte populations, identify a Treg program and a distinct expression program of liver tumor endothelial cells. We highlight specific contribution of each cell type to the tumor extra-cellular matrix composition and exposes ligand-receptor interactions between the tumor cells and specific stromal cell sub-types. Our work constitutes an important resource for identifying potential sources of vulnerabilities in liver tumors and metastases and an approach that can be applied to other tumor types as well.