The tumor microenvironment is a complex network of various interactions between immune cells and non-cellular components such as the extracellular matrix, exosomes and interleukins. Moreover, tumor heterogeneity and its constant modification may alter the immunophenotype and become responsible for its resistance regarding the therapies applied However, it should be remembered that in a strongly immunosuppressive neoplastic microenvironment, the immune system cells undergo reprogramming and most often cease to fulfill their original function. Therefore, understanding what happens within the tumor microenvironment, and which mechanisms are responsible for tumor development and progression should let us know how cancer could protect itself against the immune system. The presented review summarizes the latest information on the interactions between the tumor microenvironment and the cellular and non-cellular components, as well as their impact on cancer development, progression and immune system exhaustion.