Soil microbial communities play a key role in plant nutrition and stress tolerance. This is particularly true in sites contaminated by trace metals, which often have low fertility and stressful conditions for woody plants in particular. However, we have limited knowledge of the abiotic and biotic factors affecting the richness and composition of microbial communities inhabiting the rhizosphere of plants in contaminated sites. Using high-throughput amplicon sequencing, we studied the rhizospheric bacterial and fungal community structures of 14 woody plant families planted in three contrasting sites contaminated by metals (Pb, Cd, Zn, Mn, Fe, S). The rhizospheric bacterial communities in the given sites showed no significant difference between the various woody species but did differ significantly between sites. The Proteobacteria phylum was dominant, accounting for over 25% of the overall relative abundance, followed by Actinobacteria, Bacteroidetes and Gemmatimonadetes. Site was also the main driver of fungal community composition, yet unlike bacteria, tree species identity significantly affected fungal communities. The Betulaceae, Salicaceae and Fagaceae families had a high proportion of Basidiomycota, particularly ectomycorrhizal fungi, and the lowest diversity and richness. The other tree families and the unplanted soil harboured a greater abundance of Ascomycota and Mucoromycota. Consequently, for both bacteria and fungi, the site effect significantly impacted their community richness and composition, while the influence of plants on the richness and composition of rhizospheric microbial communities stayed consistent across sites and was dependent on the microbial kingdom. Finally, we highlighted the importance of considering this contrasting response of plant rhizospheric microbial communities in relation to their host identity, particularly to improve assisted revegetation efforts at contaminated sites.