Fitness equalizing mechanisms, such as trade-offs, are recognized as one of the main factors promoting species coexistence in community ecology. However, they have rarely been explored in microbial communities. Although microbial communities are highly diverse, the coexistence of their multiple taxa is largely attributed to niche differences and high dispersal rates, following the principle “everything is everywhere, but the environment selects”. We use a dynamical stochastic model based on the Theory of Island Biogeography to study highly diverse microbial communities over time across three different systems (soils, alpine lakes, and shallow saline lakes). Here we report for the first time a colonization-persistence trade-off in natural microbial communities. We conclude that this trade-off is mainly driven by rare taxa, which are occasional and more likely to follow independent colonization/extinction dynamics. Our work also emphasizes the fundamental value of dynamical models for understanding temporal patterns and processes in highly diverse communities.