Résumé : We investigate the clustering properties of 3<z<5 candidate passive galaxies from the Merlin et al. (2019) sample residing in the GOODS-North (35 sources) and GOODS-South (33 sources) fields. Within the large uncertainties due to the paucity of sources we do not detect clustering signal in GOODS-North, while this is present in GOODS-South, highlighting the importance of the effects of cosmic variance. The estimated correlation length in GOODS-South is r_0=12^+4_-5 Mpc, while the estimated minimum mass for a halo capable to host one of such high-redshift quenched galaxies is log10(M_min/M_sun) =13.0\pm 0.3, once also the constraints from their space density are taken into account. Both values are compatible with the results from GOODS-North. Putting the above findings in a cosmological context, these suggest no evolution of the dark matter content of the hosts of passive galaxies during the past 12.5 Gyr, i.e. during more than 90% of the age of the Universe. We discuss possible scenarios for the observed trend.