Abstract: The CO-to-H$_2$ conversion factor ($\alpha_\rm{CO}$) is central to measuring the amount and properties of molecular gas. It is known to vary with environmental conditions, and previous studies have revealed lower $\alpha_\rm{CO}$ in the centers of some barred galaxies on kpc scales. To unveil the physical drivers of such variations, we obtained ALMA Band 3, 6, and 7 observations toward the inner 2 kpc of NGC 3627 and NGC 4321 tracing $^{12}$CO, $^{13}$CO, and C$^{18}$O lines on 100 pc scales. Our multi-line modeling and Bayesian likelihood analysis of these datasets reveal variations of molecular gas density, temperature, optical depth, and velocity dispersion, which are among the key drivers of $\alpha_\rm{CO}$. The central 300 pc nuclei in both galaxies show strong enhancement of temperature $T_\rm{k}>100$ K and density $n_\rm{H_2}>10^3$ cm$^{-3}$. Assuming a CO-to-H$_2$ abundance of $3\times10^{-4}$, we derive 4-15 times lower $\alpha_\rm{CO}$ than the Galactic value across our maps, which agrees well with previous kpc-scale measurements. Combining the results with our previous work on NGC 3351, we find a strong correlation of $\alpha_\rm{CO}$ with low-J $^{12}$CO optical depths ($\tau_\rm{CO}$), as well as an anti-correlation with $T_\rm{k}$. The $\tau_\rm{CO}$ correlation explains most of the $\alpha_\rm{CO}$ variation in the three galaxy centers, whereas changes in $T_\rm{k}$ influence $\alpha_\rm{CO}$ to second order. Overall, the observed line width and $^{12}$CO/$^{13}$CO 2-1 line ratio correlate with $\tau_\rm{CO}$ variation in these centers, and thus they are useful observational indicators for $\alpha_\rm{CO}$ variation. We also test current simulation-based $\alpha_\rm{CO}$ prescriptions and find a systematic overprediction, which likely originates from the mismatch of gas conditions between our data and the simulations.