Résumé : Higher order finite volume schemes for magnetohydrodynamics (MHD) and relativistic magnetohydrodynamics (RMHD) are very valuable because they allow us to carry out astrophysical simulations with very high accuracy. However, astrophysical problems sometimes have unusually large Mach numbers, exceptionally high Lorentz factors and very strong magnetic fields. All these effects cause higher order codes to become brittle and prone to code crashes. In this paper we document physical constraint preserving (PCP) methods for treating numerical MHD and RMHD. While unnecessary for standard problems, for stringent astrophysical problems these methods show their value. We describe higher order methods that allow divergence-free evolution of the magnetic field. We present a novel two-dimensional Riemann solver. This two-dimensional Riemann solver plays a key role in the design of PCP schemes for MHD and RMHD. We present a very simple PCP formulation and show how it is amalgamated with the evolution of face-centered magnetic fields. The methods presented here are time-explicit and do not add much to the computational cost. We show that the methods meet their design accuracies and work well on problems that would otherwise be considered too extreme for typical higher order Godunov methods of the type used in computational astrophysics.