Abstract: When the body gets infected by a pathogen the immune system develops pathogen-specific immunity. Induced immunity decays in time and years after recovery the host might become susceptible again. Exposure to the pathogen in the environment boosts the immune system thus prolonging the time in which a recovered individual is immune. Such an interplay of within host processes and population dynamics poses significant challenges in rigorous mathematical modeling of immuno-epidemiology. We propose a new framework to model SIRS dynamics, monitoring the immune status of individuals and including both waning immunity and immune system boosting. Our model is formulated as a system of two ODEs coupled with a PDE. After showing existence and uniqueness of a classical solution, we investigate the local and the global asymptotic stability of the unique disease-free stationary solution. Under particular assumptions on the general model, we can recover known examples such as large systems of ODEs for SIRWS dynamics, as well as SIRS with constant delay. Moreover, a new class of SIS models with delay can be obtained in this framework.