Abstract: The optical properties of the second generation dust that we observe in debris disks remain quite elusive, whether it is the absorption efficiencies at millimeter wavelengths or the (un)polarized phase function at near-infrared wavelengths. Thankfully the same particles are experiencing forces that are size dependent (e.g., radiation pressure), and with high angular resolution observations we can take advantage of this natural spatial segregation. Observations at different wavelengths probe different ranges of sizes, and there is therefore a great synergy in multi-wavelength observations to better constrain the optical properties of the particles. We present a new approach to simultaneously model SPHERE and ALMA observations and apply it to the debris disk around HD\,32297, putting the emphasis on the spatial distribution of the grains with different $\beta$ values. This modeling approach requires few assumptions on the actual sizes of the particles and the interpretation can therefore be done a posteriori. We find that the ALMA observations are best reproduced with a combination of small and large $\beta$ values ($0.03$ and $0.42$) while the SPHERE observations require several intervals of $\beta$ values. We discuss the nature of the halo previously reported in ALMA observations, and hypothesize it could be caused by over-abundant $\mu$m-sized particles (the over-abundance being the consequence of their extended lifetime). We model the polarized phase function at near-infrared wavelengths and fluffy aggregates larger than a few $\mu$m provide the best solution. Comparing our results with comets of the solar system, we postulate that the particles released in the disk originate from rather pristine cometary bodies (to avoid compaction of the fluffy aggregates) and are then set on highly eccentric orbits, which could explain the halo detected at long wavelengths.