We present a study of the growth of the quiescent galaxy population between 0.5 < z < 3 by tracing the number density and structural evolution of a sample of 4518 old and 583 young quiescent galaxies with log(M_/M_⊙)>10.4, selected from the COSMOS2020 catalog with complementary HST/F160W imaging from the 3D-DASH survey. Among the quiescent population at z∼2, roughly 50% are recently quenched galaxies; these young quiescent galaxies become increasingly rare towards lower redshift, supporting the idea that the peak epoch of massive galaxy quenching occurred at z>2. Our data show that while the effective half-light radii of quiescent galaxies generally increases with time, young quiescent galaxies are significantly smaller than their older counterparts at the same redshift. In this work we investigate the connection between this size difference and other structural properties, including axis ratios, color gradients, stellar mass, and the intrinsic scatter in effective radii. We demonstrate that the size difference is driven by the most massive sub-population (log(M_/M_⊙)>11) and does not persist when restricting the sample to intermediate mass galaxies (10.4<log(M_*/M_⊙)<11). Interestingly, the intrinsic scatter in physical size shows a strong co-evolution over the investigated time period and peaks around z∼2 for both populations, only diverging at z < 1. Taken together, and assuming we are not missing a significant population of lower surface brightness galaxies, while the formation and quenching mechanisms that dominate at higher redshifts yield compact remnants, multiple evolutionary pathways may explain the diverse morphologies of galaxies that quench at z<1.