With the wavelength coverage, sensitivity, and high spatial resolution of JWST, it is now possible to peer through the dust attenuation to probe the rest-frame near infrared (NIR) and stellar structures of extremely dusty galaxies at cosmic noon (z~1-3). In this paper we leverage the combined ALMA and JWST/HST coverage in Abell 2744 to study the multiwavelength (0.5-4.4um) structures of 11 sub-millimeter (sub-mm) detected galaxies at z~0.9-3.5 that are fainter than bright “classical” sub-mm galaxies (SMGs). While these objects reveal a diversity of structures and sizes, all exhibit decreasing sizes and increasing central concentration towards longer wavelengths. The smaller sizes of these objects at long wavelengths indicate that their stellar mass profiles are more compact than their optical light profiles, likely due to centrally-concentrated dust obscuration. Further, we find that galaxies with higher central concentration values tend to have more extreme size ratios (comparing the rest-frame NIR to rest-frame optical); this suggests that the galaxies with the most compact light distributions also have the most concentrated dust distributions. We also find the galaxies with the most extreme size ratios do not have elevated 1.2mm flux densities compared to the rest of our sample: we argue this means compact dust geometry, rather than e.g. high total dust quantity, drives the most extreme observed rest-frame NIR-to-optical size ratios. Upcoming higher resolution 1.2mm ALMA imaging will facilitate joint spatially-resolved analysis and will directly test the dust distributions within this representative sub-mm population.