The first ($v_1^{\text{fluc}}$), second ($v_2$) and third ($v_3$) harmonic coefficients of the azimuthal particle distribution at mid-rapidity, are extracted for charged hadrons and studied as a function of transverse momentum ($p_T$) and mean charged particle multiplicity density $\langle \mathrm{N_{ch}} \rangle$ in $U+U$ ($\sqrt{s_{\rm NN}}$ =193 GeV), Au+Au, Cu+Au, Cu+Cu, $d$+Au and $p$+Au collisions at $\sqrt{s_{\rm NN}}$ = 200 GeV with the STAR Detector. For the same $\langle \mathrm{N_{ch}} \rangle$, the $v_1^{\text{fluc}}$ and $v_3$ coefficients are observed to be independent of collision system, while $v_2$ exhibits such a scaling only when normalized by the initial-state eccentricity ($\varepsilon_2$). The data also show that $\ln(v_2/\varepsilon_2)$ scales linearly with $\langle \mathrm{N_{ch}} \rangle^{-1/3}$. These measurements provide insight into initial-geometry fluctuations and the role of viscous hydrodynamic attenuation on $v_n$ from small to large collision systems.