Samsung NX300M has a
20.0MP APS-C (23.5 x 15.7 mm ) sized CMOS sensor and features DRIMe IV processor. On the other hand, Nikon D3500 has a
24.0MP APS-C (23.5 x 15.6 mm ) sized CMOS sensor and features Expeed 4 processor.
One other difference between these two cameras that is worth mentioning is that Nikon D3500's sensor doesn't have an anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moiré occurring in certain scenes.
Below you can see the NX300M and D3500 sensor size comparison.
Samsung NX300M and Nikon D3500 have almost the same sensor size, so neither of them has any significant advantage over the other in terms of providing control over depth of field when used with the same focal length and aperture.