Nikon D5300 has a
24.0MP APS-C (23.5 x 15.6 mm ) sized CMOS sensor and features Expeed 4 processor. On the other hand, Sony a3500 has a
20.0MP APS-C (23.5 x 15.6 mm ) sized CMOS sensor and features BIONZ processor.
Another difference between these two cameras is that Nikon D5300's sensor lacks anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moiré occurring in certain scenes.
Below you can see the D5300 and a3500 sensor size comparison.
Nikon D5300 and Sony a3500 have the same sensor sizes so they will provide same level of control over the depth of field when used with same focal length and aperture. On the other hand, since Sony a3500 has 21% larger pixel area (18.50µm2 vs 15.28µm2) compared to Nikon D5300, it has larger pixel area to collect light hence potential to have less noise in low light / High ISO images.