Both Nikon D5300 and Nikon D5200 have APS-C sized 24.0 MP resolution sensors so sensor size and resolution is not a differentiator between these two cameras.
Another difference between these two cameras is that Nikon D5300's sensor lacks anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moire occurring in certain scenes.
Below you can see the D5300 and D5200 sensor size comparison.
Nikon D5300 and Nikon D5200 have sensor sizes so they will provide same level of control over the depth of field when used with same focal length and aperture.