Samsung NX100 has a 15.0MP APS-C (23.4 x 15.6 mm ) sized CMOS sensor and features DRIMe Engine processor. On the other hand, Nikon D5300 has a 24.0MP APS-C (23.5 x 15.6 mm ) sized CMOS sensor and features Expeed 4 processor. Nikon D5300's sensor provides 9MP more than Samsung NX100's sensor, which gives a significant advantage in real life. You can print your images larger or crop more freely.
On the other hand, please keep in mind that Max sensor resolution is not the only determinant of resolving power. Factors such as the optical elements, low pass filter, pixel size and sensor technology also affects the final resolution of the captured image.
Another difference between these two cameras is that Nikon D5300's sensor lacks anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moire occurring in certain scenes.
Below you can see the NX100 and D5300 sensor size comparison.
Samsung NX100 and Nikon D5300 have almost the same sensor size so none of them has any significant advantage over other in providing control over depth of field when used with same focal length and aperture.