Nikon D5300 has a 24.0MP APS-C (23.5 x 15.6 mm ) sized CMOS sensor and features Expeed 4 processor. On the other hand, Sigma Quattro has a 29.0MP APS-C (23.5 x 15.6 mm ) sized CMOS (Foveon X3) sensor and features Dual TRUE III processor. Sigma Quattro's sensor provides 5MP more than Nikon D5300's sensor, which gives a significant advantage in real life. You can print your images larger or crop more freely.
On the other hand, please keep in mind that Max sensor resolution is not the only determinant of resolving power. Factors such as the optical elements, low pass filter, pixel size and sensor technology also affects the final resolution of the captured image.
Another difference between these two cameras is that Nikon D5300's sensor lacks anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moire occurring in certain scenes.
Below you can see the D5300 and Quattro sensor size comparison.
Nikon D5300 and Sigma Quattro have sensor sizes so they will provide same level of control over the depth of field when used with same focal length and aperture. On the other hand, since Sigma Quattro has 22% larger pixel area (18.69µm2 vs 15.28µm2) compared to Nikon D5300, it has larger pixel area to collect light hence potential to have less noise in low light / High ISO images.