Both Nikon D5300 and Canon 77D have APS-C sized 24.0 MP resolution sensors so sensor size and resolution is not a differentiator between these two cameras.
Another difference between these two cameras is that Nikon D5300's sensor lacks anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moire occurring in certain scenes.
Below you can see the D5300 and 77D sensor size comparison.
As seen above, Nikon D5300 has a 1.1x Larger sensor area than Canon 77D. Larger sensors give photographer more control on the depth of field and blurry background compared to smaller sensor when shot in same focal length and aperture.
DxOMark Sensor Scores
Both Nikon D5300 and Canon 77D sensors have been tested by DxoMark. DxoMark scores camera sensors for color depth (DXO Portrait), dynamic range (DXO Landscape) and low-light sensitivity (DXO Sports), and also gives them an overall score. Of the two cameras that we are comparing, D5300 has scored 83, 5 points higher than 77D.