Both Canon 77D and Nikon D5300 have APS-C sized 24.0 MP resolution sensors so sensor size and resolution is not a differentiator between these two cameras.
One other difference between these two cameras that is worth mentioning is that Nikon D5300's sensor doesn't have an anti-alias (Low-Pass) filter. Removing anti-alias filter increases the sharpness and level of detail but at the same time, it increases the chance of moiré occurring in certain scenes.
Below you can see the 77D and D5300 sensor size comparison.
Sensor Size and Resolution Comparison image of Canon 77D and Nikon D5300 Cameras
As seen above, Nikon D5300 has a 1.1x Larger sensor area than Canon 77D. Larger sensors give photographer more control on the depth of field and blurry background compared to smaller sensor when shot in same focal length and aperture.
DxOMark Sensor Scores
DxOMark is a benchmark that scientifically assesses image quality of camera sensors. It scores camera sensors for color depth (DXO Portrait), dynamic range (DXO Landscape) and low-light sensitivity (DXO Sports), and also gives them an overall score. Canon 77D and Nikon D5300 sensors have been tested by DxO and the results show that D5300 has a better overall score of 83, 5 points higher compared to 77D's score of 78.
Model
Overall
Color Depth
Dynamic Range
Low-light ISO
Canon 77D
78
23.6 bits
13.3 Evs
971 ISO
Nikon D5300
83
24.0 bits
13.9 Evs
1338 ISO
Sensor size sets the circle of confusion, so it drives how much depth of field each
of these bodies gives you at a given aperture. You can calculate it exactly for the
Canon 77D
or the
Nikon D5300 ,
including hyperfocal distance.