You mean the speaker diameter plus 1/4" is all you have for the side space?
If so, could you route out the two front part of the sides on the 3/4" MDF -
the left and right side front edges where the speaker goes into it? Looks
like a common practice for speaker construction where the outer edge or the
speaker comes almost next to the outer box edge if I'm not mistaken.

Assuming the box is not very big, I don't think too many people will notice
the sound quality difference between 1/8" and 3/4" for the side panels after
you fill up the box with insulation material.