James:

Put a spirit level on the bifold door casing. I expect you'll find that it's not plumb.

It's not that noticable. You notice it because you know it's there.

Short of redoing the bifold door now, one way to hide this is to paint that wall the same colour as the door casings. (?) That way, the wall area between the casings won't be as noticably wider at the top than at the bottom.