Is there a knack to it or am I doing something wrong?

Leave the handles on. Reassemble the door. Undo the one screw level with the
cylinder. Insert key, withdraw cylinder. You may need to rotate the key 20
degrees one way or the other. It should take all of 20 seconds.

Christian.