The Medway Handyman wrote:

You shouldn't be able to remove the olive - unless you have a specific
olive removal tool.

If the need arises you sometimes can. Put an open ended spanner round the
pipe behind the olive and drive it off with a hammer.

--
Mike Clarke