When do birds fly south for the winter?

I am renovating my rotten fascia boards and there is a birds nest in
one corner.
What is the best time to evict the birds?
I don't want the deaths of some young chicks on my conscience!

Bruce