What are you using to remove the polish? (And to dry the car as well?)
You can "fix" the swirls.
Meguire's offers a couple of products that I've used in the past; 'SwirlX', and 'Ultimate Compound'. They both work well.
You'll want to be sure and use the proper aplicator as well as the proper towels for removal. I suspect this is what is cauing you your problems.
Ed