I replaced my SS diffuser this evening and after gentle wrestling with it and deciding I didn't want to risk breaking tabs I also removed the bumper and it was worth it.
I second the recommendation of this YouTube video:
Rear Fascia Removal.wmv I had much better luck with a second set of hands to put the bumper off and back on, just removed the black structure underneath and letting each tab loose was easy.
I took the extra step to tape the paint on the seam where the bumper meets the rest of the car and it was worth it because I saved a paint scratch when one side wasn't aligned right. First time doing something like this and I was done in an hour moving a a leisurely pace. Thanks for all the advice folks.