You are doing the right thing by taking it easy.
No need to inflate the aggravation in the neighbourhood by driving like an asshole.
I know if there was someone ripping out of my neighbourhood at 4:30 a.m. every day the least of his worries would be me calling the cops on him whether he had a technically legal exhaust or not.
People on here advising different are giving you bad advice.
Even with a technically legal exhaust there are still by laws about excess noise of any type outside of 8 a.m. to 10 p.m.
At least where I live.