I would blow it off. I had one want to race me, and I said sure, but I will race him with my 4 cylinder and not my V6. And I said let's make it interesting and race for titles, if he's so confident he would win. So we went out to the drag strip later that day and I rolled up on this and he backed out.
In the end, your car looks way cooler and who cares if someone else is faster. There will always be someone faster even if it is a neon.