I've seen this happen quite a few times over dozens of different cars, honestly not worth the bet. If someone wants to trash talk your car, sure no problem. If you really can't handle the guy provoking you like that, just tell him, "You know what, your car probably IS faster. I'm down to race our times at an Auto-X event, but I'm not really up for the $500 bucks. I'll probably lose, and I don't feel like losing the money!" Laugh it off.
If he accepts, that's great! You guys get to have fun at an Auto-X. If you lose, you're not out any money. If you win, he'll probably shut up about your car and start consider getting one himself!
Basically, as others have said, this is a car to have fun in. Sure you could make it competitive but not without some mods first. More likely than not 29 years of experience (even if it's backroad driving) could make a real difference. I don't know how long you've been driving or what sort of skill you have, but in those 29 years he could have been perfecting heel-toe shifting, learning how to control his throttle input mid turn, figuring out how to save himself from mistakes, etc etc. Plus, depending on how long he's had his Audi, he could know his car much better than you do, and that could make a HUGE difference when it comes to performance driving. I've witness my friend smoke an Evo VIII at an Auto-X with a '98 Camry purely because he was more experienced(To be fair, it's a pretty awesome Camry).
Take the challenge, but don't bet any money unless you're really prepared to drop $500 bucks over a couple seconds.