Is this just with regards to performance parts, or all modding? Even most minimal performance or handling upgrades can be felt during spirited driving - even if you don't intend to track your car.
People mod their car for the same reason gamers build insanely powerful computers that are three or four times more capable of handling all of the graphics, etc of the game that their playing while streaming, or running any other apps that they desire on top of that.
I think the answer is pretty simple - it's your hobby, and no one will really understand that better than you or individuals that share that passion. I think it's insane that some people will spend thousands of dollars on the most modern, advanced kayak and equipment for it just to go dump their boat in a river a few weekends out of the year, but it's their hobby, and it thrills them and makes them happy - and that is what our spare money and time is for.
Given how much time we spend driving, I don't think it odd at all that people will pour money into upgrading various aspects of their vehicle, or customizing it to fit them better; be it changing the feel of the suspension, adding a bit of power, etc.
Now, I'm not sure if you're talking about people who dabble in modding here and there, or if you're referring to turboing, SCing, etc - in which case I presume that the later individuals simply like playing with fire
.