I agree with everything you've said in principal with a few small caveats.
Yes, Nitrogen is inert and non-flammable. Air is non-flammable as well. What normal tire fills have is oxygen which is an oxidizer. That's sort of a silly reason to condone nitrogen fills though because compressed air is exactly the same makeup as atmospheric air, just under pressure.
BUT WAIT!... tires are under-pressure which means you get lots of oxygen if a tires blows, which fuels the fire right?! Not exactly. Not sure if any of you have much experience with vehicle fires and tires exploding due to flame impingement but the main concern there is the explosive force of the rapidly expanding gas, not the sudden on rush of normal air providing an oxidizing agent and nitrogen filled tires will have exactly the same effect as a normal air filled tire in that regard. I'm quite sure that I've dealt with hundreds (if not thousands) more exploding tires due to fire than 99.9% of the forum goers here.
As for filling tires while at the track...
Weighs less, takes up almost no room, free to use, no refill cost, no electricity requirements, not limited by hose length from the trailer, etc...
Far superior to electric compressors or carrying around gas cylinders IMO.