The net effect may be as little as cutting a few thousand miles off of their overall life but it's real.
For the same reason that the vehicle weight must not rest on the wheels with the axle nut loosened, it's MUCH worse to hammer on that assembly with a loose axle nut than with it cinched up so the balls are properly loaded up in the races.
Here's a pretty good picture of what happens internally. The wheel bearings are a flavor of duplex angular contact type.
The axle nut is loose on the left.
http://www.bardenbearings.co.uk/index.cfm_articleid_717