here's a diagram
https://www.subarupartwholesale.com/...10-350-01.html
could be a problem where the pivot pins go thru bushings. Maybe one of the cotter pins is missing and the pin is backing out, allowed bushing to wear badly. Possibly loose bolts or bushing at the rear where the shifter support frame mounts to the floor pan. Could be a number of things. I think if you get the under tray removed and have someone wiggling the level while you watch those points it will be obvious.