I did my test drive in a BRZ with my wife, toddler son who both sat in the back and my younger daughter who sat in the front. We had plenty of room. Keep in mind that was only a 20-30 minute test drive and I was not in the throttle or driving hard.
Life is far too short to drive boring cars.
BTW, I have lived in Japan for the greater part of my adult life and I see so many families here that drive econo Kcars. Compared to those the BRZ is not that bad.
Just make the right choice for you because you alone will have to live with your decision, not all of the E-folks that are full of opinions...
Cheers,
-Nate