I understand exactly what you're saying. While I like my BRZ for these reasons, I fully understand what you're saying.
I said, early on, that people need to test drive this car extensively, because it may prove unsatisfactory for many reasons to a good many drivers who are unaccustomed to cars like this.
See if this resonates with you: A young member of the Honda S2000 forum said that his Dad had a new S2000 for a while, but soon sold it. He found it TOO INTENSE to drive.
The twins are like that. That close ratio gearbox keeps you shifting all the time, the low torque (relatively speaking), 2 Liter engine, normally aspirated (no turbo), and peaky performance requiring the driver to keep the revs up for any kind of performance ... conspires to make it tiring to drive for those accustomed to and preferring a more relaxed driving experience. Adding to these traits is that it's noisy, and rough riding with its stiff sporting suspension and low profile tires. (Keep those tire pressures up, or you'll bend a rim lip on today's ubiquitous potholes on our decaying roads.)
Actually, it's got good torque down low compared to many other cars with similar engines. It is quite powerful for a 2 Liter, putting out 100 hp/Liter, which is a very respectable figure. Such engines are intense, though, and typically don't offer much until they're "on the boil." (High rpms.)
But, it appears to not be what you'd expected.
By the way, you do NOT want to skip gears. You need, instead, to remain in the current gear and get the revs up higher in the power band to realize better performance. Of course, that further reduces fuel economy AND it creates more COMMOTION than you may like.

I always keep my rpms over 2000. I mosey around my village at 2000-3000 rpm, typically in 3rd gear. But there's not much grunt down there, and if you're accustomed to big engines with lots of torque at a mere 1600 rpm, you'll be surprised and possibly dismayed with the twins.
"Too intense..."
It's not Right or Wrong, Good or Bad, it's just
different from what some folks expected. It may just be the wrong car for you, is all.
Here's a car for you to go test drive: 2013 Mazdaspeed3. It's a turbo. We've had two in a row. We love ours. Give it a test drive and come talk to me.
-Porsche (a fellow old guy

)