My only problem with the bouncer is when its used against tail-enders. Even at Test level there are some really hopeless batsmen out there, and their ability should be taken into account, as they're far more likely to end up turning away from the ball and getting in a bad position.
I watched Mitchell Johnson bowl at England at the Gabba last summer, and whilst officially he only bowled one 'bouncer' per over, almost every delivery was at high chest or shoulder height. I did read some commentary about that being something umpires can and should stop against tail enders, but nothing came of it.