Writing annoying people - how accurate is too accurate?

A discussion with several friends led me to wonder where the line between accurate and too accurate may be when portraying a character who is, shall we say, not well liked. For example, I know quite a few people who stopped reading the Harry Potter books after Order of the Phoenix, because Harry as written in that book is a little too close to the average obnoxious teenager for comfort. I admit, it's very clear when some authors have more experience with, say, teenagers, and use that experience to inform their writing.

Obviously, as writers we want to create characters that are realistic and multi-dimensional, because those characters are much more interesting to read and write and they drive more exciting stories. At the same time, I've certainly put books down because a character was getting on my nerves - in general, if I want to reach through a page and throttle a character, it's probably a book I should put down.

So where to draw the line? Or is there actually a line that needs to be drawn? I'm not certain, myself; no one has ever accused any of my characters as being "too realistic," so it may be a moot point for me. But as readers and/or writers, what do you all think?