While that may be part of it, I don't think that's the most important factor. Most people's minds are tradeoffs. Good at this, bad at that, good at this, bad at that. I know I'm that way.
Yep, that's precisely the point. The tradeoff you make is in the attention you pay to something. It's not possible to spend ten thousand hours focused on something without neglecting other things. The things you are good at are the ones you focus on; the tradeoff is that you will not become good at the things you ignore while you're focused on the things you're good at.
Savant syndrome appears to be the extreme end of this kind of tradeoff; a savant becomes incredibly good at something, apparently incredibly easily, because he ignores the things that we all generally spend some degree of focus on without even being consciously aware of it--things like language acquisition, say, or learning to function in a social environment.
The fact that cognitive scientists now talk about "savant syndrome" rather than "idiot savants" is that savants aren't idiots. They aren't cognitively impaired the way someone with, say, Down's syndrome is cognitively impaired. Rather, they are so incredibly hyperfocused that they never learn the skills that most of us learn without even beig aware that we're learning them.