To the girls: does size really matter?

i was talking to my boyfreind and we were messin around and i asked him how big he was. well he seemed kind of embarresed which of course he would because he isn't one of those in your face guys, but when he finally told he said 6.5. i didn't think that was bad and he seemed kind of surprised that i didn't laugh. so my questions are, is that a bad thing that he isn't huge? and does a guys size honestly matter?