You seem perfectly fine at conversation to me. Unexpressive, perhaps, but there are other ways to read people. I think they simply consider you strange and don't care to put in the effort to make sense of someone who isn't what they expect. Most people tend toward that sort of foolish narrow mindedness.