He couldn't speak because everyone would gang up against him. He told the truth and they threw his past on his face. Trying to twist it so he's the bad one, while they are doing way worse.
He stayed away didn't seek or need their approval. It bothered them, he was his own person. he didn't need to go through them to achieve his goals. He found a way an escape