Its only those you trust with the keys who can destroy your soul. Everyone who broke your heart you gave that power to.
edited immediately after posting... to change it to statement rather than question. The "why" is really quite irrelevant.... we just do. We all will know the feeling of a broken heart or betrayed trust.