I think,
The most pain one is not who goes,
But the one who left behind.
Because,
The one who goes left the memories with the left one,
And the left one is keeping the memories with the one who goes.