I get that. I think it’s great to be in that position, and I’ve known many people who are able to compartmentalize very well, as you are. They are generally very good in leadership roles, because they aren’t easily offended.
Some people have a more difficult time coping though, and it does not always mean they doubt their own choices. I also think your perspective would be different if your faith in the Savior was shaken or destroyed. For a lot of folks, it’s not a question of how they’re being judged by friends or family, but that they never felt loved by a God or, if they thought they did, they now don’t know about this God or if he exists anymore. That type of loneliness then places all sorts of new emphasis on mortal relationships, which magnifies the smaller judgments and their effects in a way that might not have previously mattered. Who cares about what others say, so long as you are right with, and loved by, God? Until…what if there is no God…? Then it’s a whole different issue.
And it’s very difficult for people to feel and understand this if they’ve never deeply questioned God’s existence (i.e., if they’ve never believed much more strongly that there was no god than that there was one). And since it’s very difficult to put oneself in that position of fear and then empathy, I think the next best thing is just to accept and recognize that people do, indeed, feel this way, and it’s not always accurate to try to say you know why they feel this way or to imply it’s because they doubt their choices on some level. People are much more complex than that.