Is it possible to actually not care what others think?

It really used to irritate me. I would say, "I'm traveling to blah." And they would ask, "With who?" My response was always, "By myself!"...