I think from the client's viewpoint, a huge amount of whether or not they can tell a difference when you have gloves on, isn't due to the glove itself but the temperature. If the glove has warmed up to your body temperature by the time you touch the client's face, it's going to feel like being touched with bare hands. I have had plenty of facials where I didn't realize until afterwards that the esthetician had worn gloves throughout. It is only obvious if the glove is all cold and nasty.
I personally prefer not to wear gloves, except during extractions & peels. I have a sink in my room so I'm able to wash my hands whenever needed. I had a facial done on me with gloves and I could hear the crinkling noise they make throughout. It wasn't bad, I was still able to relax and enjoy it.