Although it's not specifically drawing for me, I have found that if I have some sort of stimulation other than just sitting and listening, it dramatically helps my thoughts. Sometimes I'll play games simple games on my phone while listening to a speaker and I come away with a greater recollection than times where I just sat and listened.

You're right though, it does come across as being rude/distracted.

I've been doing this for a few years now and only a few times have people commented and it's never been a big deal even then. It's a nice way to create associations with what's being said and it actually helps keep me awake and attentive when there's a lull in whatever is going on.