I've been on both sides. I'm currently on the product side as a co-founder and designer.
Before, I worked for other people's dreams, changing the world how they wanted. Now, I work for my own dreams, making the things I think will change the world. All I thought about agency-side was how I could help clients think more like us, with design at the core. Now, I have a team that values design right from the start. No need to convince anyone, they just get it.
Maybe product-side designers don't look down on agency designers. Maybe we just wonder why they'd choose to fight clients when the other side is so rewarding.