Great question, as someone who just recently switched from agency to in-house, hopefully I can provide some solid insights.

In any case, the grass is always going to seem a little greener on the other side. There is validity in having some variety in projects while at an agency, but it comes a a cost. Even the greatest agencies are going to have projects come through that aren't particularly glamorous and most definitely won't be finding a spot in your portfolio. For myself, the major downside of being at an agency is I wasn't producing work that was truly iterated upon. It was definitely shouted that we "always iterate" and "it's never done", but the reality is that projects at an agency are reliant on a approved budget from the client.

It really depends on where you are at in your career. I wouldn't suggest to a designer with 10 years of industry experience to explore an agency opportunity. For someone just starting out in their design career (1-3 yrs exp.), as others have said, it can be great for getting some clarity on what you really have interest in.

Lastly, many tech companies are drawn to candidates with agency experience because of the perceived ability to juggle multiple projects at once. So if you feel as though you could use the agency experience to then catapult you into a better in-house gig then it might be worth it.