"If I hear you right, you are saying that while downvoting solves the issue of filtering out cruft, it also sacrifices the cultivation of good vibes."

Yes, though I'd add, too, that downvoting doesn't necessarily work as described as it tends to aid in collective ignorance. If early downvoters are wrong about something, then those who are inclined to look to this social proof to decide what is good and bad information are likely to follow suit. Then a pattern is formed of a community deciding certain things are wrong and should be buried regardless of merit because, well, everyone else in the community has silently said as much. At the very least you find that communities with upvote/downvote trend toward reinforcing only a single paradigm and punishing all others. This isn't an exclusively bad thing, but if a community heads in the wrong direction it will be fully committed to that wrong direction and it will likely resist innovation of thought.

So ultimately this would require a cost/benefit analysis and some testing, I suppose. YouTube is a great example of a community that was utterly toxic with no voting system at all and improved by having upvote/downvote.