Avoid a collection and iteration on empty passes
Just some mini optimization I saw in the wild. This way, we avoid a `collect` and `map` on an empty `passes`. Honestly, I don't even think this is big enough of a change to make a benchmark, but I'd still like to see results.
Based on [this book](https://nnethercote.github.io/perf-book/iterators.html#collect-and-extend)
8e1eaddd27