When backporting patches to LTS branches, we often run into CI failures due to
changes in rust. Newer rust versions add more lints, which break CI. We really
don't want to also have to backport patches that fix CI, so instead, LTS branches
should pin the stable rust version in CI (e.g. #4434).
This PR restructures the CI config files to make it a bit easier to set a specific rust
version in CI.
