Improve integration of the http server introduced by the http-registry feature.
Now the same HTTP server is used for serving downloads, the index, and
the API.
This makes it easier to write tests that deal with authentication and
http registries.
Mark .cargo/git and .cargo/registry as cache dirs
Fixes#10457
### What does this PR try to resolve?
As we previously discussed in #10457 this marks `~/.cargo/git` and `~/.cargo/registry` as not to be included in backups, and not subject to content indexing. These directories can be fairly large and frequently changed, and should only contain content that can be re-downloaded if necessary.
### How should we test and review this PR?
I did two manual tests:
1. Using the binary built from this tree, run `cargo update` in a source tree that has a git dependency, and observe that afterwards, there is a `CACHEDIR.TAG` in `~/.cargo/git`.
2. Similarly, run `cargo update` and observe that there's a `CACHEDIR.TAG` in `~/.cargo/registry`.
(I ran this on Linux. This code should also trigger OS-specific behavior on macOS and Windows that's the same as is currently applied to `target/`.)
I added some test assertions.
- Add `ProcessBuilder::output` and ProcessBuilder::status`, which are
unopinionated version of `exec_*` (won't error out when exitcode > 0)
- Add `ProcessBuilder::retry_with_argfile` to enable trying with argfile
when hitting the "command line too big" error.
Upgrade Cargo to the 2021 edition
While I was poking around in the manifests I also went ahead and removed the `authors` entries since at least the one for Cargo itself is quite outdated and Cargo otherwise doesn't use these any more.
Otherwise symbolic links may also accidentally be resolved which may
lead to unexpected results in the case of 'coreutils', a binary
that depends on the executable name being a symbolic link.
This means a path like /bin/echo being canonicalized to /bin/coreutils
will loose all information about the desired functionality.
Test failures will occur if 'echo' is resolved that way and it's
not trivial to find the cause of it in the provided error messages.
When searching for newlines in a process output keep track which part of
buffer was already examined to avoid processing the same data again and
again.
