Credentials are always loaded, even if these are not used. If
access to confidential files such as credentials is not given,
`cargo build` fails despite not using credentials.
Fixes#7624.
A test when argument is negative is added. In addition,
`default_cargo_config_jobs` and `good_cargo_config_jobs` is moved from
`testsuite/bad_config.rs` to `testsuite/build.rs` because these tests
are not for `bad config`.
Stabilize cache-messages
This stabilizes the -Zcache-messages feature, making it always enabled.
## What is stabilized?
This feature is intended to redisplay previous warnings on a "fresh" build instead of displaying no output.
Users have occasionally indicated frustration when they know there are warnings, but no output is displayed when the build is fresh. This also improves the interaction between `cargo check` and `cargo clippy-preview`. This also simplifies the code, and opens more possibilities for `rustc` to send side-channel messages to Cargo.
Cargo will now use JSON output from `rustc` and `rustdoc` 100% of the time (`rustdoc --test` does not use JSON). Previously Cargo would only use JSON for pipelined crates.
Cargo will save the JSON output into a file named `output` in the `.fingerprint` directory. This file is only created when the compiler outputs a diagnostic message.
If a crate is being recompiled, and Cargo considers it to be "fresh", it will replay the output file to the console.
## Notable changes in this PR
- Fixed a bug where replays were erroneously including pipeline rmeta artifact json messages.
- clippy-preview is now included in the metadata hash, to force its artifacts to be separate from `cargo check`.
- clippy-preview is no longer force-enabled, under the assumption that caching and fingerprinting is accurate, and the cached messages will be replayed.
- clippy-preview's arguments are included in the fingerprint now that it is not force-enabled.
- Rustdoc colors and short messages were fixed when pipelining was stabilized, so updated tests.
Closes#6986Closes#6848Closes#6664Closes#2458
## Concerns
The only notable issue with this is that switching between short and long human messages only replays the format from the first invocation. That is, if you do `cargo build` and it generates warnings, then running again with `--message-format=short` will still show the full length human messages. I'm personally fine with that behavior, even though it is not ideal. I think this feature overall improves the situation (where before *no* output was displayed). Being able to re-render between short/long is a very difficult problem, and unlikely to be fixable in the foreseeable future.
There was some concern expressed about being able to disable this. I think that would only be necessary if a severe bug is discovered. I do not feel that this change is risky enough to warrant a configurable option. If it does cause a problem, it can be quickly reverted with a one-line change to set `OutputOptions::cache_cell` to `None`. Since pipelining has been using JSON output for a while now without complaints, I feel pretty confident in it.
First reported in rust-lang/rust#65014 it looks like our error message
on cyclic dependencies may be confusing at times. It looks like this is
an issue because there are multiple paths through a graph for a
dependency, so using the generic `path_to_top` function isn't producing
the most useful path for this purpose.
We're already walking the graph though, so this commit adds an extra
parameter which collects the list of packages we've visited so far to
produce a hopefully always-accurate error message showing the chain of
dependencies end-to-end for what depends on what.
We were sporadically but persistently seeing errors like
failed to link or copy `.../target/debugs/deps/bin-264030cd6c8a02be.dSYM` to `.../target/debug/bin.dSYM`
Caused by:
the source path is not an existing regular file
while running `cargo build`. Once the error occurs once, `cargo build`
will fail forever with the same error until `target/debug/bin.dSYM` is
manually unlinked.
After some investigation, I've determined that this situation arises
when the target of `bin.dSYM` goes missing. For example, if bin.dSYM is
pointing at `deps/bin-86908f0fa7f1440e.dSYM`, and
`deps/bin-86908f0fa7f1440e.dSYM` does not exist, then this error will
occur. I'm still not clear on why the underlying dSYM bundle
sporadically goes missing--perhaps it's the result of pressing Ctrl-C at
the wrong moment?--but Cargo should at least be able to handle this
situation better.
It turns out that Cargo was getting confused by the broken symlink. When
it goes to install the new `target/debug/bin.dSYM` link, it will remove
the existing `target/debug/bin.dSYM` file, if it exists. Unfortunately,
Cargo was checking whether the *target* of that symlink existed (e.g.,
`deps/bin-86908f0fa7f1440e.dSYM`, which in the buggy case would not
exist), rather than the symlink itself, deciding that there was no
existing symlink to remove, and crashing with EEXIST when trying to
install the new symlink.
This commit adjusts the existence check to evaluate whether the symlink
itself exists, rather than its target.
Note that while the symptoms are the same as #4671, the root cause is
unrelated.
Cargo has of #7143 enabled pipelined compilation by default which
affects how the compiler is invoked, especially with respect to JSON
messages. This, in some testing, has proven to cause quite a few issues
with rustbuild's current integration with Cargo. This commit is aimed at
adding features to Cargo to solve this issue.
This commit adds the ability to customize the stream of JSON messages
coming from Cargo. The new feature for Cargo is that it now also mirrors
rustc in how you can configure the JSON stream. Multiple
`--message-format` arguments are now supported and the value specified
is a comma-separated list of directives. In addition to the existing
`human`, `short`, and `json` directives these new directives have been
added:
* `json-render-diagnostics` - instructs Cargo to render rustc
diagnostics and only print out JSON messages for artifacts and Cargo
things.
* `json-diagnostic-short` - indicates that the `rendered` field of rustc
diagnostics should use the "short" rendering.
* `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc
diagnostics should embed ansi color codes.
The first option here, `json-render-diagnostics`, will be used by
rustbuild unconditionally. Additionally `json-diagnostic-short` will be
conditionally used based on the input to rustbuild itself.
This should be enough for external tools to customize how Cargo is
invoked and how all kinds of JSON diagnostics get printed, and it's
thought that we can relatively easily tweak this as necessary to extend
it and such.
Some tools can support rust doctests (e.g. highlighting
or launching). So it should be possible to find out if
doctests are enabled for a target or not. This commit
adds `doctest` field to `cargo metadata` output.
Implement the Cargo half of pipelined compilation (take 2)
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
Closes#6660
This commit fixes an issue when pipelining mode is used in handling
recompilations. Previously a sequence of compilations could look like:
* Crate A starts to build
* Crate A produces metadata
* Crate B, which depends on A, starts
* Crate B finishes
* Crate A finishes
In this case the mtime for B is before that of A, which fooled Cargo
into thinking that B needed to be recompiled. In this case, however, B
doesn't actually need to be recompiled because it only depends on the
metadata of A, not the final artifacts.
This unfortunately resulted in some duplication in a few places, but not
really much moreso than already exists between fingerprinting and compilation.
This commit updates Cargo to process JSON directives emitted by rustc
when we're pipelining compilation. In this mode Cargo will attempt to
start subsequent compilations of crates as soon as possible, fully
completing the features of pipelined compilations in Cargo!
This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.
Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
`DependencyQueue` structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.
A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the `DependencyQueue`.
Once that's all in place it's just a few pieces here and there to
identify compilations that *can* be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.
