Ever since the inception of Cargo and the advent of incremental
compilation at the crate level via Cargo, Cargo has tracked whether it
needs to recompile something at a unit level in its "dependency queue"
which manages when items are ready for execution. Over time we've fixed
lots and lots of bugs related to incremental compilation, and perhaps
one of the most impactful realizations was that the model Cargo started
with fundamentally doesn't handle interrupting Cargo halfway through and
resuming the build later.
The previous model relied upon implicitly propagating "dirtiness" based
on whether the one of the dependencies of a build was rebuilt or not.
This information is not available, however, if Cargo is interrupted and
resumed (or performs a subset of steps and then later performs more).
We've fixed this in a number of places historically but the purpose of
this commit is to put a nail in this coffin once and for all.
Implicit propagation of whether a unit is fresh or dirty is no longer
present at all. Instead Cargo should always know, irrespective of it's
in-memory state, whether a unit needs to be recompiled or not. This
commit actually turns up a few bugs in the test suite, so later commits
will be targeted at fixing this.
Note that this required a good deal of work on the `fingerprint` module
to fix some longstanding bugs (like #6780) and some serious hoops had to
be jumped through for others (like #6779). While these were fallout from
this change they weren't necessarily the primary motivation, but rather
to help make `fingerprints` a bit more straightforward in what's an
already confusing system!
Closes#6780
Put mtime-on-use behind a feature flag.
This places #6477 behind the `-Z mtime-on-use` feature flag.
The change to update the mtime each time a crate is used has caused a performance regression on the rust playground (rust-lang/rust#57774). It is using about 241 pre-built crates in a Docker container. Due to the copy-on-write nature of Docker, it can take a significant amount of time to update the timestamps (over 10 seconds on slower systems).
cc @Mark-Simulacrum
Rustflags in metadata
This is a small change that adds Rustflags to the `Metadata`. This means, for example, that if you do a plane build then build with "-C target-cpu=native", you will get 2 copies of the dependencies. Con, more space if you are changing Rustflags. Pro, not rebuilding all the dependencies when you switch back.
Suggested by <https://internals.rust-lang.org/t/idea-cargo-global-binary-cache/9002/31>
Fix fingerprint calculation for patched deps.
If you have A→B→C where B and C are in a registry, and you `[patch]` C, the fingerprint calculation wasn't working correctly when C changes. The following sequence illustrates the problem:
1. Do a build from scratch.
2. Touch a file in C.
3. Build again. Everything rebuilds as expected.
4. Build again. You would expect this to be all fresh, but it rebuilds A.
The problem is the hash-busting doesn't propagate up to parents from dependencies. Normal targets normally aren't a problem because they have a `LocalFingerprint::MtimeBased` style local value which always recomputes the hash. However, registry dependencies have a `Precalculated` style local value which never recomputes the hash.
The solution here is to always recompute the hash. This shouldn't be too expensive, and is only done when writing the fingerprint, which should only happen when the target is dirty. I'm not entirely certain why the caching logic was added in #4125.
Fixesrust-lang/rust#57142
This change ensures cargo will output file paths in the expected format
(C:\foo\... on Windows, /foo/... elsewhere). Previously it would output
file:// URLs instead.
To support this change, additional changes were made to the test suite
string processing such that [ROOT] is now replaced with the appropriate
file path root for the platform.
The CWD template was also updated to use [CWD] like other replacement
templates and to do the replacement on the expected value rather than
the actual value to avoid replacing things we don't expect with CWD.
.. with mutliple calls of:
fastmod --accept-all '\.cargo\("([^"]+)"\)\.arg\("([^"]+)"\)' '.cargo("${1} ${2}")' tests/testsuite/
until no changes are left.
Looks like cargo traverses the filesystem & fails if it runs into a
Cargo.toml that doesn't declare a target. I couldn't find a nice way to
re-engineer the test to avoid this issue. So I'll leave that as someone
else's exercise.
* Collapse the nested cargotest::support module into the cargotest
module (merge the mod.rs's)
* Rename the cargotest module to support
* Nest the top-level hamcrest module into support
Generally that means either switching "foo" and "bar" around (reversing
the arrow), or it means push "foo" to "bar" (and sometimes "bar" to
"baz", etc..) to free up "foo".
For trivia that leaves 80/1222 outliers, therefore 93.4% of test
project use the default. :)
By rewriting the tests, with rerast (https://github.com/google/rerast),
to use the newly introduced "at" method.
First I added the following temporary function to cargotest::support:
pub fn project_foo() -> ProjectBuilder {
project("foo")
}
Then I defined the following rewrite.rs:
use cargotest::support::{ project, project_foo };
fn rule1(a: &'static str) {
replace!(project("foo") => project_foo());
replace!(project(a) => project_foo().at(a));
}
Then I ran rerast:
cargo +nightly rerast --rules_file=rewrite.rs --force --targets tests --file tests/testsuite/main.rs
Finally I searched and replaced the references to project_foo with
argument-less project (a little awkardly on macOS with a git clean).
find tests -type f -exec sed -i -e 's/project_foo/project/g' {} +
git clean -d tests
There's a case where Cargo will recompile a project even if the fingerprint
looks like it's fresh, when some output files are missing. This was intended to
cover the case where an output file was deleted manually or otherwise messed
with. The check here was a bit too eager, however. It checked not only the
actual output destination of the compiler but *also* the location that we hard
link the output file up to.
Due to recent changes in #5460 we don't always create the hard links for path
dependencies in the top-level dir, and this meant that if the library were
compiled and then tested later on the test may recompile the original library by
accident.
The fix in this commit is to cease looking for the hardlink if it exists or not.
This way we only check for the presence of the output file itself and only
recompile if that file is missing. The reason for this is that we
unconditionally relink files into place whether it's fresh or not, so we'll
always recreate the hard link anyway if it's missing.
cc rust-lang/rust#51717
Fixesrust-lang/rust#50962.
My theory is that while copying the binary, another thread forked with the fd
open. The copy finishes and attempts to exec before the other child execs (and
closes the writeable fd).
I was able to easily repro this on linux. I ran some stress tests of this fix
locally on linux and on appveyor, and was unable to trigger it again.
This reverts commit db3328f8f7b96701faa62756a0d55ed71899d894.
We get spurious errors for thouse tests even with debug info disabled,
so let's reenable it back.
