Update assertions in LTO calculations
Turns out a case I thought didn't happen can indeed happen. Units may
depend on other units which are LTO-able because integration tests can
depend on binaries. Handle that case internally and remove a few panics.
Closes#8223
Turns out a case I thought didn't happen can indeed happen. Units may
depend on other units which are LTO-able because integration tests can
depend on binaries. Handle that case internally and remove a few panics.
Closes#8223
Rework rustc output file tracking.
This is some cleanup around how Cargo computes the outputs from rustc, and changes how `cargo clean -p` works. This also includes some minor observable differences detailed below.
**clean -p changes**
Previously `cargo clean -p` would build the unit graph and attempt to guess what all the filename hashes are. This has several drawbacks. It incorrectly guesses the hashes in many cases (such as different features). It also tends to cause bugs because it passes every permutation of every setting into Cargo's internals, which may not be prepared to accept unsupported combinations like "test a build-script".
This new implementation uses a strategy of querying rustc for the filename prefix/suffix, and then deletes the files using globs.
`cargo clean -p` should now be more complete in deleting a package's artifacts, for example:
- Now deletes incremental files.
- Deletes dep-info files (both rustc and cargo).
- Handles changes in profiles, features, (anything in the hash).
- Covers a few more files (for example, dSYM in some cases, etc.). Should delete almost all files for most targets.
**Internal changes**
There are a bunch of internal changes to make Cargo do a better job of tracking the outputs from rustc, and to make the code easier to understand:
- The list of output files is now solely computed in `TargetInfo`. The files which are uplifted are solely computed in `CompilationFiles::uplift_to`. Previously both responsibilities were awkwardly spread in both locations.
- Whether or not a file should have a hyphen or underscore is now determined in one place (`FileType::should_replace_hyphens`).
- Added `CrateType` to replace `LibKind`, to avoid usage of strings, and to use the same structure for all of the target kinds.
- Added `FileFlavor::Rmeta` to be more explicit as to which output files are ".rmeta". (Previously the `Linkable{rmeta}` flag was specific for pipelining, and rmeta was `false` for things like `cargo check`, which was a bit hard to deal with.)
- Removed hyphen/underscore renaming in `rustc`. This was mostly unused, because it did not consider hashes in the filename, so it only applied to binaries without hashes, which is essentially just wasm32 and msvc. This hyphen/underscore translation still happens during "uplift".
- Changed it so that `Metadata` is always computed for every unit. The logic of whether or not something should use it is moved to `should_use_metadata`. I didn't realize that multiple units were sharing the same fingerprint directory (when they don't have a hash), which caused some bugs (like bad output caching).
**Behavioral changes**
Cargo now does more complete tracking of the files generated by rustc (and the linker). This means that some files will now be uplifted that previously weren't. It also means they will show up in the artifact JSON notifications. The newly tracked files are:
- `.exp` export files for Windows MSVC dynamic libraries. I don't know if these are actually useful (nobody has asked for them AFAIK). Presumably the linker creates them for some reason. Note that lld *doesn't* generate these files, this is only link.exe.
- Proc-macros on Windows track import/export files.
- All targets (like tests, etc.) that generate separate debug files (pdb/dSYM) are tracked.
- Added `.map` files for wasm32-unknown-emscripten.
Some other subtle changes:
- A binary example with a hyphen on Windows MSVC will now show up as `examples/foo_bar.exe` and `examples/foo-bar.exe`. Previously Cargo would just rename it to contain the hyphen. This is a consequence of simplifying the code, and I was reluctant to add a special case for this very narrow situation.
- Example libs now follow the same rules for hyphen/underscore translation as normal libs (they will now use underscores).
- Fingerprint changes:
- Fingerprint files no longer have a hash in them. Their parent directory already contained the hash.
- The fingerprint filename now uses hyphens instead of converting to underscores.
- The fingerprint directory is now separated even if the unit doesn't use Metadata, to fix a caching bug.
- macOS: dSYM is uplifted for all dynamic libraries (dylib/cdylib/proc-macro) and for build-script-build (in case someone wants to debug a build script?).
**Notes**
- I suspect that the implementation of `clean -p` may change again in the future. If and when Cargo implements some kind of on-disk database that tracks artifacts (for the purpose of garbage collection), then `cargo clean -p` can be rewritten to use that mechanism if appropriate.
- The `build_script_build` incremental directory isn't deleted because Cargo doesn't know which one belongs to which package. I'm uncertain if that's reasonably fixable. The only option I've thought of is to place each package's incremental output in a separate directory.
- Should Cargo use `globset` to handle non-utf-8 filenames? I suspect that Cargo's support for non-utf-8 names is pretty poor, so I'm uncertain how important that is.
Closes#8149Closes#6937Closes#5788Closes#5375Closes#3530
When a unit does not have Metadata, the computation of fingerprints
depends on reusing the same fingerprint file to detect if the
output needs to be rebuilt. The previous change that put each unit's
fingerprint into a separate directory was wrong, and broke change
detection in this case (for example, executables on MSVC).
Instead, use a different approach to deal with compiler output caching
by using the same naming convention as the fingerprint file itself.
features: allow activated_features_unverified to communicate not-present
Hi there! :)
I'm currently writing [`cargo-guppy`](https://github.com/facebookincubator/cargo-guppy), a toolkit for analyzing Rust dependency graphs. As part of that I'm writing some tests to compare guppy to `cargo` (see https://github.com/facebookincubator/cargo-guppy/pull/126), to ensure that it produces the same results for the subset of analyses `cargo` can do.
While writing tests I found it useful to distinguish between packages that weren't present at all and packages that were activated but with no features. This commit adds that functionality to the `_unverified` method.
(BTW, of possible interest to @ehuss, I've also found some interesting behavior differences between the v1 and v2 resolvers. Will file issues for them soon!)
Don't force rustc to do codegen for LTO builds
This commit updates Cargo's implementation of LTO builds to do less work
and hopefully be faster when doing a cold build. Additionaly this should
save space on disk! The general idea is that the compiler does not need
object files if it's only going to perform LTO with some artifacts. In
this case all rustc needs to do is load bitcode from dependencies. This
means that if you're doing an LTO build generating object code for
intermediate dependencies is just wasted time!
Here Cargo is updated with more intrusive knowledge about LTO. Cargo
will now analyze the dependency graph to figure out which crates are
being compiled with LTO, and then it will figure out which dependencies
only need to have bitcode in them. Pure-bitcode artifacts are emitted
with the `-Clinker-plugin-lto` flag. Some artifacts are still used in
multiple scenarios (such as those shared between build scripts and final
artifacts), so those are not compiled with `-Clinker-plugin-lto` since
the linker is not guaranteed to know how to perform LTO.
This functionality was recently implemented in rust-lang/rust#71528
where rustc is now capable of reading bitcode from `-Clinker-plugin-lto`
rlibs. Previously rustc would only read its own format of bitcode, but
this has now been extended! This support is now on nightly, hence this
PR.
This commit updates Cargo's implementation of LTO builds to do less work
and hopefully be faster when doing a cold build. Additionaly this should
save space on disk! The general idea is that the compiler does not need
object files if it's only going to perform LTO with some artifacts. In
this case all rustc needs to do is load bitcode from dependencies. This
means that if you're doing an LTO build generating object code for
intermediate dependencies is just wasted time!
Here Cargo is updated with more intrusive knowledge about LTO. Cargo
will now analyze the dependency graph to figure out which crates are
being compiled with LTO, and then it will figure out which dependencies
only need to have bitcode in them. Pure-bitcode artifacts are emitted
with the `-Clinker-plugin-lto` flag. Some artifacts are still used in
multiple scenarios (such as those shared between build scripts and final
artifacts), so those are not compiled with `-Clinker-plugin-lto` since
the linker is not guaranteed to know how to perform LTO.
This functionality was recently implemented in rust-lang/rust#71528
where rustc is now capable of reading bitcode from `-Clinker-plugin-lto`
rlibs. Previously rustc would only read its own format of bitcode, but
this has now been extended! This support is now on nightly, hence this
PR.
Hint git-fetch-with-cli on git errors
Our team has struggled with making Cargo git dependencies work in CI, until learning about this setting.
Cargo doesn't have a dedicated system for error hints like `rustc`, so I've stuffed the hint into the error message.
Rename bitcode-in-rlib flag to embed-bitcode
This flag changed names in nightly, so let's rename it here in Cargo to
get our CI passing and enable the same wins for avoiding bitcode.
I'm currently writing [`cargo-guppy`](https://github.com/facebookincubator/cargo-guppy), a toolkit for analyzing Rust dependency graphs. As part of that I'm writing some tests to compare guppy to `cargo`, to ensure that it produces the same results for the subset of analyses `cargo` can do.
While writing tests I found it useful to distinguish between packages that weren't present at all and packages that were activated but with no features. This commit adds that functionality to the `_unverified` method.
Remove unnecessary loop in `maybe_spurious`
The `anyhow` library's error already does a recursive check when we use
`Error::downcast_ref`, so there's no need to explicitly do this on the
`chain` of errors.
The `anyhow` library's error already does a recursive check when we use
`Error::downcast_ref`, so there's no need to explicitly do this on the
`chain` of errors.
Fix error with git repo discovery and symlinks.
There are some cases where Cargo would generate an error when attempting to discover if a package is inside a git repo when the git repo has a symlink somewhere in its ancestor paths. One way this manifests is with `cargo install --git ...` where the given repo has a `build.rs` script. Another scenario is `cargo build --manifest-path somelink/Cargo.toml` where `somelink` is a symlink to the real thing.
The issue is that libgit2 is normalizing paths and removing symlinks, but the path Cargo uses is the path with symlinks. This was introduced in #8095.
The solution is to try to canonicalize both paths when trying to get a repo-relative path. If that fails for whatever reason, it shouldn't generate an error since this is just a "best effort" attempt to use git to list package files.
Fixes#8183
Allow failure when setting file mtime.
Some filesystems do not allow changing file mtimes (most recent example is a Windows mount in WSL). Since these operations are not critical to Cargo's operation, errors should probably just be ignored. (Currently they are used for mid-build modification protection, and the unstable mtime-on-use).
Fixes#8184
Support multiple `--target` flags on the CLI
This commit refactors the internals of Cargo to no longer have a
singular `--target` flag (and singular `requested_target` kind throught)
but to instead have a list. The semantics of multiple `--target` flags
is to build the selected targets for each of the input `--target` flag
inputs.
For now this is gated behind `-Zmultitarget` as an unstable features,
since I'm not entirely sure this is the interface we want. In general
it'd be great if we had a way to simply specify `Unit` structures of
what to build on the CLI, but we're in general very far away from that,
so I figured that this is probably sufficient at least for testing for
now.
cc #8156
This commit refactors the internals of Cargo to no longer have a
singular `--target` flag (and singular `requested_target` kind throught)
but to instead have a list. The semantics of multiple `--target` flags
is to build the selected targets for each of the input `--target` flag
inputs.
For now this is gated behind `-Zmultitarget` as an unstable features,
since I'm not entirely sure this is the interface we want. In general
it'd be great if we had a way to simply specify `Unit` structures of
what to build on the CLI, but we're in general very far away from that,
so I figured that this is probably sufficient at least for testing for
now.
cc #8156
build-std: Don't treat std like a "local" package.
This changes it so that build-std will not treat the std crates like a "local" package. This has the following changes:
- std does not build with incremental. This generally shouldn't be needed, and incremental has various bugs with std crates.
- Cargo's dep-info fingerprint tracking will not track the std crate sources, these are tracked via other means.
- Cargo's `.d` dep-info file does not include std crate sources.
- Lints are capped to "allow" for std crates, and warnings are not shown by default.
Closes https://github.com/rust-lang/wg-cargo-std-aware/issues/44
Closes https://github.com/rust-lang/wg-cargo-std-aware/issues/55
Allow `cargo package --list` even for things that don't package.
`cargo package --list` was changed in #7905 to generate `Cargo.lock` earlier. If there is a problem, then it would fail where previously it would succeed. This changes it so that file generation is deferred until after `--list`.
This also changes it so that the "dependencies must have a version" check is deferred until after `--list` as well.
Closes#8151
Use associated constants directly on primitive types instead of modules
This PR is in no way critical. It's more of a code cleanup. It comes as a result of me making https://github.com/rust-lang/rust/pull/70857 and search-and-replacing all usage of the soft-deprecated ways of reaching primitive type constants.
It makes the code slightly shorter, that's basically it. And showcases the recommended way of reaching these consts on new code :)
Clear `RUSTDOCFLAGS` before running tests
It turns out that this test will fail if `RUSTDOCFLAGS` is set in the test runner. This was found in
https://github.com/rust-lang/rust/pull/71458#issuecomment-619101858 when that PR attempted to set the proper `RUSTDOCFLAGS` to enable `./x.py doc --stage 0 src/libstd`. Run the test with no `RUSTDOCFLAGS` so that the config file always gets used.
Fix warning for `resolve` mismatch in workspace.
Using the new `resolve` field would cause a warning in a workspace that the member's resolve setting will be ignored, even if it is not set.
`warning: resolver for the non root package will be ignored, specify resolver at the workspace root:`
The code should only check the package member if the member's value is set.
