Handle LTO with an rlib/cdylib crate type
In the case that LTO is happening but we're also generating a
cdylib/rlib simultatneously that means that the final artifact will use
the rlib but the cdylib still needs to be produced. To get this to work
the cdylib's dependency tree needs to be compiled with embedded bitcode.
The cdylib itself will be linked with the linker because we can't LTO an
rlib+cdylib compilation, but the final executable will need to load
bitcode from all its deps.
This is meant to address rust-lang/rust#72268
In the case that LTO is happening but we're also generating a
cdylib/rlib simultatneously that means that the final artifact will use
the rlib but the cdylib still needs to be produced. To get this to work
the cdylib's dependency tree needs to be compiled with embedded bitcode.
The cdylib itself will be linked with the linker because we can't LTO an
rlib+cdylib compilation, but the final executable will need to load
bitcode from all its deps.
This is meant to address rust-lang/rust#72268
Gracefully handle errors during a build.
If there are certain errors like EPIPE during a build, Cargo runs the risk of hanging if the compiler emits too many messages. This happens because Cargo now uses a bounded queue for compiler messages. However, if the main loop exits while the threads are running, there is nothing to drain the queue, and thus the threads will block indefinitely.
The solution here is to be extra careful of how errors are handled in the main loop. All errors are now treated roughly the same (report the error, allow the queue to continue to drain).
I've also tweaked things so the *first* error is reported, not the last.
Closes#8245
If the readme field is not specified, assume a default of
"README.md" (temporarily). Further, if the readme field is set to
"false", then this defaulting behavior is suppressed.
Fix `cargo update` with unused patch.
If you end up with an unused patch in `Cargo.lock`, then you try to update the patch to a version that does not match the original unused patch, and run `cargo update`, Cargo refuses to update it with the error "patch for `bar` in `https://github.com/rust-lang/crates.io-index` did not resolve to any crates". At this point, Cargo seems to be permanently stuck with no way to update it (unless you manually edit `Cargo.lock`).
The solution here is to add the unused patches to the "to_avoid" list, so that `cargo update` is allowed to update them.
I am uncertain if this is the best way to fix it, but seems to match my intuition of how `cargo update` is implemented.
Rephrased error message for disallowed sections in virtual workspace
I changed the error message from `virtual manifests do not specify [target]` to `This virtual manifest specifies a [target] section, which is not allowed` because the old one confused me.
I encountered it while hacking on Rustc in CLion. I made a change to the `Cargo.toml` I had copied from StackOverflow and after the next restart code analysis and go-to-definition stopped working. After some fiddling, I tracked that down to my change in `Cargo.toml`. It took me a while because, like I said, the error message from CLion didn't make sense to me.
```
19:13 Cargo project update failed:
Execution failed (exit code 101).
/home/julian/.cargo/bin/cargo metadata --verbose --format-version 1 --all-features
stdout : error: failed to parse manifest at `/home/julian/Dokumente/Rust/Compiler/rust/Cargo.toml`
Caused by:
virtual manifests do not specify [target]
stderr :
```
I hope this change avoids future confusion :)
Ignore broken console output in some situations.
If stdout or stderr is closed while Cargo is running, Cargo would panic in some situations (usually with EPIPE). For example, `cargo install --list | grep -q cargo-fuzz`, where `grep` will close stdout once it gets a match. This changes it so that Cargo will ignore output errors in most situations. Failure to output a regular build message still results in an error, which follows the behavior of some tools like `make`.
All output, including stdout, now goes through `Shell`.
Closes#5234
Expand error message to explain that a string was found
With `opt-level = "3"` this previously said:
must be an integer, `z`, or `s`, but found: 3 for ...
The error message doesn't make that super clear.
This should now be a bit more clear.
Fixes#8234
---
We could even include a bit more saying that `"3"` should become 3 (either unconditionally or after trying to parse `"3"` into an integer?
cc @steveklabnik
With `opt-level = "3"` this previously said:
must be an integer, `z`, or `s`, but found: 3 for ...
The error message doesn't make that super clear.
This should now be a bit more clear.
Fixes#8234
Add context to some fs errors.
This adds some extra context to most fs operations that indicates some more detail (particularly the path). It can be frustrating when cargo says something generic like "Access is denied." without printing a path or information about what it is doing.
Addresses #8211, where it adds extra context to the message.
Move SipHasher to an isolated module.
This allows removing the blanket `#![allow(deprecated)]` sprinkled whenever it is used.
We could alternatively use the [siphasher](https://crates.io/crates/siphasher) crate, but I don't think it is necessary at this time.
Update assertions in LTO calculations
Turns out a case I thought didn't happen can indeed happen. Units may
depend on other units which are LTO-able because integration tests can
depend on binaries. Handle that case internally and remove a few panics.
Closes#8223
Turns out a case I thought didn't happen can indeed happen. Units may
depend on other units which are LTO-able because integration tests can
depend on binaries. Handle that case internally and remove a few panics.
Closes#8223
Rework rustc output file tracking.
This is some cleanup around how Cargo computes the outputs from rustc, and changes how `cargo clean -p` works. This also includes some minor observable differences detailed below.
**clean -p changes**
Previously `cargo clean -p` would build the unit graph and attempt to guess what all the filename hashes are. This has several drawbacks. It incorrectly guesses the hashes in many cases (such as different features). It also tends to cause bugs because it passes every permutation of every setting into Cargo's internals, which may not be prepared to accept unsupported combinations like "test a build-script".
This new implementation uses a strategy of querying rustc for the filename prefix/suffix, and then deletes the files using globs.
`cargo clean -p` should now be more complete in deleting a package's artifacts, for example:
- Now deletes incremental files.
- Deletes dep-info files (both rustc and cargo).
- Handles changes in profiles, features, (anything in the hash).
- Covers a few more files (for example, dSYM in some cases, etc.). Should delete almost all files for most targets.
**Internal changes**
There are a bunch of internal changes to make Cargo do a better job of tracking the outputs from rustc, and to make the code easier to understand:
- The list of output files is now solely computed in `TargetInfo`. The files which are uplifted are solely computed in `CompilationFiles::uplift_to`. Previously both responsibilities were awkwardly spread in both locations.
- Whether or not a file should have a hyphen or underscore is now determined in one place (`FileType::should_replace_hyphens`).
- Added `CrateType` to replace `LibKind`, to avoid usage of strings, and to use the same structure for all of the target kinds.
- Added `FileFlavor::Rmeta` to be more explicit as to which output files are ".rmeta". (Previously the `Linkable{rmeta}` flag was specific for pipelining, and rmeta was `false` for things like `cargo check`, which was a bit hard to deal with.)
- Removed hyphen/underscore renaming in `rustc`. This was mostly unused, because it did not consider hashes in the filename, so it only applied to binaries without hashes, which is essentially just wasm32 and msvc. This hyphen/underscore translation still happens during "uplift".
- Changed it so that `Metadata` is always computed for every unit. The logic of whether or not something should use it is moved to `should_use_metadata`. I didn't realize that multiple units were sharing the same fingerprint directory (when they don't have a hash), which caused some bugs (like bad output caching).
**Behavioral changes**
Cargo now does more complete tracking of the files generated by rustc (and the linker). This means that some files will now be uplifted that previously weren't. It also means they will show up in the artifact JSON notifications. The newly tracked files are:
- `.exp` export files for Windows MSVC dynamic libraries. I don't know if these are actually useful (nobody has asked for them AFAIK). Presumably the linker creates them for some reason. Note that lld *doesn't* generate these files, this is only link.exe.
- Proc-macros on Windows track import/export files.
- All targets (like tests, etc.) that generate separate debug files (pdb/dSYM) are tracked.
- Added `.map` files for wasm32-unknown-emscripten.
Some other subtle changes:
- A binary example with a hyphen on Windows MSVC will now show up as `examples/foo_bar.exe` and `examples/foo-bar.exe`. Previously Cargo would just rename it to contain the hyphen. This is a consequence of simplifying the code, and I was reluctant to add a special case for this very narrow situation.
- Example libs now follow the same rules for hyphen/underscore translation as normal libs (they will now use underscores).
- Fingerprint changes:
- Fingerprint files no longer have a hash in them. Their parent directory already contained the hash.
- The fingerprint filename now uses hyphens instead of converting to underscores.
- The fingerprint directory is now separated even if the unit doesn't use Metadata, to fix a caching bug.
- macOS: dSYM is uplifted for all dynamic libraries (dylib/cdylib/proc-macro) and for build-script-build (in case someone wants to debug a build script?).
**Notes**
- I suspect that the implementation of `clean -p` may change again in the future. If and when Cargo implements some kind of on-disk database that tracks artifacts (for the purpose of garbage collection), then `cargo clean -p` can be rewritten to use that mechanism if appropriate.
- The `build_script_build` incremental directory isn't deleted because Cargo doesn't know which one belongs to which package. I'm uncertain if that's reasonably fixable. The only option I've thought of is to place each package's incremental output in a separate directory.
- Should Cargo use `globset` to handle non-utf-8 filenames? I suspect that Cargo's support for non-utf-8 names is pretty poor, so I'm uncertain how important that is.
Closes#8149Closes#6937Closes#5788Closes#5375Closes#3530
When a unit does not have Metadata, the computation of fingerprints
depends on reusing the same fingerprint file to detect if the
output needs to be rebuilt. The previous change that put each unit's
fingerprint into a separate directory was wrong, and broke change
detection in this case (for example, executables on MSVC).
Instead, use a different approach to deal with compiler output caching
by using the same naming convention as the fingerprint file itself.
