Prevent ABI changes affect EnzymeAD
This PR handles ABI changes for autodiff input arguments to improve Enzyme compatibility. Fundamentally this adjusts activities when a function argument is lowered as an `ScalarPair`, so there's no mismatch between diff activities and args. Also removes activities corresponding to ZSTs.
fixes: https://github.com/rust-lang/rust/issues/144025
r? `@ZuseZ4`
initial implementation of the darwin_objc unstable feature
Tracking issue: https://github.com/rust-lang/rust/issues/145496
This feature makes it possible to reference Objective-C classes and selectors using the same ABI used by native Objective-C on Apple/Darwin platforms. Without it, Rust code interacting with Objective-C must resort to loading classes and selectors using costly string-based lookups at runtime. With it, these references can be loaded efficiently at dynamic load time.
r? ```@tmandry```
try-job: `*apple*`
try-job: `x86_64-gnu-nopt`
rustc_codegen_llvm: Adjust RISC-V inline assembly's clobber list
Despite that the `fflags` register (representing floating point exception flags) is stated as a flag register [in the reference](https://doc.rust-lang.org/reference/inline-assembly.html#r-asm.rules.preserved-registers), it's not
in the default clobber list of the RISC-V inline assembly and it would be better to fix it.
tests/codegen-llvm: Make rust-abi-arch-specific-adjustment portable
This test currently only runs on RISC-V and loongarch hosts, but assumes that the host target is the -gnu target. By using minicore, we can run this test on all host targets, regardless of architecture, as long as the LLVM components are built.
This also fixes this test on musl hosts of these architectures (though I've only tested on loongarch64-unknown-linux-musl).
Despite that the `fflags` register (representing floating point
exception flags) is stated as a flag register in the reference, it's not
in the default clobber list of the RISC-V inline assembly and it would
be better to fix it.
document `core::ffi::VaArgSafe`
tracking issue: https://github.com/rust-lang/rust/issues/44930
A modification of https://github.com/rust-lang/rust/pull/146454, keeping just the documentation changes, but not unsealing the trait.
Although conceptually we'd want to unseal the trait, there are many edge cases to supporting arbitrary types. We'd need to exhaustively test that all targets/calling conventions support all types that rust might generate (or generate proper error messages for unsupported cases). At present, many of the `va_arg` implementations assume that the argument is a scalar, and has an alignment of at most 8. That is totally sufficient for an MVP (accepting all of the "standard" C types), but clearly does not cover all rust types.
This PR also adds some various other tests for edge cases of c-variadic:
- the `#[inline]` attribute in its various forms. At present, LLVM is unable to inline c-variadic functions, but the attribute should still be accepted. `#[rustc_force_inline]` already rejects c-variadic functions.
- naked functions should accept and work with a C variable argument list. In the future we'd like to allow more ABIs with naked functions (basically, any ABI for which we accept defining foreign c-variadic functions), but for now only `"C"` and `"C-unwind` are supported
- guaranteed tail calls: c-variadic functions cannot be tail-called. That was already rejected, but there was not test for it.
r? `@workingjubilee`
match clang's `va_arg` assembly on arm targets
tracking issue: https://github.com/rust-lang/rust/issues/44930
For this example
```rust
#![feature(c_variadic)]
#[unsafe(no_mangle)]
unsafe extern "C" fn variadic(a: f64, mut args: ...) -> f64 {
let b = args.arg::<f64>();
let c = args.arg::<f64>();
a + b + c
}
```
We currently generate (via llvm):
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d0, r0, r1
add r0, sp, #4
vldr d1, [sp, #4]
add r0, r0, #15
bic r0, r0, #7
vadd.f64 d0, d0, d1
add r1, r0, #8
str r1, [sp]
vldr d1, [r0]
vadd.f64 d0, d0, d1
vmov r0, r1, d0
add sp, sp, #12
bx lr
```
LLVM is not doing a good job. In fact, it's well-known that LLVM's implementation of `va_arg` is kind of bad, and we implement it ourselves (based on clang) for many targets already. For arm, our own `emit_ptr_va_arg` saves 3 instructions.
Next, it turns out it's important for LLVM to explicitly start and end the lifetime of the `va_list`. In https://github.com/rust-lang/rust/pull/146059 I already end the lifetime, but when looking at this again, I noticed that it is important to also start it, see https://godbolt.org/z/EGqvKTTsK: failing to explicitly start the lifetime uses an extra register.
So, the combination of `emit_ptr_va_arg` with starting/ending the lifetime makes rustc emit exactly the instructions that clang generates::
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d16, r0, r1
vldr d17, [sp, #4]
vadd.f64 d16, d16, d17
vldr d17, [sp, #12]
vadd.f64 d16, d16, d17
vmov r0, r1, d16
add sp, sp, #12
bx lr
```
The arguments to `emit_ptr_va_arg` are based on [the clang implementation](03dc2a41f3/clang/lib/CodeGen/Targets/ARM.cpp (L798-L844)).
r? ``@workingjubilee`` (I can re-roll if your queue is too full, but you do seem like the right person here)
try-job: armhf-gnu
Implement `#[rustc_align_static(N)]` on `static`s
Tracking issue: https://github.com/rust-lang/rust/issues/146177
```rust
#![feature(static_align)]
#[rustc_align_static(64)]
static SO_ALIGNED: u64 = 0;
```
We need a different attribute than `rustc_align` because unstable attributes are tied to their feature (we can't have two unstable features use the same unstable attribute). Otherwise this uses all of the same infrastructure as `#[rustc_align]`.
r? `@traviscross`
We need a different attribute than `rustc_align` because unstable attributes are
tied to their feature (we can't have two unstable features use the same
unstable attribute). Otherwise this uses all of the same infrastructure
as `#[rustc_align]`.
This test currently only runs on RISC-V and loongarch hosts, but assumes
that the host target is the -gnu target. By using minicore, we can
run this test on all host targets, regardless of architecture, as long
as the LLVM components are built.
This also fixes this test on musl hosts of these architectures.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
Ignore intrinsic calls in cross-crate-inlining cost model
I noticed in a side project that a function which just compares to `[u64; 2]` for equality is not cross-crate-inlinable. That was surprising to me because I didn't think that code contained a function call, but of course our array comparisons are lowered to an intrinsic. Intrinsic calls don't make a function no longer a leaf, so it makes sense to add this as an exception to the "only leaves" cross-crate-inline heuristic.
This is the useful compare link: https://perf.rust-lang.org/compare.html?start=7cb1a81145a739c4fd858abe3c624ce8e6e5f9cd&end=c3f0a64dbf9fba4722dacf8e39d2fe00069c995e&stat=instructions%3Au because it disables CGU merging in both commits, so effects that cause changes in the sysroot to perturb partitioning downstream are excluded. Perturbations to what is and isn't cross-crate-inlinable in the sysroot has chaotic effects on what items are in which CGUs after merging. It looks like before this PR by sheer luck some of the CGUs dirtied by the patch in eza incr-unchanged happened to be merged together, and with this PR they are not.
The perf runs on this PR point to a nice runtime performance improvement.
Add amdgpu test for addrspacecasting global vars and the gpu-kernel calling convention
Add two tests that can now be added, as the amdgpu is merged.
- Global variables are casted to the default address space since rust-lang/rust#135026
- gpu-kernel calling convention, translatos to amdgpu_kernel rust-lang/rust#135047
Tracking issue: rust-lang/rust#135024
Fix LoongArch C function ABI when passing/returning structs containing floats
Similar to RISC-V, LoongArch passes structs containing only one or two floats (or a float–integer pair) in registers, as long as each element fits into a single corresponding register. Before this PR, Rust did not check the actual offset of the second float or integer; instead, it assumed the standard offset based on the default alignment. However, since the offset can be affected by `#[repr(align(N))]` and `#[repr(packed)]`, this led to miscompilations (see rust-lang/rust#145692). This PR fixes the issue by explicitly specifying the offset for the remainder of the cast.
Sanitizers target modificators
Depends on bool flag fix: https://github.com/rust-lang/rust/pull/138483.
Some sanitizers need to be target modifiers, and some do not. For now, we should mark all sanitizers as target modifiers except for these: AddressSanitizer, LeakSanitizer
For kCFI, the helper flag -Zsanitizer-cfi-normalize-integers should also be a target modifier.
Many test errors was with sanizer flags inconsistent with std deps. Tests are fixed with `-C unsafe-allow-abi-mismatch`.
stabilize c-style varargs for sysv64, win64, efiapi, aapcs
This has been split up so the PR now only contains the extended_varargs_abi_support stabilization; "system" has been moved to https://github.com/rust-lang/rust/pull/145954.
**Previous (combined) PR description:**
This stabilizes extern block declarations of variadic functions with the system, sysv64, win64, efiapi, aapcs ABIs. This corresponds to the extended_varargs_abi_support and extern_system_varargs feature gates.
The feature gates were split up since it seemed like there might be further discussion needed for what exactly "system" ABI variadic functions should do, but a [consensus](https://github.com/rust-lang/rust/issues/136946#issuecomment-2967847553) has meanwhile been reached: they shall behave like "C" functions. IOW, the ABI of a "system" function is (bold part is new in this PR):
- "stdcall" for win32 targets **for non-variadic functions**
- "C" for everything else
This had been previously stabilized *without FCP* in https://github.com/rust-lang/rust/pull/116161, which got reverted in https://github.com/rust-lang/rust/pull/136897. There was also a "fun" race condition involved with the system ABI being [added](https://github.com/rust-lang/rust/pull/119587) to the list of variadic-supporting ABIs between the creation and merge of rust-lang/rust#116161.
There was a question raised [here](https://github.com/rust-lang/rust/pull/116161#issuecomment-1983829513) whether t-lang even needs to be involved for a change like this. Not sure if that has meanwhile been clarified? The behavior of the "system" ABI (a Rust-specific ABI) definitely feels like t-lang territory to me.
Fixesrust-lang/rust#100189
Cc `@rust-lang/lang`
# Stabilization report
> ## General design
> ### What is the RFC for this feature and what changes have occurred to the user-facing design since the RFC was finalized?
AFAIK there is no RFC. The tracking issues are
- https://github.com/rust-lang/rust/issues/100189
- https://github.com/rust-lang/rust/issues/136946
> ### What behavior are we committing to that has been controversial? Summarize the major arguments pro/con.
The only controversial point is whether "system" ABI functions should support variadics.
- Pro: This allows crates like windows-rs to consistently use "system", see e.g. https://github.com/microsoft/windows-rs/issues/3626.
- Cons: `@workingjubilee` had some implementation concerns, but I think those have been [resolved](https://github.com/rust-lang/rust/issues/136946#issuecomment-2967847553). EDIT: turns out Jubilee still has concerns (she mentioned that in a DM); I'll let her express those.
Note that "system" is already a magic ABI we introduced to "do the right thing". This just makes it do the right thing in more cases. In particular, it means that on Windows one can almost always just do
```rust
extern "system" {
// put all the things here
}
```
and it'll do the right thing, rather than having to split imports into non-varargs and varargs, with the varargs in a separate `extern "C"` block (and risking accidentally putting a non-vararg there).
(I am saying "almost" always because some Windows API functions actually use cdecl, not stdcall, on x86. Those of course need to go in `extern "C"` blocks.)
> ### Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those?
Actually defining variadic functions in Rust remains unstable, under the [c_variadic feature gate](https://github.com/rust-lang/rust/issues/44930).
> ## Has a Call for Testing period been conducted? If so, what feedback was received?
>
> Does any OSS nightly users use this feature? For instance, a useful indication might be "search <grep.app> for `#![feature(FEATURE_NAME)]` and had `N` results".
There was no call for testing.
A search brings up https://github.com/rust-osdev/uefi-rs/blob/main/uefi-raw/src/table/boot.rs using this for "efiapi". This doesn't seem widely used, but it is an "obvious" gap in our support for c-variadics.
> ## Implementation quality
All rustc does here is forward the ABI to LLVM so there's lot a lot to say here...
> ### Summarize the major parts of the implementation and provide links into the code (or to PRs)
>
> An example for async closures: <https://rustc-dev-guide.rust-lang.org/coroutine-closures.html>.
The check for allowed variadic ABIs is [here](9c870d30e2/compiler/rustc_hir_analysis/src/lib.rs (L109-L126)).
The special handling of "system" is [here](c24914ec83/compiler/rustc_target/src/spec/abi_map.rs (L82-L85)).
> ### Summarize existing test coverage of this feature
>
> Consider what the "edges" of this feature are. We're particularly interested in seeing tests that assure us about exactly what nearby things we're not stabilizing.
>
> Within each test, include a comment at the top describing the purpose of the test and what set of invariants it intends to demonstrate. This is a great help to those reviewing the tests at stabilization time.
>
> - What does the test coverage landscape for this feature look like?
> - Tests for compiler errors when you use the feature wrongly or make mistakes?
> - Tests for the feature itself:
> - Limits of the feature (so failing compilation)
> - Exercises of edge cases of the feature
> - Tests that checks the feature works as expected (where applicable, `//@ run-pass`).
> - Are there any intentional gaps in test coverage?
>
> Link to test folders or individual tests (ui/codegen/assembly/run-make tests, etc.).
Prior PRs add a codegen test for all ABIs and tests actually calling extern variadic functions for sysv64 and win64:
- https://github.com/rust-lang/rust/pull/144359
- https://github.com/rust-lang/rust/pull/144379
We don't have a way of executing uefi target code in the test suite, so it's unclear how to fully test efiapi. aapcs could probably be done? (But note that we have hardly an such actually-calling-functions tests for ABI things, we almost entirely rely on codegen tests.)
The test ensuring that we do *not* stabilize *defining* c-variadic functions is `tests/ui/feature-gates/feature-gate-c_variadic.rs`.
> ### What outstanding bugs in the issue tracker involve this feature? Are they stabilization-blocking?
None that I am aware of.
> ### What FIXMEs are still in the code for that feature and why is it ok to leave them there?
None that I am aware of.
> ### Summarize contributors to the feature by name for recognition and assuredness that people involved in the feature agree with stabilization
`@Soveu` added sysv64, win64, efiapi, aapcs to the list of ABIs that allow variadics, `@beepster4096` added system. `@workingjubilee` recently refactored the ABI handling in the compiler, also affecting this feature.
> ### Which tools need to be adjusted to support this feature. Has this work been done?
>
> Consider rustdoc, clippy, rust-analyzer, rustfmt, rustup, docs.rs.
Maybe RA needs to be taught about the new allowed ABIs? No idea how precisely they mirror what exactly rustc accepts and rejects here.
> ## Type system and execution rules
> ### What compilation-time checks are done that are needed to prevent undefined behavior?
>
> (Be sure to link to tests demonstrating that these tests are being done.)
Nothing new here, this just expands the existing support for calling variadic functions to more ABIs.
> ### Does the feature's implementation need checks to prevent UB or is it sound by default and needs opt in in places to perform the dangerous/unsafe operations? If it is not sound by default, what is the rationale?
Nothing new here, this just expands the existing support for calling variadic functions to more ABIs.
> ### Can users use this feature to introduce undefined behavior, or use this feature to break the abstraction of Rust and expose the underlying assembly-level implementation? (Describe.)
Nothing new here, this just expands the existing support for calling variadic functions to more ABIs.
> ### What updates are needed to the reference/specification? (link to PRs when they exist)
- https://github.com/rust-lang/reference/pull/1936
> ## Common interactions
> ### Does this feature introduce new expressions and can they produce temporaries? What are the lifetimes of those temporaries?
No.
> ### What other unstable features may be exposed by this feature?
None.
pub async fn impl is monomorphized when func itself is monomorphized
Implentation coroutine (`func::{closure#0}`) is monomorphized, when func itself is monomorphized.
Currently, when `pub async fn foo(..)` is exported from lib and used in several dependent crates, only 'header' function is monomorphized in the defining crate. 'header' function, returning coroutine object, is monomorphized, but the coroutine's poll function (which actually implements all the logic for the function) is not. In such situation, `func::{closure#0}` will be monomorphized in every dependency.
This PR adds monomorphization for `func::{closure#0}` (coroutine poll function), when func itself is monomorphized.
Simple test with one lib async function and ten dependent crates (executable) that use the function, shows 5-7% compilation time improvement (single-threaded).
Consolidate panicking functions in `slice/index.rs`
Consolidate all the panicking functions in `slice/index.rs` to use a single `slice_index_fail` function, similar to how it is done in `str/traits.rs`.
Split off from https://github.com/rust-lang/rust/pull/145024
Tell LLVM about read-only captures
`&Freeze` parameters are not only `readonly` within the function, but any captures of the pointer can also only be used for reads. This can now be encoded using the `captures(address, read_provenance)` attribute.
`&Freeze` parameters are not only `readonly` within the function,
but any captures of the pointer can also only be used for reads.
This can now be encoded using the `captures(address, read_provenance)`
attribute.
