123 Commits

Author SHA1 Message Date
Zalathar
69a975faa9 Consistently import llvm::Type and llvm::Value 2025-10-06 13:09:16 +11:00
Zalathar
f8c54d24e2 Remove inherent methods from llvm::CallConv::from_conv 2025-10-04 18:47:18 +10:00
Matthias Krüger
c29fb2e57e
Rollup merge of #144197 - KMJ-007:type-tree, r=ZuseZ4
TypeTree support in autodiff

# TypeTrees for Autodiff

## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.

## Structure
```rust
TypeTree(Vec<Type>)

Type {
    offset: isize,  // byte offset (-1 = everywhere)
    size: usize,    // size in bytes
    kind: Kind,     // Float, Integer, Pointer, etc.
    child: TypeTree // nested structure
}
```

## Example: `fn compute(x: &f32, data: &[f32]) -> f32`

**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,
        child: TypeTree::new()
    }])
}])
```

**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,  // -1 = all elements
        child: TypeTree::new()
    }])
}])
```

**Output: `f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 4, kind: Float,
    child: TypeTree::new()
}])
```

## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata

## What Enzyme Does With This Information:

Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```

With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
    ptr "enzyme_type"="{[]:Pointer}" %p1,
    ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```

# TypeTrees - Offset and -1 Explained

## Type Structure

```rust
Type {
    offset: isize, // WHERE this type starts
    size: usize,   // HOW BIG this type is
    kind: Kind,    // WHAT KIND of data (Float, Int, Pointer)
    child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```

## Offset Values

### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**

```rust
struct Point {
    x: f32, // offset 0, size 4
    y: f32, // offset 4, size 4
    id: i32, // offset 8, size 4
}
```

TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
    Type { offset: 0, size: 4, kind: Float },   // x at byte 0
    Type { offset: 4, size: 4, kind: Float },   // y at byte 4
    Type { offset: 8, size: 4, kind: Integer }  // id at byte 8
])
```

Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```

### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**

#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
    offset: -1, // ALL positions
    size: 4,    // each f32 is 4 bytes
    kind: Float, // every element is float
}])
```

Instead of listing 100 separate Types with offsets `0,4,8,12...396`

#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, // ALL slice elements
        size: 4,    // each i32 is 4 bytes
        kind: Integer
    }])
}])
```

#### Example 3: Mixed Structure
```rust
struct Container {
    header: i64,        // offset 0
    data: [f32; 1000],  // offset 8, but elements use -1
}
```

```rust
TypeTree(vec![
    Type { offset: 0, size: 8, kind: Integer }, // header
    Type { offset: 8, size: 4000, kind: Pointer,
        child: TypeTree(vec![Type {
            offset: -1, size: 4, kind: Float // ALL array elements
        }])
    }
])
```
2025-09-28 18:13:11 +02:00
Nikita Popov
d226e7aa93 Use standard attribute logic for allocator shim
Use llfn_attrs_from_instance() to generate the attributes for the
allocator shim. This ensures that we generate all the usual
attributes (and don't get to find out one-by-one that a certain
attribute is important for a certain target). Additionally this
will enable emitting the allocator-specific attributes (not
included here).

This change is quite awkward because the allocator shim uses
SimpleCx, while llfn_attrs_from_instance uses CodegenCx. I've
switched it to use SimpleCx plus tcx/sess arguments where necessary.
If there's a simpler way to do this, I'd love to know about it...
2025-09-25 10:04:40 +02:00
Karan Janthe
664e83b3e7 added typetree support for memcpy 2025-09-19 04:02:20 +00:00
Zachary S
baed55ccef Remove unreachable unsized arg handling in store_fn_arg/store_arg in codegen 2025-09-12 09:49:41 -05:00
Nikita Popov
c3ab409b4f Use captures(address) instead of captures(none) for indirect args
While provenance cannot be captured through these arguments, the
address / object identity can.
2025-08-26 16:16:23 +02:00
Nikita Popov
d71ed8d19b Tell LLVM about read-only captures
`&Freeze` parameters are not only `readonly` within the function,
but any captures of the pointer can also only be used for reads.
This can now be encoded using the `captures(address, read_provenance)`
attribute.
2025-08-20 19:08:16 +02:00
Nikita Popov
ebef9d7f63 Set dead_on_return attribute for indirect arguments
Set the dead_on_return attribute (added in LLVM 21) for arguments
that are passed indirectly, but not byval.

This indicates that the value of the argument on return does not
matter, enabling additional dead store elimination.
2025-08-11 12:39:23 +02:00
Folkert de Vries
226b0fbe11
use is_multiple_of instead of manual modulo 2025-07-05 10:55:35 +02:00
beetrees
5723c9997c
Fix RISC-V C function ABI when passing/returning structs containing floats 2025-06-16 10:14:07 +01:00
Folkert de Vries
5f73ce2b7e
add extern "custom" functions 2025-06-12 20:27:10 +02:00
Matthias Krüger
644f06ec1f
Rollup merge of #141569 - workingjubilee:canonicalize-abi, r=bjorn3
Replace ad-hoc ABI "adjustments" with an `AbiMap` to `CanonAbi`

Our `conv_from_spec_abi`, `adjust_abi`, and `is_abi_supported` combine to give us a very confusing way of reasoning about what _actual_ calling convention we want to lower our code to and whether we want to compile the resulting code at all. Instead of leaving this code as a miniature adventure game in which someone tries to combine stateful mutations into a Rube Goldberg machine that will let them escape the maze and arrive at the promised land of codegen, we let `AbiMap` devour this complexity. Once you have an `AbiMap`, you can answer which `ExternAbi`s will lower to what `CanonAbi`s (and whether they will lower at all).

Removed:
- `conv_from_spec_abi` replaced by `AbiMap::canonize_abi`
- `adjust_abi` replaced by same
- `Conv::PreserveAll` as unused
- `Conv::Cold` as unused
- `enum Conv` replaced by `enum CanonAbi`

target-spec.json changes:
- If you have a target-spec.json then now your "entry-abi" key will be specified in terms of one of the `"{abi}"` strings Rust recognizes, e.g.
```json
    "entry-abi": "C",
    "entry-abi": "win64",
    "entry-abi": "aapcs",
```
2025-06-03 21:53:36 +02:00
Jubilee Young
e0b07a88a3 cg_llvm: convert to CanonAbi 2025-06-03 10:04:19 -07:00
bjorn3
865c7b9c78 Remove unused arg_memory_ty method 2025-05-28 20:55:00 +00:00
Josh Stone
12167d7064 Update the minimum external LLVM to 19 2025-04-05 11:44:38 -07:00
Oli Scherer
29440b84a9 Remove an unused lifetime param 2025-02-24 15:11:29 +00:00
Jubilee Young
2d2de18166 compiler: Stop reexporting stuff in cg_llvm::abi
The reexports confuse tooling like rustdoc into thinking cg_llvm is
the source of key types that originate in rustc_target.
2025-02-18 00:31:29 -08:00
Jacob Pratt
33c186baf7
Rollup merge of #136807 - workingjubilee:merge-gpus-to-get-the-arcradeongeforce, r=bjorn3
compiler: internally merge `PtxKernel` into `GpuKernel`

r? ``@bjorn3`` for review
2025-02-12 20:10:00 -05:00
Jacob Pratt
6153a8dcea
Rollup merge of #136721 - dpaoliello:cleanllvm2, r=Zalathar
cg_llvm: Reduce visibility of some items outside the `llvm` module

Next piece of #135502

This reduces the visibility of items (other than those in the `llvm` module) so that dead code analysis will correctly identify unused items.
2025-02-11 01:02:40 -05:00
Daniel Paoliello
5f29273921 rustc_codegen_llvm: Mark items as pub(crate) outside of the llvm module 2025-02-10 10:17:25 -08:00
Jubilee Young
e11e2b4d09 compiler: internally merge Conv::PtxKernel into GpuKernel
It is speculated that these two can be conceptually merged, and it can
start by ripping out rustc's notion of the PtxKernel call convention.
Leave the ExternAbi for now, but the nvptx target now should see it as
just a different way to spell Conv::GpuKernel.
2025-02-09 23:14:55 -08:00
bors
124cc92199 Auto merge of #136751 - bjorn3:update_rustfmt, r=Mark-Simulacrum
Update bootstrap compiler and rustfmt

The rustfmt version we previously used formats things differently from what the latest nightly rustfmt does. This causes issues for subtrees that get formatted both in-tree and in their own repo. Updating the rustfmt used in-tree solves those issues. Also bumped the bootstrap compiler as the stage0 update command always updates both at the same
time.
2025-02-09 15:44:16 +00:00
bjorn3
1fcae03369 Rustfmt 2025-02-08 22:12:13 +00:00
Jubilee Young
eddfe8f503 compiler: remove reexports from rustc_target::callconv 2025-02-07 11:25:18 -08:00
Flakebi
e7e5202978
Add gpu-kernel calling convention
The amdgpu-kernel calling convention was reverted in commit
f6b21e90d1ec01081bc2619efb68af6788a63d65 due to inactivity in the amdgpu
target.

Introduce a `gpu-kernel` calling convention that translates to
`ptx_kernel` or `amdgpu_kernel`, depending on the target that rust
compiles for.
2025-01-16 00:26:55 +01:00
Jubilee Young
b895bf4fdc compiler: Directly use rustc_abi in codegen 2024-11-03 12:30:32 -08:00
Jubilee Young
7086dd83cc compiler: rustc_abi::Abi => BackendRepr
The initial naming of "Abi" was an awful mistake, conveying wrong ideas
about how psABIs worked and even more about what the enum meant.
It was only meant to represent the way the value would be described to
a codegen backend as it was lowered to that intermediate representation.
It was never meant to mean anything about the actual psABI handling!
The conflation is because LLVM typically will associate a certain form
with a certain ABI, but even that does not hold when the special cases
that actually exist arise, plus the IR annotations that modify the ABI.

Reframe `rustc_abi::Abi` as the `BackendRepr` of the type, and rename
`BackendRepr::Aggregate` as `BackendRepr::Memory`. Unfortunately, due to
the persistent misunderstandings, this too is now incorrect:
- Scattered ABI-relevant code is entangled with BackendRepr
- We do not always pre-compute a correct BackendRepr that reflects how
  we "actually" want this value to be handled, so we leave the backend
  interface to also inject various special-cases here
- In some cases `BackendRepr::Memory` is a "real" aggregate, but in
  others it is in fact using memory, and in some cases it is a scalar!

Our rustc-to-backend lowering code handles this sort of thing right now.
That will eventually be addressed by lifting duplicated lowering code
to either rustc_codegen_ssa or rustc_target as appropriate.
2024-10-29 14:56:00 -07:00
Jubilee Young
88a9edc091 compiler: Add is_uninhabited and use LayoutS accessors
This reduces the need of the compiler to peek on the fields of LayoutS.
2024-10-28 09:58:30 -07:00
Jubilee Young
1379ef592a compiler: Factor rustc_target::abi out of cg_llvm 2024-10-08 18:24:56 -07:00
Urgau
018ba0528f Use wide pointers consistenly across the compiler 2024-10-04 14:06:48 +02:00
Michael Goulet
c682aa162b Reformat using the new identifier sorting from rustfmt 2024-09-22 19:11:29 -04:00
Folkert de Vries
1ddd67a79a add C-cmse-nonsecure-entry ABI 2024-09-21 13:04:14 +02:00
Josh Stone
6fd8a50680 Update the minimum external LLVM to 18 2024-09-18 13:53:31 -07:00
Nicholas Nethercote
410a2de0c0 Rename {ArgAbi,IntrinsicCall}Methods.
They both are part of `BuilderMethods`, and so should have `Builder` in
their name like all the other traits in `BuilderMethods`.
2024-09-17 10:24:43 +10:00
Nicholas Nethercote
61627438eb Add warn(unreachable_pub) to rustc_codegen_llvm. 2024-08-16 08:46:57 +10:00
bors
e08b80c0fb Auto merge of #128371 - andjo403:rangeAttribute, r=nikic
Add range attribute to scalar function results and arguments

as LLVM 19 adds the range attribute this starts to use it for better optimization.
hade been interesting to see a perf run with the https://github.com/rust-lang/rust/pull/127513

closes https://github.com/rust-lang/rust/issues/50156
cc https://github.com/rust-lang/rust/issues/49572 shall be fixed but not possible to see as there is asserts that already trigger the optimization.
2024-08-12 10:20:00 +00:00
Andreas Jonson
cfadfabfcd Add range attribute to scalar function results and arguments 2024-08-11 19:40:44 +02:00
Ralf Jung
273c67db83 codegen: better centralize function attribute computation 2024-08-07 19:49:48 +02:00
bors
80d8270d84 Auto merge of #125016 - nicholasbishop:bishop-cb-112, r=tgross35
Update compiler_builtins to 0.1.114

The `weak-intrinsics` feature was removed from compiler_builtins in https://github.com/rust-lang/compiler-builtins/pull/598, so dropped the `compiler-builtins-weak-intrinsics` feature from alloc/std/sysroot.

In https://github.com/rust-lang/compiler-builtins/pull/593, some builtins for f16/f128 were added. These don't work for all compiler backends, so add a `compiler-builtins-no-f16-f128` feature and disable it for cranelift and gcc.
2024-07-29 07:41:33 +00:00
Nicholas Nethercote
84ac80f192 Reformat use declarations.
The previous commit updated `rustfmt.toml` appropriately. This commit is
the outcome of running `x fmt --all` with the new formatting options.
2024-07-29 08:26:52 +10:00
DianQK
c453dcd62a
Use the aligned size for alloca at args when the pass mode is cast.
The `load` and `store` instructions in LLVM access the aligned size.
2024-07-02 06:33:35 +08:00
Nicholas Bishop
99e6a28804 Add f16/f128 handling in a couple places 2024-05-30 18:33:50 -04:00
Michael Goulet
d50c2b0a52 Make builtin_deref just return a Ty 2024-05-09 22:55:00 -04:00
bors
284f94f9c0 Auto merge of #121298 - nikic:writable, r=cuviper
Set writable and dead_on_unwind attributes for sret arguments

Set the `writable` and `dead_on_unwind` attributes for `sret` arguments. This allows call slot optimization to remove more memcpy's.

See https://llvm.org/docs/LangRef.html#parameter-attributes for the specification of these attributes. In short, the statement we're making here is that:

 * The return slot is writable.
 * The return slot will not be read if the function unwinds.

Fixes https://github.com/rust-lang/rust/issues/90595.
2024-04-25 04:31:56 +00:00
Nikita Popov
3695af697e Set writable and dead_on_unwind attributes for sret arguments 2024-04-25 11:43:47 +09:00
bors
29a56a3b1c Auto merge of #122053 - erikdesjardins:alloca, r=nikic
Stop using LLVM struct types for alloca

The alloca type has no semantic meaning, only the size (and alignment, but we specify it explicitly) matter. Using `[N x i8]` is a more direct way to specify that we want `N` bytes, and avoids relying on LLVM's struct layout. It is likely that a future LLVM version will change to an untyped alloca representation.

Split out from #121577.

r? `@ghost`
2024-04-24 03:00:44 +00:00
Erik Desjardins
f4426c189f use [N x i8] for alloca types 2024-04-11 21:42:35 -04:00
Scott McMurray
3596098823 Put PlaceValue into OperandValue::Ref, rather than 3 tuple fields 2024-04-11 00:10:10 -07:00
Scott McMurray
89502e584b Make PlaceRef hold a PlaceValue for the non-layout fields (like OperandRef does) 2024-04-11 00:10:10 -07:00