420 Commits

Author SHA1 Message Date
Zalathar
3831b60ef3 Remove inherent methods from llvm::Type 2025-10-04 18:47:18 +10:00
Matthias Krüger
c29fb2e57e
Rollup merge of #144197 - KMJ-007:type-tree, r=ZuseZ4
TypeTree support in autodiff

# TypeTrees for Autodiff

## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.

## Structure
```rust
TypeTree(Vec<Type>)

Type {
    offset: isize,  // byte offset (-1 = everywhere)
    size: usize,    // size in bytes
    kind: Kind,     // Float, Integer, Pointer, etc.
    child: TypeTree // nested structure
}
```

## Example: `fn compute(x: &f32, data: &[f32]) -> f32`

**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,
        child: TypeTree::new()
    }])
}])
```

**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,  // -1 = all elements
        child: TypeTree::new()
    }])
}])
```

**Output: `f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 4, kind: Float,
    child: TypeTree::new()
}])
```

## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata

## What Enzyme Does With This Information:

Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```

With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
    ptr "enzyme_type"="{[]:Pointer}" %p1,
    ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```

# TypeTrees - Offset and -1 Explained

## Type Structure

```rust
Type {
    offset: isize, // WHERE this type starts
    size: usize,   // HOW BIG this type is
    kind: Kind,    // WHAT KIND of data (Float, Int, Pointer)
    child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```

## Offset Values

### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**

```rust
struct Point {
    x: f32, // offset 0, size 4
    y: f32, // offset 4, size 4
    id: i32, // offset 8, size 4
}
```

TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
    Type { offset: 0, size: 4, kind: Float },   // x at byte 0
    Type { offset: 4, size: 4, kind: Float },   // y at byte 4
    Type { offset: 8, size: 4, kind: Integer }  // id at byte 8
])
```

Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```

### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**

#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
    offset: -1, // ALL positions
    size: 4,    // each f32 is 4 bytes
    kind: Float, // every element is float
}])
```

Instead of listing 100 separate Types with offsets `0,4,8,12...396`

#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, // ALL slice elements
        size: 4,    // each i32 is 4 bytes
        kind: Integer
    }])
}])
```

#### Example 3: Mixed Structure
```rust
struct Container {
    header: i64,        // offset 0
    data: [f32; 1000],  // offset 8, but elements use -1
}
```

```rust
TypeTree(vec![
    Type { offset: 0, size: 8, kind: Integer }, // header
    Type { offset: 8, size: 4000, kind: Pointer,
        child: TypeTree(vec![Type {
            offset: -1, size: 4, kind: Float // ALL array elements
        }])
    }
])
```
2025-09-28 18:13:11 +02:00
Zalathar
85018f09f6 Use LLVMDisposeTargetMachine 2025-09-25 18:10:55 +10:00
Zalathar
b17fb70d04 Add self-profile events for target-machine creation
These code paths are surprisingly hot in the `large-workspace` benchmark; it
would be handy to see some detailed timings and execution counts.
2025-09-21 21:37:15 +10:00
Karan Janthe
e1258e79d6 autodiff: Add basic TypeTree with NoTT flag
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-09-19 04:02:19 +00:00
Zalathar
8b0a254860 Move target machine command-line quoting from C++ to Rust 2025-09-18 15:25:25 +10:00
bjorn3
1991779bd4 Make llvm_enzyme a regular cargo feature
This makes it clearer that it is set by the build system rather than by
the rustc that compiles the current rustc. It also avoids bootstrap
needing to pass --check-cfg llvm_enzyme to rustc.
2025-09-15 15:31:56 +00:00
bjorn3
f2933b34a8 Remove want_summary argument from prepare_thin
It is always false nowadays. ThinLTO summary writing is instead done by
llvm_optimize.
2025-09-06 18:37:23 +00:00
bjorn3
e3d0b7d648 Remove thin_link_data method from ThinBufferMethods
It is only used within cg_llvm.
2025-09-06 18:37:23 +00:00
bjorn3
2cf94b92ca Ensure fat LTO doesn't merge everything into the allocator module 2025-09-06 13:31:41 +00:00
bjorn3
319fe230f0 Special case allocator module submission to avoid special casing it elsewhere
A lot of places had special handling just in case they would get an
allocator module even though most of these places could never get one or
would have a trivial implementation for the allocator module. Moving all
handling of the allocator module to a single place simplifies things a
fair bit.
2025-09-04 08:21:10 +00:00
Daniel Paoliello
da8f230d5f Update to ar_archive_writer 0.5.1 2025-08-29 16:37:42 -07:00
bjorn3
23e72ab41e Move ___asan_globals_registered export
All other sanitizer symbols are handled in prepare_lto already.
2025-08-28 19:36:51 +00:00
bjorn3
d76cff3f06 Only export the sanitizer symbols for LTO and move export code to cg_llvm
Don't export them from cdylibs. There is no need to do so and it
complicates exported_non_generic_symbols. In addition the GCC backend
likely uses different symbols and may potentially not even need us to
explicitly tell it to export the symbols it needs.
2025-08-28 19:36:44 +00:00
Stuart Cook
f5633098b0
Rollup merge of #145814 - bjorn3:codegen_worker_fatal_error, r=petrochenkov
Handle unwinding fatal errors in codegen workers

Also directly unwind on fatal errors at the point they are emitted inside the codegen backends.

Fixes the coordinator ICE of https://github.com/rust-lang/rust/issues/132240, https://github.com/rust-lang/rust/issues/135075 and https://github.com/rust-lang/rust/issues/145800.
2025-08-26 14:19:18 +10:00
Zalathar
b4e97e5d86 Rename llvm::Bool aliases to standard const case
This avoids the need for `#![allow(non_upper_case_globals)]`.
2025-08-24 23:09:54 +10:00
bjorn3
525c6a3562 Directly raise fatal errors inside the codegen backends
As opposed to passing it around through Result.
2025-08-24 11:20:41 +00:00
Stuart Cook
8748d8e7d5
Rollup merge of #145484 - Zalathar:archive-builder, r=bjorn3
Remove `LlvmArchiveBuilder` and supporting code/bindings

Switching over to the newer Rust-based `ArArchiveBuilder` happened in rust-lang/rust#128936, a year ago.

Per the comment in `new_archive_builder`, that seems like enough time to justify removing the older, unused `LlvmArchiveBuilder` implementation and its associated bindings.

Fixes rust-lang/rust#128955.
2025-08-19 14:18:25 +10:00
Stuart Cook
8945924d77
Rollup merge of #145432 - Zalathar:target-machine, r=wesleywiser
cg_llvm: Small cleanups to `owned_target_machine`

This PR contains a few tiny cleanups to the `owned_target_machine` code.

Each individual commit should be fairly straightforward.
2025-08-19 14:18:25 +10:00
Zalathar
cf8ec6798f Remove LlvmArchiveBuilder and supporting code/bindings 2025-08-16 16:38:12 +10:00
Zalathar
9e7d066928 Simplify the args_cstr_buff assertion 2025-08-15 20:24:13 +10:00
Zalathar
61932e1222 Avoid an unnecessary intermediate &mut reference
The `NonNull::as_mut` method returns a mut *reference*, rather than the mut
*pointer* that is intended here.
2025-08-15 20:24:13 +10:00
Zalathar
44f5ec7d56 Avoid an explicit cast from *const c_uchar to *const c_char
As noted in the `ffi` module docs, passing pointer/length byte strings from
Rust to C++ is easier if we declare them as `*const c_uchar` on the Rust side,
but `const char *` (possibly signed) on the C++ side. This is allowed because
both pointer types are ABI-compatible, regardless of char signedness.
2025-08-15 20:24:13 +10:00
Zalathar
8d0a04966c Declare module rustc_codegen_llvm::back in the normal way
Declaring these submodules directly in `lib.rs` was needlessly confusing.
2025-08-15 20:24:13 +10:00
Stuart Cook
60cb5502c1
Rollup merge of #145004 - bjorn3:remove_unused_fields, r=WaffleLapkin
Couple of minor cleanups
2025-08-15 16:16:33 +10:00
Marcelo Domínguez
e1d79b9aad Remove lto inline logic 2025-08-14 16:30:16 +00:00
bjorn3
8f648d7185 Remove bitcode_llvm_cmdline
It used to be necessary on Apple platforms to ship with the App Store,
but XCode 15 has stopped embedding LLVM bitcode and the App Store no
longer accepts apps with bitcode embedded.
2025-08-08 11:24:28 +00:00
bors
65b6cdb6a6 Auto merge of #144562 - matthiaskrgr:rollup-mlvn7qo, r=matthiaskrgr
Rollup of 7 pull requests

Successful merges:

 - rust-lang/rust#144072 (update `Atomic*::from_ptr` and `Atomic*::as_ptr` docs)
 - rust-lang/rust#144151 (`tests/ui/issues/`: The Issues Strike Back [1/N])
 - rust-lang/rust#144300 (Clippy fixes for miropt-test-tools)
 - rust-lang/rust#144399 (Add a ratchet for moving all standard library tests to separate packages)
 - rust-lang/rust#144472 (str: Mark unstable `round_char_boundary` feature functions as const)
 - rust-lang/rust#144503 (Various refactors to the codegen coordinator code (part 3))
 - rust-lang/rust#144530 (coverage: Infer `instances_used` from `pgo_func_name_var_map`)

r? `@ghost`
`@rustbot` modify labels: rollup
2025-07-28 08:50:59 +00:00
Kivooeo
43725ed819 use let chains in ast, borrowck, codegen, const_eval 2025-07-28 06:08:48 +05:00
bjorn3
948f7798d7 Remove support for -Zcombine-cgu
Nobody seems to actually use this, while still adding some extra
complexity to the already rather complex codegen coordinator code.
It is also not supported by any backend other than the LLVM backend.
2025-07-26 13:52:24 +00:00
bjorn3
fe2eeabe27 Use the object crate rather than LLVM for extracting bitcode sections 2025-07-25 11:21:28 +00:00
bors
5d22242a3a Auto merge of #144062 - bjorn3:lto_refactors2, r=davidtwco
Various refactors to the LTO handling code (part 2)

Continuing from https://github.com/rust-lang/rust/pull/143388 this removes a bit of dead code and moves the LTO symbol export calculation from individual backends to cg_ssa.
2025-07-24 12:50:26 +00:00
许杰友 Jieyou Xu (Joe)
5e3eb25125
Rollup merge of #142097 - ZuseZ4:offload-host1, r=oli-obk
gpu offload host code generation

r? ghost

This will generate most of the host side code to use llvm's offload feature.
The first PR will only handle automatic mem-transfers to and from the device.
So if a user calls a kernel, we will copy inputs back and forth, but we won't do the actual kernel launch.
Before merging, we will use LLVM's Info infrastructure to verify that the memcopies match what openmp offloa generates in C++. `LIBOMPTARGET_INFO=-1 ./my_rust_binary` should print that a memcpy to and later from the device is happening.

A follow-up PR will generate the actual device-side kernel which will then do computations on the GPU.
A third PR will implement manual host2device and device2host functionality, but the goal is to minimize cases where a user has to overwrite our default handling due to performance issues.

I'm trying to get a full MVP out first, so this just recognizes GPU functions based on magic names. The final frontend will obviously move this over to use proper macros, like I'm already doing it for the autodiff work.
This work will also be compatible with std::autodiff, so one can differentiate GPU kernels.

Tracking:
- https://github.com/rust-lang/rust/issues/131513
2025-07-22 00:54:24 +08:00
bjorn3
dadc4cae50 Remove each_linked_rlib_for_lto from CodegenContext 2025-07-21 07:58:44 +00:00
bjorn3
1a6f941d2b Move exported_symbols_for_lto out of CodegenContext 2025-07-21 07:58:44 +00:00
bjorn3
6e757354ad Merge exported_symbols computation into exported_symbols_for_lto
And move exported_symbols_for_lto call from backends to cg_ssa.
2025-07-21 07:58:44 +00:00
bjorn3
1c8dc6f440 Move LTO symbol export calculation from backends to cg_ssa 2025-07-21 07:58:44 +00:00
bjorn3
112799e637 Merge modules and cached_modules for fat LTO
The modules vec can already contain serialized modules and there is no
need to distinguish between cached and non-cached cgus at LTO time.
2025-07-21 07:58:44 +00:00
Manuel Drehwald
4a1a5a4295 gpu host code generation 2025-07-18 16:30:42 -07:00
Manuel Drehwald
634016478e add -Zoffload=Enable flag behind -Zunstable-options, to enable gpu (host) code generation 2025-07-18 16:24:00 -07:00
Nikita Popov
12b19be741 Pass wasm exception model to TargetOptions
This is no longer implied by -wasm-enable-eh.
2025-07-18 09:35:50 +02:00
León Orell Valerian Liehr
be5f8f299d
Rollup merge of #143388 - bjorn3:lto_refactors, r=compiler-errors
Various refactors to the LTO handling code

In particular reducing the sharing of code paths between fat and thin-LTO and making the fat LTO implementation more self-contained. This also moves some autodiff handling out of cg_ssa into cg_llvm given that Enzyme only works with LLVM anyway and an implementation for another backend may do things entirely differently. This will also make it a bit easier to split LTO handling out of the coordinator thread main loop into a separate loop, which should reduce the complexity of the coordinator thread.
2025-07-17 03:58:28 +02:00
Oli Scherer
d3d51b4fdb Avoid a bunch of unnecessary unsafe blocks in cg_llvm 2025-07-14 08:27:08 +00:00
Matthias Krüger
7f3204f34d
Rollup merge of #143633 - dillona:noinline-assert, r=fee1-dead
fix: correct assertion to check for 'noinline' attribute presence before removal
2025-07-11 19:45:23 +02:00
Oli Scherer
84eeca2e2f Make some "safe" llvm ops actually sound 2025-07-10 07:27:41 +00:00
Dillon Amburgey
b2299e20b2 fix: correct assertion to check for 'noinline' attribute presence before removal 2025-07-08 06:24:50 -05:00
bjorn3
7fd78df346 Move dcx creation into WriteBackendMethods::codegen 2025-07-03 14:43:09 +00:00
bjorn3
653bb64c75 Remove LtoModuleCodegen
Most uses of it either contain a fat or thin lto module. Only
WorkItem::LTO could contain both, but splitting that enum variant
doesn't complicate things much.
2025-07-03 14:28:18 +00:00
Karan Janthe
7b1c89f2b5 added PrintTAFn flag for autodiff
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-06-25 02:11:29 +00:00
bjorn3
f0707fad31 Mark all optimize methods and the codegen method as safe
There is no safety contract and I don't think any of them can actually
cause UB in more ways than passing malicious source code to rustc can.
While LtoModuleCodegen::optimize says that the returned ModuleCodegen
points into the LTO module, the LTO module has already been dropped by
the time this function returns, so if the returned ModuleCodegen indeed
points into the LTO module, we would have seen crashes on every LTO
compilation, which we don't. As such the comment is outdated.
2025-05-28 20:55:00 +00:00