[BOLT] Gadget scanner: add finer-grained modes to `--scanners=` option

[atrosinenko]

This issue is factored out of the discussion #131899 (comment). The existing --scanners= command line option defines three possible modes: pacret, pauth and all (technically, it is multiple choice):

• pacret only checks for non-protected returns
• pauth checks for any PAuth-related gadgets: those checked by pacret and additionally
  • tail calls performed with non-trusted address in LR
  • non-protected indirect calls
  • signing oracles (signing a pointer that is possibly attacker-controlled)
  • authentication oracles (authentication instructions whose success/fail result can be tested by the attacker without crashing the program)
• all is currently the same as pauth

Additionally, passing the --auth-traps-on-failure option disables the detection of authentication oracles completely and modifies the rules for other checkers.

It may be worth implementing a finer-grained control over which gadget kinds to search for.

• For example, a typical use case is to build programs with pac-ret protection enabled (which does not change the ABI, except maybe for requiring support in libunwind), but without signing function pointers (which is ABI-dependent). If this is the case, we should definitely disable "non-protected calls" check, but may or may not check for signing and authentication oracles.
• Whether we want to protect tail calls or not does not depend on the ABI, it is more of a security vs. performance trade-off. It is actually related to the pacret mode, but has to have a separate toggle.

Tagging @kbeyls, @jacobbramley.

[llvmbot]

@llvm/issue-subscribers-bolt

Author: Anatoly Trosinenko (atrosinenko)

This issue is factored out of the discussion https://github.com//pull/131899#discussion_r2012073205. The existing `--scanners=` command line option defines three possible modes: `pacret`, `pauth` and `all` (technically, it is multiple choice): * `pacret` only checks for non-protected returns * `pauth` checks for any PAuth-related gadgets: those checked by `pacret` and additionally - tail calls performed with non-trusted address in LR - non-protected indirect calls - signing oracles (signing a pointer that is possibly attacker-controlled) - authentication oracles (authentication instructions whose success/fail result can be tested by the attacker without crashing the program) * `all` is currently the same as `pauth`

Additionally, passing the --auth-traps-on-failure option disables the detection of authentication oracles completely and modifies the rules for other checkers.

It may be worth implementing a finer-grained control over which gadget kinds to search for.

• For example, a typical use case is to build programs with pac-ret protection enabled (which does not change the ABI, except maybe for requiring support in libunwind), but without signing function pointers (which is ABI-dependent). If this is the case, we should definitely disable "non-protected calls" check, but may or may not check for signing and authentication oracles.
• Whether we want to protect tail calls or not does not depend on the ABI, it is more of a security vs. performance trade-off. It is actually related to the pacret mode, but has to have a separate toggle.

Tagging @kbeyls, @jacobbramley.