🪲 CTFBench 🚩

About CTFBench

CTFBench is a benchmark for evaluating AI smart contract auditors. It uses a set of test cases where each smart contract has exactly one known vulnerability. The benchmark calculates two key metrics:

• Vulnerability Detection Rate (VDR): The proportion of detected vulnerabilities, indicating how many of the known vulnerabilities the AI auditor successfully identifies.
• Overreporting Index (OI): The number of false positives reported per line of code in contracts that are known to be free of vulnerabilities, measuring the auditor's tendency to raise unnecessary alerts.

By plotting these metrics on a graph, CTFBench allows for a visual comparison of different AI auditors, helping developers and researchers assess their effectiveness and trade-offs.

Benchmark Results for AI Smart Contract Auditors

Performance Graph