Purging is well known Software testing techniques are particularly useful for testing. Smart contracts. Traditionally, fuzzing has primarily involved testers understanding: Black box purging or more advanced, Attribute-based fuzzing. But a new and innovative approach is Manually guided purgingIt fills the gap left by existing fuzzing techniques by providing improved efficiency and effectiveness. This article explores how guided fuzzing differs from black box fuzzing, attribute-based fuzzing, differential fuzzing, and other techniques.
Basic understanding: Stateful vs. Stateless, Black Box vs. Gray Box vs. Gray Box White Box Fuzzing
Purging techniques can be classified based on their status and the level of guidance provided.
Stateful fuzzing and stateful fuzzing:
Stateless Purging: Each test case is independent of the other test cases in stateless fuzzing. The fuzzer does not maintain knowledge of previous state or inputs, which limits its ability to find bugs related to the order of operations.
Stateful fuzzing: This method takes into account the state of the system, which means that the results of one test case can affect subsequent test cases. State-based fuzzing is more effective for testing complex systems such as smart contracts where state transitions are important. See the differences below. This piece of code It is written in Wake Framework.
Black Box vs. Gray Box vs. White Box:
Black box purging: Fuzzers operate without knowing the internal workings of the system under test. They must know the interface, but rely on random inputs and heuristic techniques to explore the state space to identify unexpected behaviors or crashes. While effective in some cases, black-box fuzzing is time-consuming and resource-intensive, often resulting in incomplete coverage.
Gray Box Purging: A fuzzer has information about the internal workings of the tested system. The definition is very broad, and for example: Invariant (see attribute-based fuzzing)
White Box: In contrast, the fuzzer is provided with all information about the tested system. This approach allows for more focused and efficient fuzzing, focusing on critical code paths and specific scenarios, leading to faster and more predictable bug detection (see Manual Guided Fuzzing).
Get More: Advanced Purging Techniques
There are advanced techniques that enable testing even in more extreme scenarios.
Differential Purging: The fuzzer compares the output of the tested system to the output of a reference. The reference can be a system model implemented in a high-level language such as Python. The test subject can also be a single function that is compared to an existing library that implements the same function. Differential fuzzing is an elegant technique that mainly uses mathematical functions, can reveal many rounding and precision errors in Solidity, and becomes very powerful when combined with stateful fuzzing. An example of differential fuzzing testing is IPOR auditing, where Ackee Blockchain Security compared the implementation of continuous compound interest calculation in a contract with a custom Python model, revealing one critical issue and two serious issues.
Fork Test: When testing projects with complex dependencies or integrations (Aave, Chainlink). Fork testing involves running tests on a development chain (such as Anvil) that is forked from the live network. This allows the system to interact with real data and state without having to redeploy and mock the contract store. Using real data ensures more accurate and comprehensive testing than mocking data (a technique used in formal verification). An example of fork fuzz testing is the Lido Stonks audit, where Ackee Blockchain Security forked the Ethereum mainnet to test the protocol under realistic conditions. A medium severity integration issue was discovered using the forked USDT contract. See the full source code. here.
Attribute-based fuzzing exploration
Attribute-based fuzzingor Invariant testIt is somewhere between black-box and white-box fuzzing, where the tester (who has some knowledge of the tested system) defines certain properties such as: Invariant These are properties that the system must always satisfy, regardless of the input. The fuzzer then tries to generate inputs that violate these properties. While the tester introduces invariants, the fuzzer operates with knowledge of the system’s properties, which provides a “gray box” specification. While more controlled than black-box fuzzing, property-based fuzzing still struggles to fully explore complex state spaces, especially in state-rich systems such as complex smart contracts.
Invariants are tests that run after each state change (transaction).
@invariant()
def invariant_balance(self) -> None:
assert self.token.balanceOf(self.admin) == self.balances(self.admin)Introduction to manually guided purging
Manually guided purging It combines the strengths of Stateful Fuzzing and White Box Fuzzing and adds the following concepts: flow It provides a structured approach to testing, requiring the tester to fully understand the tested system and guide the fuzzing process to thoroughly examine critical parts of the code.
no way flow A series of actions or transactions within a system. For example, a flow might involve transferring tokens from one account to another. Testers define these flows to ensure that the fuzzer targets critical code paths.
A flow is a single test step that is executed in a test sequence. A flow is defined using the @flow decorator.
@flow(precondition=lambda self: self.count > 0)
def flow_decrement(self) -> None:
self.counter.decrement(from_=random_account())
self.count -= 1How Manual Guide Purging Works
Manual guided fuzzing using the Wake Framework involves several steps.
- Definition of invariant: Similar to property-based fuzzing, Manually Guided Fuzzing requires the definition of properties or invariants that must be true after the flow is executed. For example, after a token transfer, an invariant is that the total supply of tokens remains constant and the balance is updated correctly.
- Flow definition: Now the tester tells the fuzzer how to interact with the contract. If we stick to the previous example, the fuzzer will initiate a token transfer.
- Combining Flow and Invariants: Manually Guided Fuzzing combines random flows to create a more efficient fuzzing process. Each flow is subject to all invariant checks. This method allows testers to focus on specific areas of the code instead of randomly exploring the state space, which significantly reduces the time and computational resources required.
(Dis)advantages of Manual Guide Purging
- efficiency: Manually guided fuzzing significantly reduces the state space explored by directing the fuzzer toward specific flows and properties, thereby reducing the time required to find bugs.
- pliability: Testers have full control over the fuzzing process, allowing them to test specific scenarios that are difficult to handle with black-box fuzzing or formal verification, such as state interactions, cross-chain transactions, and complex contract dependencies.
- With great power comes great responsibility.: The tester must identify potential red flags in the code and decide to cover them with fuzz testing. Typically, the tester defines the flow for all public state-changing functions and selects the correct input values for the functions. If the tester misses red flags or has no idea of the attack vector, the fuzzing campaign will not find the vulnerability.
conclusion
Manually guided fuzzing represents a shift in responsibility from testers using brute force or heuristic algorithms to auditors and security researchers who must guide fuzz testing with attack vectors to find vulnerabilities. Manually guided fuzzing provides a more efficient, accurate, and scalable way to test complex systems such as highly integrated smart contracts. Explore the techniques mentioned using the Wake framework, which supports manual guided fuzzing, fork testing, and differential testing.
Additional Materials
The following examples demonstrate practical uses of different types of fuzz testing.