Solidity Optimizer and ABIEncoderV2 bug announced
Through the Ethereum bug bounty program, we have received reports of a flaw in a new experimental ABI encoder (called ABIEncoderV2). Our investigation revealed that the component has several variants of the same type. The first part of this announcement details this bug. The new ABI encoder is still marked as experimental, but since it’s already in use on mainnet, we think it’s worth announcing prominently.
Additionally, two low-impact bugs have been discovered in the optimizer over the past two weeks, one of which is fixed in Solidity v0.5.6. Both were introduced in version 0.5.5. Please see the second part of this announcement for more details.
that much 0.5.7 release Includes fixes for all bugs described in this blog post.
Any bugs mentioned here should be easily visible in tests that cover the relevant code path, at least when executed with all combinations of 0 and non-zero values.
Thanks to the Melonport team (Travis Jacobs and Jenna Zenk) and the Melon Council (Nick Munoz-McDonald, Martin Lundfall, Matt di Ferrante, and Adam Kolar) for reporting this issue through the Ethereum bug bounty program!
Who should worry?
If you have deployed contracts that use the experimental ABI Encoder V2, those contracts may be affected. This means that only contracts that use the following directives within their source code can be affected:
pragma experimental ABIEncoderV2;
Additionally, there are several requirements for a bug to occur. See technical details below for more information.
As far as we can tell, there are about 2500 contracts on the mainnet using the experimental ABIEncoderV2. It’s unclear how many of them contain bugs.
How to tell if a contract is vulnerable
The bug only appears if all of the following conditions are met:
- Storage data associated with an array or structure is transferred directly to an external function call. abi.encode OR AND to event data without pre-assignment to a local (memory) variable
- You have an array that contains elements that are less than 32 bytes in size, or you have a structure that has elements that share members of a storage slot or type. ByteNN Less than 32 bytes.
Additionally, your code will not be affected in the following situations:
- If any struct or array uses: Unit256 or integer256 type
- If you only use integer types (possibly shorter) and encode at most one array at a time.
- If you just return that data and don’t use that data abi.encodeExternal call or event data.
If you have a contract that meets these conditions and would like to determine whether it is in fact vulnerable, you can contact us via: security@ethereum.org.
How to avoid this type of defect in the future
To be conservative about changes, experimental ABI encoders are only available when explicitly enabled, allowing people to interact with and test them without placing too much trust before they are considered stable.
We are doing our best to ensure high quality and have recently started working on ‘semantic’ fuzzing of certain parts. OSS-Fuzz (We have crash-fuzzed the compiler before, but have not tested compiler correctness).
For developers, bugs within the Solidity compiler are difficult to detect with tools such as vulnerability detectors because tools that operate on source code or AST representations do not detect defects introduced only in the compiled bytecode.
The best way to protect against these types of defects is to have a rigorous end-to-end test suite for your contracts (checking all code paths). This is because bugs in the compiler are not “silent” and are more likely to show up as incorrect errors instead. data.
possible outcome
Naturally, any bug can have very different consequences depending on the program control flow, but this is expected to be more likely to lead to malfunction than exploitability.
The bug occurs when, under certain circumstances, a method call sends corrupted parameters to another contract.
timeline
2019-03-16:
- Report corruptions that occur when reading Boolean arrays directly from storage with the ABI encoder via the bug bounty.
From March 16, 2019 to March 21, 2019:
- Root cause investigation, analysis of affected contracts. An unexpectedly large number of contracts compiled with experimental encoders have been found deployed on the mainnet, many of which do not have verified source code.
- After investigating the bug, we found more ways to cause the bug, including using structs. Additionally, an array overflow bug was discovered in the same routine.
- We checked a small number of contracts found on Github and did not find any contracts affected.
- Bugfixes have been made to the ABI encoder.
2019-03-20:
- We have decided to make the information public.
- Corollary: It is impossible to detect all vulnerable contracts and contact all authors in a timely manner, and it would be good to prevent further proliferation of vulnerable contracts on the mainnet.
2019-03-26:
- New compiler release, version 0.5.7.
- This post has been made public.
technical details
background
Contract ABI is a specification for how data can be exchanged with a contract externally (Dapp) or when interacting between contracts. It supports many types of data, including simple values such as numbers, bytes, and strings, as well as more complex data types including arrays and structures.
When a contract receives input data, it must decode that data (this is done by the “ABI Decoder”) and encode it (this is done by the “ABI Encoder”) before returning the data or sending it to another contract. The Solidity compiler generates two code snippets for each function defined in the contract: abi.encode and abi.decode). The subsystem that generates encoders and decoders in the Solidity compiler is called the “ABI encoder”.
In mid-2017, the Solidity team started working on a new implementation called “ABI Encoder V2” with the goal of having a more flexible, secure, performant, and auditable code generator. This experimental code generator has been available to users since the 0.4.19 release in late 2017, if explicitly enabled.
flaw
The experimental ABI encoder does not properly handle non-integer values shorter than 32 bytes. This applies to: ByteNN category, boolean, enumeration Other types that are part of an array or structure and encoded directly in the store. This means that these repository references must be used directly internally. abi.encode(…), used as arguments for external function calls or event data without prior assignment to local variables. use return It doesn’t cause bugs. category ByteNN and boolean Doing so may damage your data. enumeration may lead to invalidation going back.
Additionally, arrays with elements shorter than 32 bytes may not be processed correctly, even if their underlying type is an integer type. Encoding such an array in the manner described above may cause other data in the encoding to be overwritten if the number of encoded elements is not a multiple of the number of elements that fit in a single slot. In encoding, if the array is followed by nothing (a dynamically sized array is always encoded after a statically sized array with statically sized content), or if only a single array is encoded, no other data is overwritten.
Two bugs were discovered in the optimizer that were not related to the ABI encoder issue described above. Both were introduced in 0.5.5 (released March 5). This problem is unlikely to occur in compiler-generated code unless inline assembly is used.
These two bugs were confirmed with the recent addition of Solidity. OSS-Fuzz – A security toolkit for finding inconsistencies or problems in various projects. For Solidity, we’ve included several Fuzzers that test different aspects of the compiler.
- The optimizer replaces the opcode sequence as follows: ((x << a) << b))where all and rain This is a compile-time constant. (x << (a + b)) Failure to properly handle overflow in addition.
- The optimizer handled it incorrectly. byte The opcode when the constant 31 is used as the second argument. This can happen when performing index access. ByteNN If the compile-time constant value (not the index) is 31, or if you use a byte opcode in inline assembly.
This post was co-written by @axic, @chriseth, and @holiman.