Full Report
Juno is a proof of stake based blockchain built on Cosmos and CosmWasm. While going through an upgrade of the eco-system, a bunch of issues were occurring... forks, lack of consensus and many other things. What was going on? In 2021 the Terra blockchain was dealing with the same issue. What was the problem? Different versions of libc were outputting different gas calculations. The solution? Statically link the same libc version into the binary so that everyone can use it. This didn't work though. Hmmm. At this point, they had attempted to role out the static version of the binary but this didn't work. They stopped the chain, told everybody to stop and decided to triage and analyze what was really going on. Sometimes, taking a step back is the best way to move forward. The author of this post decided to review the pending transactions of their own validator's memqueue. While doing this, they noticed a weird transaction that was a call to a CosmWasm contract with some indirection. When they started reviewing the state, they noticed that the chain halted while changing the name. Why did this occur? Every single node that the operator queried had a different name field. In particular, it had split into 125 different networks! This was at the fault of CosmWasm and led to them not reaching consensus. This vulnerability is known as indeterminatism; this is when the operations being performed do not have a deterministic path, leading to rapid forking. This is not the end of this though. More issues were still occurring. While trying to get a node up and going, they kept going down with a very similar looking transaction. It was another known CosmWasm non-deterministic bug. The bug a non-deterministic ABCI query being made from a CosmWasm smart contract. Overall, an interesting lesson on non-deterministic behavior on blockchain drastically affected the software. Thanks for the good write up!
Analysis Summary
# Incident Report: The Juno Network Indeterminism Crisis
## Executive Summary
The Juno blockchain experienced a critical network failure characterized by massive chain forking and a loss of consensus due to non-deterministic execution within the CosmWasm environment. The incident was triggered by specific smart contract transactions that produced different outputs across various nodes, resulting in the network splitting into 125 distinct versions. Recovery required a complete chain halt, manual mempool analysis by validators, and patches to the CosmWasm engine.
## Incident Details
- **Discovery Date:** Not explicitly dated (refers to a 2022-era upgrade cycle)
- **Incident Date:** During a scheduled ecosystem upgrade
- **Affected Organization:** Juno Network
- **Sector:** Blockchain / Decentralized Finance (DeFi)
- **Geography:** Global / Distributed
## Timeline of Events
### Initial Access (Exploitation)
- **Date/Time:** During the Juno ecosystem upgrade.
- **Vector:** Deployment/Execution of a malicious or buggy CosmWasm smart contract.
- **Details:** A transaction was submitted to a CosmWasm contract using indirection, which triggered non-deterministic behavior across the validator set.
### Lateral Movement
- **N/A:** As a blockchain incident, the "movement" was the propagation of the non-deterministic transaction across the peer-to-peer network, affecting all validating nodes simultaneously.
### Data Exfiltration/Impact
- **Consensus Failure:** The chain effectively "halted" because validators could not agree on the state.
- **Network Fragmentation:** The blockchain split into 125 different networks as nodes calculated different state transitions for the same block.
### Detection & Response
- **Detection:** Validators noticed forks, lack of consensus, and varying "name" fields in the state when querying different nodes.
- **Response actions taken:** The chain was officially stopped. Validators were instructed to cease operations to allow for triage. An analyst reviewed their validator's mempool to identify the problematic transaction.
## Attack Methodology
- **Initial Access:** Smart Contract Interaction (CosmWasm).
- **Persistence:** N/A (The "attack" persisted via the blockchain ledger until the chain was halted).
- **Discovery:** Mempool analysis by node operators.
- **Impact:** **Indeterminism.** By leveraging operations that yield different results on different systems (e.g., libc variations or non-deterministic ABCI queries), the attacker forced the network to lose synchronicity.
## Impact Assessment
- **Financial:** Significant, due to the inability to trade or move assets during the halt; potential impact on JUNO token price.
- **Data Breach:** None (Integrity and Availability failure, not Confidentiality).
- **Operational:** Total business disruption; the blockchain was unusable for the duration of the triage and patching process.
- **Reputational:** High; raised concerns regarding the stability of the Cosmos/CosmWasm ecosystem during upgrades.
## Indicators of Compromise
- **Behavioral indicators:**
- Rapidly diverging block hashes between nodes.
- Nodes stuck on a specific block height.
- Differing "name" fields or state variables across node queries for the same height.
- **Transaction Pattern:** A call to a CosmWasm contract involving indirection and an ABCI query.
## Response Actions
- **Containment measures:** Full network shutdown (coordinated validator halt).
- **Eradication steps:** Identification of the specific transaction causing the fork; debugging the CosmWasm engine's handling of non-deterministic queries.
- **Recovery actions:** Implementation of static linking for `libc` (initially) and subsequent patching of the non-deterministic ABCI query bug. Nodes were then restarted on a unified version.
## Lessons Learned
- **Deterministic Rigor:** Blockchain environments must ensure that every operation, down to the system library level (`libc`), is identical across all environments.
- **Observability:** Deep mempool inspection is vital for validators to identify "poison pill" transactions that bypass standard testing.
- **Complexity Risks:** Indirection in smart contracts can hide malicious or buggy behavior that leads to catastrophic consensus failure.
## Recommendations
- **Static Linking:** Ensure all blockchain binaries are statically linked to avoid environmental discrepancies between OS distributions.
- **Validation Sandboxing:** Enhance the CosmWasm sandbox to strictly forbid non-deterministic calls (like certain ABCI queries) during state transitions.
- **Incident Response Drills:** Maintain clear communication channels for validators to coordinate emergency halts and state exports.