Flow is an open, decentralized platform that provides the speed, scalability, sustainability, and safety needed to take blockchain technology mainstream.
The vision for Flow is to onboard billions of consumers on Web3. To realize such a goal requires a solid foundation at the node software level, infrastructure level and governance level. The last Flow update is a big step toward achieving this goal. It introduces several new features to improve the network scalability, performance, resilience and accessibility while making it more secure and has been several months in the making with several contributions from the Flow community members like NCC, CoinbaseCloud, BlueSign and others
Let’s explore each of these major changes and see how they take Flow closer to realizing the vision.
20% faster transaction finalization
Agreeing on a shared state is a fundamental challenge in any highly-distributed system, including blockchains. In the case of Flow, this responsibility of generating agreement a.k.a consensus, is on the consensus nodes, which currently do this by running a consensus protocol known as HotStuff. While the current implementation has served Flow well, it has some shortcomings in terms of its performance and its ability to handle situations when some consensus nodes in the network are down.
In this spork, Flow switched to using the Jolteon protocol, which is a substantially improved derivative of HotStuff (see). This new consensus protocol will result in consensus nodes reaching agreement 33% faster or, in other words finalizing blocks 33% faster. Jolteon achieves this speedup by finalizing blocks with only two child blocks (aka 2-chain finalization rule) on the happy path (as opposed to requiring 3 children for finality, aka 3-chain finalization rule, in the original HotStuff), directly resulting in 33% faster block finalization.
For a transaction to be included in a finalized block, measured as the time to finality for a transaction, there is some added latency because it has to be first submitted by an access node to the collection node, get included in a collection by the collection nodes and then finally the collection needs to be included in a block by the consensus nodes using the consensus protocol. Hence, the new implementation will translate to about a 20% improvement in the time to finality for a transaction. After a transaction has been finalized, it still needs to be executed by the execution nodes, results need to be verified by the verification node, and then consensus nodes need to seal the block, which includes the transaction. Faster finalization means transactions move faster through the pipeline and will improve the time it takes to seal a transaction.
The new implementation is also currently live on testnet, and as can be observed in the graph below, the time to finalization has improved drastically.
The new implementation will also result in fewer orphaned blocks, which should reduce cases where transactions are taking unusually long to be included in a finalized block. Moreover, Jolteon improves Flow’s censorship resilience, where malicious nodes try to orphan blocks whose transaction content they don’t like (for details, see analysis of ‘Chain Quality’ in for the 3-chain finalization rule, which improves with the 2-chain finalization rule).
Improved block production resilience
The new consensus implementation will also be far more resilient to consensus nodes in the network being down (offline), which is inevitable in a decentralized distributed system. Hotstuff as well as Jolteon are round-based algorithms, where in each round there is one consensus node designated as a leader to propose a block. If the leader proposes a valid block then other consensus nodes vote on it; if the vote passes, the block is accepted as a tentative extension of the chain and the next round ensues with a new leader. However, if the leader in a round is offline (or malicious), other nodes simply wait for the leader to propose a block, wasting time until they timeout and realize that the leader node is down. The new implementation introduces what is known as the Active pacemaker that helps nodes actively coordinate among themselves and decide if they want to abandon a leader without extensive waiting. This makes block production much more resilient to consensus nodes being down.
This also helps expedite the spork process. In the past, even when the majority of the consensus nodes were up after a spork, they were out of sync since they were not started concurrently and thus couldn’t come to a consensus, resulting in no blocks being produced by the chain till they all eventually synced up. This caused delays in spork completion. With the implementation of an Active Pacemaker, even if nodes come online out of order, they will sync up much faster since they can coordinate better.
Active Pacemaker also allows for much more aggressive tuning of consensus in the future to maintain performance even if a noticeable fraction of consensus nodes are offline. Furthermore, it unlocks a completely byzantine-fault-tolerant Epoch switchover.
Laying the foundation for hyper scaling the network to petabytes of on-chain storage
Flow will be a home to a billion users and it will achieve this by eventually supporting petabytes of on-chain data without sharding - a feat not achieved by any Layer-1.
This spork includes updates to the Execution node stack that are geared towards this goal of petabyte-scale of on-chain storage as well as optimistic concurrent transaction execution.
This sets the stage for two major future goals:
- Refactoring of the storage layer, with offloading data to disk, to enable petabyte-level on-chain storage while reducing the memory requirements to actually run an execution node making it more affordable for a node operator of any scale to run it.
- Parallel execution of transactions on an execution node to improve network throughput by several orders of magnitude.
The current set of changes already improve execution node performance by 20% resulting in an average throughput of 400TPS.
Improved protection against spamming with early account validation
This spork adds spamming protection to the network against accounts sending transactions that they cannot pay for. Earlier, if a transaction with insufficient funds to pay the fee was submitted, it executed and then failed at the end of the execution during the fee deduction, wasting resources. This has now been changed so that the payers balance is checked before executing the transaction’s body protecting the chain liveness against such spam attacks and helping Flow inch close to a 99.99% SLA goal.
Support for permissionless access nodes
Permissionless access nodes are coming soon and will enable any dApp developer who wants an exclusive non-shared access node to run a private access node. This spork marks the culmination of an extensive effort from past several months to update some of the core components of the node software to support permissionless access nodes building.
At the very core, these changes included updating the networking layer to make it more Byzantine Fault tolerant (BFT) such that if there is a BFT node in the network, the other nodes will reject traffic from that node. However, it also encompassed submitting several changes to some of the open-source libraries used at the networking layer to ensure that there are no vulnerabilities introduced by using those libraries.
The updated node software was also audited by one of the leading web3 security auditor - NCC, and the issues identified in the audit were addressed.
With this, the node software is now extremely resilient to networking layer attacks from byzantine actors that may join as a permissionless access node.
Cadence v0.30: Dynamic Contract Import and Account Inbox
This spork will update the Cadence version on mainnet to v0.30. This new Cadence version introduces two major features and twelve performance improvement changes.
Dynamic contract import
This feature enables dynamic contract imports from an address, making it possible to use patterns common to Solidity developers in Cadence. This is an alternative to importing contracts statically and improves resiliency as the import failures can be handled more gracefully.
⭐ This is a community-driven and community-implemented feature by Bluesign.
- FLIP // Pull Request
API for Capability Bootstrapping
A new feature to simplify sharing capabilities with other accounts in a single-signer transaction, replacing the old model requiring both sender and receiver to sign the transaction sharing capability.
- FLIP // Pull Request
- More on this here.
Improved network availability with faster and fewer network upgrades going forward
A spork does entail some network downtime. To improve the network availability, the spork process has been optimized as much as possible. This time the spork was completed in 60 minutes but going forward sporks that do not involve a state migration should complete within 30 minutes. If a spork involves state migration, then it should complete in approximately 45 minutes, depending on the type of migration (state migration is the process of converting existing data to the new data model). The spork cadence (no pun intended 🙂) will also be changed to one spork every quarter instead of having it bi-monthly.
However, to ensure that the pace of innovation is not slowing down, two new ways of continuous deployment will be introduced – rolling upgrades and height coordinated upgrades.
Rolling upgrades would allow node operators to update the node software without any network downtime. Height coordinated upgrades would also require updating the node software, but the changes will take effect at a particular block height requiring an execution node restart which could result in transaction execution pausing for a few minutes.
While we try to avoid downtime as much as possible, these upcoming network upgrades will make Flow more performant, secure and reliable. We look forward to continuing to work hand-in-hand with you to achieve Flow’s full potential.
Onwards and upwards! 🚀
