Better Stacks Blockchain Status Updates + Post-Mortem Roundup
by Foundation Team on October 10, 2022
In light of recent security reports via our Immunefi program, as well as congestion issues related to .BTC registrations, we took the opportunity to revisit ownership and processes related to detecting and communicating about the status of the Stacks blockchain. To start, we worked with other key infrastructure providers, Stacks entities, and the Stacks Core Developers to improve and streamline communications during incidents. We also kicked off efforts to further harden this area so that we can be sure the community always has important, timely information related to the status of the Stacks blockchain.

We build on Bitcoin because we believe it is the most secure, robust foundation for all that Web3 promises. As a layer for Bitcoin, Stacks heavily values attention to security in everything from the Bitcoin-linked consensus mechanism (PoX) to the Clarity smart contract language. As such, security is always something we’re monitoring and seeking to improve — it’s an evolving effort that grows alongside the ecosystem.

We acknowledge that during the last period of major congestion, the speed of communication around the status of the blockchain was not adequate. As the user base and audience of Stacks grows, we can’t rely on folks knowing where to look for updates as the ecosystem has been able to get away with in the past. Having a clear protocol is especially key in a decentralized ecosystem so that important activities like this don’t suffer from the ‘tragedy of the commons’ while everyone juggles multiple high-priority workstreams.
New resources and improvements
  • 1
    On-call process
    First, we connected with all the parties and core devs involved in recent incidents and firmed up our processes and communication channels around incidents. Part of the issue was a misunderstanding of who was taking lead, so we’ve updated things to ensure that doesn’t happen. Second, we are tentatively planning to establish a dedicated on-call resource at the ecosystem level. We’re unsure of the form that will take at this time but are exploring - feedback or volunteers are welcome.
  • 2
    Stacks Status Twitter account
    You may have noticed a new Twitter account: StacksStatus. The purpose of this account is provide network status updates regarding the Stacks blockchain layer. This account will identify when there’s a bug or incident related to the blockchain and provide updates as they become available. Importantly, this does not cover issues with 3rd party services such as APIs - we recommend you follow the companies behind those for status updates. If a provider reaches out to us to get the word out about an outage, we’ll certainly amplify it, but the account will primarily focus on the blockchain itself.
  • 3
    Blockchain status page
    Taking things a step further, a more detailed page we’ve already begun building will provide insight into the state of the blockchain by surfacing key metrics and statuses. Work on a portal like this has consistently been de-prioritized for other work given the relative infrequency of major issues, but as we grow this is a must-have in putting Stacks’ best foot forward. We expect to launch an initial version of this page in Q4.
  • 4
    Immunefi viewers
    A program we’ve run since March 2022 is the Immunefi Bounty Program. Immunefi has emerged as a leader in this space and brings talented researchers into the mix to review critical code. At the Stacks Foundation, we’ve been mostly handling this program alone, but want to avoid being a bottleneck to timely investigation by Core Devs. In the past couple of weeks, we’ve been able to add key contributors to the system so they can receive reports, ultimately resulting in more eyes on these issues, faster.
Post-mortem roundup
Now, let’s dive into a couple recent incidents - we want everyone that is interested or has a vested interested in the Stacks blockchain to have information like this available. For any major incidents, we’ll continue doing post-mortems and we’ll also be shifting priorities some to make sure we are publishing (or assisting others in posting) those happens faster.
Denial-of-service attack vector, upgrade to 2.05.0.3.0
Researchers from the Immunefi Bug Bounty program uncovered a denial-of-service vulnerability that was notably concerning because an exploit in the wild could have led to a chain split. Ultimately, this required that the majority of mining power upgrade before any exploitation took place. After flagging by researchers, this bug was validated by Stacks core developers and addressed through an upgrade. The bounty program ensured that this exploit was disclosed responsibly to the Stacks Foundation. Our team was able to rally contributors to identify a solution, and communicate to miners, exchanges, and others to roll out the upgrade swiftly.

This example affirms the system that we rely on. We’re extremely grateful to the researcher, everyone who contributed to the resolution, and the many integrators that quickly upgraded their software. You can review the complete post-mortem here.

BTC/Mempool Congestion
Recently, Stacks users experienced significant congestion on the network, resulting in confusion as to what the core issue was and general fear/uncertainty. Unfortunately, getting this congestion acknowledged and addressed was a slow process, making an already frustrating situation all the more difficult. We believe many of the changes and new resources outlined above will address that going forward and eliminate needless frustration and lack of clarity.

As for the underlying issue of network capacity and speed, Hiro’s Subnets are a solution in the works. They should help immensely with ‘bursty’ traffic and Core Devs have also discussed other potential ways of speeding up the main chain. We encourage anyone that wants to share needs or be part of solutions to join the open calls.
What's next?
These initial new resources will improve experiences, but we also know that these two efforts won’t immediately resolve all of the challenges associated with building on Bitcoin, so we encourage any/all feedback and ideas as we aim to build in the support that builders need - particularly as we all get ready for even more growth with the launch of 2.1.

Get involved:
Drop in your email to stay in the loop on Stacks Foundation news