Back in November I wrote about an unintentional chain split on the Ethereum network caused by a collection of clients that had not applied critical security updates. We followed this up with a call for a Long Term Service (LTS) release of Geth, even offering to maintain the branch ourselves. Unfortunately, the Geth team has refused to support any kind of LTS release.

The Geth team’s position comes from a concern that announcing security vulnerabilities highlights them for attackers, potentially inviting attackers to exploit those vulnerabilities faster than nodes on the network can upgrade to mitigate the risk. They worry that even by merging security fixes into a support branch, they would be highlighting potential attack vectors that a savvy attacker could exploit before most of the network has a chance to apply the patch.

Vulnerability Lifecycle Modeling

In 2010, I wrote my master’s thesis on vulnerability life-cycle modeling, exploring in great detail the concepts that are relevant to this question today. There are two key variables that are fundamental to the decision of how to handle patch distribution:

• t_exploit: The time it takes an attacker to identify the vulnerability, develop an exploit, and execute it.
• t_patch: The time it takes the network to reach a critical mass of nodes that have applied the patch.

When t_exploit < t_patch, the attacker is able to wreak havoc on the network by exploiting a vulnerability that is predominantly unpatched. When t_patch < t_exploit, the attacker is mostly ineffective, as their attack is ineffective against the patched nodes.

From here, making decisions on release processes can be couched in terms of their impact to these two variables. Ideal actions would increase t_exploit while decreasing t_patch. Actions that decrease t_exploit while increasing t_patch pose serious risks. Actions that move t_exploit and t_patch in the same direction should consider the scale of each of these moves.

Geth’s Current Approach

Considered in the framework of this vulnerability lifecycle model, the Geth team’s concerns are not entirely without merit. Certainly, making a full vulnerability disclosure the instant an update is made available will exert more downward pressure on t_exploit than it does on t_patch. Even if node operators are going to apply updates “quickly”, Ethereum operates around the planet, and it’s likely that ⅓ of node operators are asleep at the moment of the release. The vulnerability disclosure might spur some node operators to apply updates on an accelerated schedule, but if a vulnerability disclosure spells out an attack in detail, an attack could quite plausibly be executed before many node operators are even awake to react.

The Geth team’s current approach of not announcing vulnerabilities immediately, combined with silently mixing security fixes in with new features and optimizations, succeeds at putting upward pressure on t_exploit—as potential attackers have to dig through large code updates (which may or may not include security fixes at all) before they can even begin to develop an exploit.

Unfortunately it also puts significant upward pressure on t_patch, as organizations that run nodes must go through rigorous testing cycles (and sometimes development cycles) to make sure the new features don’t break anything in the downstream application (and update the application when they do). Depending on how you view the “critical mass” component of t_patch, the November Infura outage is a datapoint that puts t_patch on a scale of many months.

The LTS Approach

Having an LTS branch of Geth would put downward pressure on both t_exploit (which is bad) and t_patch (which is good).

The downward pressure on t_exploit comes from a reduction in the number of commits an attacker would need to sift through to identify potential security issues. Rather than having to sift through a mix of features, optimizations, bug fixes, and critical security fixes, attackers following the LTS branch would only have to sift through bug fixes (which may not always be security related) and security fixes. They would still need to take time to determine whether a vulnerability exists and how to exploit it, which leaves more time for users to upgrade than making a vulnerability disclosure along with the release, but plausibly less time than if attackers must sift through a larger volume of code changes.

The real value of an LTS branch is the downward pressure on t_patch. After the November Infura outage, Infura went on record saying

In the early days of Infura we would upgrade nodes as soon as the Geth or Parity teams cut a new release. We wanted the latest performance improvements, the latest API methods, and of course bug fixes. We stopped doing that when these changes sometimes brought instability or critical breaking issues which negatively impacted our users.

It follows that they would have updated much sooner if they could have had confidence that an update would bring bug fixes without instability and critical breaking issues that negatively impact their users.

LTS releases or stable branches are standard practice throughout the software industry for exactly this reason.

If updates are very reliably stable, businesses will generally apply those changes quickly, on a regular schedule. For a piece of software that has very reliably stable updates, an update that comes out on a Monday may be applied by Tuesday or Wednesday, especially if it is generally understood that updates on the stable branch are generally critical bug fixes.

If updates tend to cause large, breaking changes, organizations will schedule an upgrade process every couple of months, to allow for engineering resources to make the necessary changes to accommodate the change. If such an update comes out on a Monday, businesses are unlikely to pull resources from planned activities to evaluate and make any necessary changes until the start of the next sprint, likely weeks later at the soonest (especially if the organization has no reason to believe the release contains security updates).

One last factor that works to the benefit of t_patch from an LTS is that it creates a level of node diversity. If a bug is introduced, caught, and fixed between LTS releases, users running the LTS version of software were never exposed to the bug in the first place. Similarly, it is also possible that an undiscovered security bug is fixed inadvertently during a refactor to support a new feature, in which case the security fix may only be applicable to LTS users. In either of these cases, it is easier to reach “critical mass” adoption when only a subset of users were exposed to the vulnerability at the time of its discovery.

The Impact of Network Value on t_patch and t_exploit

Here, we find a factor that impacts both t_patch and t_exploit which is largely outside the control of any release process decisions a software team can make.

At the time of this writing, ETH alone has a market cap of over $195 billion. When you account for other tokens on the network, you easily get into a valuation in the range of several hundred billion of dollars. So how does that impact our vulnerability life cycle model?

First lets look at t_exploit. The more value a network holds, the more value can be derived from causing it to falter. In recent weeks we’ve seen financial exploits outside the crypto space. Vulnerabilities in Ethereum clients open up opportunities for similar financial exploits.

Imagine a hedge fund or other financial institution spending a million dollars a year employing a team of security researchers to track Geth and watch for fixed vulnerabilities. When their team finally finds something, they take out a massive short position on ETH and related tokens, then exploit the vulnerability. Even today, if they could use such an approach to extract just 0.5% of the network’s value it could yield 1000x return on the security research teams’ salaries. As the value of the network increases, so does the potential return from such an attack. Employing a team capable of closely tracking an LTS release might be 10x cheaper than employing a team capable of tracking every commit to Geth itself, but even if that’s not cost effective today, it will be if the value of the network scales the way many of the network’s believers are confident it will.

Even without a single organization that invests heavily in finding Geth bugs for the purpose of exploitation, Linus’ Law states:

Given enough eyeballs, all bugs are shallow

Applying this to Geth, as the Ethereum ecosystem continues to grow, Geth will attract more eyeballs and more opportunities for people to find exploitable bugs—especially by tracking patches.

Then juxtapose that to the impact that an increasing network value has on t_patch. As the value on the network increases, we can expect more enterprise adoption — more businesses that will need to incorporate client upgrades into their business cycle, taking longer than existing users to apply updates, especially if those updates have a tendency to come with breaking changes.

So as the Ethereum network increases in value, we can reasonably expect t_exploit to fall while t_patch can be expected to rise — a recipe for successful attacks. Given the value at stake, such an attack would be sure to attract the attention of regulators who are unlikely to be placated by claims that “most nodes have indeed updated and were not affected” while substantial losses were incurred by major institutions on a small segment of the network that had not updated.

Conclusions

As the value of the Ethereum network continues on an exciting upward trajectory, client teams need to think critically about both t_patch and t_exploit. For sufficiently well resourced attackers, the difference between t_exploit under Geth’s current strategy and t_exploit under an LTS strategy becomes negligible, while the difference between t_patch improves considerably under an LTS strategy vs the current one.

Our team is still amenable to helping the Geth team maintain an LTS release, but even if they aren’t interested in our help, we hope they will reach the conclusion that an LTS is necessary before the lack of one leads to a major disaster for the ecosystem.