print

The Ascent of the Core Infrastructure Initiative

How a bug created a superfund for cash-strapped open source projects

It’s not an uncommon occurrence for programmers to edit code that they don’t fully understand, so it wasn’t surprising that in 2008, Debian announced that they had managed to break the OpenSSL’s pseudo-random number generator (RNG) for the Debian Linux distribution—the root cause of which had happened two years earlier. The broken RNG limited OpenSSL keys to 32,767 different possible keys (of a certain kind) so that several people actually shared the same keys. The problem was reported when someone was using the Debian-based RNG to generate several prime numbers and noticed that he was getting repeats far too often.[i] Limited to the Debian, the bug and its two-year run were a result of the humanness of programmers and how difficult it is to regularly be involved with code that you didn’t create and do not fully understand. Even the best programmers make mistakes.

Figure 1: The section of a URL that includes “https://” (which stands for Hyper Text Transfer Protocol Secure) indicates that the website is secured with an SSL certificate. SSL creates an encrypted channel between the user’s computer and the web server.

Figure 1: The section of a URL that includes “https://” (which stands for Hyper Text Transfer Protocol Secure) indicates that the website is secured with an SSL certificate. SSL creates an encrypted channel between the user’s computer and the web server.

The completely free and open source Debian is one of the earliest distributions based on the Linux kernel and is used as the starting point for several other distributions. Open source code has become deeply ingrained in the fabric of a great deal of the software used to run the world as we know it, and the computing industry has become reliant upon shared code in the open source model. On the whole, open source development has provided highly secure software, and software, in general, has evolved into many complex, interoperable, and inter-dependent systems. The open source model is lauded by many and for good reason.

Not that long ago, open source communities were still skeptical of companies’ motives in open-source. Companies distrusted open source because they didn’t understand why developers spent hours coding and then gave it away for free. Companies were skeptical of participating in open source software because they would not be able to make money off something given away for free, and nothing like the open source model had been tried on a large scale in a business. Akin to the ideology of communism, the open source model provided no system of tangible compensation for its workers. Companies were also concerned about the legal ramifications of using open source software, even though most of it was protected by the GNU Public License (GPL). The desktop-friendly GPL had not been tried in courts regarding use in embedded products. Around 2005, semiconductor companies started using royalty-free Linux as board support packages for development boards, and within a handful of years, going so far as to hire full-time employees to maintain branches for their own architectures. Linux was perfect for booting development boards fresh out of the box at minimal cost to chip OEMs. Linux steadily worked its way into the embedded world. Companies initially leery of the open source model became its biggest supporters.

The Human Factor
Open source began as a complete volunteer effort and for some, it still is. But as companies have engaged in contributing paid full-time developers to open source projects, paid developers can invest 40 hours a week or more, get sent to conferences for training, and increase diversity in open source projects. Junior developers are hired to work full time on open source projects while ramping up to becoming contributors. Volunteers working in their spare time do not have as much time as paid developers to address questions or issues. Companies are more mindful of legalities, and some require a Contributor License Agreement (CLA) in order for people to contribute to their projects, partly so that they can ensure that the person has the right to submit a contribution.

Linux has been the best-known example of an open source project for a couple of decades and has withstood the test of time to become a critical foundation in a wide range of systems. However, human nature is also part of the open source model, and even critical projects can be left to wither on the vine with little attention, funding, or support. One example of critical open source software that was a major cog in the secure running of the internet, and especially e-commerce, was OpenSSL. OpenSSL is an open version of security software technology for secure sockets layer (SSL) and transport layer security (TLS) protocols which encrypts the data that users send to websites. The OpenSSL website identifies OpenSSL as a “cryptography and SSL/TLS toolkit.”[ii] To understand the ubiquity of SSL and TLS (open or not), the section of a URL that includes “https://” (which stands for Hyper Text Transfer Protocol Secure) indicates that the website is secured with an SSL certificate. SSL creates an encrypted channel between the user’s computer and the web server.


Figure 2: Secure sockets layer encrypts what users send to websites so that only the receiving website and the sender are able to unencrypt it. (Credit: Author)

Figure 2: Secure sockets layer encrypts what users send to websites so that only the receiving website and the sender are able to unencrypt it. (Credit: Author)

The OpenSSL Project began in 1998. In early 2014 there were two main developers working as full-time volunteers on the project; all of the contributing developers were volunteers. Open source projects typically have one or more full-time developers who maintain the code. The maintainers understand the code more thoroughly than others and are ultimately responsible for the state of the code, overseeing code contributions, and making final reviews before approving contributions to the project.  Maintainers typically have the majority share in guiding the direction of the code, project priorities, and even housekeeping. In a basic scenario, a process for validating code for acceptance starts when contributors send a patch in for review. The reviewer thoroughly checks the changes, compares code using diffs and so forth, and comments as needed. The commented code is sent back. The contributor then addresses any issues put forth in the reviewer’s comments, and this goes back and forth until the reviewer gives final approval. At this point, the contributor can commit the code to the trunk. This is a simplified explanation, but open source code is typically reviewed several times by different developers before a commit is accepted. This is part of why the open source model is seen as successful and is proliferating.

However, a prime example of a very visible open source failure is the appearance of Heartbleed, which hit e-commerce on the internet in April 2014 by compromising security for web users. Heartbleed exploited a memory handling bug in OpenSSL version 1.0.1, going unnoticed for three years, that allowed one to read a web application’s memory, including the sender’s private key. A fix was issued almost immediately, but it took time for the patch to filter out through the millions of websites on the internet. According to IBM, “Companies that maintained accurate asset databases and incident response plans were able to rapidly deploy patches on vulnerable systems.”[iii] Those who did not maintain tabs on their assets did not know which of their systems were vulnerable and which ones were critical. In retrospect, developers call that day in April “the day we had to re-key the internet”[iv] as companies had to revoke and reissue digital certificates that had been granted since the vulnerability was first introduced on December 31, 2011.

How did this happen?
Funding for the OpenSSL project was $2,000 for the entire year before Heartbleed was discovered. No major companies were funding OpenSSL, yet active web servers using open source Apache and ngnix did use OpenSSL and together held a 66% market share in web servers.[v] Perhaps it was human nature to take OpenSSL for granted while using it to make money, but few were aware of the fact that it was underfunded, understaffed, and so prevalent. The original archaic code in OpenSSL was difficult to understand, and the developers working on it were over-committed. Much like the roads and bridges that we take for granted daily, OpenSSL was widely used in the infrastructure of the internet and it was used often. The Heartbleed bug that compromised security so well was the impetus to make significant changes, however.

Figure 3: In April of 2014, when the Heartbleed bug was announced, the web servers using open source Apache and ngnix used OpenSSL and together held a 66% market share in active web servers. (Credit: Netcraft)

Figure 3: In April of 2014, when the Heartbleed bug was announced, the web servers using open source Apache and ngnix used OpenSSL and together held a 66% market share in active web servers. (Credit: Netcraft)

The Core Infrastructure Initiative
The antidote for Heartbleed was issued quickly, and the media was nearly hysterical in getting out the news. Ordinary people changed their passwords. Companies acted quickly to address the issue. Less than 20 days after the Heartbleed announcement, on April 24, 2014, many major companies came together and established the Core Infrastructure Initiative (CII). The CII is organized by the non-profit Linux Foundation “to fund and support critical elements of the global information infrastructure,” according to the CII website. CII “enables technology companies to collaboratively identify and fund open source projects that are in need of assistance, while allowing the developers to continue their work under the community norms that have made open source so successful.”[vi] Funding over a million dollars in projects in 2016, the first project that the CII funded was the OpenSSL project. OpenSSL now has four full time, paid developers, over a dozen team members, and a formal decision-making process.

Figure 4: How the CII Spends its Funds. The 19 founding members include Amazon, Bloomberg, Cisco, Dell, Facebook, Fujitsu, Google, Hitachi, Hewlett Packard, Huawei, IBM, Intel, Microsoft, NEC, NetApp, Qualcomm, Rackspace, Salesforce and VMWare. (Credit: The CII 2016 Annual Report)

Figure 4: How the CII Spends its Funds. The 19 founding members include Amazon, Bloomberg, Cisco, Dell, Facebook, Fujitsu, Google, Hitachi, Hewlett Packard, Huawei, IBM, Intel, Microsoft, NEC, NetApp, Qualcomm, Rackspace, Salesforce and VMWare. (Credit: The CII 2016 Annual Report)

Lessons Learned
No one individual should be relied upon to perform the role of several people without sufficient support for an extended length of time. Code reviewers are there to review code in detail; this is a requirement with the OpenSSL Project. End users and related community members who find flaws and report them are not counted upon as part of the review process. Automated code analysis tools do not replace detailed code reviews by experienced humans. Going forward, the OpenSSL team is accepting patches from those willing to sign a CLA. Due to wide-spread use of cryptography patents, the OpenSSL team has decided to re-license with the Apache license (ASLv2), one reason being that Apache includes patent protection.

After Heartbleed, at least two major forks of OpenSSL code were launched. Members of OpenBSD (a Unix derivative operating system) started LibreSSL and Google announced its fork as BoringSSL. A fork of open source code leads to a whole new version of the project with alternative priorities, perhaps different processes, and a forked community that feels it’s more important to do SSL differently. The maintainers for forked open source code must maintain the changes, however. (Every change that gets made upstream must be accommodated in the fork.)

Of the two forked entities, OpenBSD Project team members are reputed to have a security-minded culture that is reflected in the code. With security at the forefront, OpenBSD is looking quite attractive, yet in 2012, OpenBSD was faced with a $20,000 electric bill that it could not pay. The founder of a Bitcoin exchange company donated the funds that literally kept the lights on, and in 2016 the Code Infrastructure Initiative donated at least $50,000 toward the fund-raising goals of OpenBSD. The CII is now focusing on longer term strategies to address root causes and moving away from single point rescues. Had Heartbleed not made such a splash, undoubtedly something else would have come along, but the state of the OpenSSL project left a message that shamed us into addressing the issue of underfunded but critical projects.

The term “Internet superhighway” is ancient, but the analogy still holds as we recognize that the infrastructure of the internet can be neglected until the situation becomes critical. We can see rusting bridges, but we cannot see the infrastructure of the internet, nor the other bits and pieces that we unknowingly use every day. It’s somewhat surprising that the physical infrastructure is supported by gasoline taxes, but the internet infrastructure is not supported similarly by internet users.

LynnetteReese_115Lynnette Reese is Editor-in-Chief, Embedded Intel Solutions and Embedded Systems Engineering, and has been working in various roles as an electrical engineer for over two decades. She is interested in open source software and hardware, the maker movement, and in increasing the number of women working in STEM so she has a greater chance of talking about something other than football at the water cooler.


[i] Cox, Russ. “Lessons from the Debian/OpenSSL Fiasco Posted on Wednesday, May 21, 2008.” Research!Rsc: Lessons from the Debian/OpenSSL Fiasco, 28 May 2008, research.swtch.com/openssl.

[ii] OpenSSL Foundation, Inc. “OpenSSL.” /Index.html, 28 Aug. 2017, www.openssl.org

[iii] Donohue, Brian. “IBM: Heartbleed Attacks Thousands of Servers Daily.” Threatpost, 27 Aug. 2014, threatpost.com/ibm-heartbleed-attacks-thousands-of-servers-daily/107936/.

[iv] Rich Salz, OpenSSL after Heartbleed, LinuxCon Europe 2016.

[v] Netcraft. “April 2014 Web Server Survey.” April 2014 Web Server Survey, 2 Apr. 2014, news.netcraft.com/archives/2014/04/02/april-2014-web-server-survey.html.

[vi] https://www.coreinfrastructure.org/faq

Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • TwitThis