Author Archives: Alan Grau

Vending machine image

Who is Taking Cybersecurity Seriously, and why You Should be Worried

May 28th, 2015

By Alan Grau, Icon Labs

On a daily basis, I have the opportunity to interact with a wide range of companies and people involved in embedded device development and cybersecurity for embedded devices.  I was recently struck by the difference in attitudes regarding cybersecurity at three diverse companies.  Each company representative had a very different approach in evaluating how and why to implement cybersecurity in their devices, and I found the differences in their approaches more than a little surprising, and in fact rather concerning. 

I understand the flawed logic of drawing broad conclusions from just 3 data points, but I believe there are still lessons to be learned from these perspectives on security, and that they are representative of widely held opinions. 

Approach 1: Blind compliance to standards

In discussions with an engineering manager who was working on a project for the United States military, I was struck by the single-focused objective of ensuring compliance with customer supplied security standards.  Yes, compliance to standards is important and provides a baseline for building secure embedded devices.   Many companies could very well improve the security posture of their devices by adhering to industry security standards. 

That said, I am not an advocate of blind compliance with external or industry security standards.  It is critical to evaluate the risks and attack surfaces of each device and provide security solutions that address the individual needs of that device.  My criticism of this approach is not the desire to achieve compliance with security standards, but the unwillingness to consider other security threats or attack vectors.

Approach 2: ROI for including security

A manager for an industrial automation company recently asked me “what is my ROI for adding security to my device?”  I understand the need for any business to focus on ROI and make sound business decisions, but I found it surprising that security for critical control system devices was using ROI as the key determining factor on any possible investment in cybersecurity for his products. 

Clearly, a cost-benefit analysis must be made on any cybersecurity investment, but some decisions (at least in my opinion), shouldn’t come down to just ROI.  I equate this to my life insurance policy.  I did some “business analysis” to determine how much life insurance I felt I needed, but I didn’t buy life insurance for the ROI.  I hope to never use the insurance policy, and if I do there is no ROI for me, but I still purchased the policy because it is the right thing to do. 

The calculation needs to include the cost of failure.  What happens if there is a successful attack on a system using this company’s technology and the company’s solution is then later to be found at fault? With the possibility of a cyber-attack costing millions of dollars in damages, can a company afford to not have insurance provided by including security technology in their products? The ROI may be in their customers’ confidence in the security of their offerings.

Approach 3: We need this to be secure

My final example is from a manager who is tasked with building the next generation of his company’s flagship product.  The previous generation product had limited communication capabilities, utilizing cellular data communication to “phone home” and provide diagnostics and operational information to back-end servers.  The next generation products will include Wi-Fi, Bluetooth, USB and Ethernet connectivity options. 

Vending machine image

 “With all the connectivity being built into this product, we need to make sure this is secure”, was the message from this manager.  This is the approach that more companies need to take.

 This company is concerned about security standards and ROI, but they have a broader recognition of the importance of security.  They recognize the need for the device to include adequate security based on its use cases and attack vectors, and that the tradeoffs are well thought out.  What is most surprising is that this company is not building automation equipment for industrial automation or military usage, but vending machines. 


There has been a tremendous amount of press regarding cybersecurity for critical infrastructure and embedded devices, including a presidential executive order. Everyone is worried about cyber-attacks concentrating on data – accessing people’s private financial information and stealing their money. However, if a cyber-criminal successfully attacks and penetrates control and communication systems at a utility, manufacturer, transportation, medical or DoD facility system, lives are at stake. People could die. This is a much bigger threat than a few million dollars, or a few candy bars, stolen by a hacker.

Companies are now finally becoming aware of the need to secure embedded devices and some companies are doing more.  But more is not enough.  If a vending machine company can make its objective “this has to be secure”, then so can everyone else.  The fact that a vending machine company is leading the way has me worried.

Alan Grau is President and co-founder of Icon Labs, a leading provider of security software for embedded devices.  He is the architect of Icon Labs’ award winning Floodgate Firewall.  Alan has 20 years of embedded software experience.  Prior to founding Icon Labs he worked for AT&T Bell Labs and Motorola.  Alan has an MS in computer science from Northwestern University. You can reach him at 

Security Advice from the US Department of Justice: Include “Back Doors” to enable Monitoring

November 17th, 2014
DOJ recommends back doors.  How smart is this?

DOJ recommends back doors. How smart is this?

Security Advice from the US Department of Justice: Include “Back Doors” to enable Monitoring.  How smart is this?

By Alan Grau, Icon Labs

In a recent statement issued by the US Department of Justice, device manufacturers are encouraged to include “back doors” in devices to allow law enforcement to perform surveillance in the electronic world. Adding a back door can neutralize all other security measures; back doors should not be added! 

Back doors are nothing new in the world of computing.  Hackers have found, and exploited, back doors in WiFi routers and IP cameras.  In the early 1980s, Ken Thompson created a simple back door that he could have inserted into every Unix system on the planet by a cleverly disguised hook in the lowest levels of the compiler’s code generation module.  I even worked on a product early in my career that included a back door to allow developers access to any system in the field for debugging.  The intent was to remove the back door prior to shipment of production units, but it’s easy for features such as this to “accidently” slip into production code, leaving a gaping hole in the security of a product. 

This DoJ recommendation is reminiscent of my first programming job out of college.  As a fresh, bright eyed young programmer working for AT&T Bell Labs, I was working on software for their 5ESS switch, the backbone switching systems for many telephone systems worldwide.  Our team worked on the Call Monitoring subsystems, which allowed collection of phone call logs on specific phone numbers.  This allowed law enforcement to collect daily reports of everyone who was called by or who called a specific phone number.  We also supported a feature enabling law enforcement to listen in live to any call from a specified phone number. 

AT&T Bell Lab

This feature was, according to the sales people, a big selling point in the former Soviet Union, but was not included in domestic products because of conflict with privacy laws.

The new DOJ recommendation not only raises concerns over possible privacy abuses, but creates an easily exploited security hole.  With the distributed nature of devices today, devices are certain to come under cyber-attack.  For consumer devices, hackers are able to access the firmware of the device.  With the firmware in hand, a skilled hacker can disassemble and reverse engineer the code, allowing them to find back doors. 

Devices are also vulnerable to inside attacks by those with knowledge of back doors.  A single disgruntled employee or other bad actor can utilize the information obtained via off the back door to cause serious harm.  Unfortunately, once disclosed, most back doors are not readily disabled, leaving a vast number of devices vulnerable.    

While there is a legitimate need for the DOJ to monitor communication, within the bounds of our legal system, in this day of digital communication, we must not sacrifice security to do so. Regardless of the other security measures taken, don’t add back doors!

Alan Grau is President and co-founder of Icon Labs, a leading provider of security software for embedded devices. He is the architect of Icon Labs’ award winning Floodgate Firewall. Alan has 20 years of embedded software experience. Prior to founding Icon Labs he worked for AT&T Bell Labs and Motorola. Alan has an MS in computer science from Northwestern University. You can reach him at

Attack of the Spamming Refrigerator

April 17th, 2014

Updated 4-24-14: Added hotlinks

By Alan Grau, Icon Labs

Internet of Things Devices –including appliances–gets hacked and starts spamming your friends.

refrigerator2As the Internet of Things takes off and an increasing array of devices are connected to the Internet, malware, viruses and hackers will inevitably follow. Smart appliances are arriving in our homes and with them, so has the first confirmed cyber-attack against a home appliance. As reported by (, a botnet spam attack infected a number of TVs and at least one refrigerator. This report raised several questions in my mind. Is this an anomaly or is this just the beginning? Should we care? After all, this is just a refrigerator. And if we do care, what should we do about it?

[A botnet is a number of Internet computers that, although their owners are unaware of it, have been set up to forward transmissions (including spam or viruses) to other computers on the Internet, or perform other operations as directed by the hacker that controls them. Any such computer is referred to as a zombie - in effect, a computer "robot" or "bot" that serves the wishes of some master spam or virus originator.]

Is this an anomaly, or just the beginning?

If we let history be our guide, the answer is clearly that this is just the beginning. When hackers first began developing viruses and other malware, many exports dismissed these as fads that would be short lived. It was commonly believed that as operating systems and application software became more advanced with time, they would become less vulnerable to threats of all kinds. Similarly, when spam first showed up in the email world, it was commonly dismissed as a weak threat that would be eliminated by future, more advanced email platforms. Anyone using email or a PC today can attest to the fact that just because a system is more “advanced” it is not necessarily more secure.

Should you care?

There are an estimated 6 million to 24 million computers that have been infected with botnets, depending upon who you ask. Given the prevalence of botnet infected computers, should we care if a few Internet connected Smart home devices are infected with malware and conscripted into the services of a hacker? There are several reasons why we need to care.

1. Discovering that an IoT device has been compromised is more difficult than discovering that a laptop or desktop computer is comprised. The user of a Smart TV or Internet connected refrigerator probably won’t notice that performance has bogged down or if tasks are failing and then be able to infer that something is wrong. Even if they did, they can’t fix the problem – they cannot run anti-virus or anti-malware software to help diagnose the problem. With an IoT device, the user really has no means to determine that a machine has become infected.

2. IoT devices are not built with a method to recover from malware infections. PC systems have built in capability to tolerate, mitigate and recover from security breaches. Backup systems exist. Anti-malware tools are ubiquitous. And if all else fails, you can reinstall the software. However, if your fridge is taken over by malware, you have no way of reinstalling the OS and application software.

3. If an IoT device can be taken over by botnet software, it is also susceptible to attacks that could cause the device to fail. If this happens to your PC, you can reinstall software and recover. Most IoT devices are not designed to allow end users to reinstall software or recover from system failures. Device failures, no matter the cause, are extremely damaging to consumer loyalty, brand reputation and ultimately, corporate profits for the device manufacturer selling the device.

4. IoT devices interact with the world in concrete ways that could be exploited by hackers. A hijacked high performance TV, with its built video camera, microphone and motion sensor, could be used to spy on family activities.








Smart Home Technology is becoming pervasive

How do we respond?

The main lesson is that we need to hold IoT and embedded devices to a higher standard of security and reliability than we hold PCs to. IoT technology adds communication capability to existing devices (and in some cases enables the creation of new types of devices). Adding communication, cloud enabled services and other IoT capabilities must not result in sacrificing the performance, safety, reliability or security of the origin device or the market will not accept the updated devices. While consumers we have learned to accept a certain amount of performance impact, data loss and other impact of security breaches on PCs, consumers are unlikely to accept this on IoT devices. Technology exists to block, mitigate and recover from these breaches on PCs, but recovering from security breaches will not be as easy, or even possible with IoT devices.

In many cases, IoT devices will perform critical functions where security breaches can have serious impacts. If the device were a robotic manufacturing device or medical device, the impact of a security breach could impact factory production or the well-being of patients.

If a business lunchroom’s refrigerator was discovered to be the source of damaging computer virus attacks or even just tons of malicious spam, aside from the bad publicity, there possibly could be legal and business repercussions.


The assumption that future IoT devices and systems will be more advanced and therefore immune to security threats is naïve. Future IoT devices and systems will certainly be more advanced and it is possible they will be highly secure. But this will only happen if there is a dramatic change in the approach to security by everyone involved in the development and deployment of IoT devices. Engineers, managers and executives need to take security seriously. Security needs to be designed in from the early stages of the engineering process and must include multiple levels of security. Only then can we avoid spamming refrigerators or hijacked Smart TVs and ensure that our IoT devices are secure.

Icon Labs Iconfidant SSL Product Not Vulnerable to the Heartbleed SSL Bug

Alan Grau is President and co-founder of Icon Labs, a leading provider of security software for embedded devices. He is the architect of Icon Labs’ award winning Floodgate Firewall. Alan has 20 years of embedded software experience. Prior to founding Icon Labs he worked for AT&T Bell Labs and Motorola. Alan has an MS in computer science from Northwestern University. You can reach him at

Are you taking embedded security seriously?

February 4th, 2014

A lock with the key still in it isn’t much of a solution. Sadly, the security in many embedded devices isn’t much better.

By: Alan Grau, Icon Labs

At the gym last week I noticed a locker secured with a padlock but with the key still in the lock. One of the assumptions of locking something up is that there is something worth protecting. If the contents of the locker weren’t worth protecting, then don’t bother with the lock. But if you are going to bother with a lock, or with adding security to your embedded device, you might as well make sure the security will work. A lock with the key still in it isn’t much of a solution. Sadly, the security in many embedded devices isn’t much better.

Is this your embedded security solution?

Is this your embedded security solution?

Many embedded devices have been built with very little, if any security. Too often what security is provided is insufficient. In June 2013, ICS-CERT (Industrial Control Systems Cyber Emergency Response Team) published a list with more than 300 medical devices from approximately 40 vendors that used hard coded passwords, almost the literal equivalent of leaving the key in the lock.

Other embedded devices are using out of date crypto standards. For example, keyless remote vehicle entry systems utilize encrypted RFID signals. Most implementations use a 40 bit encrypt key. When the standard was first introduced in the 1990s 40 bit encryption keys would take days to crack, but not today. A google search of “keyless vehicle entry systems hacked” turns up a number of articles describing how thieves are exploiting this weakness to steal cars.

Security – Priority or Afterthought

While embedded devices are very different from standard PCs and require unique solutions, the biggest challenge to embedded security isn’t technical.

Too often, security is simply not a priority. I recently received a call from a company putting together a requirements document for a critical new piece of technology for a Fortune 100 company. This was a core piece of technology that was to be used across a majority of the company’s products. As this document was being finished they realized they had not addressed security and they didn’t have the expertise in house for this task. They wanted to see how they could drop in some security at the last minute. While they were still in the requirements phase and there was time to address it, security was clearly not a priority despite the fact that this would go into a product that virtually everyone in the developed world uses on a daily basis – and a security breakdown could literally result in the loss of life. Sadly, this is not the only example I have seen in which security is an afterthought.

Security for Modern Embedded Devices

A security solution for embedded devices must ensure the device firmware has not been tampered with. It must also secure the data stored by the device, secure device communication and protect the device from cyber-attacks. This can only be achieved by including security from the early stages of design.

There is no one one-size fits all security solution for an embedded device. Security requirements must take into consideration the cost of a security failure (economic, environmental, social, etc.), the risk of attack, available attack vectors, and the length of time the device will be in service. Many embedded devices will still be in service 10 or even 20 years from now. Given Moore’s Law, an encryption key that provides strong encryption today may not hold up for the next two decades.

Embedded security features that need to be considered are:

  • Secure Boot – Achieved using cryptographically signed code from the manufacturer along with hardware support to verify code is authenticated. This ensures that the firmware has not been tampered with.
  • Secure Access – Access to the device (remote login, access to a web interface, etc.) need to be secured with encryption and a strong authentication method. A strong password or an authentication method such as Kerberos can be used.
  • Secure Communication – Communication to/from the device needs to be secured using encrypted communication (SSH, SSL, etc.). Care must be taken to avoid the use of insecure encryption algorithms.
  • Embedded Firewall – Embedded firewalls provide a critical layer of protection against attacks. A firewall can limit communication to only known, trusted hosts, blocking hackers before they can even launch an attack. A layer of defense to protect against common attacks such as packet flood attacks, buffer overflow attacks and known protocol exploits.
  • Summary

    If security is made a priority, I believe engineering teams are more than capable of building much higher levels of security into their devices. Solutions are available for including encryption, secure boot, authentication and firewalls into embedded devices. The key (no pun!) is that security needs to be considered early in the design of a new device or system. Support for secure boot or device tamper detection requires specific hardware capabilities. Since hardware is typically selected early in the design phase, this capability must be considered very early in the process. Since many embedded devices are deployed outside of the standard enterprise security perimeter, it is critical that security be included in the device itself.

    Alan Grau

    Alan Grau is President and co-founder of Icon Labs, a leading provider of security software for embedded devices. He is the architect of Icon Labs’ award winning Floodgate Firewall. Alan has 20 years of embedded software experience. Prior to founding Icon Labs he worked for AT&T Bell Labs and Motorola. Alan has an MS in computer science from Northwestern University. You can reach him at

    Embedded Device Security – Not My Problem

    October 23rd, 2013

    By: Alan Grau, Icon Labs

    Anyone who seriously studies embedded device security will quickly realize that embedded security is very different than enterprise or desktop security.  The challenges for securing enterprise and PC networks are different from the challenges for securing a network of fixed function devices.

    A few of the factors that make embedded device security different are:

    • Most embedded devices are fixed function devices.  Once they are shipped they cannot be upgraded to add security after the fact.
    • Embedded devices are special purpose devices, not general purpose devices like PCs or servers, and as a result require special purpose security solutions.
    • Embedded devices frequently run real-time operating systems such as VxWorks or INTEGRITY and cannot run security solutions designed for Windows or Linux based systems.
    • Once deployed, the devices cannot be upgraded to add security fixes unless the device manufacturer provides an upgrade.  The end user cannot buy security software from a third party and install it on the devices.
    • There are millions of devices already deployed that lack protection and for which there is no readily available security solution.

    Since end users of embedded devices cannot install security software onto the device after the fact, the responsibility for security would seem to fall squarely onto the shoulders of the OEMs who build the device.  Security for embedded devices has to be built into the device itself.  All too often, however, OEMs push off the responsibility for the security of the device to the operating system running on the device.  They argue that the OS is responsible for the security of the device.  Or even worse, that security is either not a requirement or provides no competitive advantage and therefore can be ignored.

    Since you were interested enough to read this article, you likely already agree that security is an important requirement for most embedded, network enabled devices.   Granted, the security requirements differ from device to device.   Security is more critical for a heart pacemaker with wireless communication capability than for Wi-Fi/GPS system that is used to track and locate  lost hospital beds (and yes, according to an engineer with a company building hospital beds, this is a real problem).

    If security is a requirement – whose responsibility is it? 

    Unlike enterprise and PC security, where responsibility lies mainly with the end user organization, security for embedded devices is primarily the responsibility of the OEM building the devices.  However, the OEM is not solely responsible.  Everyone involved in the development and deployment of the device plays an important role.

    The role of the OEM

    The OEM plays the primary role in embedded device security.  They are ultimately responsible for specifying security requirements, implementing security in the device and testing the device to ensure it meets the security requirements.  The OEM is responsible for selecting an OS and processor that meet the security requirements of the device. They are also responsible for ensuring that the device uses security protocols, secure authentication and protections mechanisms such as an embedded firewall to protect the device from attack.  Unfortunately, we have seen too many cases of OEMs taking the “not my problem” approach to security far too often.

    The FDA and Department of Homeland Security recently issued an alert urging medical device makers and medical facilities to upgrade security protections to protect against potential cyber threats. This came in response to an ICS-CERT publication of a list of more than 300 devices with hard coded passwords.

    The role of the OS vendor

    While it is the role of the OEM to select the OS, the OS vendor (or open source community creating and maintaining the OS) is responsible for ensuring the security of the OS.  Typical communication protocols and services are bundled with the OS and these provide one of the main attack vectors for cyberattacks.   The OS vendor should be responsible for ensuring the security of each of the components they provide.

    The role of the chip vendor

    Chip vendors also play a key role in embedded device security.  They are building processors with built in code verification capability, physical tampering detection capability and encryption engines.  These tools allow OEMs to develop and deploy devices that can verify they are running authentic code and to detect when someone has physically opened a device. Once these events are detected, they can then shut the device down or report the event to prevent tampering.

    The role of the specialized security companies

    OEMs, end users, and even RTOS companies do not always have the expertise to ensure all aspects of device security are addressed.  Companies specializing in embedded device security provide expertise, tools, specialized security solutions and security audit and verification services.  These companies can play a critical role in embedded device security by ensuring compliance with security standards, providing education to OEMs and end users, and testing devices to ensure they are not vulnerable to cyberattacks.

    The role of the end user

    The end user is depending upon the OEM and the OEM’s suppliers for providing a device with adequate security capability.  It is up to the end user to ensure the device is deployed in a secure manner.  The end user must properly set passwords, enable authentication and perform any other steps required to deploy the device in a secure manner.  Far too often, devices are deployed with weak or default parameters or without security enabled, leaving the device open to attack.

    The end user is also responsible for demanding that OEMs add security to their devices.  Just last week I was in a meeting with Fortune 1000 OEM and was told that “our customers are not requiring security”.  Even if this is true, the OEM needs to understand that security is an unstated requirement.  However, more end users need to stand up and demand security.

    Security Responibility


    The only way to ensure embedded devices are secure is through the coordinated effort of everyone involved in the development and use of the product.  Unfortunately, no one group can by their efforts alone ensure that a device is secure.  However, a failure to implement or properly use security at any stage in the process can result in a device with a significant security loophole.  No one can afford to say embedded device security is “not my problem” any longer.

    Alan Grau

    Alan Grau is President and co-founder of Icon Labs, a leading provider of security software for embedded devices.  He is the architect of Icon Labs’ award winning Floodgate Firewall.  Alan has 20 years of embedded software experience.  Prior to founding Icon Labs he worked for AT&T Bell Labs and Motorola.  Alan has an MS in computer science from Northwestern University. You can reach him at