How To Protect Customers From A Fortune 500-Destroying Vulnerability

By Tatu Ylonen, founder and SSH Fellow, SSH Communications Security

The most critical and vulnerable part of an enterprise is its information systems. Fortune 500 companies store highly-coveted data in those systems, making them priority targets for cybercrime. Even though they spend billions on cybersecurity solutions, a largely ignored bit of infrastructure could — if compromised — circumvent security measures and bring a business giant to its knees. VARs that understand this vulnerability can save their customers, whether Fortune 500s or start-ups, a great deal of woe and secure more business in the process.

Circumventing Secure Access

Fortune 500s go to great lengths to safeguard their critical data, including access to their tens of thousands of servers and disaster recovery data centers. Servers are managed by system administrators and various automated tools which need access credentials for other systems to conduct daily communications and operations. They usually use SSH keys, also used by developers to do their work internally, to log in from their workstation to access servers without having to type their password all the time.

Here’s the problem: roughly 90 percent of an organization’s SSH keys are unused. That means there is privileged access to critical systems and data that has never been terminated, violating policies, regulations, and laws. It is almost as if employees’ user accounts were never removed when they left, and they had the capability to create new accounts for anyone they like.

To make matters even riskier, about 10 percent of those keys grant root access (highest-level administrative access). Such keys are used to make backups, install patches, manage configurations, and implement emergency response procedures, often using automated tools.

Spreading Throughout The Enterprise

A cybercriminal typically starts by accessing a company computer and stealing passwords or other credentials to gain access to a set of servers — often involving malware. Once on a server, the attacker obtains elevated privileges using locally exploitable vulnerabilities to read private SSH keys from the server. Many of these keys grant unrestricted access to other servers and systems. The attacker uses these keys to gain access to those other servers and repeats the process to move undetected within the enterprise.

The attack can easily spread to nearly all data centers in the enterprise because of the high number of keys (10 to 200 per server, on average) in most enterprises. Some companies with more than 100,000 keys are granting access from low-security test and development into production servers alone. Key-based access between data centers is almost always present. Usually, there are also many SSH keys granting access from individual user accounts to privileged service accounts, bypassing systems that were supposed to monitor privileged access.

The attacker can avoid detection by monitoring the server for days or weeks to see which SSH keys are actually used with what servers, and then piggyback on legitimate connections to move undetected.

Going In For The Kill

At this point, the attack can disable the enterprise in a variety of ways. He or she can modify database records in subtle ways, corrupt backups or render every penetrated server, storage device, and router inoperable. For example, the attacker can reprogram the firmware on routers and switches, install malware into disk drive firmware, network adapter firmware or bios firmware, and wipe any data on the affected servers and storage systems, including any penetrated backup systems and disaster recovery systems.

It would take the enterprise weeks or months to rebuild and reinstall its systems, and it would likely lose a good number of recent transactions. How many hours, days or weeks can a typical Fortune 500 be down before the reputation damage is irreparable?

There are a variety of criminal types who already have engaged in such behavior: nation-states in a cyberwar, a terrorist organization trying to create chaos, a hacktivist with an agenda or a criminal organization looking for a ransom.

No Stone Unturned

Ultimately, this is not an infrastructure problem; it’s an administrative one. There is no simple patch or quick fix. Enterprise operations totally depend on automation made possible by SSH keys. Essentially, enterprises must establish proper management of automated access, as well as sort out the mess of legacy keys.

VARs can assist their customers in this process by helping them establish a controlled provisioning process as well as get rid of unused and policy-violating SSH keys. Because this task is too big to do manually, finding and offering the proper solutions will make this process effective and efficient for customers. Finally, recommend a careful review of SSH key-based access into backup systems and disaster recovery data centers. Your diligence in leaving no SSH stone unturned will result in happy, secure, repeat customers.

Tatu Ylonen is the creator of the SSH protocol and the founder of SSH Communications Security. He is an experienced entrepreneur, manager and engineer. He still keeps up to date with technology and loves the technical side and inventing new technology. He participates in product architecture design and occasionally writes code when he has time or when he thinks that’s where he can bring the most value.

His primary current interests relate to broader cybersecurity priorities and how to design systems to be more secure. He understands both the big picture and the deep technical issues. He also wants to solve the massive gap in identity and access management in relation to SSH key based credentials.