Overcoming The 3 Stages Of SSH Key Management Grief
By Matthew McKenna, chief strategy officer, SSH Communications Security
There’s a kind of grief that occurs when small and mid-sized organizations realize that have a gaping hole in their network security because of inconsistent or non-existent SSH user key management. And, like the familiar stages of grief, once the denial is over there are three stages of coming to grips with and rectifying poor key management.
First, a quick assessment to gauge the amount of denial at work in your organization. Who is responsible for the provisioning, revocation, and recertification of SSH user key-based access in your organization today? If you have answered this question with a shoulder shrug or forced grin, you may be in denial or partial denial and should consider reading further.
Much like a password, SSH user keys are used for authentication. Whether it is access to our Oracle databases, SAP financial systems, payment processing systems, trading platforms, network devices, Linux based IOT devices, or our new agile DevOps processes or Amazon cloud, SSH and SSH user key-based access is pervasively used. It is the threading that connects our processes together for application-to-application data transfers, and it is the means by which our administrators and developers can conveniently access their environments without having to retype their passwords each time.
Why are SSH user keys so important? Here are three primary reasons:
- You can’t easily assess which person or what process a key pair is associated with because SSH user keys don’t necessarily correlate themselves to an identity.
- Since SSH user keys don’t expire, they continue to provide access unless they are removed from the locations where they reside.
- Anyone in your organization with an SSH client and access to a server can generate an SSH key pair.
This means you have lost control of the access to your most critical infrastructure. But those who are in denial will ask the following question: “If SSH user key-based access is such a big problem, why don’t I hear more about breaches caused by their use?”
Stage 1: Denial
SSH user keys played a major role in the Snowden and Sony breaches. In both cases, there is significant evidence SSH user keys were extensively used to gain access to critical servers and data, move laterally to additional assets within the network, and exfiltrate data. Poor to non-existent SSH key management and encrypted access control were large contributing factors to both of these major headlines.
It would seem, at this point, it would be impossible or nearly so to know if SSH user keys had been used to gain unauthorized access to servers and exfiltrate data if we don’t have any inventory of SSH user keys and are not monitoring their usage and how they are provisioned, revoked, and recertified. The truth is that we wouldn’t and we don’t.
A contributing factor is many SMBs do not have visibility into their encrypted SSH and SFTP traffic, thereby rendering their intrusion detection systems, data loss prevention systems, anti-virus, and SIEM tools ineffective to identifying threats in real time where SSH is potentially being used maliciously.
Stage 2: Resistance
Resistance follows on the heels of denial and usually entails the following sentiments:
- I already use Kerberos tickets or this or that privileged access management solution, so we don’t have a problem.
- There are too many stakeholders involved to solve this.
- Don’t scan my environment to show me the size of the problem, because then I will have to fix it.
- There are so many other more important initiatives.
- We are moving to the cloud, so this won’t be an issue.
In light of these seemingly insurmountable hurdles, resistance is an understandable reaction. But let’s go through these step by step to be able to move to the phase of adapting our organizations to acceptance of this massive hole in our access management postures.
1. I already use Kerberos tickets or this or that privileged access management solution, so we don’t have a problem. When it comes to managing interactive access and role-based access to infrastructure assets, Kerberos tickets and traditional PAM solutions are excellent tools. However, what these solutions do not cover well is the management of automated machine-to-machine and application-to-application-based access. When we look at the greater picture of SSH user key-based access, over 80 percent of the SSH user key-based access in our environments is application to application in nature.
2. There are too many stakeholders involved to solve this. With regard to access and data flows, we can think of SSH as a unifying force in our organizations. It cuts across our on-premises assets and is used by third-party suppliers to gain access to our systems remotely and transfer data to our systems. It is used by our developers to move code throughout the DevOps supply chain as we create new applications. It is used to gain access to our cloud and to transfer data and connect applications between our midrange servers and our mainframe.
First and foremost, SSH user key and encrypted access is an identity and access management issue. However, cryptographic services, security operations, and infrastructure all play a significant role in effectively addressing the issue. It’s not really an issue of too many stakeholders; it’s more about identifying who has the ownership to solve the problem.
3. Don’t scan my environment to show me the size of the problem, because then I will have to fix it. Scanning the environment is an eye-opening and humbling experience, but it must be endured. It will provide insight into risk around segregation of duty access controls from non-production to production environment. It will demonstrate gaps in how we manage third-party access to our environments. It will give us insight into access that has not been recertified in two or more years and demonstrate where we have key-based access with weak encryption. It will highlight how deeply a malicious actor could penetrate into our environment with a single stolen private key and show us how big the gap is. But the size of the issue is honestly the lesser concern. The greater concern is how we gain control of this access in terms of how we provision it, revoke it, and recertify it.
4. There are so many other more important initiatives. It’s time to re-examine priorities. After all, what could be more important than securing the primary back-down in terms of access to our most critical infrastructure? We are often too focused on what is coming through the front door when we should be thinking what is happening at the back door. How can we continue to ignore hundreds of thousands to millions of access credential to our most critical infrastructure? What is more important than securing the crown jewels of our businesses?
5. We are moving to the cloud, so this won’t be an issue. SSH will continue to play a key role when it comes to access and the movement of data, irrespective of your plans for migrating services and applications to the cloud. Many businesses engage in what is known as a “lift and shift” when it comes to the first step of cloud migration. When it comes to SSH user key-based access, this means that we have simply migrated the access challenges we faced on-premises into our cloud.
SSH user key-based access is the developers’ tool of choice, when in the automation stack of the DevOps process, to move code from source code repositories like Github, to test and build automation tools like Jenkins, all the way through to how we upload the code to a cloud application. We must take into consideration, in the quickly evolving world of cloud transformation, is that developers continue to have access through the supply chain closer than ever to development environments. This access also needs to be controlled effectively.
Stage 3: Adaptation
The third stage of coming to terms with poor key management is adaptation. At this stage, there are effective ways to gain visibility, control, and ongoing governance of our SSH user key-based access.
Visibility is the first challenge that must be solved because, in most cases, SSH user key-based access has not been part of the overall provisioning processes of organizations’ identity management frameworks. Visibility consists of developing a mapping of private and public keys to their corresponding interactive or application-to-application connection. This can be achieved through scripted approaches, some of the privileged access management solutions and those dedicated to managing SSH user keys.
In tandem, we can implement tools to gain an understanding of how frequently keys are authenticating and from where. This is known as verbose logging and can be activated in the SSH daemon. This information is essential because it provides us the needed intelligence as to what key-based access in our environment is valid and what access may no longer be needed. It also can help us identify if private keys are authenticating from IPs, which are not recognized by our network.
What is most likely to biggest challenge to overcome is preventing the free flow of unauthorized SSH user key-based provisioning. Through a process called lockdown, where authorized keys are relocated from users’ home directories to a central root-owned location, we can create an effect where now only users with root-level access are able to generate new SSH user keys going forward. But be careful here; if you lock down access without having a provisioning process in place, you may frustrate many administrators, developers, and application owners in your organization. Communication of the process and oversight is essential to the success of this process.
What is the ultimate goal? Automation. It’s essential due to the sheer volume of SSH user keys. On average, if a skilled user sets up an SSH user key-based trust and validates that the connection is working, it may take them five minutes. An unskilled SSH user, on the other hand, may take 30 minutes or more. If we do the math, we are generating thousands, tens of thousands or hundreds of thousands SSH user keys a year, and the costs really begin to add up. Automation can be achieved through scripting, solutions dedicated to SSH user key management, and even orchestration tools.
When we are reminded of something we don’t what to think about or learn something that we should have already known, we can experience a sort of grieving process. Denial and resistance are the first two stages, but we can’t stay there. We must move on to adaptation which, in the case of SSH user key management, means gaining visibility into the environment and all its processes and procedures. This lays the groundwork for a clearly defined system that creates greater network security and peace of mind for security professionals.
Matthew brings over 15 years of high technology sales, marketing, and management experience to SSH Communications Security and drives strategy, key account sales, and evangelism. His expertise in strategically delivering technology solutions that anticipate the marketplace has helped the company become a market leader. Prior to joining the company, Matthew served as a member of the executive management team of ADP Dealer Services Nordic and Automaster Oy, where he was responsible for international channel operations and manufacturer relations. In addition, he was responsible for key accounts including Mercedes-Benz, General Motors, and Scania CV. Before this, Matthew played professional soccer in Germany and Finland.