Introduction:
Cloud instance forensic acquisition presents certain challenges to forensics teams. Traditional forensic methods usually are not effective in the cloud. Access and networks are designed differently than in an on-premise Data Center. Forward thinking strategies need to be implemented so that Incident Response Cyber teams can effectively use forensically sound methods to examine artifacts on hosts.
This article is about how to prepare your organization for forensic acquisitions in a cloud infrastructure. I will quickly cover how to prepare a fleet of systems for memory and physical disk forensics. The targets are AWS EC2 instances but could be applied to any other cloud providers host provisioning infrastructure. My talk will focus on the process and infrastructure required to do this level of inspection. By the end you will be able to apply these strategies to activities such as Threat Hunting.
Many organizations struggle with implementing Threat Hunting programs with orchestration in mind to capture memory and disk level forensics. How does a Cyber team respond to an alert they receive from a cloud host? How can they quickly collect artifacts for further forensic inspection? Last, how can you best secure the forensics infrastructure from where you launch the orchestrated forensic examiner systems?
The first part of this article will describe the infrastructure required to be in the place to support forensic orchestration. I will outline a strategy: servers, tools, storage, and protective measures to ensure that forensic activities are conducted behind a cloud of secrecy. Maintaining stealth mode is critically important to enabling the forensic team to do their job while the business is not impacted by the investigative activities.
In the second part, we will examine the pipeline process to implement solutions in EC2 instances with pre-configured memory and acquisition tools ready to be tapped into by the forensic team. I will discuss some of the challenges encountered when conducting forensics with the different AWS hypervisor solutions.
Preparation:
Building a hidden infrastructure that the forensic team can conduct systems acquisitions and pull-down forensic artifacts is important to the success of the defensive team’s architecture. The forensic infrastructure needs to have its own environment that is operated and owned by
that team. This often means that the forensic teams need to have staff with systems experience so that they can act as their own internal I.T. The forensic infrastructure includes storage, forensic examination systems, and a fleet of acquisition systems. Preferably these systems should be designed with orchestration in mind so that in times of heavy workloads the environment can expand to meet demand.
The infrastructure can be on premise or in the cloud. What is important is that only forensics members have access to the infrastructure. The forensic infrastructure needs to be protected behind a firewall and other IDS and IPS protective security devices. This presents a challenge for cloud infrastructures. Often the devops teams are in a constant state of flux. They build and destroy new VPC infrastructure environments all the time. Thus, there is a need to design the network access systems that defaults to allowing the forensics team to use their tools on the ports required to access any systems in the cloud. So if a new VPC is brought up an automated process enables access from the forensic infrastructure. The objective is to avoid trying to open security groups or ACLs during an emergency situation. All of the access should be in an open state but limited to the forensics environment from the beginning. This does require working with the cloud architecture teams to coordinate the proper designs.
Using cloud services such as S3 buckets, OneDrive or Google Drive to facilitate the delivery of forensic artifacts is a common usage. Though it is recommended that they be only used temporarily as a medium to transfer data to its final resting place. The forensic examination systems need to be located separately from the business infrastructure. Using cloud tools to temporarily transfer forensics evidence between the two infrastructures.
Baking Process:
The key to success is to bake into the AMI creation process the tools required to enable quick analysis of an instance by the threat hunting and forensics team. File system and memory capturing tools are the two most useful. You can purchase commercial or use open source tools. These tools need to be incorporated into the overall design of the base image.
For memory acquisition tools use of AVML or Lime are the most common. These are open source though it takes a lot of effort to maintain with Linux systems as new versions of the kernel are released. Once installed you will want to build in automation scripts so that the threat hunting team can produce a memory acquisition. This can be accomplished by staging scripts that can be executed from the cloud CLI console interface. By using the console you can ensure integrity of the memory dump by avoiding to login into the system. The process can be scripted to drop the memory dump onto an S3 bucket or similar Cloud remote storage device.
There are numerous file system acquisition tools that can be deployed as a solution. This is an area that can be challenging due to the constant hypervisor design changes by Cloud provides. The file system acquisition tool needs to be flexible as well as effective. AWS has two Hypervisor
solutions that fundamentally changed how file acquisition tools can be used on an EC2 instance. The Xen hypervisor has been around since 2003. The newer Hypervisor called Nitro was launched in 2017. The major difference from a forensics perspective are the changes implemented at the device level.
Example of Nitro - using an nvme device.
[root@ip-x.x.x.x ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 4G 0 disk [SWAP]
nvme2n1 259:1 0 10G 0 disk
└─nvme2n1p1 259:2 0 10G 0 part /r01
nvme0n1 259:3 0 8G 0 disk
├─nvme0n1p1 259:4 0 8G 0 part /
└─nvme0n1p128 259:5 0 1M 0 part
The Nitro Hypervisor uses NVMe storage devices for its EBS volumes. In our research to create forensic images with F-Reponse we enchanted issues finding the disk device. It turns out that F-response was not picking up the naming schema change when a device query is performed. “In Linux, NVMe device names follow the pattern /dev/nvme<x>n<y>, where <x> is the enumeration order, and, for EBS, <y> is 1. Occasionally, devices can respond to discovery in a different order in subsequent instance starts, which causes the device name to change.” Reference https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html
They have since provided a patch and will release a fix in the next release of the product. It took several trials and errors to reach a final solution. It was not expected that changes below the AWS management plane at the Hypervisor level would have an impact on the forensics operations. But they did and we would have had to resort to using different methods to create a forensic image of a host. Such as using the DD command. The downside of using a tool like DD is that it is hard to scale at an Enterprise level. It would have also involved changing tactics that were already known to be working with the XEN technology. Not an ideal situation.
Conclusion:
As a result, testing each design of the Linux instances with your forensics tools is an important part of the process. Do not expect the forensic tools to work seamlessly when the architecture teams switch fundamental infrastructure designs. Each phase of the AMI delivery pipeline needs to be tested and verified that the Cyber team can continue to perform their investigations without running into challenges during a real incident. Do not wait until forensics is really needed to only find out that the tools designed did not perform their job.
Defcon 28 Cloud-Village Presentation
Comments