Amazon EBS and EFS Summary Notes

An Amazon EBS Volume is a durable, block level storage device that you can attach to your instance.

EBS is Availability Zone specific. To move EBS volume from one AZ to another: -EBS volume–Snapshot S3—Copy to another AZ –Create new Same EBS volume

When you create an EBS Volume, it is automatically replicated within its Availability Zone, to prevent data loss due to failure of any single hardware component.

You can attach multiple EBS volumes to a single instance.  The Volume and instance must be in same Availability Zone. You can attach an EBS Volume to any EC2 instance in the same Availability Zone.

You can use EBS volume as primary storage for data that requires frequent updates, such as the system drive for an instance or storage for a database application. Also use them for throughput intensive applications that perform continuous disk scans.

Constraints on Size and configuration of an EBS volume: EBS currently supports a maximum volume size of 16 TiB.

Amazon EBS provides following volume types:

General purpose SSD (gp2 and gp3), Provisioned IOPS SSD(io1 and io2), thourhput optimized HDD (st1),Cold HDD (sc1) and Magnetic (standard).

Benefits of EBS Volumes:

1.Data availability: When you create an EBS Volume, it is automatically replicated within its Availability Zone, to prevent data loss due to failure of any single hardware component.

2.Data persistence: An EBS volume is off-instance storage that can persist independently from the  life of an instance. EBS volumes attached to running instance can automatically detach from the instance with their data intact when the instance is terminated if you uncheck the Delete on Termination checkbox while configuring EBS volume  on EC2. The Volume can be reattached to new instance, enabling quick recovery. EBS backed instance, data is unaffected when EC2 instance is restarted. Volume remains attached throughout start to stop cycle .Data persist on Volume until deleted explicitly.

By Default, root volume attached to instance is deleted when instance is terminated.

Additional EBS volume that are created and attached to an instance at launch are not deleted when instance is terminated.

These behaviors can be changed by modifying the settings.

3.Flexibility:

EBS volume support live configuration changes while in production. You can modify volume type, volume size, and IOPS capacity without service interruptions.

4.Snapshots-

Amazon EBS provides ability to create snapshots (backup) of any EBS volume and write a copy of the data in S3, where it is stored redundantly in multiple Availability zones.

Incremental Volume: In EBSM Snapshot created only for consumed part.

For next snapshot, only updated part stored in next snapshot. First snapshot is clone, subsequent snapshot is incremental.

Incremental snapshot helps in saving cost.

Deleting data will only delete data exclusive of that snapshot.

If first snapshot is deleted, Amazon will move the first snapshot data, to another snapshot data.

You are charged for Data transferred to S3 from EBS volume snapshot stored in S3

5. Data Encryption-You can create encrypted EBS volumes with Amazon EBS encryption feature.

Encrypting data at rest is vital for regulatory compliance to ensure that sensitive data saved on disks is not readable by any user or application without a valid key. Some compliance regulations such as PCI DSS and HIPAA require that data at rest be encrypted throughout the data lifecycle. To this end, AWS provides data-at-rest options and key management to support the encryption process.

Data encryption at rest means encryption of data stored in EBS volume.

Data in transit means encryption of data moving from EC2 to EBS.

You can attach an encrypted and Unencrypted Volume to the same EC2 instance.

Data cannot be directly transferred between encrypted to unencrypted volume.

Encryption is always at EC2, and then moved to EBS. Data is encrypted prior to moving to EBS.

EBS volume is not physically attached to EC2, but is network connected.

Data in transit between EC2 and Encrypted EBS is also encrypted.

There is no direct way to change the encryption state of volume. To change the Encryption state indirectly, follow the below steps:

1.Method 1

• Attach a new encrypted, EBS volume to EC2 Instance that has the data to be encrypted.
• Mount the new Volume to EC2 instance
• Copy the data from unencrypted volume to new Volume.
• Both the volume must be on the same EC2 instance

2.Method 2

• Create a snapshot of unencrypted volume.
• Copy the snapshot and choose encryption for new copy-This will create encrypted copy.
• Use this new copy to create new volume which would be encrypted too.
• Attach the volume to EC2 instance.

_____________________________________________________________________________________

Root Volume Encryption-

Root volume is not encrypted by default.

There is no direct way to change the encryption of a volume

There is an indirect workaround to Encrypt Root Volume-

1.Launch the instance with the EBS volume required

2.Do patching or install the application

3.Create an AMI from EC2 instance

4.Copy the AMI and choose encryption while copying

5.Result is an encrypted AMI that is private

6.Use the encrypted AMI to launch new EC2 instance which will have EBS root Volume encrypted.

_____________________________________________________________________________________

How to share EBS snapshot—

To Encrypt a volume of a snapshot, you need an encryption key , these keys are called Customer Master Key (CMK) and are managed by AWS key management service.

When encrypting first EBS volume, AWS creates a default CMK key. After that each newly encrypted volume is encrypted with a unique AES-256 encryption key. This key is used to encrypt the volume, its snapshot and any volume created of its snapshots.

A customer master key (CMK) is a logical representation of a master key. The CMK includes metadata, such as the key ID, creation date, description, and key state. The CMK also contains the key material used to encrypt and decrypt data.

Before you share a snapshot, the following considerations apply to sharing snapshots:

• By default, only account owner can create volume from account snapshot.
• Snapshots are constrained to the Region in which they were created. To share a snapshot with another Region, copy the snapshot to that Region and then share the copy. — EBS volume—create Snapshot in S3—Copy to another AZ –Create new Same EBS volume.
• You can’t share snapshots that are encrypted with the default AWS managed key. You can only share snapshots that are encrypted with a customer managed key.
• You can share only unencrypted snapshots publicly. You can share unencrypted snapshots with the AWS community by making them public.
• You can also share unencrypted snapshots with other AWS accounts by making them private and selecting the accounts to share them with.
• When you share an encrypted snapshot, you must also share the customer managed key used to encrypt the snapshot.
• You cannot make encrypted snapshots public.

When encrypting first EBS volume, AWS creates a default CMK key.

Default CMK key is used for your first volume encryption, encryption of snapshot created from this volume, volume created from these snapshots. If your snapshot has default CMK key-it cannot be shared.

Steps to Change encryption key:

• You cannot change the encryption key directly which is used to encrypt an existing EBS volume
• If you want to change the key, create a copy of snapshot and specify during the copy process that you want to re-encrypt the copy with a diff key
• This comes handy when you have a snapshot that was encrypted using default CMK key and you want to change the key in order to share the snapshot with other account.

Steps to share unencrypted volume with specific account:

1.Make sure you use non default key to encrypt the snapshot, not the default CMK key

2.Configure cross account permissions in order to give the account with which you want to share the snapshot, access to custom CMK key to encrypt the snapshot. Without this the other account will not be able to copy the snapshot, not will be able to create volume of snapshot.

AWS CloudHSM is a cloud-based hardware security module (HSM) that enables you to easily generate and use your own encryption keys in the AWS Cloud. It helps you meet corporate, contractual, and regulatory compliance requirements for data security by using FIPS 140-2 Level 3 validated HSMs.

EFS:

EFS: Amazon Elastic File System (Amazon EFS) provides a simple, serverless, set-and-forget, elastic file system that lets you share file data without provisioning or managing storage.

• EFS is a fully-managed service that makes it easy to set up and scale file storage in the Amazon Cloud.
• Uses NFSv4.1 protocol.
• EFS has Elastic storage capacity that grows and shrinks as you add or remove data and need to pay only based on usage.
• Multi-AZ metadata and data storage.

How does EFS Work:

• With Amazon EFS, you can create a file system, mount the file system on an Amazon EC2 instance, and then read and write data to and from your file system. 
• You can mount an Amazon EFS file system in your virtual private cloud (VPC), through the Network File System versions 4.0 and 4.1 (NFSv4) protocol.
• Can be mounted from on-premises systems ONLY if using Direct Connect or a VPN connection.
• Alternatively, use the EFS File Sync agent.

Use case–EFS is good for big data and analytics, media processing workflows, content management, web serving, home directories etc.

EFS Features:

• Pay for what you use (no pre-provisioning required)
• Can scale up to petabytes.
• EFS file sysatem can be accessed simaltanioesly from 1 to 1000s of EC2 instances, from multiple AZs.
• By default you can create up to 10 file systems per account.
• Access to EFS file systems from on-premises servers can be enabled via Direct Connect or AWS VPN.
• Can choose General Purpose or Max I/O (both SSD).The VPC of the connecting instance must have DNS hostnames enabled.
• Read after write consistency.
• Amazon EFS file systems are resilient to one or more Availability Zone failures within an AWS Region.Data is stored across multiple AZ’s within a region.
• Need to create mount targets and choose AZ’s to include (recommended to include all AZ’s).
• Limited region support currently.
• Instances can be behind an ELB
• Can also be mounted on an on-premises server (via Direct Connect)
• EC2 Classic instances must mount via ClassicLink
• EFS is compatible with all Linux-based AMIs for Amazon EC2
• Using the EFS-to-EFS Backup solution, you can schedule automatic incremental backups of your Amazon EFS file system

How do you mount an EFS volume?

• Mount EFS on an existing EC2 Instance
• On the Elastic File system console, select the EFS you created. Click attach. This opens a page with mount instructions for the EFS. Select Mount via DNS or Mount via IP.

How Amazon EFS Works with Amazon EC2-
The above illustration shows an example VPC accessing an Amazon EFS file system.

Here, EC2 instances in the VPC have file systems mounted.
In this illustration, the VPC has three Availability Zones, and each has one mount target created in it. We recommend that you access the file system from a mount target within the same Availability Zone. One of the Availability Zones has two subnets. However, a mount target is created in only one of the subnets.

Amazon EFS vs EBS:

• EFS–Data is stored redundantly across multiple AZs.
• EBS-Data is stored redundantly in a single AZ.

• EFS-Upto thousands of Amazon EC2 instances from multiple AZz, can connnect concurrently to a file system.
• EBS-A Single Amazon EC2 instance in a single AZ can connect to a file system.

• EFS-Use cases–Big data and analytics, media processing and workflows, content management, web serving and home directories.
• EBS–Boot volumes, transactional and NoSQL databases, data warehousing and ETL.

EBS is cheaper than EFS,

Performance

There are two performance modes:

• “General Purpose” performance mode is appropriate for most file systems.
• “Max I/O” performance mode is optimized for applications where tens, hundreds, or thousands of EC2 instances are accessing the file system.

Amazon EFS is designed to burst to allow high throughput levels for periods of time.

Amazon EFS file systems are distributed across an unconstrained number of storage servers, enabling file systems to grow elastically to petabyte scale and allowing massively parallel access from Amazon EC2 instances to your data

Access Control:

• When you create a file system, you create endpoints in your VPC called “mount targets”
• When mounting from an EC2 instance, your file system’s DNS name, which you provide in your mount command, resolves to a mount target’s IP address.
• You can control who can administer your file system using IAM.
• You can control access to files and directories with POSIX-compliant user and group-level permissions.
• POSIX permissions allow you to restrict access from hosts by user and group.
• EFS Security Groups act as a firewall, and the rules you add define the traffic flow.

EFS Encryption:

• EFS offers the ability to encrypt data at rest and in transit.
• Encryption keys are managed by the AWS Key Management Service (KMS).
• Data encryption in transit uses industry standard Transport Layer Security (TLS) 1.2
• Enable encryption at rest in the EFS console or by using the AWS CLI or SDKs
• Data can be encrypted in transit between your Amazon EFS file system and its clients by using the EFS mount helper.

EFS File Sync:

• EFS File Sync provides a fast and simple way to securely sync existing file systems into Amazon EFS.
• EFS File Sync provides the following benefits:
• Efficient high-performance parallel data transfer that tolerates unreliable and high-latency networks.
• Encryption of data transferred from your IT environment to AWS.
• Data transfer rate up to five times faster than standard Linux copy tools.
• Full and incremental syncs for repetitive transfers.
• When deploying Amazon EFS File Sync on EC2, the instance size must be at least xlarge for your EFS File Sync to function.
• Recommended to use one of the Memory optimized r4.xlarge instance types
• Can choose to run EFS File Sync either on-premises as a virtual machine (VM), or in AWS as an EC2 instance
• Supports VMware ESXi

Compatibility:

EFS is integrated with a number of other AWS services, including CloudWatch, CloudFormation, CloudTrail, IAM, and Tagging services

Pricing and Billing

• EFS is billed only for the amount of file system storage you use per month.
• When using the Provisioned Throughput mode, you pay for the throughput you provision per month.
• There is no minimum fee and there are no set-up charges.
• With EFS File Sync, you pay per-GB for data copied to EFS.