Storing Mail-in-a-Box Backups in S3

Current Mood: “So Easy” by Röyksopp


One of the things that I left unfinished when I set up my Mail-in-a-Box server was backups. Mail-in-a-Box will automatically create backups of your data, but as far as I know it doesn’t have any easy way to transfer these backups to another location for safekeeping. Because Mail-in-a-Box encrypts its backups, it should be OK to store these backups in S3. In this post, I will describe how I set up my server to automatically transfer backups to S3.

Step 1: Copy the Secret Key

The default path for backups is /home/user-data/backup/encrypted. If you look in /home/user-data/backup, there should be a file named secret_key.txt. scp this file off of the server, or even better, copy the contents and store it in an encrypted password vault on your local machine (which should also be backed up remotely somewhere).

Step 2: Create an S3 Bucket

Create an S3 bucket to store backups. Open S3 and click “Create Bucket”. Enter a name for the bucket - for example, “mail-server-backups”. For the Region, select a region that is different from the one where your mail server resides. For example, my mail server is in us-east-2 (Ohio), but I used us-east-1 (Northern Virginia) for my backups. If something catastrophic happens to AWS’ data centers in either region, this should buffer against such events (if a catastrophic event affects both regions, you’ll probably have bigger concerns than your mail server backups). Continue to the “Set permissions” options. Verify that the S3 bucket will have the following settings (they should be selected by default):

  1. Owner account has full access
  2. Manage public permissions is set to “Do not grant public read access to this bucket (Recommended)”
  3. Manage system permissions is set to “Do not grant Amazon S3 Log Delivery group write access to this bucket”

Click “Next” to continue, then click “Create Bucket” to create the bucket.

Step 3: Create an IAM Role

Open Identity & Access Management. Click on “Roles” on the side menu, then click “Create role”. Select “AWS service” and then click on “EC2” for the role type. For the use case, select “EC2”, then click “Next: Permissions”. The next page is “Attach permissions policy”. We are not going to use any of the AWS managed policies. Instead, click “Create policy”, then click the “Select” button next to “Create Your Own Policy”. The page will open in a new window - keep the other tab open. Set the “Policy Name” to “mail-server-backup”. Copy the following policy document, replacing both instances of your-bucket-name with the name of your S3 bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Save the policy. You will be redirected back to the IAM dashboard. Switch back to the other tab where you were creating the role. Click the “Refresh” button to refresh the list of policies, then filter for “mail-server-backup”. Enable the checkbox for the policy, then click “Next: Review”. Name the role “mail-server-backup”, then click “Create Role”.

Step 4: Add Role to EC2 Instance

Open the EC2 Instances page. Select your instance, then click the Actions button and select “Attach/Replace IAM Role” under “Instance Settings”. Select “mail-server-backup” from the “IAM role” dropdown, then click “Apply”.

Step 5: Install awscli

SSH into your mail server. Execute the following commands to install awscli:

sudo apt-get update && sudo apt-get install -y awscli

Step 6: Verify that Role is Configured Correctly

Let’s verify that your IAM role and bucket are configured correctly:

echo "This is a test file" > ~/test-file && aws s3api put-object --bucket your-bucket-name --key test-file --body ~/test-file

Execute the command above, then open your S3 bucket and verify that the key test-file exists in the bucket. Click on the object, download it, and verify that it has the contents “This is a test file”.

Step 7: Upload Backups on Cron

We’ll use awscli’s s3api put-object operation to copy an archive of the files in /home/user-data/backup/encrypted to S3. This command will need to run as root because your normal account won’t have sufficient permissions to view the files in /home/user-data/backup/encrypted. Execute sudo crontab -e. If you haven’t set up a crontab for root and also haven’t configured a default file editor, you’ll see the following screen:

ubuntu@box:~$ sudo crontab -e
no crontab for root - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/ed
  2. /bin/nano        <---- easiest
  3. /usr/bin/vim.basic
  4. /usr/bin/vim.tiny

Choose 1-4 [2]:

Select whichever editor you’re most comfortable with (insert proselytizing for vim here). After you’ve selected an editor, you will see an empty crontab file (if it didn’t exist already):

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

Add a single command on a new line at the end of the crontab:

30 3 * * * tar -czf /tmp/backup-$(date '+\%m-\%d-\%y').tar.gz /home/user-data/backup/encrypted; aws s3api put-object --bucket box.meanly.io --key backup-$(date '+\%m-\%d-\%y').tar.gz --body /tmp/backup-$(date '+\%m-\%d-\%y').tar.gz; rm /tmp/backup-$(date '+\%m-\%d-\%y').tar.gz

This command means “create an archive of /home/user-data/backup/encrypted named backup-mm-dd-yy.tar.gz and upload it to this S3 bucket every day at 3:30 AM”. Note that the time for backups might be different if you’ve changed this - check the timestamp on the files in /home/user-data/backup/encrypted. Each file should have roughly the same creation time. If the time is different on your server, just adjust the minute/hour fields (the first two numbers) in the line above.

Save the crontab and exit your editor to install the new crontab.

(Optional) Step 8: Automatically Transition/Delete Backups

Over time, your backups will probably become much larger, which means you’ll be storing a lot of data in your S3 bucket. One way to minimizie costs is to set up automatic Lifecycle rules for your S3 bucket. To do this, open your S3 bucket in the AWS console and click the “Management” tab. Click “Lifecycle” (it should already be selected), then click “Add lifecycle rule”. Enter a lifecycle rule name, e.g., “Delete Old Backups” on the first page, then click “Next”.

The next page allows you to configure automatic transitions. We didn’t configure versioning for your S3 bucket above, so you only need to select the checkbox for “Current version”. Next, click “Add transition”. From the “Select a transition” dropdown, select “Transition to Amazon Glacier after”. In the “Days after object creation” field, enter 15. Then click “Next”.

The next page allows you to configure automatic deletion of objects. Again select “Current version”, then select the checkbox for “Expire current version of object” and enter “180” for the “After ___ days from object creation”. This will give you about 6 months worth of backups should something go wrong. Then click Next. On the next page, review your settings, then click “Save” to create the lifecycle rule.

More …

How to Set Up Mail-in-a-Box on AWS

Current Mood: “Mountains, Pt. 2 & 3” by Djrum


Last week I decided to replace my Gmail account with a Mail-in-a-Box instance hosted on EC2. Mail-in-a-Box is a one-command script that installs and configures all of the software necessary to run your own mail server. The setup guide is great, but setting it up on EC2 requires some extra work. In this blog post, I share instructions for setting up Mail-in-a-Box in AWS.

1. Create an SSH Key Pair

You will need to have a key pair configured on the EC2 instance so that you can SSH into it and run commands. To generate the key, execute the following command from a terminal prompt:

ssh-keygen -t rsa -b 4096 -C "Your Name"

ssh-keygen will create the key. It will ask you where you want to place the key pair files. By default, the path will be ~/.ssh/id_rsa (and the public key will be stored in ~/.ssh/id_rsa.pub automatically). Even if you don’t already have a key file stored at that path, I suggest using a more unique name, e.g., “MailServer”. Next, ssh-keygen will ask you for a passphrase. If you provide a passphrase, you will have to type this passphrase in every time you want to use your key to SSH into a server that is configured to use the key pair. If your private key is ever stolen, this can help prevent your key from being used by an illegitimate user. If you do not want to provide a passphrase, press return twice to continue. ssh-keygen will create your key and store it at the path you specified.

To add the key pair to your AWS account, login and go to the EC2 dashboard, then click on “Key Pairs” on the left (or click here). Click “Import Key Pair”, then go back to your terminal. cd to the path where you created your SSH key and copy the contents of the .pub key file (example: if you stored your key at ~/.ssh/MailServer, the public key file would be at ~/.ssh/MailServer.pub). You can output the contents of the file with cat ~/.ssh/path_to_key.pub and copy from the terminal. If you are you using OS X, you can use the pbcopy command to copy the contents of the file to your clipboard by executing pbcopy < ~/.ssh/path_to_key.pub. Once you have copied the public key, go back to your browser and paste the contents into the “Public key contents” field. Set the “Key pair name” field to something like “Mail Server” and click “Import”.

2. Create an EC2 Instance

Open the EC2 dashboard and click “Launch Instance” or click here to load the Launch Instance wizard.

Step 1: Select an AMI

The first step is to select an Amazon Machine Image for your EC2 instance. Mail-in-a-Box only support 64-bit Ubuntu 14.04. Click on “Community AMIs”, then select the filter checkboxes for “Ubuntu”, “64-bit”, and “EBS”. Enter “14.04” into the search field above the list of AMIs. Look for the most recent Ubuntu 14.04 AMI you can find and click its “Select” button once you’re ready.

Step 2: Select an Instance Type

On the next page, you will select the instance type. Mail-in-a-Box requires at least 1 GB of RAM. You can use a t2.micro if you want (it is free-tier eligible, which could reduce your costs significantly). However, I picked a t2.small since the t2.micro instance type is pretty bare-bones. When you are ready, click “Next: Configure Instance Details”

Step 3: Configure Instance Details

Select a VPC and subnet for your EC2 instance. If you don’t already have a VPC and subnet available, you can use the “Create new VPC” and “Create new subnet” links to create a VPC and subnet. Instructions for how to configure your VPC and subnet are beyond the scope of this guide, but you can follow the instructions in Amazon’s docs here to create the VPC and subnet: Scenario 1: VPC with a Single Public Subnet. Set “Shutdown Behavior” to “Stop” and select the “Protect against accidental termination” checkbox. Make sure that the “IAM Role” dropdown is set to “None”. Click “Next: Add Storage” at the bottom of the page to continue.

Step 4: Configure Storage

On this page, you will configure the amount of storage reserved for your EC2 instance. I suggest using 50 GiB of General Purpose SSD storage (gp2) at minimum. The costs for this are fairly minimal. For example, in the US East (Ohio) region, this costs me about $5 per month. You can use less if you want, but over time you will use up this storage and need to expand the mount. Using 50 GB will buy you some extra time before you have to do this. Once you have modified the Size field, click “Next: Add Tags” to continue. On the next page, we’re going to skip the tags and continue to configuring the Security Group for the instance, so click “Next: Configure Security Group” once the page loads.

Step 5: Configure Security Groups

In this step we will configure a security group for your new EC2 instance. Think of a Security Group as a kind of firewall. Basically, it allows you to block requests to ports except for the ones you configure the Security Group to allow access to. Select the “Create a new security group” option. For the “Security group name” field, enter “Mail Server”. You can leave the description as-is or modify to something you prefer. Then, configure the following rules. You will need to add a new rule for each item in the list below:

Type Protocol Port Range Source Description
SSH TCP 22 My IP SSH
SMTP TCP 25 Anywhere SMTP
HTTP TCP 80 Anywhere HTTP
HTTPS TCP 443 Anywhere HTTPS
Custom TCP Rule TCP 587 Anywhere External SMTP Submission
IMAPS TCP 993 Anywhere IMAPS
POP3S TCP 995 Anywhere POP3S
Custom TCP Rule TCP 4190 Anywhere ManageSieve (mail filters)

You will only need to manually set the protocol and port range for rules with type “Custom TCP Rule”. For the other types, AWS will automatically fill in the port and protocol. In the case of SSH, it’s generally best to only allow access from your current IP address. If your IP changes, you will need to update the rule, but you probably won’t need to access the server over SSH very often anyway. When you finish configuring the rules, click “Review and Launch” to continue.

Step 6: Review and Launch

Double check the information on the “Review Instance Launch” page to make sure that everything is correct. When you’re ready, click the “Launch” button. The “Select an existing key pair or create a new key pair” window will open. On the top dropdown, select “Choose an existing key pair”. From the “Select a key pair” dropdown, select the “Mail Server” key pair we created in the “Create an SSH Key Pair” section above. Check the checkbox at the bottom and then click the “Launch Instance” button.

Step 7: Sit back and wait!

Congratulations, you’re the proud owner of a shiny, new EC2 instance. It will take several minutes for AWS to provision the instance and get it up and running.

3. Allocate an Elastic IP Address

Navigate to the Elastic IPs page in EC2. Click “Allocate new address”, then “Allocate” to allocate a new Elastic IP address. An Elastic IP address is an IP address that is reserved exclusively for your account and can be transferred between different resources in the account as needed.

4. Attach Elastic IP to EC2 Instance

Go to the Elastic IPs page. Select the Elastic IP you allocated in the step above, then click the “Actions” button at the top and select “Associate address”. Select the instance you created from the “Instance” dropdown, then select the instance’s private IP from the “Private IP” dropdown. Finally, click “Associate” to associate the Elastic IP with your EC2 instance.

5. Configure DNS

I used Route53 for this. Note that there is a downside to using Route53 - it does not support DNSSEC for DNS service. If you are concerned about the security of your domain, I suggest following the Mail-in-a-Box instructions for configuring your domain.

Open the Route53 dashboard. If you haven’t already registered a domain, click on “Registered Domains”, then on “Register Domain”. Find a domain you want to register and then purchase it. It will take a few minutes for AWS to acquire the domain and configure a hosted zone. Once AWS has done this (you should receive an e-mail when it’s finished), click on “Hosted Zones” on the left menu and click the link in the “Domain Name” field in the list of hosted zones to open the record list for the hosted zone. You will need to add two records: one for your server’s DNS address, and another for the mail exchange record. Click “Create Record Set”, then enter the following values in the panel on the right:

Name Type TTL Value Routing Policy
box A - IPv4 address 300 IP address Simple
(leave empty) MX - Mail exchange 300 10 box.example.com Simple

For the first record, paste the Elastic IP address you allocated into the Value field. The second record’s value should be box. plus the domain name you just registered (I just included example.com to demonstrate the full URL).

6. Install Mail-in-a-Box

SSH into your server with the following command:

ssh -i ~/path/to/your/key ubuntu@1.1.1.1

Replace ~/path/to/your/key with the actual path to your private key (e.g., ~/.ssh/MailServer - not the .pub file). Replace 1.1.1.1 with the Elastic IP address associated with the EC2 instance. Once you have connected, you’re ready to set up Mail-in-a-Box by following the instructions in the “Machine Setup” section of the Mail-in-a-Box setup guide.

7. Ask AWS to Remove Email Sending Limitations

There is one more step to get your mail server working. You need Amazon to unblock port 25 for your EC2 instance, and you need them to configure reverse DNS so that other mail servers can lookup the DNS value for your Elastic IP address and get your box.example.com domain in return. Fortunately, this is pretty easy. Log in to AWS as your root account and then go to the Request to Remove Email Sending Limitations form. Paste your Elastic IP into the “Elastic IP Address 1” field, paste your box.example.com domain into the “Reverse DNS Record for EIP 1” field, and click “Submit”. It will probably take about a day for them to respond to your request. Once that’s done, you’re ready to login to your Mail-in-a-Box instance and start using it for email!

More …