LargeFS and Offloading Media to Amazon S3

LargeFS (Large Files System) is a product created by WP Engine, designed to store and transfer virtually unlimited amounts of data and integrate it into the familiar WordPress experience. It is especially useful if you need to use large amounts of media on your website and can help reduce local storage usage.

Be aware that while the WP Engine LargeFS service to offload your content is free, an S3 bucket must be purchased and paid for with Amazon. Prices for this bucket can vary based on the level of service.

For more information on what LargeFS is, read our announcement post, Unlimited Storage with WP Engine’s Proprietary LargeFS.

---

Setup LargeFS

Configuring LargeFS to work with WP Engine to automatically offload your media to Amazon S3 consists of four steps: Create bucket, give WP Engine access, make the bucket readable and provide the bucket to WP Engine Support.

If you want to use LargeFS for multiple sites on the same account, you do not necessarily need to use separate buckets for each website. Be sure to let our Support team know which sites will be using which bucket in the final step.

Create Bucket

1. Login to your AWS console
2. Choose S3 from the service list
3. Click Create Bucket

4. Give the bucket a name, that meets the following criteria
5. Select a region you want to host your bucket in
   • Remember this! You will need to provide this later to WP Engine Support
6. Click Next

7. Make any desired changes to the Set properties area
8. Click Next
9. From the Set permissions page, uncheck all 4 boxes:
   • Block new public ACLs and uploading public objects
   • Remove public access granted through public ACLs
   • Block new public bucket policies
   • Block public and cross-account access if bucket has public policies

10. Click Next
11. Review your bucket configuration against the settings above
12. Click Create

Give WP Engine Access

Next we need to assign the required permissions to the bucket, so that Large FS can begin placing files in the bucket. These permissions grant our WP Engine LargeFS user the rights to read and write objects in your bucket on your behalf.

1. Find and select the bucket you just created in the list of buckets
2. Click Permissions
3. Click Bucket Policy
4. In the Bucket policy editor, copy and paste the policy below
   • Replace the placeholder your-bucket-name with the bucket name you’ve created, there are 3 locations.
   • Be sure to keep the additional surrounding characters intact when replacing the placeholder text.

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowLargeFSUploads",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::902500896138:user/wpengine-largefs"
},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
},
{
"Sid":"PublicReadGetObject",
"Effect":"Allow",
"Principal": "*",
"Action":["s3:GetObject"],
"Resource":[
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}

5. Click Save in the Policy Editor

Make Bucket Readable

These policy and settings make your bucket world-readable. This is because when WP Engine goes to retrieve your files from the bucket, it does this as if it was a user on the internet.

1. Click Properties
2. Click Static Website Hosting
3. Click Use this bucket to host a website
4. Specify index.html as the Index Document
5. Click Save

It will take some time for AWS to propagate permissions from here.

Provide Bucket to Support

Contact Support and include the following:

1. Name of the environment
2. Bucket name
3. Bucket region
4. Path
   • Default setting is /wp-content/uploads/ however it can be any directory you keep media in
   • You may include multiple paths
   • Paths cannot be excluded, only included

The Support technician will then walk you through the process of validating your S3 bucket with our system by having you upload a file. This file will be titled with a random string and end in a largefs file extension, ex: 123abc.largefs

It’s important to upload this file exactly as provided, do not add .txt, .doc, .html or any other file extension.

Once this process is completed, our system will start copying over files from your local /wp-content/uploads/ to your new Amazon S3 bucket.

We will continue to regularly move new files that are over 10 days old to this bucket.

---

Backup Logs to Amazon S3

LargeFS can also be configured to automatically offload your Nginx website’s access and error logs be stored to your own S3 bucket. This feature allows you to maintain your log files as long as you wish, and access them without needing to login to your website or the WP Engine User Portal.

To have your logs offloaded, you must first ensure you have configured a bucket as normal, and have applied the following bucket policy to the bucket. Be sure to replace your-bucket-name with the name of the bucket.

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowWpeLogsBackup",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::902500896138:user/wpengine-remote-logs"
},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}

Then, simply reach out to WP Engine Support and let us know you’d like your logs configured to offload to LargeFS and the following information:

• s3 bucket region
• s3 bucket name
• s3 path (optional, see note below)*

Once access is verified, the logs will begin exporting a copy to your S3 bucket each night. If you encounter any errors or issues with this process please contact Support (available 24/7) in your User Portal for more help.

Custom Log Directory

By default, the files stored to your S3 bucket will be stored in the following format:

Nginx Log:

/wpe_logs/environmentname/nginx/201807051234-environmentname-apachestyle.log.gz

Apache Log:

/wpe_logs/environmentname/apache/201807051234-environmentname-access.log.gz

Error Log:

/wpe_logs/environmentname/error/201807051234-environmentname-error.log.gz

If you wish to use a custom folder name for your logs, use the bucket policy below instead–remember to replace your-bucket-name with the name of your bucket and your-log-folder with the name of the folder you would like to use:

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowWpeBucketRead",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::902500896138:user/wpengine-remote-logs"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
},
{
"Sid": "AllowWpeLogFolderWrite",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::902500896138:user/wpengine-remote-logs"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-bucket-name/your-log-folder",
"arn:aws:s3:::your-bucket-name/your-log-folder/*"
]
}
]
}

---

Known Limitations

When using LargeFS, you are storing your website images in a separate service via Amazon S3, those files no longer exist on WP Engine servers. For this reason the following items are known to not work with LargeFS.

• Gravity Forms Plugin
• Woocommerce Product Downloads
• WordPress Download Manager Plugin
• BuddyPress
• Image Compression Plugins
• ​Image Resizing Tools
• Hard or Absolute Server Paths
• Duplicate media file names
• Copying To and From Environments
• Deleting Images
• Network Out Bandwidth
• Any files which require APACHE PROXY PASS

Below further information can be found on these limitations and incompatibilities.

Gravity Forms

If you use Gravity Forms for uploads this will not work with LargeFS. This feature uses hardcoded server paths which means assets must load directly from the WP Engine server, rather than from LargeFS.

BuddyPress

If you are using the Avatar feature for BuddyPress, let our Support team know at setup that the BuddyPress Avatar and Group Avatar directories needs to be excluded from LargeFS. This feature uses hardcoded server paths which means assets must load directly from the WP Engine server, rather than from LargeFS.

The standard directory paths are:

wp-content/uploads/group-avatars/
wp-content/uploads/avatars/

Image Resizing Tools

TimThumb and other PHP-based dynamic image resizing tools. These systems use a PHP script to manually resize images as they are requested, requiring the images to be on the server itself. As a result, these dynamic resizing tools will not work with LargeFS.

However, WordPress has image resizing built in natively now via srcset, so these additional tools should no longer be needed. The built-in image resizing in WordPress works normally with LargeFS.

Image Compression Tools

WP Smush, EWWW Image Optimizer Cloud, and other image compression tools typically attempt to compress images when they are uploaded. Our LargeFS system waits 10 days after uploading the files to send them to your S3 bucket, so in most cases there are no conflicts.

However, if you ever change thumbnail sizes or need to regenerate thumbnails, this could require pulling your images back down from your S3 bucket to your server. You can use the steps in our Moving Buckets guide to assist in copying files to your local machine and then uploading them via SFTP.

Copying Environments

The assets are offloaded to your LargeFS bucket and removed from WP Engine servers. This means if you copy one environment to another images will be missing on the site you’ve copied to. There are two options:

1. Support can assist by adding a fallback rule to copied environment
2. Set up LargeFS on the new environment and copy the files through S3

It is not typically recommended to use LargeFS on more than one environment within the same site (STG, DEV, PRD).

Hardcoded Server Paths

If you are using a plugin or theme which references images as a path on the server rather than a URL, this may also conflict with LargeFS. Remember, LargeFS sends your images to your Amazon S3 bucket, so they do not live on a hard server path to be referenced in this way. However, calling the images as a URL path will work normally with LargeFS.

An image referenced as a server path that will not work: /nas/content/live/environment_name/wp-content/uploads/myimage.png

An image referenced at a domain path that will work: mydomain.com/wp-content/uploads/myimage.png.

Files Requiring Direct Access

LargeFS should be used for uploads that are called in a page rather than accessed directly. Files that need to be executed or opened directly in the browser should not be uploaded to LargeFS, spefically HTML.

Instead, store these files in a directory outside of those configured to offload to Amazon S3, typically outside of wp-content/uploads/.

Duplicate Media File Names

Media files should have unique names prior to uploading them to your website. This is to prevent files using the same name from being overwritten after the original has been offloaded to S3.

If a file is uploaded to your website as image.jpg it will be offloaded to S3 shortly after and no longer be found on the local server. If you proceed to upload a new image a few days later also titled image.jpg it will be offloaded to S3 overwriting the previously uploaded version.

The best way to ensure your media files remain unaffected is to develop and use a unique naming structure for media files prior to uploading them to the website.

Deleting Images

If a asset is deleted from the WordPress media library, the image must manually be deleted from the S3 bucket as well. The LargeFS sync off of WP Engine will not remove files that were deleted from WordPress.

Network Out Bandwidth

When configuring your Amazon S3 bucket, you agree to some terms and conditions with Amazon. One of these terms is accepting terms of payment for Network Out bandwidth usage for your bucket. Another is the storage usage with Amazon. Be sure to inspect the terms and conditions for your agreement with Amazon closely, to ensure you understand all charges.

---

NEXT STEP: Learn how to copy files between Amazon S3 buckets