Evaluating Acquia storage limits - "Emergency upsize" notification

July 19, 2024

Ashraf Abed

Recently, a client who hosts their Drupal 9 website on Acquia reached out to ask us to investigate an alert they had received from Acquia. Acquia sent an email which said "This email is to inform you that we emergency upsized your storage from 200GB to 300GB FS in Case # [redacted]. The cost to keep this upsize in place [is..]"

Acquia Emergency Upsize Alert

The emergency upsize alert is sent when one or more of your applications runs out of space. In this case, instead of allowing your application to 'break', Acquia may temporarily increase the storage space available to your application. They may or may do this at no cost to you temporarily, but if you decide to keep the "emergency upside" in-place without addressing the root cause, your costs are likely to increase.

So how do you "address the root cause" in this case..? First, you should understand the source of the problem.

Understanding Acquia's storage

On Acquia, there are multiple "pools" / types of storage that could run out of space.

Types of storage on Acquia hosting

At a high-level, there is typically at least two types of storage:

  • File Storage
  • Database Storage

You can view either one by:

  • Logging in to Acquia
  • Clicking on your application
  • Clicking on an environment (such as Prod)
  • Clicking "Stack metrics"
  • Clicking either the "File System" tab (for File storage) or "Database" tab (for Database Storage)
  • Scroll down, view the Storage used (e.g. 105GB out of 200GB)

Understanding different storage "pools" on Acquia

The "prod" environment gets its own isolated storage for both the File Storage and Database Storage. This means when you review the available storage for the prod environment, you're looking only at prod's available storage.

A common mistake is that people stop looking when they reach this point.

In fact, Acquia typically has a separate storage allocation for the non-prod environments. So your dev and stage environments will typically share storage space.

In practice, this ends up looking something like this:

  • File storage
    • Prod has its own file storage allocation (e.g. ~105GB out of 200GB used)
    • The other environments (dev, stage, etc) share the "non-prod" file storage (e.g. 195GB out of 200GB used)
      • Checking the storage space for any non-prod environment will typically show the same values as the rest of the non-prod environments, because they share storage space.
  • Database storage
    • Prod has its own database storage allocation (e.g. 20GB out of 100GB used)
    • The other environments (dev, stage, etc) share the "non-prod" database storage (e.g. 30GB out of 100GB used)
      • Again, checking the storage space for any non-prod environment will typically show the same values as the rest of the non-prod environments, because they share storage space.

Analyzing storage limits

When investigating storage space issues, you should typically look at all of the following:

  • File Storage on prod over a period of time
  • File Storage on dev over a period of time
  • Database Storage on prod over a period of time
  • Database Storage on dev over a period of time

You can view a chart of storage used over a period of time as follows:

  • Click (e.g.) prod environment > Stack metrics >

    • Click the date dropdown near the top of the page.
    • Change it from a 1 day range to a longer range (e.g. 2 months) to view trends

  • Do the same thing for the dev environment

  • After choosing a date range, click the "File System" tab

    • View the "Storage % used" chart
      • In this example, I found the following chart which shows that the File system storage % used hit 100% in early January before dropping. It increased again in April before eventually leveling out.

  • Do the same for the "File system" and "Database" tabs prod and dev environments, both the "File system" and "Database" tabs

The initial drop from 100% utilization was caused by Acquia increasing the allocated storage space for the environments. But is that a long-term solution?

Long-term solution for running out of file storage

Acquia temporarily upsized the storage space. Is that an appropriate long-term solution? As any experienced architect knows, the answer is always "it depends"!

There's nothing wrong with increasing the storage space (aside from incurring additional costs). However, we should first determine whether we actually need to be using the storage space. Here's how to investigate it further (warning: connecting via SSH is very powerful and very dangerous. Proceed with caution, and consider hiring a Drupal expert to investigate for you):

  • In the Acquia UI, visit the environment (e.g. dev) which has the storage space problems
  • Get the SSH information from the environment and use it to connect via SSH
  • Check how much space is being used by each environment's files directory:
    • du -sh /mnt/gfs/*
  • Review the output list. It will typically look something like this:
    • `1.1M home

88G debugsclientdev

0 debugsclient.dev

98G debugsclientstg

0 debugsclient.test

1.1G debugsclientpredev

0 debugsclient.predev`

In the example above, we can deduce that the "dev" site's files folder is using 88GB of space, while the "stage" site's files folder is using 98GB of space. That's a lot of space for two lower environments to be utilizing!

In this case, it seems the developers of the "debugsclient" website most likely copied all files from "prod" to "stage" environment, and again to the "dev" environment.

Saving space on Acquia by sharing files across environments

Instead of physically duplicating all files from prod to multiple lower environments, did you know you can "share" files across environments instead of duplicating them? There are a few ways this could be accomplished, but one popular solution is the "Stage files proxy" module:

https://www.drupal.org/project/stage_file_proxy

This module allows e.g. the "dev" site to load files from e.g. "prod" when the files can't be found on the dev server. That means you don't need to place the dev files on the dev server at all! Instead, skip copying the files from prod to dev altogether, and let stage file proxy fetch the files on-demand directly from prod - no more duplicated storage!

In this instance, we can save almost 200GB of storage space on the lower environments with this solution, and save ourselves from an unnecessary cost increase. Thanks, Acquia, for the emergency upsize - but our client doesn't need a permanent account upgrade. They can use the extra savings on Drupal courses (or an extravagant outing, if you're into that sort of thing).