Grow worry-free: storage autoscaling on Timescale Cloud

Grow worry-free: storage autoscaling on Timescale Cloud

Available today in Timescale Cloud: Automatically increase database storage as your application grows, while instituting cost constraints to make sure you stay within budget.

Time-series data is everywhere, and it drives decision-making in every industry. Time-series data collectively represents how a system, process, or behavior changes over time. Understanding these changes helps us to solve complex problems across numerous industries, including observability, financial services, Internet of Things, and even professional American football. Analyzing this data across the time dimension enables developers to understand what is happening right now, how that is changing, and why that is changing.

And time-series is also relentless, often at high rates and volumes. To help developers keep up, we built Timescale Cloud: a database cloud for relational and time-series workloads, built on PostgreSQL, and architected around our vision of a modern cloud service:  easy, scalable, familiar, and flexible.

We recently announced larger disk plans of up to 16TB per node on Timescale Cloud, enabling developers to effectively store and analyze 200+ TB of data (thanks to TimescaleDB’s hybrid row-column native compression, with compression rates of 94-97%).

Today, to continue with this theme of delivering a worry free platform for time-series that scales with your data needs, we’re announcing the release of storage autoscaling on Timescale Cloud.

What is storage autoscaling?

Storage autoscaling helps you keep your database operations running smoothly and avoid the hassle of dealing with full disks, by automatically increasing your database storage, with user-defined limits to keep costs under control (and avoid unexpected bills). Storage autoscaling (and manual disk resizing) happens fully online, without service downtime.

Storage autoscaling is on by default. You now create a database service on Timescale Cloud with a single click. Your service starts small, but the platform continuously monitors your storage consumption and automatically increases your capacity when needed. From 10GB up to 16TB (or 200+TB of effective storage). Easy to use and highly cost effective.

But Timescale Cloud also empowers developers with greater control. The platform allows developers to easily set configurable limits for auto-scaling (or turn it on or off on a per-service basis). When enabled with limits (which can be modified at any time), your storage will automatically scale up to your specified limit.  And then other full disk protection mechanisms will kick in if and when you hit your limits.

The screenshot of Timescale Cloud UI showing that storage autoscaling feature is enabled, your Timescale Cloud service will increase its storage once it reaches 95% of the current storage plan until it reaches its storage autoscaling limit.
With storage autoscaling enabled, your Timescale Cloud service will increase its storage once it reaches 95% of the current storage plan until it reaches its storage autoscaling limit.

While managed database services save you the hassle of administering your own database, storage autoscaling helps you save even more time by avoiding dealing with operational blockages caused by full disks, and at the same time ensuring that your users and customers experience uninterrupted service. Furthermore, storage autoscaling gives you full control over your future costs, with transparent pricing and user-defined limits to ensure that you never scale unexpectedly beyond your budget.

Testing storage autoscaling

As an experiment, we inserted more than 300 billion rows of data over 14 days into a single-node service, autoscaling from 500GB to 2.5TB of storage, thanks to TimescaleDB’s automated compression and data retention policies. In the experiment, 2.1 million rows of data were inserted into a hypertable every 5 seconds, compressing any chunks that no longer had data being inserted into them. Every time the disk was 95% filled, Timescale Cloud automatically increased its storage to the next available plan, as indicated by the discrete jumps of the yellow line in the graph below: from 1TB, to 1.5TB, to 2TB, and to its pre-configured limit of 2.5TB in this experiment (you can set this limit as high as 16TB).

The graph with green and yellow lines showing disk usage (green line) and disk capacity (yellow stepped line) for the experiment of inserting 300B rows in 14 days on Timescale Cloud. Every time the team filled 95% of the disk, Timescale Cloud automatically increased storage to the next available plan, as indicated by the discrete jumps of the orange line.
Graph of disk usage (green line) and disk capacity (yellow stepped line) for the experiment of inserting 300B rows in 14 days on Timescale Cloud. Every time the team filled 95% of the disk, Timescale Cloud automatically increased storage to the next available plan, as indicated by the discrete jumps of the orange line. (Note: The graph above is cropped to illustrate storage autoscaling which occured within the first 9 days of the experiment. The full graph showing disk usage over 14 days is available later in Case Study section of the blog post.) 

In addition to automatically scaling up storage as required, Timescale Cloud also includes full disk protection mechanisms, which puts the database into read-only mode once the auto scaling limit is reached and the database is full, ensuring that the database doesn’t crash from new data attempting to be inserted into it (read more here about Timescale Cloud’s storage safety mechanisms.)  It’s another example of how Timescale Cloud is focused on providing a worry-free platform for developers.


Learn more about storage autoscaling

Read on for more about what storage autoscaling is, the problem it solves, and why and how to use storage autoscaling in Timescale Cloud. You can also find more details about this experiment in the Case Study section of this blog post.

For existing Timescale Cloud users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.

If you’re new to Timescale, create a free account to get started with a fully-managed Timescale Cloud service (100% free for 30 days, no credit card required).

Once you are using TimescaleDB, please join the Timescale community and ask any questions you may have about time-series data, databases, and more.

And, for those who share our mission and want to join our fully remote, global team: we are hiring broadly across many roles.

If you'd like to connect with Timescale community members, get expert tips, advice and more, tune in to Timescale Community Day later this month for talks (and demos!) about time-series data and TimescaleDB tips.

An image with two light spheres on each side and text in the middle says October 28, 2021, Timescale Community Day, #TimescaleDay. Yellow button with dark text saying Subscribe for updates.

Finally, special thanks to Ivan Tolstosheyev and the entire Timescale Cloud team behind the development of this capacity 🙏.

Full disks are a pain

We’ve all experienced it. You’re trying to take that latest puppy video or photo to share in your group text, but your storage is full. You’re left trying to figure out which caches to clear or other pictures to delete.

Nobody likes a full disk, including your database and the operating system it uses. You try to insert more data into your database, and the “write-ahead-log” (WAL) it uses to ensure all data is reliably and atomically written has no place to write its log. Try to add an index, and there’s no place on disk to store the index pages. And, even if you don’t directly write any new data to the database, things are happening (or, perhaps more accurately, not happening) in the background. Temp files can’t be written. File system blocks can't be allocated. Unexpected things go wrong.

As a database cloud which prioritises ease and scale, Timescale Cloud provides built-in safety mechanisms to deal with these issues on your TimescaleDB services. Timescale Cloud will alert you whenever you approach storage limitations on your account, put your service in a read-only state so that your data is not lost, and give you an opportunity to configure your service so that you can resume collecting everything that matters. Many of these mechanisms are transparent and “just work” (as one example, Timescale Cloud employs system balloon files for an additional layer of “defense in depth” to full disks) and you can read more about the existing measure in our blog post announcing automated disk management.

But today, we wanted to share a bit more about our new capabilities for storage autoscaling and why we built them.

There is plenty of friction involved in triaging a database with a full disk (even with protections in place to ensure a database doesn’t crash, like putting it into read-only mode). It requires manual intervention from developers in order to provision more storage or perhaps to move, delete or compress data to save space. This intervention at worst takes away from engineering time that could be used for feature development, or at a minimum is just plain inconvenient.

Moreover, in the case you choose to provision more storage, it’s difficult to know exactly how much storage you’ll need with most DBaaS or serverless solutions. If you choose a disk that’s too small, you run the risk of having to manually intervene again in the near future, but if you choose a disk size that’s too big you end up paying for more capacity than you actually need as you wait for your disk to fill up.

Storage autoscaling on Timescale Cloud was designed to solve these problems, by removing the pain of dealing with full disks and minimizing the time that developers spend worrying about their database.

Enter storage autoscaling

How it works

With storage autoscaling on Timescale Cloud, you can start with as little storage as you need but set an autoscaling limit to give you the peace of mind that you’ll be able to scale up smoothly to storage levels you may need in the future.

With storage autoscaling enabled, when a service hits 95% of the current storage capacity, the storage plan will autoscale to the next available plan. (For example, if you’re on the 25GB storage plan, you’ll be autoscaled to the 50GB. And if you’re on the 9TB plan, you’ll be autoscaled to the 10TB plan.)

The screenshot of the Timescale Cloud UI showing the storage autoscaling feature, your disk capacity is automatically increased to the next largest tier when 95% of the current capacity is used.
With storage autoscaling, your disk capacity is automatically increased to the next largest tier when 95% of the current capacity is used.

This means that you’ll never pay for more than the storage plan your service is currently using and only pay for larger storage plans when you autoscale to that tier. You can start at 10GB and automatically scale up to 16TB as your workload increases over time with no manual intervention. Moreover,  storage autoscaling (and manual disk resizing) happens fully online, without service downtime.

Autoscaling limits

You set an autoscaling limit, which is the maximum disk capacity that your service’s storage will reach with autoscaling enabled. This gives you full control over your future costs and helps keep your operations running smoothly with predictable, transparent pricing.

The screenshot of the Timescale Cloud UI with the autoscaling option in the Timescale Cloud console shows you your storage autoscaling limit and the next storage plan it will autoscale to.
The Autoscaling option in the Timescale Cloud console shows you your storage autoscaling limit and the next storage plan it will autoscale to.

This saves you from potentially nasty billing surprises that many users have experienced on serverless data platforms, where their services automatically scale opaquely without any insight into limits or why their costs increased.

What can you do if your limit is reached?

Once your autoscaling limit has been reached, your disks enter “read-only” mode, meaning data can no longer be inserted. From here, you’ll be notified to take action. You have three options: first, you can reduce disk usage by compressing data and altering your native compression settings, second, you can enable data retention policies or manually drop chunks, or third, you can update your autoscaling limit to a higher tier. You can alter your autoscaling configuration anytime under your service’s operations tab.

Video tour

If you’d like to see how to use storage autoscaling in the Timescale Cloud UI, watch this 5 minute demo video from Timescale team member Carlota, where she takes you on a tour of how storage autoscaling works:

Case study: Autoscaling to 300 billion rows in 2 weeks

As an experiment, we inserted more than 300 billion rows of data over 14 days into a single-node service, autoscaling from 500GB to 2.5TB of storage. Here are the details:

The experiment

To test the new auto-scaling feature in Timescale Cloud, our Developer Advocacy team set out to insert at least 250 billion rows of data into a single-node server in less than 14 days. To do this, they created a stored procedure that inserted 2.1 million rows of data into a hypertable every 5 seconds, with data timestamps starting from January 1, 2000. They also enabled hypertable compression and automatically compressed any older chunks that no longer had data being inserted into them.

Autoscaling to 200B rows

Once they had an idea of how much data was being generated, they enabled auto-scaling on the Timescale Cloud service to a maximum of 2.5TB of disk. This allowed them to test two features of Timescale Cloud: disk auto-scaling and automatic read-only mode should they fill our allowed disk - to make sure they could still query the data.

As illustrated by the graph below, they let this procedure run for a little over 14 days, inserting 2.1 million rows of data every 5 seconds. Every time the disk was 95% filled, Timescale Cloud automatically increased its storage to the next available plan, as indicated by the discrete jumps of the yellow line in the graph below: from 1TB, to 1.5TB, to 2TB, and to its pre-configured autoscaling limit of 2.5TB, which was the limit in this experiment (you can set this limit as high as 16TB).

The graph with green and yellow line showing the disk usage (green line) and disk capacity (yellow stepped line) for the experiment of inserting 300B rows in 14 days on Timescale Cloud. Every time the team filled 95% of the disk, Timescale Cloud automatically increased storage to the next available plan, as indicated by the discrete jumps of the orange line.
Graph of disk usage (green line) and disk capacity (yellow stepped line) for the experiment. Every time the team filled 95% of the disk, Timescale Cloud automatically increased storage to the next available plan, as indicated by the discrete jumps of the orange line.

Testing read-only safety measures

Once they had inserted more than 200 billion rows of data (in only 1.8 TB of storage), they wanted to verify that the read-only safety measure would kick in if they stopped compressing chunks.

The graph with yellow and green line on the dark background showing the disk usage (green line) and disk capacity (yellow stepped line) for the experiment of inserting 300B rows in 14 days. The blue box indicates when the disk reached full capacity and the database entered read-only mode. Our experiments showed that the safety mechanism was effective in preventing the database from crashing, despite the disk being full and new data attempting to be inserted.
Graph of disk usage (green line) and disk capacity (yellow stepped line) for the experiment of inserting 300B rows in 14 days. The blue box indicates when the disk reached full capacity and the database entered read-only mode. Our experiments showed that the safety mechanism was effective in preventing the database from crashing, despite the disk being full and new data attempting to be inserted.

On October 1, they stopped the compression task from running (as indicated by the blue box in the graph above), quickly filling the remaining storage with ~10 billion rows of data. At this point, Timescale Cloud switched their instance to read-only mode, which they let remain over the weekend. On Monday morning, they re-enabled the compression job, regained space, and started inserting data again.

Experiment Conclusion

In total, our Developer Advocacy team inserted more than 300 billion rows of data using only 2.5TB of storage. With data retention and compression enabled, there's almost no limit for how much data you can store in Timescale Cloud!

Get started today

For existing Timescale Cloud users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.

If you’re new to Timescale, create a free account to get started with a fully-managed Timescale Cloud service (100% free for 30 days, no credit card required).

Once you are using TimescaleDB, please join the Timescale community and ask any questions you may have about time-series data, databases, and more.

And, for those who share our mission and want to join our fully remote, global team: we are hiring broadly across many roles.

If you'd like to connect with Timescale community members, get expert tips, advice and more, tune in to Timescale Community Day later this month for talks (and demos!) about time-series data and TimescaleDB tips.

This post was a collaboration between
11 min read
Announcements & Releases
Contributors

Related posts

TimescaleDB - Timeseries database for PostgreSQL

Explore TimescaleDB

Learn more about how TimescaleDB works, compare versions, and get technical guidance and tutorials.

Go to docs Go to products