Introducing one-click database forking in Timescale Cloud
Today, we are excited to announce the support for database forking in Timescale Cloud. Database forking enables you to create a copy of your database in a single click, simplifying your development and testing process.
Timescale Cloud is a database cloud for relational and time-series workloads, built on PostgreSQL, and architected around our vision of a modern cloud service: easy, scalable, familiar, and flexible. To that end, we’ve recently released larger disk plans of up to 16TB per node and features like one-click multi-node deployments and storage autoscaling.
Today’s release of database forking enables development teams to perform a variety of important tasks in their workflow in less time and with more flexibility, such as creating testing environments, performing safer upgrades, maintaining up to date staging environments, creating snapshots of data, and providing access to production data without access to the production database itself (e.g for data science or business intelligence teams).
Forked services are created with the same data and configuration as the parent services. After the time of forking, your forked service will be a completely independent database, meaning changes in the forked service won’t be reflected in the parent database and changes to the parent database won’t be reflected in the forked service. However, you’ll always know which database a service was forked from, as we maintain an internal relationship between forked databases and their parent database.
To fork a service in Timescale Cloud, pick the service you’d like to fork, navigate to the “Operations” tab, and click the “Fork service” button. That’s it! A copy of your database will be created and displayed on your “Services” page, with a message indicating the parent database that it was forked from.
Read on for more information on how forking works under the hood, some ideas for how to use forking to simplify your development workflows, and details of using forking in Timescale Cloud.
Database forking is available immediately to all current users of Timescale Cloud. If you’re new to Timescale Cloud, you can create a free account to get started (100% free for 30 days and no credit card required).
Once you are using Timescale, join the Timescale Community and ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team, we are hiring broadly across many roles!
A huge thank you to all the engineers and designers who worked on this feature: Anthony Dodd, Camila Hirthe Memelsdorff, Feike Steenbergen, Hardik Maheshwari, Ivan Tolstosheev, James Hong, Oleksii Kliukin, Rob Kiefer, and Sam Gichohi.
Using database forking to simplify your workflows
The ability to easily fork your database comes in handy for multiple scenarios. Here are four common ones:
Testing. If your team needs a shared set of data for running tests but you want to prevent that data from being modified, here’s our recommendation:
- Pause your service. When your services are paused, you only pay for storage (no CPU costs); this will help reduce your bill during testing.
- Create a fork of the paused service.
- Run the tests against the running fork.
- Once your team is done, delete the fork.
- Resume the main service.
You can repeat this process as regularly as you need in your testing environment.
Safer database upgrades. When doing a major upgrade in a production database (e.g. upgrading from PostgreSQL 12 to 13), we recommend forking your service first. This way, you can perform the upgrades on the forked service first, ensuring that there are no issues related to this change. Once you’re sure the upgrade was successful, you can be confident that everything will work well when running it on your production service.
Create and refresh staging environments. An important aspect of a good testing procedure is having a staging environment with production data, so you can test the quality of your new features in real-world conditions. Through database forking, you can spin up an exact copy of your production data without affecting the actual production service. Also, as your production data changes over time, it is good practice to refresh your staging service, as the conditions may change - to have an easy procedure for forking makes this task painless.
Create a snapshot of your data. Keeping database snapshots can be very useful for auditing and reporting, and also for doing potential analysis or forensics after carrying out an important change in your service configuration.
Provide access to production data (but not to the production database). Many times teams of data scientists or business intelligence analysts might want access to production data to query and analyze. Database forking enables you to provide access to production data, without having to provide access to the database itself. This is especially useful for cases where implementing access control via Postgres might be too complicated. Simply fork the database and provide teams access to a copy of that data, while your production database continues to operate unaffected.
Forking: Under the hood
Every service running in Timescale Cloud has a backup that we regularly test. This backup is more than a snapshot of the data directory: powered by the continuous archiving feature of PostgreSQL, it contains all database changes at a given point in time. It can be used to restore a database even when the original volume containing the data directory is unavailable. This is how we fork: instead of restoring this backup in-place, it is restored to a new instance that becomes a “clone” of the original one.
See our blog post on continuous backup/ restore validation on Timescale Cloud for more details.
Forking a service in Timescale Cloud
Here’s how to create a fork of your database in Timescale Cloud:
- Navigate to the service you’d like to fork.
- Go to “Operations”.
- Click on “Fork service”.
- Voila! You will immediately see a copy of your database on your Services page. It may take a few minutes for your database to be fully available (in some cases longer if you have a lot of data). Once it’s ready, you will see a green “Running” status.
At Timescale, we like to move fast (without breaking things), so this release of database forking is just the beginning. We are working on releasing the following improvements in the near future:
- One-click forking for multi-node services. (Timescale Cloud fully supports multi-node deployments, but for now, forking is only available for single-node databases.)
- Forking from a certain point in time (PIT) so you can fork older states of your database.
- Programmatic API to automate forking.
- Forking with an alternative configuration than the parent service (different CPU, memory, or disk).
- Forking to a different region than the parent service.
Stay tuned! And let us know if there’s an improvement or feature you’d like to see.
Database forking is available immediately to all current users of Timescale Cloud. See the Timescale documentation for information on how to get started with database forking on Timescale Cloud.
If you are new to Timescale Cloud:
- Create an account to get free access to Timescale Cloud for 30 days, with no credit card required - and start forking today!
- Join the Timescale Community Slack and ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. Join us: we are 7,000+ and counting!
- Read our vision for Timescale Cloud: a database cloud for relational and time-series workloads, built on PostgreSQL and architected around our vision of a modern cloud service: easy, scalable, familiar, and flexible.
- Check out our Getting Started documentation. These articles walk you through the basics: creating your first instance, accessing your database, loading your data, and so on. Get familiar with key TimescaleDB concepts like hypertables and chunks, compression, or continuous aggregates. Understanding these key features will allow you to use Timescale Cloud up to its maximum potential. ✨
And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team, we are hiring broadly across many roles!