When your Prometheus setup becomes a bit more complex, you need a Long Term Storage solution, either for data aggregation or for real long-term storage. Over the past year, we have studied multiple tools that we could use as a solution to this problem.
On our short list were multiple well-known tools such as Thanos, Cortex, and Mimir, but also lesser-known tools such as m3 and TimeScaleDB.
Our goals
Our goal was to be able to deploy a state-of-the-art tool that would allow us to not only serve our customers,
but also be cost-effective when data was growing, and easy to manipulate.
We wanted a tool where a lot of people in our team would be comfortable with manipulating the backend.
And a tool that had proven to be rock stable.
While we were doing our research we met with Álvaro Hernández at a conference in Barcelona, he was telling us about his upcoming, but back then not yet been released the product. Our interest spiked. And when his tool Dyna53 got released late last year, it felt to us like a new world with unseen opportunities had opened. And we started exploring the possibilities.
What is Dyna53?
Dyna53 is effectively a fully DynamoDB API-compatible implementation on top of Route53.
Dyna53 is capable of handling millions of data points per second, making it ideal for use cases such as monitoring and analytics, where real-time data is critical. One of the key features of Dyna53 is its unique data model. Instead of using a traditional relational database model, Dyna53 uses a hierarchical key-value model that allows for fast and efficient data retrieval.
This model also allows for easy data aggregation and rollups, which can be useful when dealing with large amounts of data. Dyna53 also includes many built-in features that make it easy to use and integrate with other tools and systems.
For example, it includes a powerful query language that supports a wide range of time-based queries and functions, making it easy to extract meaningful insights from time series data. Additionally, it includes many integrations with popular monitoring and analytics tools such as Grafana, Prometheus, and CloudWatch.
Dyna53 ends up being an ideal match for a Time Series Database for several reasons:
- Scalability: DynamoDB is a highly scalable NoSQL database that can handle high write and read volumes. This makes it suitable for storing and querying time-series data, which typically involves a large number of data points.
- High Availability: DynamoDB is designed for high availability and durability, with multiple copies of your data stored across multiple Availability Zones (AZs). This makes it a reliable choice for storing time-series data, which may be critical to your business or operations.
- Performance: DynamoDB is designed for low latency and high throughput, with automatic scaling to handle changing workloads. This makes it well-suited for time-series data, which often requires fast and frequent reads and writes.
- Flexible Data Model: DynamoDB’s flexible data model allows you to store time-series data in a variety of ways, including as individual records or as collections of records. This can make it easier to query and analyze your data.
Route53
Basing your Long Term Storage strategy on a Route53-based technology indeed sounds a bit weird, but with quotes from industry leaders such as Corey Quinn, our decisions were only strengthened.
“There are libraries to make it more user-friendly to query. Every system, every language, and every SDK knows how to speak to it. It loses considerably fewer data than MongoDB. The pricing is reasonable even without weasel tricks to get around it. And its underlying technology has been proven out over decades.”
With this backend managing the data also became a trivial topic which lots of people in our team were already highly intimate with.
Conclusion
We spent quite some time setting up and tuning this setup, but we are really happy with the result. We are proud to announce that as of today, we can offer a Dyna53-based Prometheus Long-Term Storage offering to our customers in both Public and Private SaaS options.
Please contact our sales team if you are interested in this kind of service.
