Why Dynamo should be your next web database

One of the larger revelations I've had over the years has been around data relationships and the web. Very, very many applications that you work with today have a very hierarchical structure to their data, and that kind of hierarchy shows up everywhere on the web. Users have posts. Posts have likes. Users have many-to-many relationships. Sites have pages; pages have content slots; content slots have content. When you really look around the web, you start to see patterns with how sites represent data.

SQL is a fantastic language and can represent these data relationships very easily. Add a foreign key of the page id to your content slot, and you have a list of content slots, queried quickly with a WHERE pageid={foo} clause. SQL has been the backbone of the web for many, many years, and choosing MySQL or Postgres or MS SQL Server or what have you is always going to be a safe choice for applications.

Except when it isn't a safe choice. Which is today.

We're in the 21st century, where an application can see a million new users overnight. Getting featured on an app store homepage, or having a popular concept that people can share with their friends, or whatever other OOB event that would cause traffic to explode to your application is a real possibility, and getting that kind of traffic will almost certainly expose the bottlenecks in your infrastructure and app itself.

That number one bottleneck is going to be your SQL server.

Is it possible to scale SQL? Yes, absolutely! You can pick from Amazon's Aurora, Google's Cloud Spanner, or start building out your MySQL/Postgres clustering. However I'm here to say that there is a better way, and that better way is called Dynamo.

OLTP, OLAP, and the web application.

Stop me if you've heard this one. You've authenticated a user's API request, then you go and run a SELECT with a few joins, marshall those into plain-old-(java|javascript|python|whatever)-objects, do a quick transformation on those objects, and finally marshall that result into JSON output to throw back to the browser. Or you do an INSERT. Or an UPDATE. Or a DELETE.

I feel like I've just done requirements def for 90% of nearly every web application out there. CRUD methods alongside some specialized queries make up the vast majority of server-side web engineering; there are hundreds of frameworks out there to make this job easier. (When was the last time your friendly neighborhood Java engineer didn't use Hibernate or Spring boot? Or Rails with ActiveRecord?)

The thing is, in order to build out your data model in your application using one of these frameworks, you have to understand the relationship between your data. You can't just do OLAP style querying using these kinds of frameworks without overriding a huge chunk of what makes these frameworks attractive, so you're mostly doing OLTP work to grab data.

But you don't have to - enter Dynamo.

In your head, answer the question "what would it take for my application to grab a user's profile, and a list of their blog posts." You're looking at a join query, and parsing through records, and ultimately using up a lot of CPU for what should be a simple query. Now multiply that by a million simultaneous users. Ouch.

Dynamo lets you issue the query "give me everything where the partition key is my username", and bam, you have all of your user data, with blog posts, comments, whatever, in only a few milliseconds. Pipe the resulting Items from your query to a JSON.stringify call, and your backend CRUD methods are done. As long as you know your data relations, and you should, using Dynamo for this kind of OLTP work is fantastic.

The curse of connection limits and pools.

If you've used any kind of SQL server in your applications, you have absolutely run into this issue. Load testing immediately starts throwing a "too many connections" error, and you start pulling your hair out trying to figure out how to reduce your application's footprint to your database. You plan to use a global database connection pool, then you start dealing with connections not being released.

"Lambda connection pooling" is an incredibly popular Google search term, and for good reason. How on earth do you have 100 simultaneous lambda functions running hitting your database? "Lambda pgbouncer" is just as popular for you Postgres users. SQL servers have incredibly low connection limits, and they have them for a reason-- you're going to exhaust a CPU when you start issuing that many simultaneous queries.

A huge number of solutions architects have made their careers based on effective database sharding, connection pooling, and managing proxies back to databases. There's a huge amount of infrastructure planning, thinking, and testing when it comes time to scaling these SQL servers out to a large set of users, but guess what? All of that headache goes away with Dynamo. Dynamo gives you an infinite number of connections when it's in on-demand mode, and autoscaling Dynamo's provisioned throughput gives you a ramp of connections for you to use as necessary.

You can't throw too many connections or transactions at Dynamo.

Modeling your data for Dynamo.

The trick here is understanding your data's relationships, and how you want to query your data. Dynamo's speed comes from it being much more of a specialized kv store with indexes. If you know how you're going to look up data, building out those indexes and columns in Dynamo gives you that capability. You can't, however, query on some random column.

Writing down your data access patterns becomes critical here, because you can usually fit your entire application within one Dynamo table.

Yes, one table. If you think I'm crazy, just watch this video from start to finish, and you'll see the power of Dynamo. Rick Houlihan does an incredible job explaining how to use Dynamo to its full potential.

A SQL database simply can't match this kind of raw speed and power.

Dynamo and Lambda as a high-velocity pair of tools.

The video touched on the relationship between Dynamo and Lambda a little bit, but I'll expand on the explanation. Once you have your data model set up in your Dynamo table (or tables), creating CRUD methods in Lambda is extremely fast. Authenticate, do simple POST object validation, then take that data and put it into Dynamo. Read whatever's in Dynamo and spit it back out to the user. Done.

You now have an infinitely scaling data repository. In just a small amount of code, with no extra infrastructure needed. Having this sort of capability within your grasp gives even the smallest of teams the ability to write an application that would have taken dozens of support engineers only a few years ago. You can spin up a fully-functioning data backend for your application within a sprint, and concentrate on your application rather than your infrastructure.

The power of Dynamo and Lambda together doesn't stop there, though. Traditional SQL stored procedures have been messy, mostly allowing just for modifying additional data or doing super-hacky callouts to the underlying operating system. With Dynamo, you can run Lambda functions when data changes. This is stored procedures done right, and done in a 21st-century way. Want to send out a welcome message to a user? Don't bother doing it inside a web route, just run a Lambda when a new user record is created in Dynamo. Get that code running, and bulletproof, and you never need to look at it again.

Again, this is a 21st century database with infinite potential for your application. Your datastore becomes just another place to initiate events, against which you can write infinitely-scaling functions. This is what I like to call the democratizing power of the cloud - it gives the smallest of teams the capabilities of large organizations and it gives large teams unimaginable velocity.

Dynamo conclusions and future direction.

Dynamo has had some growing pains; for the longest time, you had to provision Dynamo with what you thought would be your expected traffic, and autoscaling those limits never worked fast enough to be able to handle sudden spikes in traffic. Most of the Dynamo libraries dealt with the database errors that would get thrown while Dynamo was scaling up, but that's a band-aid over the problem. Amazon dealt with this problem in November of 2018 by introducing on-demand mode for Dynamo, where you pay per millions of reads and writes.

Similiarly, the number of global secondary indexes (the way you can query individual columns) recently was upgraded to 20 indexes; but if you need that many indexes, you're likely modeling your data incorrectly. (Go watch the video again!)

Today, though, Dynamo is a mature solution that some extremely-high-trafficked sites use reliably for all of their data. Its growing pains have been dealt with, and it is an attractive solution for storing data (and acting on that data with Lambda!).

I really enjoy using Dynamo as a high-powered web database. Discarding all of that pain around planning for database sharding, clustering, etc., in favor of concentrating on business logic has really increased my teams velocity, and can give your teams the ability to stand up new applications quickly without having to worry about what happens when a million users hits your application the day it launches.