scaling

Serving 20 million objects for nearly nothing

Using AWS serverless to build a COPPA-compliant save-and-share feature that has run for ten years.

20M objects · 10 years · 100% uptime

Situation: Our client wanted a way to save and share the current state of the application.
Constraint: The audience was four-year-olds, so the feature had to comply with COPPA. Technically it had to scale from zero users to thousands while keeping storage and maintenance costs to a minimum.
Decision: I chose AWS serverless for the application and S3 for storage, deployed through AWS SAM — which was new at the time.
Tradeoff: Hosting the service on a dedicated machine was the comfortable route. It would have meant a beefy box, or several boxes and a load balancer, sized for a peak we could not predict. AWS's managed services ticked every box, and the free tier made it cheap to find out whether an unproven system would hold.
Result: Prototyped in 2016, launched in 2017, and still running mostly the code I wrote then. Seven contributors across five releases, most of them maintenance or language-compatibility updates. Ten years at 100% uptime. The monthly bill is S3 storage and bandwidth on more than 20 million objects and roughly a terabyte. For a well-architected system at a certain scale, serverless is unbelievable.

What the product had to do

Our client has a series of cross-platform math apps. They let teachers and students drop counters and other pieces onto a board to support a visual math curriculum. Every app opened on a blank slate: an empty board and a palette of pieces waiting to be placed.

Save and share was designed around four requirements:

The share codes had to be simple
They had to work cross-platform
They had to work without a student account
Specific codes had to be settable as evergreen

Share codes are eight-character alphanumeric strings, displayed with a dash in the middle: A0B2-5DF3. That is simple enough for a four-year-old to use with guidance, and no obstacle at all for an older student. Students might be on an iPad or on the web, and a code generated on one had to work on the other, in any browser. Because the service could not sit behind a user account, it had to be robust on its own: these apps are free, they have a large user base, and the feature simply had to work. One lesser-used capability was that certain codes could never change, so an app could be preloaded for a curriculum unit or set up as a demo on a trade-show floor. Those required a separate back-end procedure.

Beyond the functional goals, the service had to:

have low operating and maintenance costs
be easy to support
be developable locally, with a minimum of “works on my machine”

What COPPA actually constrained

To work within COPPA, there could be no user accounts at all. No accounts means there is no identifiable information to store or to share. What goes into a share is the state of the app and nothing else: the position of the pieces on the board. What the teacher receives is thousands of near-identical finished boards, turned in as share codes in Google Classroom. There is no classroom integration and no way to correlate a code back to a student. Later versions of the API allowed educator saves for logged-in teachers in one of the apps, but never student information.

The biggest infrastructure hurdle was retention and replication policy. It is one thing to keep every state forever; it is another to delete them after eighteen months and replicate production states down to the development environments.

A code is a SHA hash of the state, truncated to eight characters. We had discussed the possibility that a hash might spell something vulgar, and had written it off as very unlikely. Several days before launch, one of our testers got back the code RAT5-H0L3. Given the age of the audience, that was close enough to call me off vacation to write a vulgarity filter. I found an official list of banned words and expanded it with variants, transliterating parts of each word into leetspeak equivalents. After hashing, the generated code is checked against that list; on a hit, I perturb the state in a benign way so it hashes differently, then check again. It would have been a fun problem to solve had it not been three days from launch.

The architecture

To the user it is seamless. On the front end, a user taps the share button; the app wraps up the JSON, sends it to the HTTPS API, and receives a code in response. The app also takes a screenshot, but that never leaves the device. To retrieve a save, the user taps load and types the code. The app will also load a save if the code is appended to its URL, and iOS devices use deep links to open the app straight to the save.

On the back end, CloudFront points at API Gateway, which sends the payload to Lambda. The obscenity filter runs in the Lambda, and the JSON is written as an object in S3. Retrieval is the same path in reverse, with one exception: evergreen codes are stored in a separate bucket, so the Lambda looks there first and returns that save if it exists.

Deployment is handled much the same way as when I first wrote it ten years ago. Some things have been streamlined — we use Makefiles instead of deployment scripts — but the process is unchanged. Releasing a new version ends with changing a few environment variables and pointing the CloudFront distribution at it. That could be automated, but there is little to be gained by taking the human out of the deployment loop.

Why not a box

This was not the first system of its kind we had built. We were primarily a Drupal shop, and we had built something similar for teachers a year earlier. It was not messy, but it had many moving parts, and the potential scale here was not close to comparable, so we looked for another way.

Going serverless is why the service has run for ten years on minimal maintenance. The biggest hurdle to an update is researching language deprecations, and Python moves slowly enough that this has been relatively painless. We have had to update the tests more often than the code.

SAM was new when we tried it. The documentation was thin and there were few examples. I had put Lambda through its paces and experimented with API Gateway, but this was the first genuinely production deployment we had done on it. The AWS free tier made it a gamble worth taking. I prototyped the development environment and proved out repeatable deployments before signing off on building the full system.

I still have a hard time believing what this system costs. It is several orders of magnitude cheaper than our client’s other services, and it sees more day-to-day use than any of them.

What it cost

The biggest cost of this route is lock-in. Writing for Lambda means writing around the way Lambda works. There is no easy way to use the standard library for the handler, or to wrap it in a common framework like Flask.

Local development was not as easy as I had hoped. SAM emulates only the Lambda runtime and the API Gateway endpoint; reaching any other AWS service means mocking its API or talking to the real thing. I used an open-source tool called MinIO for local S3. It worked for years, until they dropped support for our use case and we had to find an alternative.

Maintaining a full local AWS stack is hard. Set against how light every other cost is, the robustness is worth it.

Ten years later

A product and a solution do not often fit this well. Sometimes a project finishes so cleanly that you have to ask why it went as well as it did. The problem was simple, and a simple problem allows a simple solution. A vendor was offering a near-exact match to that solution. Everything lined up, and I was in a position to see it.