Pretzel Rocks is a music streaming service from Pretzel Tech, a company that does things a little differently. You won't find the latest top 40 or your favorite deep cuts in its catalog. Instead, on Pretzel Rocks you'll hear hour after hour of music that is a perfect fit for live streamers on platforms such as Twitch. Pretzel Tech is also home to Ninety9Lives, a record label that specializes in music content for online video producers. But that's just the start of what makes Pretzel Tech unique.

Serving the needs of live streamers –– whether on Twitch, Mixer, or elsewhere –– brings with it a highly volatile usage pattern. When a popular streamer goes live, the demand on Pretzel Rocks can go from zero to hundreds of thousands of users in just a few seconds. From day one, Heroku has been home to Pretzel Tech, and today, Heroku remains at the heart of a platform that brings together services from AWS, content distributors, and the streaming platforms themselves.

Streaming music for streamers

Music plays almost constantly in the background of most live gaming streams. For some channels, it fills the silences while the host focuses on gameplay. Others integrate music into the format of their streams, letting viewers choose tracks.

But, of course, most music isn’t licensed for streaming in that way and the streaming platforms actively monitor for copyright violations. Even some of the most popular live streamers have been suspended by streaming platforms for using music without permission.

Former game developer Nate Beck saw a solution. Fresh from launching the Ninety9Lives record label to provide YouTubers with suitably licensed music, Nate already had a roster of music tracks ideal for live streaming. There was, though, a significant difference between YouTube and live streaming. While YouTubers carefully select one or two tracks during their edit, streamers can get through hours of music in just one session.

Nate recognized that streamers needed a service that would play stream-safe music automatically, and so he set about building Pretzel Rocks.

The best tool for the job

Let’s take a moment to look at some numbers. Twitch, the most popular live streaming platform, streamed 609 billion minutes of content from 3.7 million streamers in 2019. The average stream viewership was around 26 people, yet popular channels frequently attracted live audiences in the hundreds of thousands.

For the Pretzel Rocks platform, those numbers inform almost everything about its architecture, falling into three areas:

• Providing an interface that serves the millions of live streamers
• Streaming the music itself to those users
• Integrating with platforms, such as Twitch, and serving the viewers of streams

From day one, the Pretzel Tech design philosophy has been to use the best tool for the job. At the heart of the platform is a Heroku-hosted Ruby on Rails application that acts as the source of truth and marshal data for multiple other services. Today, the Pretzel Tech platform includes services running on:

• Heroku
• Netlify
• AWS
• the Stackpath CDN
• and the Twitch platform

Each one was selected for its suitability to the job in hand.

Heroku is at the center of our infrastructure, tying everything together. That makes it easy to choose the best service for each part of our tech stack, with Heroku delivering a solid base from which to build a multi-platform architecture. Nate Beck, Founder, Pretzel Tech

Heroku at the center of a multi-platform architecture

For the most part, the Pretzel Tech platform looks like many other Rails-based web apps, with components including:

• Artists’ Application: an internally focused Rails app that runs on Heroku and manages the business aspects, such as artist royalty payments.
• Vanguard: the Pretzel Rocks music catalog that stores metadata about each track that is available to stream.
• API server: the core Rails app from which all interfaces to Pretzel Rocks gather music information and trigger streaming. The team recently updated the API server to leverage GraphQL.
• Background Job Queue: using the open source Resque project, Pretzel has a fleet of dynos always running on Heroku. These jobs include tracking who is currently live streaming, delivering messages to chat, as well as tracking play data and analytics.

These apps each run with Heroku Postgres as their core database, supported by Heroku Redis for ephemeral data and caching.

Heroku Redis has been essential to making more data available more quickly, and because Heroku handles everything, it adds virtually no ops burden. Nate Beck, Founder, Pretzel Tech

Pretzel Rocks has a number of front-end apps, with two in particular that deliver the Pretzel Rocks player to live streamers:

• Web front end: a ReactJS app hosted on Netlify that consumes the API served by the core Rails app.
• Desktop app: an Electron-based desktop player that also consumes the API served by the Heroku-hosted Rails app.

The Pretzel Rocks team chose Netlify to host their web front end. As a ReactJS app that relies entirely on Pretzel Rocks’ Heroku-based API app, the front end has very simple hosting needs. When a browser requests the front end’s URL, Pretzel Rocks needs only to serve the React app as a static file.

Working with Netlify was immediately familiar to the Pretzel Tech team. Netlify takes a similar approach to Heroku, albeit for serving static content, in that Netlify sits at the end-point of a CI/CD pipeline that builds the React app from source.

So far, so normal, but that’s where “normal” ends. Let’s talk first about getting the music to streamers. Like any streaming service, including Netflix, there’s a distinction between Pretzel Rocks as an application and the infrastructure required to deliver the content itself. The music files and artwork graphics reside in Amazon S3, with the Stackpath CDN distributing the streams to end users. With 100,000 Pretzel Rocks users, that translates to around 30 terabytes of streamed data each month.

That’s a lot of data, but the requests per second are relatively modest. For most other music streaming services, this would be the extent of the platform. However, Pretzel targets Twitch streamers, and these streamers have an audience of viewers who also need to interact with the Pretzel platform.

Complementing Heroku with AWS Lambda

Pretzel Rocks allows users to stream with all major platforms, but it’s Twitch that remains its closest partner.

When a Twitch streamer uses Pretzel Rocks, they can add a Pretzel Rocks widget to their streaming page. That widget, a Twitch extension, lets viewers see what’s playing, tip to choose the next track, and even compete with other viewers for whose preferred track plays next.

This is a source of extreme traffic spikiness. For Pretzel Rocks, a popular streamer going live can mean that hundreds of thousands of widgets suddenly start polling for “now playing” data. Then, once the stream ends, those thousands of widgets fall silent.

Like all Twitch extensions, the Pretzel Rocks widget lives on the AWS CloudFront CDN and it was in the AWS ecosystem that the Pretzel Tech team found the solution to handling those peaks and troughs in demand. Rather than direct all those thousands of concurrent requests to the core Rails app, the Twitch widget polls AWS API Gateway, which routes requests to an AWS Lambda function. This can be a stretch even for AWS Lambda, which has a maximum number of 20,000 concurrent tasks. Nonetheless, the Lambda function serves a cached version of the Heroku-hosted API, meaning that smoothing out that traffic volatility is virtually effortless.

Future rearchitecting will focus on serving even larger scales as Twitch’s popularity grows. The key, though, is that the Heroku-hosted Rails app remains the source of truth. Lambda simply offsets the extremes of demand.

Heroku helps Pretzel Tech execute faster

As a small team, the Pretzel Tech developers have a laser-like focus on minimizing distractions. For them, the right tool is the one that helps them to execute fastest. By putting Heroku at the center of the Pretzel Rocks application architecture, they have been able to minimize how much time they spend managing ops, and are better able to prioritize delivering features into production.

And it's not only the core Heroku platform that has made Pretzel Tech a low-touch operation. By using Heroku Redis, Pretzel Tech has a low-ops solution to reduce the read-load on its Heroku Postgres database. Similarly, the Scout APM application monitoring add-on and Papertrail logging add-on have been crucial in identifying inefficiencies both in Pretzel Tech's application code and database architecture.

I hate doing ops, which is why I love Heroku. Heroku lets me focus on writing code that delivers value to my users. Nate Beck, Founder, Pretzel Tech

Heroku is a natural hub for integrating with third-party services

Pretzel Rocks has grown from a niche music streaming service to being a central part of the experience offered by many live streamers. Through its partnership with Twitch, the platform faces remarkable extremes of demand that would put a strain on even the most experienced infrastructure architects. The key to Pretzel Tech’s success has been a pragmatic approach to choosing the best tool for the job and an openness to mixing solutions from different vendors.

Throughout each architecture decision taken by Pretzel Tech, Heroku has been a constant at the center of a platform that combines Netlify for a ReactJS front end, a global CDN for data streaming, and AWS Lambda to offset extreme peaks in demand.

Listen to the Code[ish] podcast featuring Nate Beck: Streaming Music to Livestreamers.

Inside Pretzel Tech on Heroku

At the heart of the Pretzel Tech platform are several Ruby on Rails applications running on Heroku, backed by Heroku Postgres and supported by Heroku Redis. Pretzel Rocks makes extensive use of Heroku Add-ons, including Scout for application performance monitoring, SendGrid for transactional email, and Papertrail for log aggregation. The platform also uses Sentry for error tracking and monitoring.

The Pretzel Tech web interface runs as a ReactJS application on Netlify, making calls to the Heroku-hosted API server, while the Stackpath CDN takes care of delivering music to listeners. To serve the volatile usage patterns resulting from the platform's partnership with Twitch, AWS Lambda serves a cached version of the Heroku-hosted API.