Nineteen thousand shipping containers enter the U.S. each day. For most of them, that's just the start of another journey to their final destinations across the country. Until recently, managing that process was largely manual, which resulted in a range of inefficiencies across rail yard operations. In 2018, a collaboration between digital agency GNAR and freight yard operator RMS Intermodal delivered a digital transformation solution to help the intermodal industry solve many age-old problems.

GNAR built a rail yard management platform for RMS called Intrmodl. Running on Heroku, the IoT platform delivers a real time view of the location of each vehicle, crew member, and container, along with workload and efficiency data to help site managers optimize their operations. From day one, Heroku has been integral to Intrmodl's success, enabling GNAR's lean team to deliver features rapidly without losing precious headcount to DevOps duties.

Improving on a truck, a CB radio, and a good memory

Rail terminal sites are big, and crews tend to be spread out across a wide area. When a locomotive pulling 200 rail cars enters a facility, the right staff and vehicles must be on hand straight away to start unloading the freight. Traditionally, that required a manager driving around the rail yard corralling staff to head immediately to the right track. Lucky managers might have a CB radio to help them spread the word. Either way, simply getting people and vehicles to the correct place quickly was both time consuming and labor intensive. Such operational inefficiencies were rife throughout the intermodal industry.

For the GNAR team, this presented both an opportunity and a challenge. The opportunity was clear: a digital management system could transform the way yard managers coordinate staff, workloads, and container movement. The challenge was that whatever they produced had to deliver an immediate improvement in order to convince people to change decades-old work practices.

Tracking vehicles with real-time GPS data

Most logistics-focused IoT projects are built using RFID (radio frequency identification). When scanned, RFID tags emit a unique code via low power radio. They’re cheap, small, and don’t need a power supply.

However, RFID does require significant infrastructure throughout each site where it is used. Even then, RFID systems reveal only which zone an item is in rather than its precise location. To enable rollout at yards across the U.S., the GNAR solution had to be more agile. Instead of RFID, GNAR chose to build Intrmodl using real-time GPS coordinates streamed from Android tablets in each vehicle. By turning the vehicles themselves into connected IoT devices, it not only simplified on-site infrastructure, but it also provided much finer grained location information. However, the flipside was that it put greater demand on the platform’s data layer.

Managing industrial IoT data with Apache Kafka on Heroku

Processing streams of real time data from IoT devices presents a challenge for traditional app infrastructures. Each vehicle monitored by Intrmodl sends back ten sensor readings, including its GPS coordinates, every two seconds. There’s no room for lag. Intrmodl must ingest and process that data immediately so that yard managers can act on it in a timely manner.

At launch, Intrmodl relied on a single Node.js app backed by Heroku Postgres. This was more than capable of managing a single facility. However, RMS Intermodal wanted to use the new platform at multiple freight terminals, which required a more scalable data architecture.

After experimenting with different options, the GNAR team chose an event streaming approach to handling sensor data and built their own Kafka cluster. Stress testing showed that Kafka itself was an almost perfect solution. It could ingest, transform, and route data with virtually no lag. However, administering it would be a full-time job for a DevOps specialist. With their commitment to rapid feature delivery, GNAR wanted to ensure that their limited headcount budget was entirely dedicated to software development roles rather than DevOps. At the same time, they wanted to ensure that their existing engineers focused on feature delivery rather than becoming sidetracked by DevOps tasks.

Having built their existing platform on Heroku, the GNAR team were keen to investigate Apache Kafka on Heroku. The benefit of Heroku's managed Kafka service was more than simply removing the DevOps burden from the GNAR team. With thousands of instances under their management, Heroku's team of Kafka experts would be able to draw on deep experience of tuning Kafka in order to get the absolute best out of the streaming platform. Similarly, having relied on Heroku's support in the past, the GNAR team were reassured that there would always be a friendly answer to their Kafka related questions.

Today, Intrmodl running with Apache Kafka of Heroku provides yard managers across the United States with the precise activity and location of each vehicle type working within their rail terminal. When a train comes in, they each have a clear view of which crew members are closest to the right track and also available to work. Rather than waiting for a shout from a truck or a call via CB, workers receive alerts on their in-vehicle tablets telling them where they need to be and when.

Apache Kafka on Heroku gives us expert fine tuning of the platform, along with day to day management and metrics, that we simply couldn’t achieve by ourselves. Yuri Oliveira, Full Stack Engineer, GNAR

Heroku Postgres is the data heart of Intrmodl

While Apache Kafka on Heroku gets inbound data to the right place, it is up to Heroku Postgres to ensure that data is available when the platform needs it. At the heart of Intrmodl’s data layer is an operational Heroku Postgres database, while a follower instance is available for analytics and other secondary tasks without putting load on the primary instance.

Heroku Postgres is crucial to enabling Intrmodl to improve operational efficiency at RMS Intermodal’s terminals over the long term. In particular, the system tracks the hours worked and the container movements of each team member. Heroku Postgres queries then take that data and generate productivity metrics, such as number of container movements per hour worked. Managers can then take both a macro and a micro view of the efficiency of a particular efficiency. For example, are some days of the week more productive than others? Or perhaps there are individual crew members who need encouragement to bring their own productivity up to the standard of their colleagues.

As Intrmodl usage has grown, such queries have put a greater load on the platform's Heroku Postgres instance. Rather than risk decreasing the responsiveness of their apps, the GNAR team turned to Heroku Redis to cache the most commonly used database queries. This relieved pressure on Heroku Postgres by ensuring those queries ran only when the results were likely to have changed, rather than for each page load of Intmodl's management app. With Heroku Redis serving data directly from memory, rather than disk, this delivered a significant performance boost while avoiding the need to increase the size of the Postgres instance.

Heroku is more than just a deployment platform. With its amazing documentation and responsive support services, Heroku has become an extension of our team. Yuri Oliveira, Full Stack Engineer, GNAR

Heroku powers Intrmodl’s growth to meet new challenges

To date, five of RMS Intermodal’s freight terminals have fully adopted the Intrmodl platform, with another 45 beginning their journey to transform rail yard operations.

For GNAR, the Intrmodl project showcases what a relatively small development team can achieve when they are free to focus entirely on delivering value. With Heroku taking care of DevOps across the entire Intrmodl platform, from app servers, through the data layer, to metrics and monitoring, GNAR has been able to concentrate its resources on feature development.

In the near future, GNAR will be exploring how to apply Intrmodl to managing a similar type of site operations for the automotive industry. The company’s immediate focus, though, is to transition day-to-day management and future development of the platform to RMS Intermodal’s newly hired internal development team. Heroku plays a vital role in simplifying that transfer of responsibility, as the new team needs to learn only the software code rather than the underlying complexity of keeping the platform running.

Thanks to Heroku, we’ve been able to focus entirely on delivering functionality that makes a measurable improvement to our client’s business. With Heroku, DevOps is not, and never will be, our problem. Brandon Stewart, Founder, GNAR

Read more about GNAR's story on the Heroku blog: How to Transform a Heavy Industry, One Sensor at a Time.

Listen to the Code[ish] podcast featuring Brandon Stewart and Yuri Oliveira: Monitoring Productivity Through IoT.

Inside Intrmodl on Heroku

The Intrmodl platform runs eight Node.js applications on Heroku, backed by an operational Heroku Postgres database and a follower Heroku Postgres instance for analytics. Heroku Redis caches commonly used queries, to ensure fast end-user response times, and Apache Kafka on Heroku ingests data from hundreds of industrial IoT devices across the United States.

The GNAR team makes extensive use of Heroku's Application Metrics and the Librato Heroku Add-on to monitor the status of their platform. Heroku Scheduler takes care of regular jobs, while the Logentries add-on provides additional insights.

Heroku helps firms like GNAR grow their business. Learn more about Heroku for digital and software development agencies.