June 09, 2020

How we migrated our website from Handlebars to React in 12 weeks

Jamie Kelly

Updated on June 19, 2020

TL;DR: We built an automated tool that let us incrementally migrate, while keeping up regular feature work and maintenance.

As a company, Plaid has reached critical mass and is rapidly expanding our product offerings and website to serve new markets. Today, we connect over 11,000 financial institutions across the United States, Canada, and parts of Europe. We power thousands of applications that users rely on to manage their financial lives and our website has scaled to serve millions of visitors every year. As we continue to grow, so do the requirements for plaid.com.

Our website is the nexus of the Plaid user ecosystem. Customers like Venmo and Robinhood visit it to learn about new features and products, developers use it to reference API documentation in order to build new applications, and end users seek details about our data practices and policies. Making this information readily available to global audiences is imperative to the success of our international growth.

Scaling problems

As we began our expansion into new markets, we noticed the number of incoming change requests increased with each additional locale. Growth experiments no longer pertained to North American markets and what historically worked in the United States was not working in the United Kingdom. Serving users at a global scale demanded faster iterations, more experimentation, and localized content. As internationalization marched on, our team would need to effectively handle the ever increasing complexity of maintaining a globalized static website.

At the start of the year, plaid.com consisted of over 25,000 lines of markup, 108 static pages, and nearly 300 UI components. The website was built with the Handlebars templating language and flat file generator for compiling to static HTML. We used Foundation, a dated CSS framework, and most of our client-side logic was one-off jQuery scripts that did not effectively leverage modern JavaScript features. Maintenance was arduous, and the static nature of the website prevented us from optimally serving market-specific content. There wasn’t much in the way of code reuse; with each new market added we were left hacking together solutions by duplicating pages to quasi-represent new locales. This wasn’t scaling, tech debt was piling up, and our legacy dependencies limited new innovations.

The problem was evident: what worked for a scrappy startup a few years ago, was no longer working for a maturing business. We needed to migrate to a fully extensible, zero configuration, production ready framework that would allow the team to hit the ground running. We also wanted a system that could easily attract talent by using JavaScript on both the frontend and backend. So we evaluated React frameworks like Gatsby.js and Next.js. We loved the performance benefits of Gatsby.js, but ended up deciding on Next.js due to its maturity and ability to easily server-side render pages.

Migration approaches considered

We’re constantly shipping updates to plaid.com in order to deliver the best experience for our users, so it wasn’t acceptable for us to pause work while undergoing the migration. We needed to approach this in a way that would allow the team to keep up momentum, while avoiding any unnecessary downtime. This is how we thought about it:

1 // Build a new website from scratch

Building a new website from scratch is often an extremely time consuming challenge for a small team. At the time, our web development team consisted of only three engineers, and we couldn’t allocate resources to starting over. Our hands were tied with regular maintenance and new feature requests, making it impractical to consider taking on such a massive initiative.

2 // Compile Handlebars templates to HTML then wrap the output in React

Using a flat file generator (e.g. Panini), we could compile all of the Handlebars templates to HTML and then wrap the output in a React component. This is a better solution than starting from scratch. It’s a cheap and easy way to get the ball rolling, but at what cost? Compiling to static HTML would defeat the purpose of using a templating language. We would lose all of the benefits of Handlebars, such as conditionally rendering components, and iterating over data. Our website would be frozen in its current state and we’d be left to manually abstract all of our data and components from each compiled page.

3 // Server-side render the website and incrementally migrate each page

Similar to Hulu’s solution, we also considered rendering the templates server-side with the Handlebars view engine for Express.js. This would allow us to unblock the international expansion by enabling dynamic content sooner. However, as we incrementally migrated pages, we would need to maintain several top-level components like primary navigation, supporting dynamic internationalization logic, in both Handlebars and React. This context switching would present new challenges in how duplicate components are maintained making our website susceptible to errors and inconsistencies across pages.

4 // Compile Handlebars templates directly to React

All of the options considered above came with tradeoffs that we couldn’t justify at the time, so we asked ourselves, “could we build a script to handle most of the migration for us and manually migrate only a small number of complex pages?”.

Building a compiler

The engineering teams at Plaid prefer to write technical specifications for larger projects that generally take more than a month to complete. We do this because it forces engineers to think through all of the details of the design, gather feedback early, and accurately scope the task at hand. Part of that process sometimes involves building a proof-of-concept in order to test theories early, and ensure the project will succeed.

While thinking through the different approaches to the migration, we discovered the handlebars-to-jsx library. It did a great job of compiling some of our Handlebars expressions, but could only assemble very simple components. When it came to more complex templates that invoked other components (i.e. partials), the library was unable to compile without throwing an error.

Since our website contained over 900 invocations of partials throughout 146 files, the library, in its current state, was only capable of compiling 35% percent of our website. This inspired us to build our own compiler to use in conjunction with the library.

Since, for our purposes, we were only dealing with simple markup and a few Handlebar helpers, there was no need to parse our code into an Abstract Syntax Tree before compiling. In fact, the compiler we built was actually quite simple. At a high level, it just read the contents of a file into an array and applied several reducer functions to transform each line of code. Below is an example of the first compiler function we wrote for our proof-of-concept:

1/**
* Replaces all {{> partials}} with a <Component />
*
* @param {string[]} jsx
* @param {string} html
* @returns {string[]}
*/
8const compilePartials = (jsx, html) => {
// matches {{>component-name}}
const wholePartialPattern = new RegExp('{{>(.*?)}}', 'g');

// matches everything inside of {{> }} so in this case,
// "component-name" is matched
const partial = new RegExp('(?<={{>).+?(?=}})', 'g');

if (wholePartialPattern.test(html)) {
 html = html.replace(wholePartialPattern, matched => {
   const [fileName] = matched.match(partial);
   const [camelComponent, ...props] = fileName.trim().split(' ');
   const componentName = capitalize(_.camelCase(camelComponent));

   const component = [
     componentName,
     ...(props.length ? [props.join(' ')] : []),
   ];
   return `<${component.join(' ')} />`;
 });
}

return [...jsx, html];
31};
32

This function coupled with the Handlebars to JSX library, would turn a Handlebars template like this:

1---
2meta-title: Homepage
3meta-description: "Develop the future of fintech with Plaid, the technology layer for financial services. Plaid enables applications to connect with users’ bank accounts."
4layout: default
5---
6 
7{{> hero-homepage }}
8{{> homepage-bottom-layer }}
9{{> homepage-what }}
10{{> horizontal-mobile-scroll }}
11{{> feature-section-homepage }}

Into a React component like this:

1import React from 'react';
2 
3import DefaultTemplate from 'src/templates/default';
4import HeroHomepage from 'src/components/headers/hero-homepage';
5import HomepageBottomLayer from 'src/components/homepage-bottom-layer';
6import HomepageWhat from 'src/components/homepage-what';
7import HorizontalMobileScroll from 'src/components/horizontal-scroll/horizontal-mobile-scroll';
8import FeatureSectionHomepage from 'src/components/features/feature-section-homepage';
9 
10const metaData = {
11 'meta-title': 'Homepage',
12 'meta-description':
13   'Develop the future of fintech with Plaid, the technology layer for financial services. Plaid enables applications to connect with users’ bank accounts."',
14};
15 
16export default () => (
17 <DefaultTemplate {...metaData}>
18   <HeroHomepage />
19   <HomepageBottomLayer />
20   <HomepageWhat />
21   <HorizontalMobileScroll />
22   <FeatureSetionHomepage />
23 </DefaultTemplate>
24);

We had to write an additional 10 reducers to handle other helpers and edges cases not included in the handlebars-to-jsx library. After that, we were able to compile almost the entire website using custom directives to ignore erroneous blocks of code. With more than 400 files processed, this resulted in:

82% compiled with no errors
3% compiled with minor warnings
9% partially compiled
5% required manual migration

Incrementally migrating

One of the biggest challenges of a website migration is the transition from old to new systems. By automating the brunt of the migration, we estimated the team would save a minimum of 120 hours of work while enabling a smooth transition from new to old. Manually rewriting hundreds of files from Handlebars to JSX was shaved off the migration process by our compiler, not to mention a lot of stress.

We wanted to keep the migration within the same repository as the legacy website so that we could share dependencies, make it easy to reason about, and run both websites concurrently while we migrated the remaining templates and tested our changes in production. However, Next.js required that we upgrade some of our legacy dependencies like Babel and Webpack. Upon upgrading, we discovered that Browsersync was no longer compatible with the latest version of Babel. We didn’t want to change our regular developer workflow, so we forked the unmaintained dependencies, fixed the errors, and published them on npm.

By keeping the legacy and migrated websites under one repository, we were able to share dependencies like styling, static assets, and JavaScript between both websites. We were also able to wire the migration script into our existing Gulp task runner to incrementally migrate for us. We had the Gulp watch task detect changes to legacy files and automagically migrate the new changes to Next.js. This allowed other contributing teams to continually iterate on our API docs without the two sites diverging much.

Without the Gulp task runner watching for incremental changes and recompiling them into the new website, contributors would be left making duplicate changes while both websites existed. One changeset for the legacy codebase, and the same again for the new site. They would also need to deal with the mental overhead of “what has been migrated, and what has not?” Over time, this list would inevitably begin to slip our minds and make it very difficult to keep abreast the migration.

Results

In 12 weeks we were able to successfully migrate 25,000 lines of code to Next.js without duplicating feature work. From early feedback and acknowledgements in our #praise Slack channel, we’re already finding that engineers prefer to work with React components and vanilla JavaScript over Handlebars and jQuery. Additionally, our international team is excited to start working with modern, proven technologies for their future endeavors.

While interviewing new candidates for the team, we often pitched that the migration to Next.js would enable us to write UI components using React. In fact, during our migration we were able to double the team to 6 members, thanks in part to this effort. Interview candidates often expressed a lot of interest in working with Next.js and our newest hires are happy they did not have to endure the legacy tech stack.

What we learned

While migrating the website, we learned a lot about our legacy code base:

The preexisting code was littered with invalid syntax which caused the compiler and React to choke. By spending more time testing our API documentation we uncovered code samples that were often missing important tokens because of Handlebars markdown compilation bugs. After testing the website for launch, we ended up documenting 49 low priority bugs unrelated to the migration, and we were able to resolve all these bugs before going live.
Our API docs were a mixbag of Markdown, markup, and Handlebars that required an additional custom compilation step on top of the Handlebars-to-JSX compiler and our hand rolled compilers. Compilers all the way down.
A thorough audit of dependencies led to deleting several HTML, CSS, and JS artifacts. Additionally, we were using dependencies that are no longer maintained and we plan to replace as we continue to iterate.

What we would do differently

Migrating plaid.com off of these legacy technologies was long overdue. In fact, it probably should have started before expanding into new markets so that we were well equipped for our speedy growth. Now that we’ve migrated, we will need to retroactively patch up some of our temporary workarounds from initiatives aimed at expanding into parts of Europe.

The Next.js migration is but a small fraction of some of the amazing work we’re doing on the Plaid WebDev team. If you’re interested in solving the challenges we’ll face as both our team and the company continue to grow, consider applying for one of our open roles.

Join us!

engineering

For consumers