In search of higher engineering productivity: A data first remote working perspective
At Echobox, we constantly strive for maximum productivity. We regularly review our existing processes and are open to exploring new ways of working as a team. It’s part of our DNA at Echobox to be data-driven and take a scientific approach to everything we do. We take inspiration from anywhere but remain cautious of the Halo effect, which is “the tendency to focus on the high financial performance of a successful company and then spread its golden glow to all its attributes”, even the negative ones.
A recent study about technological productivity that spanned six years with over 31,000 responses across a number of industries showed that the choice of technological methodologies or processes could lead to a hundred fold difference in the delivery of new business value confirming such exploration can reap huge rewards.
Remote working was one of those processes we decided to review. At the beginning all Echobox employees worked out of our beautiful London HQ in Notting Hill. As everyone enjoyed working together the demand for remote working was low but we were aware of anecdotes from other leading technology companies about how they increased their engineering productivity through remote working.
There is a plethora of existing research that attempts to provide insights into the impacts of where employees spend their day on productivity. These simultaneously point to the benefits and drawbacks of more remoteness between coworkers, a somewhat unhelpful picture when developing your own corporate policies. In one controlled experiment Stanford Professor Nicholas Bloom reported that getting a randomised half of call centre staff at CTrip, a 16,000 employee NASDAQ-listed Chinese travel agency, to work entirely from home over a 9 month period, resulted in a 13% performance increase (https://ebx.sh/fjiuVV, https://ebx.sh/hlNmTz). Conversely colleagues Ben Waber et al “showed that when a salesperson increased interactions with coworkers on other teams — that is, increased exploration — by 10%, his or her sales also grew by 10%” (https://ebx.sh/VZJWui). In this work, Ben Waber demonstrated that a reduction in distance between team members had the opposite effect as that observed by Nicholas Bloom. These two examples are representative of an extensive body of seemingly contradictory studies. Moreover, neither measure engineering productivity at a technology company.
This ‘paradox’ of contradictory studies may be understood if at a deeper level the results are in fact due to the non trivial interplay of many underlying factors. To clarify, remote working practices can only indirectly impact productivity. Such factors, alluded to by Bloom and many others, which include “distraction vs. focus” are loosely summarised in Appendix 1.
Importantly the contribution and strength of these factors to overall productivity, unique to each set of starting conditions, would reasonably explain why two companies within the same industry, or different departments within the same company, can experience different remote working outcomes. For example, in one of those companies the benefits of additional focus might significantly outweigh a proportional loss of collaboration, as was observed by Nicholas Bloom at CTrip.
Whilst this complexity prevents anyone from accurately forecasting the impact of remote working on productivity, the evidence that an optimal amount can be significant is clearly established in peer reviewed literature. When considering your own corporate policies the biggest risk would therefore be to avoid all experimentation, preventing an evidence based decision from being made. Sandy Pentland observed that the best-performing and most creative teams in their study sought fresh perspectives constantly, from all other groups in, and some outside, the organization — so it’s important that such experimentation is ongoing. For example, a new technology or piece of software might significantly impact any of the factors presented above.
Given the lack of representative studies on remote working, we conducted our own experiment. Although we hadn’t originally planned on sharing our results we felt it would be helpful if we shared the results of our research with the wider community, especially as most technology companies are now forced to let employees work from home due to the coronavirus pandemic.
For our experiment, our software engineers were able to self-select a single remote day per week. After six months, we increased it to two remote days per week. Remote days couldn’t conflict with team meetings or events, for which we’d require everyone in our office.
In their book Accelerate and in further detail in their peer reviewed paper Nicole Forsgren et al established a statistically significant relationship between the frequency of deployments, lead time, mean time to restore and change fail percentage to productivity. In this work, due to it’s easy availability, we have reused the frequency of deployments as our primary metric for reviewing the productivity impact of remote working. We normalise this to ‘per engineer’ to account for our team size varying between 3 and 5 members over the period of the experiment. We also contrast this against productivity of the engineering team measured as average story points completed per day per engineer. Whilst no predictive relationship has been established with story points we include it as it is a more familiar industry measure of productivity, also referred to as “velocity”.
Figure 1 shows that the proportion of remote working amongst engineers doubled between January 2018 and November 2019. Engineers felt this was primarily due to finding better ways to take advantage of the remote working time, for example by improving their work from home environments.
Figure 2 shows the number of deployments per month per developer against the proportion of remote working during this same period. We observe no statistically significant trend suggesting productivity remained static over the levels of remote working explored in this experiment. Figure 3 compares the remote working proportions, figure 1, against the average story points completed by the engineering team, per engineer, again showing no statistically significant trend with productivity over this 23 month period.
Based on these results, we concluded that at the levels of remote working observed during this experiment at Echobox there was no statistically significant long term relationship against engineering productivity at Echobox. This outcome was still considered a net positive as team members reported being more motivated and less stressed due to a reduction in commuting time.
And what about Echobox’s corporate remote working policies you might ask? Accepting that remote working didn’t have a negative impact on engineering productivity at Echobox, we decided to cautiously integrate more remote working practices into our processes for non-managerial and non-leadership roles within the engineering department, primarily due to the added flexibility in hiring and retaining amazing talent from outside of our HQ in London. And we definitely plan to continue experimenting whilst forging an environment that promotes mutual trust, experimentation, improvement, knowledge transfer and education into the different factors that influence company productivity, so that everyone can make informed choices — which goes well beyond remote working practices.
If you’re interested in engineering jobs at Echobox we’re always hiring for outstanding candidates, please get in touch at https://careers.echobox.com/
About the author …
Marc Fletcher graduated with a PhD in Physics from the University of Cambridge and has been the CTO at Echobox since 2014. Whilst not jumping out of planes he’s particularly passionate about maximising productivity in high performance cross functional technology teams.
Appendix 1
Underlying factors that impact productivity of remote workers:
- Distractions and focus. For example, personal productivity, “the effectiveness with which a worker applies his or her talents and skills to perform work, using available materials, within a specific time”, and deep work (https://ebx.sh/rLpXYp) (https://ebx.sh/7OHrM8).
- Creativity, collaboration and group learnings. For example, employee collisions grouped by energy, engagement and exploration. MIT Media Lab Professor Sandy Pentland, inventor of the sociometric badge, demonstrated that 35% of a team’s productivity can be accounted for simply by the number of exchanges among team members (https://ebx.sh/TqqIyC). Preferably these are face-to-face but closely followed by phone or teleconferencing providing the number of participants is small. Importantly there’s an optimum rate of collisions and “going beyond that ideal number decreases productivity”. Sandy also observed that the type of collisions matters greatly — swapping a young software startups “beer meets” and other events for longer tables in the lunchroom, “so that strangers sat together”, had a huge positive impact.
- Motivation, levels of isolation, stress and employee health. For example, the quality and quantity of an individual’s social relationships have been linked to mental health, morbidity and mortality (https://ebx.sh/UytSXy).
- Retention and costs of training replacements. For example Bloom et al reported retention rate improved by 50% for fully remote workers.
- Recruitment. For example, it might be possible to hire significantly more skilled or cheaper employees in certain locations or geographies, entirely offsetting other productivity tradeoffs.
- Amount of additional time worked due to not having a commute. For example, remote workers log more hours than employees who work in the office.
- Office space cost savings. For example, stats from PGI news suggest the average annual real estate savings for employers with full-time remote workers is $10,000 per employee (https://ebx.sh/RliKSH).
- Proximity of customers. For example, incompatible timezones making regular customer communication inefficient, common within sales or consultancy organisations.
- Employee beliefs and attitudes. For example, Derrick Neufeld and Yulin Fang showed in their study the most important determinant of remote worker productivity “were beliefs and attitudes about telecommuting and social interactions with manager and family members” (https://ebx.sh/ntb00d).
