Episode #73: Optimizing for Maintainability with Joe Emison

About Joe Emison

Joe is a serial technical co-founder, recently launching his fifth company, Branch, in 2017. His previous ventures have been BuildFax, Spacefu, BluePrince, and EphPod. Additionally, he has consulted with many other companies on software development and cloud migrations, including many in the DMGT portfolio. Joe graduated with degrees in English and Mathematics from Williams College and has a law degree from Yale Law School.
Twitter: twitter.com/JoeEmison
LinkedIn: www.linkedin.com/in/joemastersemison
Blog: emison.orgBranch Insurance: ourbranch.com
Watch this episode on YouTube: https://youtu.be/tGwGxfczaJ4

Transcript:

Jeremy: Hi, everyone. I'm Jeremy Daly and this is Serverless Chats. Today I'm chatting with Joe Emison. Hey, Joe. Thanks for joining me.

Joe: Hey. Thank you for having me.

Jeremy: So, you are the co-founder and CTO of Branch Insurance. I'd love it if you could tell the audience a little bit about your background and what Branch Insurance does.

Joe: Absolutely. I am a serial technical entrepreneur. Branch is my sixth adventure. Branch is a home and auto insurance company and we also sell renters and umbrella. There are two things that make us completely unique in the United States. One of them is we sell the home and auto as a bundle much more easily than anybody else. So really, anywhere else you would go to buy the home and auto bundle, you'd have to buy one and then the other, and it would take you somewhere between 45 minutes and two hours or maybe a week, and you'd get a lot of fake prices along the way. You'd get like, “Well, we think it's about this.” Okay, I need more information. Then, “It's about this.” We will sell you the home and auto together and for most people, we just need your name and address to give you a real price that you can buy instantly.

Jeremy: Awesome. Well, the thing I think that is probably the most interesting to the listeners is the fact that Branch Insurance is entirely serverless.

Joe: It is. This is actually the third company I've started fully serverlessly. Branch, from the beginning, has been built on Amazon's AppSync service. It uses Lambda, DynamoDB, CloudFront, and a bunch of other third-party services that we love.

Jeremy: Awesome. All right. I have been watching your presentations. I've seen you at Serverless Conf. I've seen you at a couple of other conferences. One of the things that I always thought was fascinating is how you're always recommending some third-party service. You've got a third-party service for everything. Now, obviously some of those are our AWS services, but then you use other things like BigQuery and Stripe, and these other ones, and all these other different things. I was trying to think of something clever. We always talk about stuff like serverless first or static first or whatever. I was thinking that your approach was very much so like third-party first, but then after talking to you about it, it's more about optimizing for maintainability than it is just using some third-party service. What is that that you do, especially at Branch, to optimize for maintainability?

Joe: We think about optimizing for maintainability having two central categories. The first is the less you have to maintain, the easier it is to maintain and so in that bucket goes things like a code is a liability, so less code to maintain is easier to maintain. Or running VMs and containers and having DevOps as a core competency that you need. If you don't need it, you run serverless or you need less of it, then that's also easier to maintain. Then there's another bucket which is how do we not be blockers for all of the other departments in the company? We constantly ask this question of how do we empower other departments to go do what they need to do without talking to tech, without talking to product.

Jeremy: I love that idea of empowering other departments. One of the things that I often see when I'm collecting links from my newsletter is people who write these posts that say they're using Google, for example, I'm sorry, Google Sheets as a backend database for something. Some people criticize it, but for the right application, it makes a lot of sense because then someone can actually go in as somebody who doesn't have the ability to query my SQL database or go into DynamoDB or something like that. Can go in and just manipulate some cells in a database and it gives them the ability to run some business process, which can be really, really effective.

Joe: It is fantastic. One of the companies I started, which I did as a weekend project for a friend, was an angular app that sends a bunch of data to a Google Sheet for all these calculations. Since I built that in 2015, 99% of the business logic of that application has been the main founder on that project making changes in Google Sheets. He onboards a new customer. He sets up their logic in Google Sheets and it does everything that they need to do.

Jeremy: Which is amazing. Then there's so many other applications that you can do that with, that make it very, very simple for … I mean that's when you think of things like maybe Airtable or even integrating things. I know in the past I've integrated into services like Asana. Just tie into the API so that you already have those interfaces built because that's one of the hardest things to do sometimes as a developer, is to build a really good admin and if that's already built-in for you, you're off to the races.

Joe: I think people too often will disregard using a service like Zendesk, which is this amazing Swiss Army knife. They'll disregard it because they'll say, “Well, I can envision a world in which we're going to want to do this thing and it won't do that properly,” but the problem is you're a lot better off getting off the ground, getting things going, having a system that works, and then discovering what things really matter because I agree with you that probably you're going to run into cases where this third-party service like SAS tool with an API doesn't completely solve what you want. The bigger problem is what you think that is at the beginning is definitely not going to be what it actually doesn't do later. We live in a world where we don't have enough people who are constantly thinking.

More than 50% of the things that we think we need to build in the way we're going to build them, actually aren't going to work. We're going to build them. We're going to put them out there and they're not going to be usable. They're not going to do what we need to do. We're wrong about these things. It's so much better to get it out, experience it, even if you know that it's only 60, 70% ideal because that information that you'll get will enable you to make much better decisions. Often, what you'll find is when you weigh the priorities, you'll say, “Well, Zendesk is imperfect, but I'd much prefer to keep using Zendesk and use my development time on this other thing that I now realize is critical for our success.”

Jeremy: That's one of the things too that I love about serverless, just being able to string some things together quickly to get a prototype up and running because you do not know actually, never mind what's going to work, but what's going to be used. I can't tell you how many times I've implemented some process or some system that just doesn't get adopted by the organization because either the workflow is too complicated or whatever, but again, being able to test that quickly is a really powerful thing.

Joe: Absolutely.

Jeremy: One of the problems when you are building new applications and you're trying to, as you said, optimize for maintainability, one of the things you need to maintain or what you need to perform maintenance, I guess, is people, so you need to have engineers or software devs that can go back and they can look at that code. They can continue to upgrade it if they need to. How do you make those choices? I know you have a software development guide at Branch. You put everybody through those paces. What does that guide look like? You had mentioned optimizing for maintainability, but what else is part of that?

Joe: Really, it starts off completely on optimizing for maintainability and understanding that when you have multiple choices to make in the world, you choose the one that's going to be easier to maintain, even if it disagrees with whoever luminary on Twitter says to do. Recently, we had a discussion about whether the right way to convert strings to numbers in JavaScript is I guess the preferred way as a plus sign in front of the variable name, but you can just use the number function which is just much more readable. We were like, I don't care if so-and-so says you should use the plus sign. Number and then in parenthesis is a much easier way to read it. Then the rest of the document, the rest of the software development guide really talks about two things.

One of them is we practice Agile, I think Agile in the way that I have heard Alistair Cockburn, who's one of the signatories of the Agile Manifesto, thinks about which is every week we come in peace to ship software and every other week we do a retro to figure out how and do it better and nothing else is sort of set. You talk to a lot of people about Agile and they'll say, “Oh, well, we're Agile. We do Scrum and every two weeks we do a release, and we do a demo on this day.” Literally, number one is "people over processes." If you just describe your Agile as a bunch of processes that aren't changing with ceremonies at the same time, that's not Agile. This is process over people. Part of the software development guide, after maintainability, really goes into helping understand like, “We're trying to figure out how to do this better.”

We don't know how to do it or we have an idea on how to do it, but we're not saying we're perfect on it and so we need everyone's input. We use … Retro is such a critical thing to do to figure out how to do all these things better. Then the rest that we talk about in the guide is really about how what's most important for you as a developer is to think about how you could be doing things better, how you could be shipping software better, what tools you could be using to be more effective. We don't do enough of this. We actually hire people into jobs like Oracle DBA. If you ever wanted to get rid of Oracle, you're probably going to have to get rid of all the people who have the vendor's name and their title first.

We have this problem, where people ... Ben Keough talks about this, where people identify as experts in the tools they use and that's really bad. You should identify in being able to ship software. It shouldn't really … You should identify as someone who can learn things. This is the most common debate I'm having over Twitter these days, really, where people want to say, “The right language is the language you know.” I'm like, “Okay, but what that says is you can't learn.” The reality is if your job is figuring how to ship software and you're like, “I'm never going to learn another language or another framework. I'm only going to use the stuff I know,” you're going to be bad at it.

I'm sorry. You might be good at a college, but, oh my God, if you're still using the thing you learned in college and you're 35 or you're 40, you're not doing the best thing. It's just true. The rest of what we talk about in this guide is like, “It's important to learn and it's important to treat people very kindly and with respect.” There's a lot of discussion, actually about how to do a proper code review, like, “This is how you review someone's code.” This is how we make sure we have great code, but we don't make people feel bad.

Jeremy: I think that's something too where, again, as people grow as developers and they start using new things, it's very, very easy, especially moving into serverless. There's a lot of information out there, but it's a whole different way to think about software development, so it's very easy for people to make mistakes or to do something wrong. And again, having a team that doesn't criticize, but just helps you, you know what I mean, and helps you ship better software, I think is a hugely important piece of it.

Joe: I think about it a bit as, how do you get to a world where everyone is relentlessly trying to do things better and is dissatisfied or is willing to be dissatisfied with what they just did? How do you do that in a world where everyone respects everyone and treats everyone as humans who are great and wonderful people? How do you separate those two things? That's magic, when you can do that.

Jeremy: Well, certainly not on Hacker News. I can tell you that.

Joe: No.

Jeremy: All right. With this idea of optimizing for maintainability, you mentioned this idea, the code is a liability: the more things you write, the more code has to be maintained, the more that has to be upgraded later, maybe or whatever. Of course, you have potential issues with using third-party tools. They might change something. Maybe there's a breaking change or whatever and you have to go back and change your code, but just in terms of that technical debt that builds up, how do you tell your developers to think about technical debt as they're building something?

Joe: We have no roadmap, essentially no deadlines, and unlimited refactoring. We tend not to do a lot of … We do have some like, “Hey. We should refactor this when we get back to it,” but there's a lot ... It is a very friendly environment in which if you, as a developer, are going to work on something and you say, “Hey. I really think I need to spend a couple of weeks refactoring something before I get into this,” we're never going to say no to that. Now, a lot of times there'll be a discussion about the best way to do that and that doesn't happen with any sort of high frequency, but it's not infrequent that you'll get, “Okay, now that I'm going to tackle this thing, I'm going to go ahead and do this refactor as part of it.”

And I think that's really it. Again, this is maintainability. We say, “We optimize for maintainability.” If we don't refactor things that we think need to be refactored, it's not going to be maintainable. I was just going to touch briefly on the … You mentioned third-party breaking changes. My experience is that that doesn't happen with a high frequency. I tend to find that the third-party services, either APIs or SAS, that we use tend to be way better at change management than we would be internally.

Jeremy: That's right.

Joe: It tends to be. I tend to think of all of them as, what if I had a bunch of super-expensive engineers who were really good, documented things really well, had amazing change management, amazing uptime, and I only paid them by use of the thing in a business context? Wouldn't that be great? Shouldn't I do that for everything? So I tend not to have … It happens from time to time, but I never … Anybody who is telling me, “We would never use Algolia over Elasticsearch because we're worried about breaking changes.” That person is just making that up. That's not a problem.

Jeremy: Well, I think I would worry less about breaking changes, but maybe new features that are added, things like that that you're going to have to go and potentially do that, but I totally agree with you. That's one of the things that's nice about having a SAS company, is they are focused on that one specific problem, that business domain, and that's their job. Whereas when you have a team of developers, especially if you're a small team, just maybe four or five of you or something working on a project, if you're building all this stuff yourself, you are across a lot of different disciplines. With the third parties though, obviously, if I said, “Okay, should I have Algolia or should I use Elasticsearch? Should I build my own search there or should I just use Algolia?" You say, “Okay. That is the easier choice because it is easier to maintain that.” Obviously, third-party services are going to be easier to maintain. What is the deciding factor? Because why wouldn't you just choose third-party services for everything?

Joe: In general, I do choose third-party services for everything. My general view is, prove to me that this third-party service won't work. Now, again, I have a very strong difference though between a third-party service that's serverless and one that isn't. You can find third-party services where they want you to go into the AWS marketplace and run it on a VM. That's not serverless and I'm not interested in that. Or like, “Oh, it's an open-source project. Run it yourself.” Again, I'm not interested in that, but when it's serverless … My short definition of serverless is it's not my uptime. I literally can't influence uptime. Beyond, I could put bad configuration or a bad code in, but if some server fails, it's not on me to bring it back up. I think if you can have a serverless third-party API, I think your default should be to use that unless you can prove that you shouldn't use it.

Jeremy: All right. Then what about proof of maintainability? We can say, “Oh, it's easier to maintain this code because I'm using some third-party service or it's easier to maintain because again, it's an Algolia versus Elasticsearch.” How do you actually prove that, though?

Joe: Well, the easiest way to prove maintainability of your codebase, does it have terrible technical debt or do you have terrible other issues with it or some issues with it, I think there's two things that you can do all the time. One, if every developer has his or her own Amazon account or whatever, provided you're using accounts, you can check out the code and run a command and deploy it into a fresh environment, and it works just the same. Everyone has production. You can onboard a new developer, ideally a junior developer and they can start being productive as a member of the team very quickly. I don't know. Within, say six weeks, they can understand front end, backend, how to commit infrastructure as code. They get all of those things. You give them tickets. They can work in them.

That's a maintainable code. I think part of your question was also like, “How would you know if, let's say you used a really bad third-party service and it was really terrible?” I think it's, again pretty easy to ask what percentage of your downtime or what percentage of bugs do you have that are relating to these things? You'll know. If there's a lot of pain, you'll know. Back in 2012, at a prior company, we used a recurring billing service. At this point in time with third-party services, we would do random testing where we would just hit their development endpoints randomly. This service would return 500 errors about every fifth day out of their dev service. The lead developer was like, “I don't feel comfortable using this service.”

We contacted them and they said, “It's totally normal for that to happen,” and we were like, “We totally disagree.” A 500 server should be a defect. We were a really squeaking wheel about it. They had a couple of outages. We complained about those and we got fired as a customer, for bringing that up. I woke up one September morning to get an email. It was like, “You have 30 days to get off our service because you're an obnoxious twit.” The joke was on them because six months later, due to poor engineering, they lost all of their customers' data.

Their whole business was a recurring billing service that was supposed to store all the credit card information so you didn't have to be PCI compliant. Who knows? I don't know what was causing the 500 server error. They had a poor database replication design where they replicated an error in the master. But I've always relied on that at this point. If you do regular testing and you understand the services you use, you can get a sense of which ones are good or not if you're actually using them. My past is a history of services I will never use again because of bad experiences with them, but we found it. We discovered it.

Jeremy: That's a risk too, though. I mean as soon as you use a third-party service, a lot of that control is out of your hands. How do you mitigate against that? When you're integrating with third-party services, do you say, “How easily can we swap this out if we decide to go in a different direction?”

Joe: Yeah. You can do that or you can load test it and availability test it, and see how long it's been in business, and go look at … All of these services that are any good have status pages with lots of history, have customers who are references and so I don't think it's that hard to vet these, to figure out if they would be okay. I don't know. In the high-risk category, I do think … I'm a Fortune 500 company. I'm about to send a million transactions a month through this service that just got out of Y Combinator last month and it doesn't have a website? I wouldn't do that. I think there are standard vetting things you could do. I think a lot of these services, I would certainly recommend to anyone. Algolia or Cloudinary or Auth0, these services have been around for years and years and years. They're big.

Jeremy: If you did have to swap out a service or … Even not that. Maybe you just say, “Look, we need to upgrade some piece of the system. Something's a little bit out of date.” Maybe there's a new feature that's added. You want to go in and refactor. Or maybe you just have something that's not running efficiently and you want to refactor. You've mentioned refactoring a number of times. Now, I know … I've had a CEO in the past. I've had some interesting CEOs that I've worked for.

One of them who still loves the idea of counting lines of code as if that was some sort of an asset and not a liability. Then I had another one who would say that if we leave engineers to their own devices and we don't give them specific projects, that they would just refactor all day. Obviously, you can't refactor all day. You need to build some new things every once in a while, but from your point of view, refactoring is a really important part of the software development cycle.

Joe: Absolutely. Look, I think there's something that's higher level in this conversation, which is that if you have good technical leadership in your organization, your technical leaders will be able to … I mean what they're trying to is get enough trust from the rest of the organization so that you can do things the way you think is best. All of these things, any sentence that starts like, “If you left the engineers to themselves,” is just an example that there's no leader who's managing the trust in the organization across the organization. At Branch, we do not have a roadmap. We do not have deadlines. I mean people can ask like, “When do you think this thing is going to be done?” There aren't company-wide burn-up and burn-down charts. We do size things, but there isn't what day or is everything that everyone's working on going to be done on?

We don't do that. We don't do that because it takes time. Would you rather have it done sooner or would you rather have estimates that are wrong? That's the reality. That's the choice that you're making. It's the only way that you get the rest of the organization to say, “Okay. I'm not going to ask you for a roadmap.” I ran into this a lot at Branch because we've hired a lot of people who've worked at very large organizations than Branch. They're like, “Well, I want to see the roadmap.” I'm like, “Well, there is no roadmap” and they're like, “What do you mean? You have to have a roadmap.” We sit down and we talk and we say, “Well, if I had a roadmap, if you really needed something done, it's going to go at the end of roadmap.” I'm sure that's what you experienced. You ask for something, how long will it take?

“Well, in my last company, it would take nine months to get it done.” I'm like, “Okay. We'll get it done in two or three weeks, but the trade-off for that is there's a roadmap. Are you okay with the trade-off, where you can get stuff done in two to three weeks instead of nine months, but there's no roadmap? Do you really want a roadmap more than you want things done eight months early?” People are like, "Hhmm." They're not sold on it, like, “I don't know. This is new, ” but if you can do this, if you can gain the trust by delivering software … This is why our Agile is we come every week to deliver software. This is also a lot of the accountability of developers. A lot of what serverless gives you is developers have control about, “We're going to deliver this stuff.” We don't have a DevOps team that blocks things.

We don't have a DevSecOps team that blocks things. We don't have an operation team that blocks things. We build it. We have automated tests. We have some manual testing people who are asynchronous and services that are asynchronous, that will look at things occasionally. We have smoke tests. We put things live right. Everybody knows sometimes there will be bugs. Sometimes you can only catch these bugs in production and we will fix them quickly. We will shift things quickly for you. In that world then, if you live in that world of trust like that, then you get to do what you want to do on the tech side.

I totally agree with you. Developers don't want to refactor all day, but developers also don't want somebody breathing down their neck, trying to make them into a feature factory. That they have no say in what's being done and they're not allowed to clean up their room, basically. I mean this is what refactoring is. It's like, “This is bad." Some new features came out in Reapp, let's say they'll let us use them, whatever they are. This is the crazy thing about it. This is no different from management in general, where you try to empower people to have control over their space so that they can be more effective and happier. That's all it is, really.

Jeremy: You mentioned three things that all tie something together for me. One of those was tests. One of those was trust and the other one was feature factory. I had a consulting client. I went into this consulting client. They had maybe 10% code coverage and the CEO and the COO did not trust the CTO because they had recently had a number of failures where they deployed something and the database broke. There was a database change in there that was in staging, but didn't make it to production. Standard stuff happens if you don't have a good process in place. I go in and I'm looking at this test coverage. I said to the CEO, I said, “You need to give the development team time to add test coverage and add some additional tests in so that you can be confident, when we launch a new update, that we're not going to break something and then people are going to get mad,” but the important thing was they wanted to pump out features.

They had a ton of different clients that needed very specific things. It's like you quickly went from being a software company to being a custom web development company, it almost seemed like, but again, those things, where not giving your team time to build the things they need to make sure that the process is solid, that's going to erode trust. Then when you don't have the trust and then you keep pushing them to deliver feature after feature, after feature, without the time to go back and clean those up, without the time to ensure that everything is automated, that's a huge recipe for disaster. Honestly, I don't know many companies that I've consulted for or worked for, that haven't seen that type of vicious cycle over and over again.

Joe: I agree. I think the vast majority of companies that do software development do it very badly. Another way to say this is, I think the gap between the top 1 to 5% of organizations in software development are dramatically … There's a chasm between them and the average. Absolutely.

Jeremy: As a co-founding CTO or as a CTO, you've done this a number of different times. You mentioned things like no roadmaps, giving users, or giving your developers time to refactor, things like no deadlines, no burn-downs, a couple of notes I have here. What else? What else do you do? If you're running a company, which you are, but if you are running that, if you're advising someone else to manage their developers and their development team, what are those things that we have to change, I think maybe as a culture so that we can, again, build up that trust, get to a point where you're delivering good software, but also not stressing people out.

This idea of developers working 70, 80 hours a week, hey, that was great in my 20s. I did that in my 20s and my early 30s. I don't want to have to do that anymore. I just want to be efficient. I want people to know that they can trust me to work on something and that I'll get it done. What's that advice? What does it need to look like? What does a modern development department need to look like, a modern engineering department?

Joe: I love this question. I feel like this question isn't talked about enough. A huge problem with this question and the answer is that people want to find excuses for how they do things today. That's all they want to do. They also want to … Nobody wants to make judgements in this space. They just want to say, “The way we do it is different and that would never work here.” Now, I will say I've come in and consulted with many organizations and my results, consulting with them, have ranged anywhere from utter failure where I failed to affect any useful change to ones where I think I was impactful, but it took years and years to have really good, effective change. I don't think that it is easy at all to go in and make changes and I certainly don't claim to be that good at it. I think the main thing that I would say to anyone, as a starting point, is that we now have the best book ever written about this, which is Accelerate by Nicole Forsgren.

Jeremy: A great book.

Joe: It lays out four metrics and it gives you good ranges for those metrics. I do think it is a fantastic way to manage organizations if there are good metrics to manage them to metrics. In marketing organizations, sales organizations, finance organizations, managing the metrics works and they're happy to do it. I think and I believe that the four metrics in Accelerate are four really excellent metrics and I think they're a great way to align. If you could get … The last few times I've been asked this, I've said to the CEO, “You get on board with these metrics. Let's the CTO on board with these metrics and let's optimize to the metrics.” Then that will at least allow … That empowers the technical teams to figure out how they're going to get there.

Those metrics are, you need to be releasing multiple times a week. You need to resolve errors quickly. You need to have a low rate of failure and low rate is just less than 50%. They give you these metrics. I do actually think that the most important one is deploying multiple times a week. I think that if you're not deploying multiple times a week, you should be. I also think that you should be trying to get to a point where Friday deploys don't scare you or at least, I'll leave it this way, you should be deploying frequently enough and have good enough testing so that most of your bug reports are not in the thing you just deployed.

In my experience at Branch, most of the bugs we find are bugs that have been in the code for a couple of releases before they're found, even regressions, and so because of that, you don't need to fear the Friday deploy because it's already in the code. The next bug you find is already there. Also, you're deploying such a small thing that it's not that big of a deal. I'm totally sympathetic to, maybe you shouldn't deploy something right before you go to sleep or something. I'm not opposed to that. We don't do automated deploys at Branch. We deploy several times a week, but I think we've deployed two or three times today, for example, totally fine. That's where I would start from. That book is fantastic. Everyone should read it. It's really important if you're trying to make an organization run better.

Jeremy: At Branch, part of your development process, I think it's really interesting because again, you try not to overwhelm your engineers. There's no working nights and weekends. I'm sure if maybe there's something critical, then maybe that would happen, but what about using ticketing systems? How do you use ticketing systems at Branch?

Joe: I really don't like Jira. I've used Jira a lot in my past. There's a really good article by John Evans, I think he's at Rezende, that he wrote in TechCrunch. He wrote about how Jira is terrible. When I first read it, I was like, “He's totally wrong,” and then over time, I've come to think that it's one of the better articles on it. We use GitHub projects and issues. This is a classic case of where we started using it in a certain way and it didn't work that well. We used retro. There was a lot of like, “Well, maybe we should use Jira” and I was like, “Okay. Just prove to me that this won't work. I am down with using …”

We got into the linear, beta, and stuff like that, but I was like, “I really don't want to use Jira, but I will use Jira if guys say to use Jira. Just ... force it.” I actually hired a PM who had never been a PM before because I didn't want to bring … I generally have this thing. I don't like senior developers or senior PMs because in general, I find that all they're going to do is bring whatever process they had instead of looking at what's great about what we do. I know it's not all senior people everywhere, but I've experienced that in my career. Senior people don't look at what you're using. They're just like, “We're going to change over to however I did it the last time.”

So, I brought in a PM that had never been a PM and I was like, "Look, you're going to use retro." We're going to use retro to figure out what we don't like about what we're doing and we're going to try to fix it. We're going to try to use GitHub and GitHub projects and issues and if it doesn't work, we'll know why. We'll know the limitations. We're like iteration, I don't know, six and it works really well. We have a super-functional way of using GitHub projects and issues to do everything we need to do, look at statuses, everything.

Jeremy: You mentioned that you don't do sprints and burn-down charts, and some of those other things. When somebody gets a ticket, how does that work? Do you assign someone a ticket? Do they get a ticket themselves? Do they take multiple tickets? Are they dumped in a batch and say, “Get these done by the end of the week?” Again, you said, “No deadlines,” but how does that work?

Joe: One thing that I'll add is that the thing that I don't like about Accelerate is that it is only concerned with work being done. When the work is done, the development work is done. How does that get live and stable? That's Accelerate. There's an entire other book to be written on ... there is a business problem that needs to be solved with custom software development or something by the team that does software development. How do you get to the code is done, that work is done? It's useful to talk about it. The process that we use for this is we really require the business to state problems, not solutions. You can give solutions as examples. Again, this is an example of we have trust, so we get to define the process. When the trust breaks down, we lose control over the process, but our process is you help us understand the problem you're trying to solve and potential solutions to it, maybe if you want to give them.

That goes to our designers. Our designers are going to come back, and multiple times a week, there's a designer review meeting, and you're going to get to go look at the designs and you can approve them or not. You don't get to bring designs. As a business stakeholder, you can't go say, “I need a button here and it's this color. It needs to say this.” You don't get to say that. You can say, “I want to solve this problem and one way to solve the problem might be to put a button here in this code or whatever,” but eventually, you learn it's not even worth telling them that because they're going to design what they want to design. I'm just going to tell you … I'm going to get really good, which is what you want the business stakeholders to do. You want them to get really good at helping you explain what problem they're trying to solve.

Then the designers deliver designs. Once the design is accepted, it could be the first design. It could be the 85th design. The process is the designers get to propose whatever, but the business stakeholders accept the design. Either they accept them or they don't and sometimes it gets a little tense. It's okay. It works, as long as you learn how to disagree with people, which is a skill everyone needs to have. Work through disagreements. Once the design is set, our designers design and sketch. They use Envision for interactions. They export to Zeplin. Zeplin is amazing because it gives you all of the margins and fonts and things. You don't have to eyeball something and try to figure out what the CSS should be, but when a developer gets a task, that ticket has the designs.

They just work the designs and they've probably been … Actually before they get it, there are weekly, basically sprint planning meetings, except there's no sprints. Every week, we try to go and size and talk about all of the tickets that might be worked by someone before the next weekly meeting in which we will do the same thing again. That's all. It's not that … We've got a method that's working. It's Kanban-E, but we steal the ceremonies we need, but this is Agile. We're just trying to figure out what's the best way to get what we want. Then anybody who, … Then what's great about retro is concerns with things, maybe like in a demo, sometimes the business stakeholders seem like they're shocked by what they got. What happened?

It seems like they should have accepted that design. You'll get those. Then that prompts somebody to think about, how could I revise this process? How could I clean this up? Was that a one-off? Those are the types of things that we have in retro, as opposed to retro being … I think when we started retro and when GitHub projects wasn't fully working, it was a lot like, “I don't know what I'm supposed to be working on. I did this thing. I did it and totally wasn't set.” People were like, “That's not what we want at all.” As a developer, you do a lot of work. You throw it away. We don't actually do that. I would say today, most of the work gets done. It's done. It's right. It goes out the door. It's small in scope. We watch it and then it iterates on it.

Jeremy: That's what I was going to ask you about in terms of scoping. One of the biggest problems I've seen when you use a ticketing system is someone will say something. There'll be a job and it'll say, “Move this button five pixels to the right or rename this button,” something simple, small scope. Then the other one would be like, “Build a new billing system,” a very large scope project. What's your process in terms of down-scoping things to get to the right level of work so you don't have an engineer stuck in a back room for three months?

Joe: There are a bunch of tactics here. One of them is when you have trust, you can convince your stakeholders to down-scope themselves. At Branch, we have a saying. We actually have our "Branch Roots," which are the principles, cultural principles that we have, that we say a lot. One of them is, “What's the V1?” This is like endemic language throughout Branch, but essentially, somebody comes in and we ask, “Well, is this enough for the V1?” Everyone's on board with this because they understand that, one, it'll get out faster and two, when they want to make changes to … We all recognize you're going to want to make changes to it. This is the delightful thing about everyone bought into, “What's the V1?” If you can get that, that helps to down-scope. Of course, sometimes the V1 is large.

I think it's very important. One of the rules that we have is that anything large, you've got to build a release schedule and you got to release at least every two weeks, preferably every week. We will have branches that live for two weeks, let's say. Now, we run a monorepo and we do a lot of … We use GitTown, which is an open-source thing that makes it easy to … I mean it's like any of those Gitflow or whatever, but it allows you to merge trunk to branch very easily. Obviously, we have to resolve some merge complex. We have a couple of different teams. We do our best to make sure that there aren't multiple teams working on the same code at the same time.

Then we have people collaborating. You do some sort of planning so you don't collide into everybody. We use feature flags to release things so that they're out there, but not live. I think there are a bunch of tactics there, but in general, that's the strategy. I think another thing that we usually do is if you are working on something that's going to be a week or two that you put at least two developers on it, they'll pair, they'll split up the work or whatever. That tends to be very good in a lot of different ways, to have that extra number of people if somebody gets sick. Then also to have focus. "We care a lot about this. It's big and so we're devoting a lot of time to it."

Jeremy: It sounds like that provides a lot of transparency where everyone can look. Those business stakeholders, they can go in and they can say, “This project is 50% done.” Or it's in someone's queue or whatever. How do you push back against business stakeholders, though, or the stakeholders when they do push something like, “Oh, we really need this feature? I really need you to prioritize that.” Is that something where …? Again, I know the trust is a big piece there, but as the CTO, I don't know if you have engineering managers, is that something that you really push back on the business side of things?

Joe: No. This is the benefit of not having a roadmap. If you're like, “I really need this,” then sometimes there's a conversation with the other business stakeholders. I always find that transparency solves so many of these problems. I've worked with so many CTOs who like to hide information. I was consulting for this CTO, he was, like, “I really don't like to let my vendors know everything that's going on.” I was like, “You're an idiot. Tell everybody everything and if they're bad people, don't work with them.” It's crazy to hide information. If someone's like, “I really need this,” then I go make sure that the other stakeholders are okay with us moving that ahead. One rule we tend to have is I won't interrupt a developer working on anything, so the best I'm ever going to do for you is put it up next.

It's going to go up next for the appropriate developer or developers, which could be lots of them. In some cases, that might be like, “There's one person who should do this” and so you're just going to have to wait until they're ready. I think very occasionally, for each developer, maybe once every six months or something, we might interrupt them. We might say, “Hey. Can you just put down what you're working on? We really got to fix this thing.” Usually, it would be a bug, but that's very infrequent. Everybody gets that, but in general, everyone's primed. I got to tell you. If you're a business stakeholder and you know that something you really care about can be done in about a week, if it needs designs maybe two weeks, and if you've worked anywhere else at any other organization, you're like, “I'll take that. I don't know what other weird processes you have, but that's fast.”

Jeremy: I don't know. I think that's just amazing, that idea of having business stakeholders that understand what the processes are within the engineering organization and having that leadership. I think that's the most important thing, having the leadership within that engineering organization to really go to battle for your engineers. Build up that trust. Have that history of delivering things. Of course, I think it's a fantasy world in a lot of organizations. I think they can get there, but I think the vast majority of organizations aren't quite at that level yet.

Joe: The one organization that I really helped go from no ticketing system, all tickets on whiteboards, terrible communication, business stakeholders walking into the developers' offices and sitting over their shoulder while they worked on stuff. It was no testing, a million and a half lines of PHP in the application. One page took seven minutes to load on average, stuff like that in a SAS product that lots of people were using. I mean it was not in a good place. Now, three years later, we brought in a consulting firm that was really good, that really collaborates with you, but everybody there was like, “I want help.” I know that the biggest challenge to fixing and making an engineering organization go from, let's say not compliant with the Accelerate metrics to compliant, is most of the people in that organization saying, “I want help. I know it's not good. I'm willing to listen to people who aren't me.”

Until you get to that point, obviously, you're not going to change anything. There's so much around trust here that matters. This thing … It really helps to be the technical co-founder. It really helps to be one of the first two people in the company on day one and so you predate everybody else. That helps a lot, but that doesn't mean … I've held lots of meetings in which I have explained like, “This is how we do things. What questions do you have?” People are going, “I don't get why you do this. Why can't I do this thing?” Explaining like, “Well, you're right.”

If you look at it from that perspective, it seems bad. Here's why. One of the concerns that I get a lot is, “Because you won't let me specify designs, I'm reliant on those designers to come back. They get to make whatever designs they want. I could be waiting forever for something.” I was like, “But have you? Has this happened in practice?” Yes, you're right, it's a theoretical concern. In the past two or three years, I've probably spent fifteen to twenty hours, not a ton, but in meetings, just asking people to vent about what they didn't like about it and just explaining why we did what we did or why we do what we do.

Jeremy: Well, that's why I like junior devs because you can just bend them to your will usually and say, “No. This is happening.”

Joe: It's awesome. It's so awesome to take someone who has not had a lot of experience. They come in and they're like, “Well, everyone's happy here. We only work 40 hours a week. No deadlines. I get to just work. I'm not in a lot of meetings every week. Okay. I'll do it, whatever the hell you tell me to do.”

Jeremy: Exactly. All right. Awesome. I just want to ask you one more question because this is something I am always interested to see and again, because you use so many third-party services and you've been fully serverless, like you said, for a couple of companies now. Where is serverless going? This is a cliché question, but five years from now, is serverless just going to be part of the cloud? I mean are we just going to not even think about it? Is there going to be a distinction between what we consider serverless and what we don't?

All of these major cloud providers are creating these Kubernetes management systems where you're not even going to manage Kubernetes anymore. It's just going to be Kubernetes, but you're not technically going to be managing Kubernetes. You're just going to be using K8 then on top of it or whatever it is. Is it going to go away? Is it going to abstract away? Are we getting close to that, anyways? We're just these public cloud providers that are going to take over the management and serverless is just going to be the way forward?

Joe: No. I don't think so. One, never underestimate an engineer's desire to keep doing things the way they have been doing them or to make iterative improvements. If you were managing VMs with Terraform and your job was to manage VMs with Terraform … This is the problem. I talked about the Oracle DBA. I think Simon Wardley talks about this a bit, but we now have a whole … The new Oracle DBA is probably DevOps, so you just built up a big DevOps team. Well, I don't think you're leaving containers or VMs until that team is not and your prod infrastructure runs on those, so you're in a bad place. If you hired me as a CTO in an organization, you dropped me in and all your prod was running on Kubernetes, and three people understand how it's set up and it's brittle in places, that's a tough place to be in.

You're going to be there for many years. A lot of people are jumping into that right now. I think VMware is out there and has a lot of incentive to drive that and is a very competent organization that will continue to proliferate that. I also think there's a huge debate about what is serverless and what are good serverless architectures. When I go out and talk about serverless, there are a lot of senior devs in the Hacker News world who are like, “Well, you just stitched a bunch of APIs together and I don't want to do that.” I live in a world where all … At Branch, we only hire front end developers. I went out to try to see, can I find a senior developer I wanted?

I advertised for a software development leader and 100% of the resumes … I said, “You need to have a year of experience, not necessarily in your job, but just a year of experience developing a modern JavaScript framework.” No applications that I got had that. I was like, “Okay.” All the front-end developers are selecting out and all of the backend developers are selecting in and they're not qualified. I really believe in this full-stack front-end developer. I really believe in stitching these API APIs together. I believe very strongly in all of those things, but I don't think that there is a lot of acceptance on that at all. You asked me about the future. I think the future is a service that is like Draftbit. I think this is … I don't really believe in no code for interfaces because I think in the end, you're going to want to customize the interface. That is the one thing everyone is going to want to do and that is a place which is going to pay off in every organization, to customize the interface to your specifics.

The idea you can do that through drag and drop, I haven't seen it. Draftbit is this interesting product that lets you almost drag and drop and build a React Native app, but it compiles to React Native, so you can inject it to React Native. You can build it there and when you reach a point where you're like, “I got to customize this more,” learn React Native. Then you keep going. Another way to say this is if you look at the future, the future is just better, higher-level abstractions that compile down to lower-level functional code that you can work with because if you build the version that you can't compile down to code, but it's totally proprietary, which is the Airtables of the world, what you have is fundamentally proprietary. Until you give me the ability to do extensive modification in code, I'm going to have to leave you.

I'm going to outgrow you or you're going to be an internal tool. I think this is the future. I think AppSync built on what Firebase did, it's a great high-level obstruction, but you have full access to customize everything you'd care to customize about it. EventBridge, another version of this that's fantastic. We have this great history of these things. We're seeing them on the front end more and more. That's what I think the future of highly effective development is, but I think the challenge is that 80 plus percent of organizations are going to have a bunch of people working for them who are going to resist this tooth and nail because it means they will have to throw out what they have invested so much in today.

Jeremy: I totally agree. That's amazing. Well, listen, Joe. Thank you so much for chatting with me and sharing your immense knowledge on not just serverless, but also just managing an organization. Honestly, if there's any, I guess technical leader out there right now, read Accelerate. Listen to the advice Joe just gave. Push back on your CEO. It's not about lines of code. It's not about a feature factory. It's about having a healthy, productive, trusting, transparent organization where everyone is working together to, just like you said, make themselves better. If people want to contact you, how do they do that?

Joe: Twitter, @JoeEmison.

Jeremy: All right. Then you have a blog too, emison.org, right?

Joe: Yup.

Jeremy: Then Branch Insurance is just ourbranch.com?

Joe: That's right.

Jeremy: Awesome. All right. Well, I will get all that in the show notes. Thanks again.

Joe: Thank you.

This episode is sponsored by New Relic and Epsagon.

Episode source