Episode #48: Serverless Developer Culture with Linda Nichols

About Linda Nichols:
Linda Nichols is a Cloud Solution Architect at Microsoft. In addition to creating software solutions, she has a passion for community involvement and education. She is a co-founder of Norfolk.js, NodeBots Norfolk, and RevolutionConf. She also enjoys teaching local classes and workshops.

• Twitter: twitter.com/lynnaloo
• LinkedIn: linkedin.com/in/lynnaloo/
• GitHub: github.com/lynnaloo 

Watch this episode on YouTube: https://youtu.be/e5oFSIMuvcM

Transcript:
Jeremy: Hi, everyone. I'm Jeremy Daly and this is Serverless Chats. Today I'm chatting with Linda Nichols. Hey, Linda, thanks for being here.

Linda: Hello.

Jeremy: So you are a cloud native technical specialist, and a member of the global black belt team at Microsoft. So why don't you tell listeners a little bit about your background and what you do at Microsoft.

Linda: Sure, sure. So first of all, I have like the most awesome title at Microsoft. And most people don't understand, but it sounds great. Especially you join a call with a customer and they're like, the global black belt is here. But, essentially, we're problem solvers. If someone has a problem, or they want to know how to build something, or they want to have a conversation about maybe like, something that's out of the norm, like, it's not your typical service that a lot of the cloud architects within Microsoft know, or it's something more in the open-source side, something that's heavier into serverless or Kubernetes, or just something that's maybe out in the community. There's a lot of open-source tools and things that we talk about, we could be just the open-source blackbelt team. And my background is development. I was thinking about this today. Like, I'm not afraid to say I'm in my early 40s. And so now, half my life has been developing.

I've had a professional job as a developer for half my life. So it's really like kind of ingrained in me being a developer, even if I'm not coding every single day now, because I'm on the phone a lot, just like chatting with people, but I still, like really enjoy kind of hacking at things and thinking about methodologies. And, that's part of what we do too on our team I mean, maybe someone calls us up and says, I just can't get this working and we help them through it. But also, maybe we just kind of talk about, like why are you doing this this way? And, what you think about this? And how about these tools? And that sort of thing.

Jeremy: Awesome. Alright. So I've seen you give a number of presentations actually in a lot of the presentations that you give are around DevOps and serverless, right? And kind of how those things connect. And speaking about being in your early 40s, one of the things I love about your presentation, I'm in my early 40s, as well, I love your, like 80s and 90s References because I get all of them and it is absolutely amazing. So, but your talks usually are around DevOps and how it kind of intersects with serverless. And a lot of times about the serverless developer themselves. And I remember back at Serverlessconf, it was like serverless developers are developers or something like that and it was great talk. So I kind of want to talk to you today, though, about the culture, right? Like this culture around the serverless developer. Because, if you look at people using things like Amplify, there's this whole new thing like a full-stack serverless developer.

And then you've got some people who are kind of focused more on the, I guess, on the infrastructure side of serverless, which is maybe a bit of an oxymoron, but maybe understanding at least how some of these configurations work. So maybe you just give us a quick overview like, what is the overall culture look like for serverless developers?

Linda: Sure, sure. Well, first of all, you threw like an AWS term at me. And I was like, Amplify, which one is that? But, yeah, I mean, I think what I keep trying to kind of drill in, is it like, yeah, serverless developers are developers. And I keep saying too, serverless was made for us, right? I mean, serverless wasn't really it didn't come out and become popular because ops people were like, "No we don't really want to do our jobs." Like we hate infrastructure. No, no, they love it, they've been skeptical this whole time. They're like," Oh, so the developers are going to push to production now. Okay, have fun with that." so I mean, it's essentially for us, so we shouldn't be the ones that are distrustful, we should be the ones that are saying, okay, here's our process, which is what we love to do. These are the things that we've done to be really successful at development all these years. And now we're carrying it over into this ecosystem where we have a little bit more control, but also less control kind of. I mean, we're not having to hand as much over to the ops people. But we don't have to worry about things.

Like, I think there was a period of time there for a while, where I started to have to care about Docker containers more than I wanted to for a while in development. And I mean, I'm at the point now, where I kind of, I understand the process a lot more, because I've just been in cloud for so long at this point. But there was a point where I just, I really, I had a lot of strong opinions about my development environment and like and libraries and tools, and then suddenly I'm like, Okay, well, I'm just going to push to, whatever past service and then the ops people are like, okay, but like, we're going to need you to like, put a Docker file in there. And I'm like, "Hmmmm." And then there's suddenly there's all these troubleshooting steps. So when serverless kind of took hold, I was like, oh, okay, everyone, this is now the way forward. Because I don't have to care as much, I'm just using some command line tools. And just as simple as I push to GitHub, I push to the cloud.

And I really got on board with a lot of tools like serverless framework especially, too that even abstracted some other things away. Now, I think we're at the point where like, you can use different IDEs, and push different cloud platforms and be really successful too.

Jeremy: Right. And that's actually one of the things I want to ask you about, too, is that again, I've been developing for 20 some-odd years, I think I started in 1996, or something like that. So, this idea of having your tools, right? Your IDEs, I mean, I remember way, way back when using Eclipse and things like that, and some of those other ones. Obviously, there are a lot more now. But what are those tools that the developers were using in the past and are still, or can they still use those now?

Linda: Yeah, well, it's funny, I was talking to my husband yesterday, so my husband works for GCP so we're like a multi cloud household anyway. But we have a very similar background in that we were both Java developers, and we moved to kind of...

Jeremy: I'm sorry to hear that. I sorry to hear that.

Linda: Imagine that. I still love Java. I don't know .NET at all, which is funny. I always like, I'll tell customers, I'm like, "Hey, Microsoft hires traitors like me too." like, I don't know anything about .NET. But I can look at it and say, like, okay, this is enough like Java that I can figure things out. But, so we still talk about kind of development culture a lot. And we're both in cloud now. And so we were just talking about Eclipse yesterday, because I just loved Eclipse. And when I moved to Node, I was like, Okay, so now to kind of assimilate into this hipster culture, I need to like use just a text editor with no highlighting. Okay. Alright. So I struggle with IDEs for a long time. And I was saying yesterday, I just got to the point where I feel like VS Code is at a point that I love it as much as Eclipse. And I'm not like, a VS Code salesperson, but like I was already loving it quite a bit before I even joined.

Jeremy: I was actually, I used Atom for quite some time that was sort of my go to and everyone was like, oh VS Code VS Code. And I'm like and then I started using it, then I got my new laptop, I only installed VS Code and other than needing to create an alias in my CLIs so that when I type Atom that it opens code right? That was, other than that, the transition has been relatively easy.

Linda: Yeah. So and I loved Eclipse, I think VS Code is there. But there's all this other like additions too. Like I said, there's so many great extensions and like trustworthy extensions, you can really push it to anything. And, I think, because my interest is developer culture and deployments and just thinking about how people interact with cloud, like I think about multi cloud all the time. So I work for Microsoft, and I love that Azure but I think a lot about okay, well how would AWS approach this problem? How would GCP approach this problem? My husband and I talk a lot about, okay what's like your priority? Or like, how would you tackle this if you were using Cloud Run or Knative or something like, how do you approach this problem? And really, as developers, that part's the same. I mean, once it gets to the cloud, things are architected slightly differently. But as developers, we still have the same sorts of opinions. And we're still essentially using our same process that we did when we were both developers, but we just pushing to some other place.

Jeremy: Well, and that's something interesting too where when I get into the mode of sort of enterprise development, or you're developing code, you're checking it into a CIC, or you're checking it into some sort of code repository. It's going through code review, going through CICD, you got Jenkins doing builds and tests and all these kind of things, and doing the deployments for you. That was something that became very, I just think became ingrained in a lot of developers that were working for larger enterprises. And then serverless comes along, and all of a sudden you've got the serverless framework, it's like serverless deploy and next thing you know, it's in production. Is that something... Obviously, there's good practices around code repositories and GitFlow and things like that, and CICD and testing, but we can talk about that later. But is that something you see is part of the culture where just people are I mean, I guess they're skipping some of those things?

Linda: Yeah, yeah. And I mean, that's kind of what I talked about a lot. And I was making fun of we as developers in my talk at Serverless Nashville and also at Serverlessconf. But, it's just funny to me, because especially coming from a Java background, I mean, Java developers are just notorious for process. And I mean, your static code analysis and going through I mean, it would take forever for a build to run. I mean, literally, it's just like the XKCD comic, where the guys just sword fighting while the build's going on. I mean, it really was like that. I remember sitting there and being excited when smartphones came out because I could get on my smartphone while my build is running. And it was because it was just running through so many tests, unit tests and integration tests. And like this whole process of even being able to check code in and then moving through all the different environments was a pretty elaborate pipeline. I mean, I didn't have access to testing or production, I only had access to development.

And so when I moved to Node.js, it was a little bit more relaxed, just because like Node was super brand new when it started and we just didn't really know what to do exactly. There wasn't even really like testing frameworks out. And so things were a little bit wild, wild west, but people still knew that they were doing things wrong. Like they were like, Okay, this feels really dirty that I'm not unit testing. This feels... Like how do I move things through the stages? But I started noticing though, when I was in consulting before I started working for Microsoft, that our customers were not feeling that guilt at all. That something, there was some like switch that flipped. And like they're in the cloud, but more importantly, like they're in serverless, and suddenly it's okay to just push to production all the time.

And so we talked to customers, and they would say, "Oh, I have 40 or 50 Lambdas." And, okay, great, great. Okay, how are you keeping track of them? Or how are you testing them? Or how do you develop them? "Oh, well I log into AWS and I go to the portal and I type some code in, I hit submit." And I'm like, "Okay, so you don't lint anything, you don't test anything. That's production. You don't have a separate subscription or account for these different?" "Nope, mm-mmm no." And then they want help debugging something. And I'm like, "Okay, well, first of all, like we need to start from scratch." Because I mean, at that point, I think I was kind of saying during my talk it's not just even pushing to production, but you're essentially like using the cloud as your revision control, like, yeah, your portal i, AWS or Azure, GCP. Like that little box that has a certain amount of protection in there, that's not your revision control. Like GitHub needs to be involved or GitLab or like, whatever your choice is, but.

Jeremy: I remember when I first started developing websites, and I was uploading Perl scripts to cgi-bins, but what was great about that was, you make a change, you upload it, and it's immediately available because it's running on one server, right? We didn't see the volume that we had now. And you do things like that, and it was fast, like, oh, something's not working, right. Okay, fine, change it, upload it, it's there. And then we went to this culture, like this Java culture, like you talked about where we're essentially like, okay, you need something changed, it may take four seconds to make the code change, but it's a good three hours before that thing's going to make its way into production. Then serverless comes along, and then all of a sudden, I feel like I'm uploading things to a cgi-bin again and it's like great, because I'm like, "Oh, I can just put this into production really quickly."

And that's obviously a better practice, I think, than just using the console. But certainly being able to just write quick scripts, especially if they're like DevOps Scripts. Things that aren't really client facing production, just being able to push those up is a lot of fun. But now we go back to this thing where it's like, we have all this control, the power is in the developers' hands, but the importance of going through that CICD process, that's something we have to get back to, right?

Linda: Yeah, yeah, absolutely. And I mean, I will say if you're using serverless framework, or you're using VS Code to push, you're at least slightly better than the portal developer, because you're at least in your IDE, you're in your comfort zone. And you probably have some sort of linter or static code analysis running because, in VS code, or atom or whatever, there's all these extensions. So you're like, you have a little bit of protection engaged there. So that's at least somewhat good. But then yeah, of course, if you're using serverless framework, just pushing production, then you're going to possibly run into some issues, especially because you're offline there too, right? So you're not considering all the other services

There are so many tangents that I could go off on here, but I think that even if you have like a baby CICD process, like, even if you're pushing to revision control, and you're keeping track of those changes that you're pushing, and then maybe you have something simple from there, like it's a GitHub action, maybe you go with like Travis CI or pipelines and Azure DevOps and then you have like a really simple pipeline and that pipeline, just runs some additional checks for you. Maybe it does some additional linting or runs through your test suite or something. Like that's better. Like you don't have to start out, you don't want like to get analysis paralysis, right? Like, okay, I'm writing a Twitter bot and so what is my path to production look like? No, not really but like, but you need to like at least have that muscle memory engaged where you're developing.

And I think that's kind of the thing is with these developers, like when they, you cross languages or you cross tools, you still kind of keep that muscle memory of this is what it feels like to develop and this is the right way to do things. But there's just something about going to serverless that just became like processless. It's like, it removed that muscle memory because it suddenly felt like it wasn't development anymore, but it's like more development than ever. It's like most development you can do in the code or even the cloud is serverless, essentially.

Jeremy: But that's the other thing though that I think that's kind of interesting, too. And maybe we can get into just the testing bit a little bit more in a minute. But this idea of being able to write code on your local machine, and then test it locally has always been a thing. We'd always be running a JVM or something locally on our machine where we could test that code, and we knew exactly how it looked. And then we moved to Docker and we'd run things in Docker containers and if it ran on the Docker container on your machine then it was even more you never get that it works on my machine TM statement all the time. But the thing about serverless that is sort of interesting now is that you oftentimes need the cloud and you need all of those interconnections in the cloud to actually see if something is running correctly. And so that's why I really like this rapid development thing of sort of like SLS deploy to Dev. Like just keep putting stuff up there and then be able to test that, but because that's so easy, it seems like it's also like well, wait, I could just change dev to prod and then all of a sudden it would be live. So, getting rid of that process.

But I mean, that's the thing, too, is like, how are people supposed to just iterate? You know what I mean? Especially when you need the cloud for a lot of the nuances and the interconnectivity that you're going to have to deal with.

Linda: Yeah, I mean, you still need your playground, right? And that's fine. And you can still do whatever the heck you want in your playground. But the problem is, is that, and we've all seen this, as developers to POCs always become production apps. Always. Every time a customer says, "Oh, we're just making a POC." Oh, yeah, That POC just-

Jeremy: Oh in production. It means, the P stands for production.

Linda: That's right. It always turns into some enterprise application. So I mean, that's fine if you're truly playing around, you're truly testing things, you're getting things going in Dev. And Dev is Dev for a reason. But once things start going to test is when you start thinking, okay, I'm building something real here. This, like, there's going to be an end user, there's going to be some consequences here. And then that's kind of when you really need to start to buckle down on that process there. But, I have seen that so many times, oh, this is just a POC, we don't need to worry about, DevOps not for our POC.

Jeremy: Well, I always find that as soon as you put something up there, that anybody who's not a developer can play with, it's automatically in production. They just assume it's productized and it's ready to go and they start pointing customers there so I've had that experience as well. So the other thing about serverless, I think that is really interesting is less as a technology and more as a mind shift or mind change, or whatever it is, or what's the other word that we usually use for it? Like, I guess it's sort of a culture shift in the sense. But this idea of saying, look, there's all these services out there for you, right? So if you're in Azure and you want to use a database, use Cosmos DB or whatever, right? Like, you've got all these other things that exist for you, why would I create these things ourself, or recreate these things ourselves? But that, I guess that's less serverless developer culture and more overall developer culture. Everybody wants to build their own things. But is that something you think might shift with serverless?

Linda: Yeah, absolutely. And I mean, I think in the cloud in general, because when we were talking about standing on the shoulders of giants before, we were talking about using NPM packages or Ruby gems or whatever. And like you kind of just don't even know who's written those. And there's some security kind of considerations there. And also, like, you just don't know how much they're tested, you don't know if that maintainer is just going to go find another job somewhere. When you're talking about cloud services. This is true for all the major clouds. I mean, they are tested by millions of people, they are used billions of times. So I mean, if you... No one's going to say like, oh, I just don't think that... Fill in the blank servers, like Cosmos DB. Like oh, I just don't think it's really tested that much. Or like, what if the person who works on it leaves. Well, there isn't one person, right? It's a whole team of people, and there's a company that supports it.

So I mean, I think it's somewhat it's kind of ridiculous when people don't trust cloud services. If you don't want lock in that's a whole other discussion, right? But you if you're already in a cloud, I mean, a lot of people are multi cloud also, and they kind of spread things around to try to minimize that sort of lock in feeling. But really you get locked into libraries too right? If you're writing... If I'm writing a Node.js app, and I'm using some NPM package, yeah that thing's going to stick around forever. How often do you go back and switch out your NPM package?

Jeremy: Right, and if the NPM package, speaking of the developer going getting another job or something, if the NPM package is used by thousands and thousands of or has thousands of dependencies, and they make a change that breaks a whole bunch of stuff like that just happened a couple weeks ago, I mean, that's what I think was the isPromise package or something like that. I mean, that's kind of scary. I mean, then lock in, like you said, that's a probably a longer discussion. But I mean, for me, I feel like there's this sort of 80% rule, right? where if another service gets you sort of 80% of the way there, to me that just makes a lot of sense. Like, I like building my own stuff too like, yeah, I mean this system is good, but maybe I could make it a little bit better. And you might get to that point, but I just, what are your thoughts around just this idea of this, I don't know, I call it the 80% rule, and maybe it's called something else in actuality. But just this idea, like, just good enough. Is that something we should be embracing?

Linda: Yeah, yeah. I mean, yeah. So I mean, I have so many thoughts here, I have to give a plug to Forrest Brazeal for his talk at ServerlessDays Virtual this week, because his talk was basically a talk that I also was submitting to conferences about. But my talk was like, don't write code, because I just started thinking about the fact that if I was developing something for the cloud, or just in general, if I start typing a lot, I pause and I go, okay, somebody's already written this. I'm not that clever. Not really, there's a lot of smart people in the world. There are a lot of people that code all the time. This is already done somewhere. It's either in a library or it's a service. And I talk to so many customers and people who they're like, "Oh, here's my great idea of a thing." And almost always I'm like, "No okay, so that is this." And I mean, even like messaging systems like Service Bus on Azure. I mean, there are so many developers that have tried to write messaging systems. And there are so many out there, there's so many people that tried to write Kafka. And they still are.

And sometimes I talk to people that are trying to create something, and they will say, "Okay, well, I'm going to put this in a function, this in a function." I'll say, "No, you don't need functions here. This is already a service, or you can already use something like Logic Apps, like you don't have to write any code." And, you just kind of connect some things together, or there's already built in services and that's still serverless, right? Like serverless is not just fast. Like, I don't have to write 100 Lambdas to be a serverless developer.

Jeremy: Could just write figuration code. I mean, in many cases, it's just, it's YAML or JSON or something like that, right?

Linda: Yeah. And for me, if that's the case, I've won. I feel, I'm like wow, I just saved all this time, I don't have to test anything, because I didn't write anything.

Jeremy: You don't have to maintain it. I mean, that was the point that Forrest made, I think that was, is pretty smart to think about that as the amount of technical debt that you take on. He basically said something like the first version of anything you build is the worst version of that thing that's ever been built, because it hasn't been tested, it's missing all those features. I mean, and that's brilliant, right? And then not only that, but then even if you keep working on it and adding features, and so forth, someone's got to maintain it. And you know that there's always that one developer who leaves and you're like, "Hey, how do we manage this project? Or how do we control this server?" Whatever it is. And it's that one person who left who did it and all that institutional knowledge is gone.

Linda: Yeah, exactly. And I mean, what if that person left and everyone was like, "Oh, look at this architecture that he built, we only have to change it slightly and swap out a few services or Microsoft or AWS came out with something else that deprecated this thing he used and we just have to swap something in and out. And we don't have to worry about any bugs hidden down in code. I mean, what about that, that sounds beautiful, right? Instead of just hating Larry, who like left the company and like, why did he write all this code? And, I mean. I just, it's funny. I mean, this was, it was funny. It was like, the week that we got married, we went to Canada, and we're on vacation, and I was cooking up this idea for this "Don't code" talk the whole time we were up there.

I mean, it's just how my brain works, because there was some conversation I had before we left for Canada, and then the whole time I was there, and I'm like, and I'm sort of talking it through and sort of writing notes in the hotel because I'm just sort of thinking about, okay you're not that clever; your codes not that important. Like code doesn't define you, you're not a coder, you are an engineer. And like when you're given a bunch of building blocks that are solid, then use those building blocks to build something more efficiently and something that's stronger than if you built your own blocks that are potentially full of technical debt essentially. And so I think it just... It's a matter of changing our idea of like, what a developer means. And a lot of developers are still like running by this idea of like, Okay, well, my salary is based on how many lines of code I write or something stupid like that.

Jeremy: And the CEO who asked me how many lines of code we wrote, I said, "Why?" Why would you want to know that? The more lines of code, the more of a risk we are, right? I think that's-

Linda: It's like a Dilbert comics, right? That's like the pointy eared boss, like, well. So I think it's like, it's a matter of... So like my background, my original original background was art. I was an art major in college. And I spent many years struggling to figure out how the heck I was going to like pay off my student loans. And then I was repairing computers on the side, I was like, oh, okay, this is how I make money. But I'll be an artist. My identity as an artist, but like I'm going to make money doing this, like computer thing that like is stupid. Well, and so I remember when visual artists started transitioning to digital. And it was really difficult because they really felt like they were losing their identity or this is like I'm cheating, and I'm not getting my hands dirty and I'm not like using paint, and there's not a canvas and someone can't own a physical thing and so this isn't real. And there was like this division between the digital artists and the visual artists.

And so I feel like it's a little bit similar. But it's the same thing that you're still using your same part of your brain. You're still an artist, you're still an engineer. It's just, you're just evolving essentially. And you're embracing technology.

Jeremy: Yeah. And so there's certainly a shift, though. And I think this is something that you mentioned, where you write less code. But it certainly as a serverless developer, you might write less code, but you write more config. And it's more about, and I guess it's less about knowing how to code a switch statement and more about knowing how to connect, I'll use a AWS terms, because I'm more familiar with those, but how to connect a Lambda function to an SQS Dlq or something like that. So I mean, and knowing how that works, there are scaling characteristics, right? There's still stuff that you need to know like, how many concurrent functions do I want running? Or what are the redrive policies? I mean, there's a lot of stuff that you need to know that is not "if this equals that then do this."

Linda: Mm-hmm. Well, I mean, and that's when we bring the ops people back in like, "Hey, hey, guys, you still have a job. We didn't put you out of business with serverless because there's still servers and there's still config and there's still some infrastructure." So I think a lot of developers are still only going to go but so far. And I think really it's still our job to write like the infrastructure as code or to make sure that infrastructure as code is existing and that's part of our process too. Maybe we're not just writing Terraform or whatever all day long. But that's part of it too, is when you have all these configs and you have all these pieces together then that's when this repeatability becomes more important. It becomes as important as unit testing or integration testing, is make sure that you have a repeatable deployments and that includes your, all the ways these systems connect together.

Jeremy: Yeah, I mean in certainly infrastructure as code is just the new normal right? Like I can't even react... I remember again, this is how old I am but again, if you had to move it to a new system or you had to transfer something to where it was like everything from the database configurations to the patchy config mean, like all that stuff, you had to duplicate it on all these other systems. I mean, actually, not that long ago I, that so I guess it does make me that old. But even just using like OpsWorks, or any of these Chef scripts and Puppet and those sort of things, in order to recreate these infrastructures that just to me was this crazy idea where now with serverless, it's just like, serverless remove, and it pulls down the whole, or tears down the whole stack, and then serverless deploy, and it recreates a new one and it's just there for you.

So I do think learning infrastructure as code and at least the developers, they need to embrace that first piece of it, right? Like he or she has to say, "I am willing to at least give a rough architecture, and then write any code that has to support that." But then I do think Ops people come in there, that is something where understanding the scaling of that and monitoring it. Like that's a whole other thing that we didn't talk about. But I do think that developers need to really embrace that IAC stuff.

Linda: Yeah, I think so too. I think so too. And I mean, these are lots of conversations I've had as well because there's always that struggle between the development team and the ops team of like, something's broken. And whose fault is it? Is it the fault of the code? Is it the fault of the infrastructure? And if you're always deploying everything the same way, if your infrastructure looks the same on Dev, as does test, and then prod, it doesn't matter, like you should be deploying the same way you're deploying the same code, and that repeatability just sort of, like keeps you honest. And I guess that kind of goes back to like writing that code in the portal or whatever. You're just completely bypassing everything and you're not even considering, like, what configuration changes that someone might have made in other environments which you just give your Dev subscription or whatever. So yeah, there's just, there's so much outside of that.

I mean, it feels good when you get something working, when you just like you type something in there and like, I mean my talk at Serverless Nashville, I had a video of the doofus developer that's like typing in the portal. And then at the end he's like, "Yeah!" Because it like it works, that feels great. But for longevity reasons, like you need to embrace your process. And process feels good too right? When you have a process that just works, and like a bug comes up, and you know exactly where it is, you know exactly what the problem is and or like, tests fail, and you're like, oh, okay, that's why these tests were here, because it's going to catch this particular problem before it slips through to production, that feels good too.

Jeremy: Well again, I mean, again, I do want to get into testing just quickly, but that is one of those things too where, knowing that if you change something, I have seen spaghetti code, companies I've worked with or clients I've worked on code for, where there's just so much like, I don't know, if I change this, what happens, right? Like, why is it checking for a string and an object you know what I mean? And trying to figure out why it does that and then having no test coverage at all to know what breaks in production? So I think, yeah, I totally agree with that. But one of the thing though, about the Dev ops or the ops side of things. So obviously, a lot of configuration and like you said, the less code you have to write the better.

So what about going the other way? What about ops people that are very familiar with infrastructure and have been using Terraform, have been using CloudFormation, things like that, over the last couple of years, have gotten very familiar with this, the configuration piece of it. Now then moving into a developer role, only having to write a few lines of code in order to make an entire an entire system.

Linda: Yeah, and that's funny, because I've had a lot of DevOps conversations with infrastructure teams. And a lot of them are like, "I don't need this. We're not a development team. I don't need to have this discussion." And a lot of the discussions have been around our tool, which is Azure DevOps and really and like, I mean, I guess like, that could be seen as a developer only tool. Because you can run builds and you can run your static code analysis and unit tests and all that. But I mean, you can also deploy with your IAC of choice. And so there are a lot of... You can and there's so many like plugins and things, even if you're not using Azure DevOps, if you're using like some other completely different lesser tool. There's a lot of like we have a ton of plugins, so like, if people want to use like, Ansible, or whatever, we support all that. So a lot of discussions I've had, I've had to have conversations about tools that are way outside of the realm of like, what Microsoft produces, or even that I've used all that heavily, because I'm talking to entirely infrastructure teams, and I'm trying to teach them about the culture of DevOps, which isn't necessarily around code development.

You can have no developers whatsoever, like let's say you have some off the shelf tool, but you just need to make sure that that tool is getting deployed in the same way in a way that like supports, I guess, whatever configurations, this like code needs, this software product let's say. And that's sort of a bit... Like in healthcare, that's a huge use case. There's a lot of industries like legal or finance, where they have software products that they use and even have in house developers, but they do have an Ops team that has to make sure that this gets deployed. And as they make upgrades, as cloud platforms change, as maybe they're migrating to a different cloud platform, maybe they're going multi cloud, they still have to make sure that this software product keeps getting deployed and keeps working, and keeps integrating with all these other tools. So they have to start now thinking about testing, they have to think about repeatable deployments, writing terraform.

It's a little bit of a struggle because I've heard of it tons of times. Well, "I'm not a developer." Like, well that's, okay, like, I mean, anybody can be a developer and, it's not like you don't just get a stamp at birth "developer," "not developer." I mean, it's like, goes back to that identity that I was talking about before. But, and I think like, obviously, they have this great benefit of knowing infrastructure really well, and understanding like networking and security way better than I do even in the cloud. But it's more like teaching them like, you can't log into the portal and just even change a little configuration setting. No, because you don't know that you're making that same change in every environment. And then what if you make that change, and then you leave. And then everyone's forgotten that like Sally, like set some flag once upon a time, and then everything breaks the next time you deploy.

Jeremy: And it takes forever to hunt those things down. So I mean, another thing and this is maybe a little bit off topic, but, this idea of like low code or no code solutions and things like that. I do you see those, I mean, I think some people equate serverless to that idea of like what happens when we don't have to write any more code? And now it's just configuration, and why can't we just put that in UIs and things like that? I mean, do you see these tools becoming popular?

Linda: Yeah, absolutely. And I mean, I think serverless kind of started this, at least for me with things like API Gateway. Because for the longest time I was writing these Node apps, and I was using like, Happy or Xpress. And there was so much code that I was just repeating constantly, just to set up like an HTTP server, and to set up routes for like an API. And I mean, a lot of it, I was just copying and pasting code and copying pasting code into handlers and just like to add to like an API and it just like, it took a long time. And now either those types of things are just becoming separate services. Like, we have API Management in Azure. A lot of the things that I was repeating and a lot of the problems I had, I can kind of just do in the portal now and it's just kind of its own service. Or I remember the first time I wrote a web app that was like a Lambda with API Gateway. And I was like, wow, I don't have to worry about any of this HTTP BS. Like, I just wrote my code.

And also too, I mean, I would just take the libraries that I had written before, and I could just sort of call those still from a Lambda. And I really like didn't have to rewrite that much to make a serverless application. And that was really nice. And I still kind of write things like that, like, I'll write a library first. And then the actual amount of code that's in my function is really small. Because I'm just calling either other services in the cloud or I'm calling some library or something. but now I almost never feel like okay, I got to copy and paste this again, and copy and paste that again. Like maybe I have a scaffolding in GitHub and I just have a like a project template that all reproduce to start out with things. But, I never feel like, oh man, like, it really sucks to be a developer to have to, like, make sure you don't like copy paste this thing wrong.

So that I mean, so that was how it kind of started for me with serverless is like, okay, I'm writing way less code, and the things that are replacing the code that I was writing these services, they were so well tested and trusted, and I don't have to worry about it, like I never would be concerned that like, API Gateway or API Management, like just aren't working correctly, it's probably some configuration that I've made. But I think like in your next point about like, sort of low code, no code, then you start to move into tools, like we have Logic Apps. If you look at kind of on the Office 365 side, we've got Power Platform and Flow. And so that's kind of interesting because not only are you is it low code, but we're now taking the shadow IT people and the citizen developers kind of, end like, pulling them out and saying, "Hey, you can now create applications in the cloud."

So it's kind of like democratization of coding now. And so that like, this is even a bigger topic. But, okay, now coders are now engineers. And then citizen developers are developers. And then it opens it up to a lot of people being able to create things really quickly, get them into the cloud, faster. And like on... I don't know what the sort of equivalent would be kind of in the other clouds, but for us, like, I mean, with Logic Apps for example, I've built a ton of Twitter bots. I have so many Twitter bots out there. People have no idea. Like if there's an account out there is tweeting like adoptable cats it's probably mine. And who knows what cloud platform it's on because I've deployed them all out to different cloud platforms. They just run forever, they don't cost me any money.

But for the longest time, though, I was having to like copy paste my Twitter API code across all these apps. But now I just, I'll just use Logic Apps. And I just sort of like, drag it over my Twitter connector, and then just connect that and then maybe I'll write like a function to do something really simple and then... So in it, I have tons and tons and tons of bots now coming out of Logic Apps. And, you can still, I still set up, I have like an ARM template for my Logic Apps and so I still have like, somewhat of a, like a baby CICD process where I deploy those. And I don't really have much code, so I'm not really writing any unit tests there.

Jeremy: And I think that's interesting, too, because I mean, if you look at services, maybe like a kind, I don't know, like Airtable, for example, where you can just drag it and you create a spreadsheet, essentially, or a database and then create forms for it, and then reports. I mean, you mentioned API Gateway earlier. And that's one of those things where, API Gateway and then AppSync, which is the GraphQL, sort of I guess version of that. You right now you can set up your authentication, right? And you use Cognito, or OcZero or something like that. And then you can set up all your routes, so it knows where it needs to go. And you can do service integration. So you can say just pull data, right from the database and or put it right into an SQS queue or something else. And I think that that is really interesting, because a lot of those things that we do now where we write a Lambda function, or we write an Azure function or something to process some data just to move it somewhere else. All of that stuff is starting to go away.

And so that's why I just I bring it up, because I think that those are really interesting tools. And if you can solve those problems, where it's like, you don't need a developer now to write a simple API to maybe, put data in and things like that, but you need them to do something more interesting afterwards. I mean, I'm not saying developers are going away. I think there's more need for us now than there ever was. But it was I find that interesting. And then I think, with your Logic Apps example you know like AWS has like step functions, which is like their state machine type things. And they also have their serverless application repository where you can create like little reusable snippets that you can do all kinds of processes for you. And so I think that's really interesting, because you're right, you don't want to be using the same Twitter API over and over and over again, copy and paste that in, because then it changes in one. And then all of a sudden, you're adopted cats things breaks because the Twitter API...

Linda: Yeah. And like, all those cats don't find homes because I had a terrible deployment process.

Jeremy: And that would be a tragedy.

Linda: Like if there's any reason to have a good developer process, it's for those kitties okay?

Jeremy: It's for the cats. Make sense. Alright so, one other thing. So I do want to talk about testing and I know there's holy wars on this. Especially now with testing in the cloud and things like that. So we don't have to get into the specifics of exactly how we might do it, but just testing overall. I mean, obviously a good idea, and I just don't think enough people are doing it.

Linda: No, I mean, like, a lot of people aren't doing it at all. Maybe most people aren't doing it at all. I mean, I think I mean, I guess most things that are public on GitHub are not production applications. I mean, the production, especially enterprise applications, like those are going to be private. But you almost never see unit tests on GitHub. Like when I go to people's projects, and the usual things that I'm using I don't...

Jeremy: You do on mine. My projects, I write tests, I write too many tests.

Linda: For my demos, like for my conference demos I'll have a test that runs but usually it's just like printing something out. I'm like super lazy about conference talk demos which goes against everything I'm going for here, right? Because like if that demo breaks before a talk, that is devastating. I mean, that's my equivalent of a production app breaking. But yeah, I mean, I think that kind of goes back to, this like muscle memory. I mean, you don't have to go full on test driven development or I think Forrest was saying you need to, like, have those tests in there first. And I mean, that's really good practice. But I think it goes into that analysis paralysis, where people are like, I don't know how to write a test for this. And so you just put something in there, and then it will start to come to you, as you're like writing code, or you're connecting services or you're building something, it will start to occur to you, okay, if, you don't need to test the services themselves. But you start to say, okay, this is where things might break down and then you kind of insert tests there. You don't necessarily have to have it planned all out it in the beginning from my opinion, I mean, I think that's great.

But, I've fallen into a lot of traps with development teams where you try to teach them best practices, and it just gets to a point where they can't move. Because they're like they can't develop because they don't know how to write the test first or... Like I worked on a team that tried to implement required pair programming. And all the introverts were like, "No, we're calling in sick." So I don't... You have to understand human nature there to some extent. But yeah, I mean, as much as you would test any application, it's no different with serverless. And if you've been testing zero, then okay, you should at least test 5% or something. But, I think it is really important. And I think people get into a real rabbit hole too, about unit testing versus integration testing, and the cloud and testing frameworks and all that. And that's like, the thing that I don't want to argue with people about but just do something.

Jeremy: Do something. Yeah, I totally agree. And the funny thing is, is that, for a while I was big into TDD. And I was like, I write my tests up front, and I was very good. Did it for a big project and it worked really, really well. But the problem is, is that you're just... You're right, it traps you, you're like, now I'm writing tests, like I just wrote of these tests and I haven't written any code yet, right? But I just spent like five days writing tests. And often what I do and this is just something that if you have tests are you haven't written tests right now, this is just some advice I'll give. I love using like a code coverage tool to just go back and show you which statements and which branches haven't been run, and then just go in and just write some tests that make sure that those different parts of the code are tested.

As I know, for me, I'll stare at something and I'll have like these nested ternary operators, and I'm like, wait a minute, how do I even trigger that branch, like I don't even know how to get there. And so being able to write that and reason about and then sometimes it's good to go back and look at your test to figure out how code works sometimes, which is always a good thing. Alright. So one more thing about testing in general. Like do you think that the testing culture has changed? I mean, is this due to the cloud? Is it due to serverless? Containers?

Linda: Oh, it's definitely changed. I mean, but it's changed in the same way of people selling their soul essentially as I was saying in my talk. I mean, if people weren't testing before, then they're not testing now. But there are a lot of people that were TDD enforcers that just suddenly don't write anything, because they just like, don't know how kind of. But I mean, really, I think if you start from the beginning, if you start with unit tests, you can still use whatever your testing framework of choice is like JUnit or Mocha or whatever. If you start there at least writing unit tests and writing them to test your code, then you can at least that's something, that's like a good building block, and you can work from there to start and say, okay, what does integration tests look like? How do I test these interactions?

And this is sort of back to what we're talking about before too. Like, the basis of serverless, in my opinion, Is that it's event driven? And so for me serverless isn't Lambda or functions or Logic Apps or whatever. It's about like this event driven paradigm. And it just happens to be fully managed. And it happens to be kind of to support developers writing less code. There's all these other features. But it comes back to the to event driven architectures. And I think that create... That's kind of a different, like kind of a mind switch, I think for a lot of people, too. And so that makes it a little bit harder to wrap their brain around how they might test those things. But that also allows for you to go more service to service to service with events and not have to write a lot of code, but at some point maybe you're ingesting or using some code to kind of figure out like what's inside of this event? What you want to do with it and pass it on. And those are the cases where you definitely want to be testing that.

You don't want to lose some parts of your message because like you didn't test your message in gesture.java or whatever.

Jeremy: Well, the black box of I think that's one of the reasons why people love to write functions in serverless applications, is because there's just, it seems like a black box, like if it goes from service A and service B to service C, and nobody ever sees it I mean, it's kind of like, what happened there? And it's funny...

Linda: Like, why do I commit to GitHub if I'm just doing ABC? I mean, I've fallen into that before, too. And It's just like a Terraform.

Jeremy: It's just a config file, exactly. That's kind of crazy. I mean, and I think the advice always there has been like use functions to transform not to transport, right? So if you're just moving down from service to service that's not the best way to do it. So speaking of the best way to do things, just maybe wrap this up. I mean, off the top of your head, what are the sort of the most important, I guess, processes or best practices for a serverless developer?

Linda: Well, I think it's the same as a developer developer, I think that all the things that we've been kind of drilling into developers forever. And I mean, you're the same age as I am, so like, we've seen everything. We've seen every shift possible and even shifts within like Scrum and how Scrum affects development. And there's all these weird trends in development. I was thinking earlier, when you're talking about testing, that there was a trend where developers would have the customers write the unit tests for them. And that so that way, the developers would know what to write. And I mean, that was weird. I mean, I understand why they were doing that, but that was weird. And so but I think like, do what works for you, right? You're a developer, like you're a trustworthy individual who knows how to write applications and you're transitioning to the cloud.

Everything that you've been doing, it is probably correct. And now we just need to tweak your process. We want you to be more efficient. We want you to use services that are trusted. And but ultimately, I think if you're using your IDE that you love, you're doing your static code analysis in one way or another, you're writing some amount of unit tests and integration tests, and you have a repeatable CICD process, revision control, of course, is always in there. I have to stress that too because that's like, there are a lot of people that write serverless applications that never commit code. And that just blows my mind. Because if you're writing a traditional web app that maybe you deploy to a pass or something, you'd always commit it somewhere. So I think just yeah, I mean, go with what you know. I mean, that like, Forrest talks about the familiarity bias, like there's some good to that too, because like you've been doing things in a repeatable way that have been very successful, continue those, even though you're in this scary cloud environment.

Jeremy: Awesome. Alright, well, listen, Linda, thank you so much for being here today, and sharing all of this awesome knowledge. If people want to get a hold of you or find out what you're working on, how do they do that?

Linda: So I'm pretty much "lynnaloo" everywhere. L-Y-N-N-A-L-O-O. I don't I'm... I find that I'm like bad about checking Twitter DMs, but like, and they're not open because there's just like, not nice people in the world. But if you just mentioned me on Twitter, I'll find it. You can email me. GitHub. I'm out, out in the world. I think probably just a Twitter mention is probably the best way to poke me. And I'll find it.

Jeremy: Alright. Well, I will get all that into the show notes. Thanks again.

Linda: Alright. Thanks.

THIS EPISODE IS SPONSORED BY: Datadog

Episode source