Episode #102: Creating and Evolving Technical Content with Amy Arambulo Negrette

About Amy Arambulo Negrette

With over ten years industry experience, Amy Arambulo Negrette has built web applications for a variety of industries including Yahoo!, Fantasy Sports, and NASA Ames Research Center. One of her projects modernized two legacy systems impacting the entire research center and won her a Certificate of Excellence from the Ames Contractor Council. She has built APIs for enterprise clients for cloud consulting firms and led a team of Cloud Software Engineers. Currently, she works as a Cloud Economist at the Duckbill Group doing bill analyses and leading cost optimization projects. Amy has survived acquisitions, layoffs, and balancing life with two small children.

Website: www.amy-codes.com
Twitter: @nerdypaws
Linkedin: linkedin.com/in/amycodes

Watch this episode on YouTube: https://youtu.be/xc2rkR5VCxo

This episode sponsored by CBT Nuggets and Lumigo.

Transcript
Jeremy: Hi everyone, I'm Jeremy Daly, and this is Serverless Chats. Today, I'm joined by Amy Arambulo Negrette. Hey, Amy thanks for joining me.

Amy: Thank you, glad to be here.

Jeremy: You are a Cloud Economist at the Duckbill Group, so I'd love it if you could tell the listeners a little bit about yourself and your background and what you do at the Duckbill Group.

Amy: Sure thing. I used to be an application developer, I did a bunch of AWS stuff for a while, and now at the Duckbill Group, a cloud economist is someone who goes through cost explorer and your usage report and tries to figure out where you're spending too much money and how the best to help you. It is the best-known use of a small skill I have, which is about being able to dig through someone's receipts and find out what their story is.

Jeremy: Sounds like a forensic accountant, maybe forensic cloud economist or something to that effect.

Amy: Yep. That's basically what we do.

Jeremy: Well, I'm super excited to have you here. First of all, I have to ask this question, I've known Corey for quite some time, and I can imagine that working with him is either amazing or an absolute nightmare. I'm just curious, which one is it?

Amy: It is not my job to control Corey, so it's great. He's great to talk to. He really is fully engaged in any conversation you have with him. You've talked to him before, I'm sure you know that. He loves knowing what other people think on things, which I think is a really healthy attitude to have.

Jeremy: I totally agree, and hopefully he will subtweet this episode. Anyways, getting into this episode, one of the things that I've noticed that you've done quite a bit, is you create technical content. I've seen a lot of the talks that you've given, and I think that's something that you've done such a great job of not only coming up with content and making content interesting.

Sometimes when you put together technical content, it's not super exciting. But you have a very good way of taking that technical content and making it interesting. But then also, following up with it. You have this series of talks where you started talking about managing FaaS, and then you went to the whole frenemies thing with Fargate versus Lambda. Now we're talking about, I think the latest one you did was about Lambda and the container support within Lambda. Maybe we can just go back, or start at a point where, for people who are interested in maybe doing talks, what is the reason for even creating some of these talks in the first place?

Amy: I feel a lot of engineers have the same problem, just day-to-day where they will run into a bug, and then they'll go hit the all-knowing software engineer, which is the Google search engine, and have absolutely either nothing come up or have six posts that say, I'm having this problem, but you won't ever get an answer. This is just a fast way of answering those questions before someone has to ask.

Jeremy: Right. When you come up with these ... You run into this bug, and you're thinking to yourself, you can't find the answer. So, you do the research, you spend the time digging through, and finding the right way to solve it. When you put these talks together, do you get a sense that it's helping people and then that it's just another way to connect with the community?

Amy: Yeah. When I do it, it's really great, because after our talk, I'll see people either in the hallway, or I'll meet someone at a booth, and they'll even say, it's like, I ran into this exact same problem, and I gave up because it was such a strange edge case that it was too hard to fix, and we just moved on to another solution, which is entirely possible.

I also get to express to just the general public that I do, in fact, know what I'm talking about, because someone has given me a stage to talk for 30 minutes, and just put up all of my proofs. That's an actually fun and weirdly empowering place to be.

Jeremy: Yeah. I actually think that's really interesting. Again, for me, I loved your talks, and some of those things are ... I put those things at the back of my mind, but I know for people who give talks, who maybe get judged for other reasons or whatever, that it certainly is empowering. Is that something where you certainly shouldn't have to do it. There certainly should be that same level of respect. But is that something that you found that doing these talks really just sets the tone, right off the bat?

Amy: Yeah, I feel it does. It helps that when someone Googles you, a bunch of YouTube videos on how to solve their problem comes up, that is extremely helpful, especially ... I do a lot of consulting, so if I ever have to go onsite, and someone wants to know what I do, I can pull up an actual YouTube playlist of things that I've done. It's like being in developer relations without having to write all of that content, I get to write a fraction of that content.

Jeremy: Right. Unfortunately, that is a fact that we live with right now, which is, it is completely unfair, but I think that, again, the fact that you do that, you put that out there, and that gives you that credibility, which again, you should have from your resume, but at the same time, I think it's an interesting way to circumvent that, given the current world we live in.

Amy: It also helps when there are either younger engineers or even other younger professionals who are looking at the tech industry, and the tech industry, especially right now, it does not have the best reputation to be able to see that there are people who are from different backgrounds, either educationally or financially, or what have you, and are able to go out and see someone who has something similar being a subject matter expert in whatever it is that they're talking about.

Jeremy: Right. I definitely agree with that. That's that thing, where the more that we can amplify those types of voices and make sure that people can see that diversity, it's incredibly important. Good for you, obviously, for pushing through that, because I know that I've heard a lot of horror stories around that stuff that makes my blood boil.

Let's talk to some of these people out here who potentially want to do some of these talks, and want to use this as a way to, again, sell themselves. Because I can tell you one thing, once I started writing blog posts and doing talks and doing those sorts of things, clearly, I have a very different background, but it just gave me a bunch of exposure; job offers and consulting clients and things like that, those just become much easier to get when you can actually go out there and do some of this stuff.

If you're interested in doing that, I think one of the hard things for most people is, what even makes a good talk? You've come up with some really great talks. What's that secret sauce? How do you do that?

Amy: I think it can also be very intimidating since a lot of the talks that get a lot of promotion are always huge vendor events that they're trying to push their product, they're trying to push a solution. That usually takes up a lot of advertising real estate, essentially, where that's what you see, that's what you see all the threads and everything. When you actually get to these community conferences, or even when I would speak at AWS Summit, it was ... I had a very specific problem that I needed to solve. I ran into a bug, the bug was not in the documentation, because why would it be?

Jeremy: Why would you put that in there, right?

Amy: Of course. Then Google, three pages down, maybe put me on the path to finding the right answer, and it's the journey of trying to put all of the bug fixes in place to make it work for your specific environment and then being able to share that.

Jeremy: Right, yeah. That idea of taking these experiences that you've had, or trying to solve a problem, and then finding the nuances maybe in solving the problem as opposed to the happy path, which it's always great when you're following a blog post and it says, run this command, then run this command, then run this command. Well, what happens on that third command when the thing blows up, and you have no idea what to do? Then you end up Googling for five hours trying to find your way out of that.

You take this path of, find those bugs or find that non-happy path and solve it. Then what do you do around there? How do you then take that ... You got to make that interesting somehow.

Amy: Yes. A lot of people use gifs and memes. I use pictures of food and screencaps from Dungeons and Dragons. That's usually just different enough that it'll snap someone just out of their phone going, "Why is there a huge elf on my screen trying to attack people screaming elf errors." Well, that's because that's what they thought it would be great to call it. It's not a great error code. It doesn't explain what it is, and it makes you very confused.

Jeremy: Right. Part of that is, and again, there's that relatability when you create talks, and you want to connect with the audience in some way. But you also ... This is the other thing that I've always found the hardest when I'm creating talks, is trying to find the right level. Because AWS always does this thing where they're like, it's a 200 level, or it's a 400 level, and so forth. I think that's helpful, but you're going to get people of all different skill levels, and so forth. How do you take a problem like that, and then make it relatable, or understandable, probably? Find that right level?

Amy: The way I see it, there's going to be at least one person of these two types in the room that are not going to be your target audience, someone who doesn't know what you're talking about, but sees that a tool that they're considering is going to pose a problem, and they want to know how difficult it is to fix it. Or there's going to be a business person who has no technical background, and they just want to know if what they're evaluating is worth evaluating, if this error is going to be so difficult to narrow down and try to resolve that, yes, why would we go through something that my engineers are going to spend hours to try to fix something that's essentially a configuration issue?

When I write any section of a talk, I make sure that it addresses a person who may not have come into that with that exact problem in mind. For the people who have, they'll understand the ... In animation, it's called key images, where there are very specific slots where you understand the topic of what is happening and the context around it. I always produce more verbose notes that go with my presentation. I usually release it either at the end of the day, or later on that week, once everyone has had time to settle, and it provides a tutorial-esque experience where this is what you saw, this is how you would actually do it if you were in front of a screen.

Jeremy: Yeah.

Amy: There are people who go to technical talks with a laptop on their lap because they're also working while they're trying to do it. But most of the time, they're not going to have the console open while you're walking through the demo. So, how are you going to address that issue? It's just easier that way.

Jeremy: I like that idea too, of ... I try to do high-level bullet points, and then talk about the bullet point. Because one thing that I try to do, and I'd love to hear your thoughts on this as well. Here I am picking your brain trying to make my own talks better. But basically, I do a bullet point, and then I talk through it. I actually animate the bullet points coming in.

I'm not a huge fan of showing an entire slide with all the bullet points and then letting people read ahead, I bring a bullet point in, talk about the bullet point, bring another bullet point in. Is that something you recommend doing too? Or do you just present all the concepts and then walk people through it?

Amy: I think it depends. I tend to have very dense slides, which is not great for reading, especially if you're several rows back. I truly understand that. But the way I see it, because I also talk very fast when I'm on stage that I want there to be enough context around what's happening, so that if I gloss over a concept, then you visually can understand what's happening.

That said, if that's because the entire bullet block on my slide is going to be about a very specific thing that's happening. It's not something that you have to view step-by-step. Now, I do have a few where, especially in a more workshop scenario, where you're going, I want you to think about this first and then go on to this next concept. I totally hide stuff. I just discovered for a talk that I was constructing the other day, that there's an animation that drops them down like index cards, and that's now my favorite animation right now.

Jeremy: When you're doing that, like because this is the other thing, just for people who have ever ... If you're out there and you've ever written a talker or you've given a talk, the first iteration of it is never going to be the right one. You have to go through and you have to revise. It is sort of weird, and I don't know, maybe you felt this way too, in the pre-pandemic world, when you would give talks in person, most of the time, you'd give it to a relatively small audience, a couple of hundred people or whatever, as opposed to now, when we do talks, post-pandemic, and they're online, it's like, they're immediately available online.

It's hard to give the same talk over and over and over and over again, without somebody potentially having seen it. A lot of work goes into a single talk. Not being able to use the same time over and over again, is not great. But, how do you refine it? Is it that you tested it with a live audience, or do you use a family member or a friend, or a colleague? How do you test and refine your talks?

Amy: I'm actually an organizer at a meetup group, and specifically built around giving people of marginalized gender identities, and a place to stage and write technical content. It is a very specific audience.

Jeremy: I can imagine.

Amy: But it addresses that issue I had earlier about visibility, it also does help you ... If you don't have a lot of contacts in this industry, just as an aside, technical speaking is a way to do it, because everyone loves talking to each other after the stress has worn off, and you become the friendliest person after you've done that.

But also, there are meetup groups out there, specifically about doing technical feedback, or just general speaking feedback. If you want to do something general, Toastmasters is a great organization to do. If you want to do strictly technical, if you do any cloud-related stuff, the DevOps communities are super friendly, even if it's not specifically about DevOps. I'm not a DevOps person, but I have a lot of DevOps friends. Some of my best friends are DevOps people.

And you can get on a meetup or a Zoom call and just burn through your slides for about 10 or 15 minutes and see ... Your friends will be very honest with you, in a small group.

Jeremy: Right. One of the things I did notice, too, giving a speech in person or giving your talk in person versus giving a talk via Zoom call, is sometimes when you don't hear any laughs or chuckles from a little joke that you make in there, it can feel very lonely in that space after you're waiting for something in there, but. It's a little bit ...

Amy: It's worse when there are people in the room. I assure you, it is so much worse.

Jeremy: That is very true. If something falls flat, that's a good point. Just going back to more this idea of creating good talks, and what makes a good talk. Where do you find ... You mentioned, maybe it's a vendor conference or something and you maybe install the vendor stuff, and you find the bugs and so forth. But is there any other places that you get inspiration from? Are there any resources you use to sort build some of these talks?

Amy: Again, the communities help. The communities will tell you, really, it's like, I don't understand this thing, can someone hop on a call with me for real quick minute and explain why this concept is so hard? That's a very good place to base your talk off. As far as making them engaging, and interesting, I tend to clone video gaming videos, just because that's what I watch. I know, if it's going to be interesting to me, then it will probably be at least different than the content that's out there.

Jeremy: Right. That's a good way to think of things too, is if it's something that you find interesting, chances are, there are lots of other people that will find that interesting. All right, let's go back to just this idea of creating new talks. You had mentioned this idea of, again, finding the bugs and so forth. But one of the things that I think we see quite a bit is always that bleeding edge stuff. People always want to write content about something new that happened.

I'm guilty of this, I would think from a serverless standpoint where you're talking about things that are really, really bleeding edge. It's useful and they're interesting. Certainly, if you go to a conference about serverless, then it's really nice to see you have these talks and what might be possible. But sometimes when you're going to more practical type things. Again, even DevOps Days, and some of those other things, I think you've got attendees or talk listeners who are looking for very practical advice.

I guess the question is like, how do you take a new piece of content, one of these problems, whatever it is. I guess, how do you keep finding new content is probably the better way to ask that question?

Amy: Well, to just roll back just a little bit. My problem with bleeding edge content, I love watching it, but bleeding edge content will almost always be a product demo because it's someone who developed a new solution, and they want to share with everybody, which is just going to walk you through how it's used, which is great, except, and this is just a nature of what the cloud industry is like, all of this stuff, it changes day-to-day.

These tools may not be applicable in a few months, or they may become the new standard. There's no way to tell until you're already six months out, and by then, they've already gone through several product revisions. I once did a talk where I was talking about best practices, and AWS released their updated best practices the day before my talk, and I had to update three slides. It threw off my timing, it was great.

That's just one of those kind of pitfalls that you have to roll with. As far as getting new content, though, especially if you're dealing ... It depends who your audience is, because my audience tends to be either ICs or technical leads, and by then you're usually in a company ... If you're not developing these bleeding edge solutions, you're just using the tools that's out there already.

You had brought up my "Serverless Frenemies," which is still my favorite title of any talk that I've ever made, because when I did the managing containers one, and I love all my Devro friends, but they all got into my mentions about why don't you just use Fargate? If you're at the containerization stage, why don't you just use Fargate, because it's not even close to the same thing, it is closer to Kubernetes than it is to Lambda, and I'm looking for a Lambda-like solution. That's what that whole deal was about, and I was able to stretch that out into I think 30 minutes because Twitter will tell you what's wrong, whether or not it's accurate or not, and whether or not they're actually your friends. They are my friends, but come on.

Jeremy: Twitter can definitely be brutal. I think that, and maybe unpack a little bit what you were saying, is you're creating content around existing tools. One way to do it is, you're using existing tools, you're creating content around that, or you can create content around that. Looking at those solutions, you introduce a new solution to something, or you're even using an existing tool, nothing's perfect. You had mentioned that idea of bugs and so forth. But just, I guess new solutions, or just solutions, in general, maybe higher-level abstractions, everything creates some new type of problem that you have to deal with, and that's probably a pretty effective way to generate new content.

Amy: It is. If you ever have to write down an RCA, which, for those who have not had the pleasure of doing one is called a root cause analysis, where you took down production, and you had to explain why.

Jeremy: Yep.

Amy: Or you ever did this, hopefully, in stage, or hopefully, in development where you ran into a situation where ... I had a situation once where Lambda would not delete itself. I call it my Skynet problem where it just hit a stage where it was both trying to save and delete at the same time. It would lock itself and I had to destroy the entire stack and send that command several times just to force that command through.

If you ever have a problem like that, that is a thing that you write up instantly, and then you turn it into slide decks, and then you go to SlidesCarnival, you throw a very flashy background on it, and next thing you know, you have a TED talk, or a technical talk.

Jeremy: Right. The other thing too, is, I find use cases to be an interesting, just like ... Non-traditional use cases are kind of fun too, how can I use this in a way that it wasn't meant to be used, and do something like that?

Amy: I love those. Those are my favorite. I love watching people break away from what the tutorial says you have to do, and I'm going to get a little weird with it, and that to me is totally fascinating. When the whole, I fed these scripts into a computer meme came out, I thought that was super fascinating because that was something a company I had worked for did, they used analytics ... I used to work for Fantasy Sports, to write color commentary for your Fantasy Football team, and they would send it out.

If you did really well, you would get a really raving review, and if you did really poorly, you would get roasted by a computer, and then that gets sent to everyone in the league, and it's hilarious. But that is not a thing that you would just assume a computer would do, is just write hot takes on your Fantasy Football team.

Jeremy: That's ... Sure, go ahead.

Amy: It's so much fun. I love watching people get weird with the tools that are there.

Jeremy: There are times where you could do something like that, you could maybe create a content around some strange use case or whatever, and I love that idea of getting weird with that. The other part of it, though, is that, I guess, if you're sitting through a talk, and it's some super interesting problem that you're listening to, and again, I don't know, maybe it's some database replication thing, that you're just really into, whatever. That makes sense. But I think the majority of problems that developers have, are not that interesting, they're just frustrating.

Probably the worst thing to do is wanting to sit through a talk that talks about some frustrating issue you have. Is there a way to basically say, "Look, I have a problem that I want to talk about. It's not the most interesting problem, but how can you flip that and take a problem that's not interesting and make it interesting?

Amy: The batching containers and the frenemies talk was all based off of a bin library error from within the Lambda AMI. That, on paper is extremely boring, and should be a thing that you can easily look up, it is not. When I went around it trying to make tracking down library errors interesting, just saying it is very slow and can drain the energy out of your voice.

But, I put a lot of energy into my work in general, and that's just how I had to approach pulling these talk is like, I like what I do, just, generally. When I try to explain what I do to people, it sounds super boring, and I own that. Now I'm doing it with spreadsheets, which is much, much worse. But when I tell people, it's not about the error itself, it's about everything that happened to make this one particular error happen. The reason why this error happened was because Lambda uses AWS's very specific Linux AMI when they did not used to, and they left stuff out for either security or performance purposes.

Whether or not we as a group agree with that, that's a business decision that they made. How does their business decision affect your future business decisions and your future technical ones? Well, that becomes a way more interesting conversation, because it's like, we know this is going to break at this part, do we still want to use SSH? Do we still need it for this reason? You can approach it more from a narrative standpoint of, I wasted way too much time with this, did I need to? It's like, well, you shouldn't have, this should not have happened, but no bug should have happened, right?

Jeremy: Right.

Amy: You work through your process of finding a solution instead of concentrating on what the solution is because the solution they can look up in your show notes later.

Jeremy: Right. No, I love that idea of documenting your process as opposed to just the solution itself. You find the problem, you pull the thread and where does that take you? I think to myself, a lot of times I go down the rabbit hole on trying to find the solution to a problem that I have or a bug fix, whatever. Sometimes, the resolution is underwhelming. Maybe it's not worth sharing. But other times, there's a revelation in there. I think you're right, with a little bit of storytelling, you can usually take that and turn that into a really interesting talk.

Amy: One of the things it will also do, if you look at it from a process and from a narrative standpoint, is that when you take this video, and you send it to either a technical lead or a product manager, they'll understand what the problem was because you did not bog it down with code. There's very little live code in mine because I understand that people build things differently, just because every code is as different as every person. I get that and I've come to terms with it. This is the best way to share that information.

Jeremy: Absolutely. All right, let's wrap up the idea of building talks. What is your advice to someone who is starting out new? What's the best way for them to get started, or what's just some general advice for people starting to build talks?

Amy: The best content new engineers can do, and that's mostly because this is never the standpoint from which tutorials are ever written in, is that, as someone who knows very little of the way a language or a framework should work, write down your process, the entire thing on you getting either a framework onboarded, how you build, and a messaging system, things that people have written a billion times because chances are, one, you got that work from someone else's blog post or their documentation, and you can cite that. And two, when you do it that way, you not only get into the habit of writing, but you get in the habit of editing it in a way that makes it more palatable for people who are not in your specific experience.

When you do it this way, people can actually see, from an outsider's perspective, exactly what is hard about the thing that they built, or what people who do not have a different level of experience are going through. If a tutorial is targeted at engineers who know where the memory leaks in PHP are, that's the thing that comes with experience, that is not the thing that can be trained.

When a new engineer hits that point, and they found it in a new framework where you fix it, then you start knowing where to fix other problems. That way more senior engineers and more vetted people can learn from your experience, and then they will contact you and they will teach you how to find these issues, so you don't run into them again, and you end up with someone you can just bounce ideas off of. That's how you get pulled into these technical communities. It's a really self-healing process.

Jeremy: Yeah. I love that. I think this idea of you approaching something from a slightly different angle, your experience, the way that you do it, the way that you see it, the way that you perceive the word or the next prompt that comes back, or how you read an error message or any of those things, you sharing your experience around that is hugely valuable to the people that are building these things. But also, you may run into problems that other people like you run into, and it's just ... Sometimes, all it takes is just a tiny twisting of the words, rearranging a sentence in a way that now that clicks with somebody where the other time it didn't. I love that.

That's why I always encourage people, just even if somebody has written his content 100 times before, whatever slight difference there is in your content, that could have a powerful effect on someone else.

Amy: Yeah, it really can.

Jeremy: Awesome. All right, let me ask you a couple of questions about Lambda and Functions as a Service because I know that you spent quite a bit of time on this stuff. I guess a question, especially, maybe even from a cloud economist, what's next for Lambda and Functions as a Service? Because I know you've written about the Lambda containers, but what's maybe that next evolution?

Amy: What AWS did recently when they released Lambda Containers is basically put it at feature parity with Azure and GCP, which already had that ability, they had either a function service or a function to Json service where you could upload your own container. They finally released the base image, where, granted, if you knew where to look, you could get it before, but they actually released it, and announced it to the general public, so you don't have to know someone in order to be able to use it.

What I see a lot of people being able to do with this now is they really want to do local development testing, so they don't have to push anything to their account and rack up those charges, when all that you want to do is make sure that whatever one line update you made, actually worked and you didn't put the space or the cab in the wrong place, which is, I guess, how it works now and it takes down the entire stack, which again, we've all done at least once, so don't worry about it. If you've ever taken down production, don't worry, you're not the only one, I promise you. You can't throw a t-shirt into an empty conference room and not hit a dude who took down production. I'm going to save that for later.

Local development testing, live simulation is a really big thing. I've seen asked to do full-on data science just on Lambda containers, so they don't have to use Kubernetes anymore, because speaking of cost stuff, it's easier to track cost-wise than Kubernetes is, because Kubernetes is purely consumption-based, and you have to tie a bunch of stuff together in order to make that tracking work. That would be great.

I think from here on, and a lot of the FaaS changes, they're not going to be front ends anymore, it's all going to be optimizations by the providers, you're not going to see much of that anymore. It's not like before, where they would add three more fields and make a blog post about it. I think everything is just going to be tuning just from Lambda's perspective now. That and hooking it to more things, because they love their integrations. What good is Lambda if you can't integrate it yourself?

Jeremy: Right, if you can't hook it up to events. It's interesting, though, this move to support containers as a packaging format. You're right, I think this has been available in IBM, it's been available in Google, it's been available in Microsoft, these capabilities have existed for a while to use a container, and again, that's a very overloaded word, I know, but to use that as a packaging format. But moving to that, the parity there with the other cloud providers is one thing, but who's that conversation for? Whose mind does that change about serverless, or FaaS, I guess.

Amy: The security team.

Jeremy: Security, okay.

Amy: Because if you talk to any engineer, if it's a technical problem, they'll find a way to fix it. That's just the way, especially at the individual contributor level, that's how the brain works is like, oh, this is a small thing, I bet I can fix it with a few days, or a weekend. Weekend turns into a month, but that's a completely different problem. I've had clients who did not want to use Lambda because they could not control the containerization system. You would be pushing your code into containers that were owned by Amazon, and the way they saw that, they saw that as liability.

While it does have some very strong technical implications, because you're now able to choose the kind of runtime you do, easier than trying to hamstring layers together, because I know layers is supposed to fix this problem, but it's so hard. It's so hard for something that you should be able to download off of Docker and then play with it and then put it back. It's so unnecessarily hard, and it makes me so angry.

If you're willing to incur that responsibility, you can tweak your memory and you have more technical control, but also you have more control at a business level too, and that is a conversation that will go way easier as far as adoption.

Jeremy: Right. The other thing, in terms of, I guess the complexity of running K8s or running Kubernetes is one of those things where that just seems like a lot of complexity. You mentioned the billing aspect of it and trying to track cost. Not that everyone's trying to narrow down exactly how much this Lambda container ran them, maybe you have more insight into that than I do, but the idea of just the complexity.

It seems to me that if you start thinking about cost, that the total cost of ownership of running a container and a Lambda function or running it in Fargate, versus having to install and maintain ... I would say, even if you're using one of the managed services like EKS, or something like that, that the total cost of ownership of going down the serverless route has got to be better.

Amy: Yeah, especially if you're one of these apps that are very user generater based. You're tracking mostly events and content, and not even a huge amount of content, you're not streaming video, you're sharing pictures, or sharing ... If you were trying to rebuild Foursquare, you would just be sharing Geo data, which is comparatively an extremely small piece of data.

You don't need an entire instance, or an entire container to do that. You can do that on a very small scale, and build that out really quickly. That said, if you go from one of these three-person teams, and then there's interest in your product, for whatever reason, and it explodes, then not just your cost, but if you had to manage the traffic of that, if you had to manage the actual resources of that, and you did not think your usage would stick with your bill, that's not great.

Being able to, at least in the first few years of the company, just use Lambda for everything, that's probably just a safer solution, because you're still rapidly iterating, and you're still changing things very quickly, and you're still transmitting very small bits of data. That said, it's like there are also large enterprise companies that are heavy Lambda users, and even their Lambda bill compared to their Kubernetes bill, it is ... If you round it to down there Kubernetes bill, you would get their Lambda bill.

Jeremy: Right. Gotcha. I think that's really interesting because I do ... I actually would love to know your thoughts and whether you even see this. I don't know if we have enough data yet to know this, but this idea of using Lambda, especially early on in startups, or even projects within an enterprise, being able to have that flexibility and the low operational overhead and so forth, I think is really great. But do you see that, or is that something that you think will happen is, you'll get to a point where you'll say we've found some sort of stability point with this product, where we now need to move it over to something like Kubernetes, or a container management system because overall, it's going to end up being cheaper in the long run.

Amy: What usually happens when you're making that transition from Lambda to either even ECS or Fargate, or eventually Kubernetes is that your business logic has now become so complex, or your infrastructure requirements have become so complex that Lambda can't do it cleanly anymore. You end up maxing out on either memory or CPU utilization, or because you're ... Apparently Lambda has a limit on how many times you can invoke it at the same time, which some people have hit in real life.

Those are times when it stops being a cheaper solution, and it stops being a target solution because you can run your own FaaS environment within instances, and then you can have a similar environment to what you're building so you don't have to rebuild everything, but you don't have to incur that on-demand cost anymore. That's one path I've seen someone take, and that's usually the decision is that Lambda, before, when it was limited, can't hold it.

Now that you can put your own container, so long as it fits in that requirement, you can pad that runway out a little bit, and you can stretch out how long you have before you do a full conversion to ECS environment. But that is usually how it is because you just try to overload or you have, maybe, 50 Lambdas trying to support one application, which is totally a thing you can do, it may not be the best ... Even with Step, even with everything else. When that becomes too complex, and you end up just going through containers, anyway.

Jeremy: Right. I think that's interesting, and I think any company that grows to the point where that they need to start thinking about that next little infrastructure, it's probably a good thing. It's a good point to start having those conversations.

All right, I got just one more question for you, because I'm really interested. You mentioned what you do as a cloud economist, reading through people's bills and things like that. Now, I thought Corey just made this thing up. I didn't even know this thing existed until, Corey comes out, and he probably coined the term. But in terms of that ...

Amy: That's what he tells people.

Jeremy: He does tell people that, right. I think he did. So, I will definitely give him credit there. But in terms of that role, of being a cloud economist and having to look through people's bills, and trying to find them ways to save it, that's pretty insane that we need people like you to do that, isn't it?

Amy: Yes, it's a bananas job. I cannot believe this is a job that I'm actually doing. It's also a lot of fun. But if you think about it, that when I was starting out, and everything was LAMP stack, when I started. That was a hot new tech when I started, was the LAMP stack. The solution to all of those problems were we're going to throw more hardware at it. Then the following question was, why are we spending so much on hardware?

Their solution to that problem was, we're going to buy real estate to store all of the hardware on. Now that you don't have to do that, you still have the problem of, I'm going to solve this problem by throwing more hardware at it. That's still a mindset that is alive and well, and you still end up with the same problem, except now you don't have the excuse that at least we own the facility that data is in because you don't anymore.

Since you don't actually own the cases and the plates and everything, you don't have to worry about disposing of them and having to use stuff that you don't actually use anymore. A lot of my problems are, one of our services has gone out of control, we don't know why. Then I will tell you, who is spending that money. I will talk to that team to make sure that they know that it's happening because sometimes they don't even know what's happening. Something got spun up into their account, and maybe it was a testbed, maybe it was a demo, maybe they hired a vendor to load something into their environment and those costs got out of control.

It's not like I'm going out trying to tell you that you did something wrong. It's like, this is where the problem is, let's go find out what happened. Forensic cloud bill person, I'm going to workshop that into a business card, because that sounds way better than the title that Corey uses.

Jeremy: Forensic cloud accountant or something like that.

Amy: Yes.

Jeremy: I think it's also interesting that billing is, and the bills you get from AWS are a leading indicator of things that are potentially going wrong. Interesting, because I don't know if people connect this. Maybe I'm underestimating people here, but the idea that a bill that runs, or that you're seeing EC2 instances cost spiking, or you're seeing a higher load or higher bandwidth or things like that. Those can all be indicators of poorly written code, it can be indicators of the bad compression or missing compression settings, all kinds of things that it can jump out at you. Unless somebody is paying attention to those bills, I don't think most developers and most teams, they're not going to see that.

Amy: Yeah. The only time they pay attention when things start spiraling out of control, and ... Okay, this sounds like an intuitive issue, and first thing people will do, will go, "We're going to log everything, and we're going to find out where the problem is."

Jeremy: It'll cost you more money.

Amy: There is a threshold where cloud watch becomes very expensive.

Jeremy: Right, absolutely.

Amy: Then they hit that threshold, and now their bill is four times as much.

Jeremy: Right.

Amy: A lot of the times it's misconfiguration, it's like, very rarely does any product get to the point where they just can't ... It's built so poorly that it can barely hold itself up. That's never been the case. It's always been, this has been turned off, or AWS also offers S3 analytics. You have to turn them on per bucket, that's not a policy that's usually written in anyone's AWS config. When they launch it, they just launch it without any analytics. They don't know if the thing is supposed to be sending things to Glacier, if it's highly used data, there's no way to tell.

It's trying to find little holes like that, where it seems like it shouldn't be a problem, but the minute it becomes a problem, it's because you spent $20,000.

Jeremy: Right. Yeah. No, you can spend money very, very fast in the cloud. I think that is a lesson learned by many, many people.

Amy: The difference between being on metal and throwing hardware at a problem and being on the cloud and throwing hardware at a problem is that you can throw hardware at a problem at scale on the cloud.

Jeremy: Exactly. Right. There's no stopping point like we have to go by using servers ...

Amy: No one will stop you.

Jeremy: No one will stop you. Just maybe the credit card company or whatever. Anyways, Amy, you are doing some amazing work with that, because I actually find that to be very, very fascinating. I think, in terms of what that can do, and the need for it, it's a fascinating field, and super interesting. Good for Corey for really digging into that and calling it out. Then again, for people like you who are willing to take that job, because that seems to me like poring through those numbers can't be the most interesting thing to do. But it must feel good when you do find a way to save somebody some money.

Amy: Spreadsheets can be interesting. Again, it's like everything else about my job. If I try to explain why it's interesting, I just make it sound more boring.

Jeremy: Awesome. All right. Well, let's leave it there. Amy, thank you again, for joining me, this was awesome. If people want to find out more about you, or maybe they have horribly large AWS cloud bills, and they want to check out the Duckbill Group, how do they do that?

Amy: Honestly, if you search for Corey Quinn, you can find the Duckbill Group real fast. If you want to go talk to me because I like doing community engagement, and I like doing talks, and I like roasting people on Twitter just about different stuff, you can hit me up on Twitter @nerdypaws. If you want to be a professional, I'm also on LinkedIn under Amy Codes.

Jeremy: All right, and then you also have a website, Amy-codes.com.

Amy: Amy-codes.com is the archive of all my talks. It's currently only showing the talks from last year because for some reason, it's somehow became very hard to find a spot for the past year. Who knew?

Jeremy: A lot of people doing talks. But anyways, all right, Amy, thank you again. Appreciate it.

Amy: Thank you. Had so much fun.

Episode source