DevOps Decrypted Ep. 5 - Crazy Corporate Container Conundrum Conversations C5s

In this month's episode of DevOps Decrypted we discuss how to scale for online shopping demands using Kubernetes, the changes to Docker licensing, and take a look at our annual Adaptavist Hackathon... What brilliant new ideas did we see this year and who won?

Hello, everyone. Welcome to another episode of DevOps Decrypted. This is episode five, the Crazy Corporate Container Conundrum Conversation, C5s. Yeah, say that fast three times backwards.

I'm your host, Romy Greenfield and I've got with me Jobin, Matt, and Lisa today.

So today we are going to be talking about... What are we going to be talking about? Black Friday? Do you want to start off with that? Black Friday and DevOps?

So first of all, well done, Romy, for managing to come out with that name without tripping over it and honestly, for those of you listening in this is the first take. Fantastic.

So yeah, Black Friday. Yeah, apparently we are recording this on the 11th of November, so just before Black Friday, and by the time the episode goes out, it would've been Black Friday.

We've been looking in quite some depth at things that have gone wrong on Black Friday in the past and how we can solve them with technology, process and people and technology. So I think that's what we wanted to talk about, wasn't it?

Yep.

Yep.

Definitely don't want to say that name out loud, so let's talk about containers.

So containers are great for doing this sort of thing for Black Friday. It's almost like the ideal use case where you look at doing containers and container orchestration.

How do you scale up your services given that there's going to be 5X or 10X the user demand? And so I think it fits quite nicely into this container world, and hopefully people are taking advantage of doing things like that for Black Friday.

Black Friday is the perfect use case for it, isn't it, I mean when is another time when the usage spiked so high.

Yeah.

I think we have seen cases where company websites go down during that specific time of the year. I believe there was a blog post or was it an interview, Matt, you have done where we had listed out few examples of company websites going down and obviously costing a lot to the business.

Yeah. And I think in previous years it used to be most of them, anyone retailing a high volume of stuff, particularly consumer electronics. Yeah, people going down, hard down and over the last few years it seems like Black Friday's got bigger and bigger, but also in terms of the tech that people use, the resilience and scaling of things, that seems to have got better and better.

So I think you still hear the odd outlier and I'm not going to mention the one I'm thinking of, because I think everyone suffers from this to some degree. But containers are brilliant for doing this, end of story, right?

Yeah.

I would say so, but let me ask you this, I mean it's not all about containers, even before we talk about containers I think AW Storage changed the game with its auto scaling capabilities and you could do that even without worrying about containers spending a busy to instances as needed using an Auto Scaling group and obviously hard to still figure out what are the different criteria for scaling, but that's true for containers as well.

I think the biggest thing for me is Kubernetes changed the game. It makes it a lot more easier. That's why I'm confused, I mean, Auto Scaling groups, I mean, they do a pretty good job as well, right. So what is the advantage for me to use Kubernetes over standard [inaudible 00:03:41] and Auto Scaling groups, Matt?

Well, I don't ever shut up about Kubernetes, sorry. It's just like I dream about it and but I'm very aware that I've got this echo chamber of how things are done. Honestly, I think we were solving these problems before that, and I think it comes back to things like getting away from monolithic applications.

So I've been in many retailers, well, a few, where you go into them, you look at like why they can't deliver things quickly, while they can't respond quickly to outages, the mean times to recovery and why they're terrible.

And it's because they've got these big bloated apps that do everything. And over the years, it seems that those have died a death a little bit. And people are focusing a bit more on writing smaller bits of code, microservices. And as soon as you start down that path and you can do things like saying... You couldn't say like, "Oh yeah, we are big retailer.com. And we've got this big application that runs on our server. We'll just have five of them," because it wouldn't work that way. It would take them months or years to buy more servers.

And that is even if the massive model list could work together in the first place.

But now we've got all these microservices. So you've got one of these things running to, I don't know, talk to a payment gateway. Well, instead of just having one, that's have 100. These things become stateless. So you can't do that, when they can communicate with each of other.

And yeah. So maybe, maybe the five CS here, the containerization bit is a bit of a red herring like you say, yeah. Auto scaling and Amazon and all the equivalents in other Clouds yeah, can solve these problems for you, right?

It is interesting. You mentioned about the monolith versus microservices, because earlier we used to have this scenario, or if I want to add more capability into the application, you have one big application that you're basically interesting you around from maybe 25 gigabyte to 250 gigabytes for the Black Friday. So such a huge increase for the entire application.

Now we are talking about microservices, where you're probably thinking about, "Hey, you're billing microservice actually needs to go from 10 instances to 100 instances and it's auto scaling, whereas your profile app is not that hit heavily. So it may still remain at 10 and maybe go up to 20 instead of billing going from 10 to 100.

It's interesting, the way you can scale these different microservices in different ways. I think that's another one advantage. Yes.

Just with, I know that we are recording on the 11th of November, and by the time you would've heard this, it would have been Black Friday already.

But what I wanted to ask, the two of you, Matt and Jobin is we only have a few weeks. We can't change these big legacy product products into microservices, et cetera. What are some top tips for ensuring that Black Friday goes smoothly for some of these organizations with little time left?

Oh, you should have started planning this long ago, Lisa. So I would be, so this is about... Yeah. Where do you start? Because it's so easy to say, "Oh yeah, microservices just do everything in microservice."

Exactly.

If you had all the time in the world, yes but if you don't.

All the time in the world, because of course everyone's got all the time in the world to do this sort of stuff because you're working in a large retail organization and there's no product owners who want new features, or new products or new capabilities, none of that exists, right? So yeah. How do you start?

I think there's a few things here. Cashing is a good one. You look at things that are here frequently. How much can you cash things? If you can serve things from your edge, from CDN or some sort of proxy layer saves you going back to the origin service. So you get scaled that way.

Doesn't really work when you're dealing with things like stock allocation. Because if you have 10 people through a cash allocate the same Tamagotchi or whatever on off-site, I'm showing my age, then that ain't going to work.

So after that, where are you looking? So there's-

Matt. I would argue though that cashing, itself, that also is going to take time, right? I mean, implementing proper cashing, that's a big initiative also. Is it going to happen in three weeks, in time? Probably not.

I don't know. Don't know.

Yeah. I'm thinking maybe what you can do is, I mean obviously monitoring. Provided you have monitoring already implemented. You look for what is your current usage, right? Usage limits in terms of CPU, memory, find out where you are, how many users you're getting today and how to anticipate how many you might be getting during Black Friday, which can be a huge task, but again, based on the past few years, you can probably anticipate.

And then depending on that, arrive at a new number, right? For your memory, for your CPU, for the number of instances that you might want. Yeah, just do your best calculations.

Obviously you don't have auto scaling, so you have to rely on your engineers to do that math for you. I have done that math many times in the past experience. So at some point you have to do that, look at the statistics you have, figure out what you might be hitting during Black Friday, then have your engineers ready to act if something goes wrong.

Yeah. I feel like I've heard a story in the past as well from someone that worked in retail where they'd made all those calculations. And so they thought they were well prepared.

And then some influencer on Instagram happened to tweet about their Black Friday sale. And it was maybe three times what the maximum that they'd accounted for, hit them, and they went down instantly and couldn't work out why someone eventually found the Instagram post.

So I feel like even if you've done the calculations, maybe just to throw through a bit extra in there, in case.

That's likely why Matt is actually so lyrical about Kubernetes, right? And containers and scaling.

Yeah. Yeah. That the thing. I, I don't think when, when you're doing this sort of work, I don't think you're ever done. There's there's always something unexpected that's going to happen.

And yeah. I just want to get behind what Jobin said there around, around like monitoring, not just CPU and this, but the time that things take. You can be looking at, if you can get a good overview of all the constituent parts even just understanding what all the constituent parts are for, I don't know, the function of adding something to a shopping basket that could be hitting five or six different services, start measuring them.

You can use good application performance monitoring tools to look at those, and you look at them and you see, it goes through the chain. This one takes one millisecond. This one takes two milliseconds. This one takes three milliseconds. This one takes 40 millisecond. That one takes three milliseconds... "Oh, hang on that one took 40 milliseconds. That it's probably going to start to be a bottleneck then." If once you get under big loads, you can start to find where you might want to investigate some time and effort into making things a bit better.

And those sort of things might be candidates for early microservices stuff. You got three weeks before Black Friday, you're not going to replace your model with a microservice. But an old mentor of mine once said to me about microservices, like "Just consider it to be a big ball of mud and find the bits of mud on the edge that are a little bit less sticky. You can maybe pull them off and make a nice small microservice, bit of mud out of that," that sort of thing. I don't know why I come up with the ball of mud, but…

It's a beautiful analogy.

Yeah, lovely.

Cool, so-

Yeah, one other thing on that, sorry Romy. Yeah. Sorry. I'll shut up in a minute, is I want to talk about circuit breakers.

So if you accept that things are probably not going to go your way on Black Friday, which is good, it means you've got lots of demand and there's probably going to be some problems. You want to minimize the customer impact on those things.

So I'll be looking at things like if you have problems, for example, if you're having to talk to a database that allocates stock, and everyone has to wait for that because you you can't do that in parallel. If you've only got 100 Tamagotchis to sell, if you like cash the selling of 200 of them, then you've got a problem.

Yeah.

So, what tends to happen is you get this thundering herd of activity waiting on database, perhaps this mythical database, where each request starts queuing up and it waits and it waits and it waits and you probably find it waits for 60 seconds before, finally something times out somewhere.

And the user gets like a blank screen or an error 503. Why not look at these things and, and trip it at 10 seconds, perhaps, because if you've got loads of like requests building up waiting 60 seconds, that's just going to hammer the database.

So if it's not going to work, then just bail on it and write client code that does things like, "Sorry, we're in a queue." You can put a nice little, I don't know, a piece of tumble weed rolling across the screen and mitigate that way.

What makes it worse is when the client is actually timing out at 10 seconds or something. And then the database is actually timing out at 60 seconds and the client will keep clicking after the 10 seconds, but is sending a 10 different instances of one. And all of them are now going to queue in the database, making things even worse, right?

Matt, that's a great suggestion because according to the news, there's going to be a lot of shortages around toys. I mean, I've been reading some articles around tips for how to shop on Black Friday and it's, "Get up early, do your shopping early hours in the morning, if you can do that," if you can buy your toys or your gifts over the telephone or on a, on a mobile app versus getting onto their website, there's different avenues to try and get the tools and gifts that you want to put the smiles on those loved ones' faces.

So what you're saying is 10 years back, I had to wake up early, go to Walmart and stand in the queue, and now I have to do the same thing online?

Yep. Pretty much, pretty much. But you can do from the comfort of your home, you can do it from the comfort of your home, with a hot cup of coffee in hand.

Sounds good to me.

And they call this progress?

They call it progress. But, Matt, coming back to the topic, is Kubernetes the only answer to Black Friday?

Definitely. No, I'm joking.

Bias.

It's not to get to that point where you can just go, "Oh yeah, well, we'll just spin up more containers in Kubernetes."

I think you need to be on a two or three year path. Very, very dedicated to properly containerizing things from the fundamentals, properly microservice-alizing, if that wasn't a word, it is now, everything. And it takes quite a lot of effort to get to that in any organization.

The only solution... No, the solution is to patterns. It's about encapsulating things, containers and Kubernetes fit them very, very nicely. But there's many different ways of doing it.

And yeah, most of them are container-based, and you see a lot of innovation in that world, but equally you got to pick your battles and coming back, Lisa, to know what you said about where can you start.

Maybe you are running an infrastructure where everything's running on virtual machines, but you've done some work to use tools like having golden images. So you can actually replicate virtual machines. In which case you can be using things like Auto Scaling in Amazon, which... You can do things like if all your instances are more than 75% busy, then create a whole load more and have the thing automatically doing that for you.

So, yeah, it's all the same concepts. Kubernetes is brilliant. And I won't hear a word said against it. I'm kidding.

He smells.

Right. OK. 

If you could, if you could offer... So obviously the clock is ticking and for those who haven't had their two-year plan in place two years ago, Black Friday is coming around the corner are some things that our listeners can do in order to learn some good lessons from, maybe not having implemented microservices at this point or as many as they needed to, what could they take away from a Black Friday where they weren't actually prepared?

I would say monitoring, right? I mean, see what's going on and have the data ready, right, for next time when you are going to, say that "Okay, hey, we need Kubernetes, this is why, right?

See how many people came on board. What was the unexpected volume of traffic that you got hit with? And then what would've happened if there was that extra memory or extra containers running, which would've helped you scale up, right.

I was going to say, I just suppose that it's actually quite good to have a bad Black Friday example if they're not getting the buy-in from executives or higher-ups in the company.

If it all goes terribly wrong, then at least they have that evidence now, that "If we had the X, Y, and Z, or if we at least tried to implement that, then this might not have happened," or "This is a very strong use case for why we should start implementing this strategy."

Yeah.

Almost all the engineers [crosstalk 00:18:36] sorry, Matt. I was just going to say that almost all the engineers and architects are just saying, "I told you so," right?

Yeah.

Of course, of course.

Yeah. Romy you basically set up at my point there because yeah, I was going to talk culturally about this.

I think if you accept to some degree that you're going to fail or create a culture of psychological safety where that is an acceptable situation, then you can learn from it.

The reality is that Black Friday 2021 will be different to Black Friday 2020, will be different to Black Friday 2022, et cetera.

So if you're not always learning about this stuff, then you're not going to succeed. So, the best possible outcome here is that you get a 10X increase in orders and revenue, et cetera, every request gets fulfilled successfully, there's no outages, no slowdowns, et cetera.

If so, brilliant, give yourself a gold medal, give a star. If it isn't like that, then I would context, I would couch out into principles.

Number one, congratulations on getting all the traffic.

Number two, what can we learn from this? And if you are doing... If you're putting monitoring, if you are thinking of things that... If you are accepting that you think something is going to break, but you can't actually fix it before Black Friday, then make sure that your organization is in a position for that to be seen as a good thing, rather than like, "Look at those useless people. They knew this was going to happen, and yet still let it happen."

If you can be couching those in the right kind of constructive forward-thinking, learning in innovative ways for next year, then you're going to win, or do better than the people who have got these horrible pathological cultures where failure is not tolerated.

Yeah.

Awesome. Talking about Kubernetes and containers, right. Previously what I would've said is, especially in light of what Lisa was saying, "Okay, you don't have containers now, but you want to actually move to containers, containerize your application, maybe microservices."

I would've told my team in the past, "Hey, download Docker Desktop, right. Start playing with the containers and Kubernetes and whatnot." Right? But what I'm hearing now is Docker Desktop is not free anymore.

Yeah.

Yeah.

I don't understand. So what's going on there?

Yeah. So I think there's been this change in the Docker licensing recently.

Yeah.

So yeah, it's only now free for individual developers or open-source communities, which is great that it's still free for people that want to use it on a small scale, but now having to pay to use it from an enterprise level.

Yeah. I wonder what the logical thinking behind that is, obviously it's a lot of money. Because I know a lot of teams are actually using Docker Desktops.

And to be honest, there are a few competitors out there too, and there are alternatives that engineers are coming a bit. Matt, I would like to hear your thoughts on that.

Yeah. So, I've been... Yeah, I've tracked this one quite a lot, because working in the CIO function, internal IT is a team that I sit quite close to and we're very lucky at Adaptavist in that we don't have mandated golden image installs of our laptops. We get to use decent kit.

I've got like latest generation, no previous but one generation MacBook with an M1 chip in it, the new shiny fast Silicon MacBook. So, the result is we don't actually have a department that says, "Right, well we're installing Docker on all these laptops. We're installing the same GUI for developments on all these laptops."

So it was a little bit of a surprise when we started counting up how many people are actually using Docker Desktop, especially as we do a lot of work in containers and we know that we're running Docker and other containerized solutions for CI, building container images and doing some good work there in pushing things out as container images to production. And some of our busiest services are run out of containers.

But yeah, you forget that Docker Desktop is where it all starts. Especially developers are running, building containers, the same container images that end up getting built in CI and then pushed out to production on the Docker Desktop. Docker have, traditionally, and I say traditionally, like they're an ancient retailer or something, like the ones we were talking about in the previous segment, been around since 2014 or whatever it is, they struggled to make any money.

So they get some bad press because you could argue that Docker took a load of open source, free technologies and smashed it all together and make something and then tried to charge money for it.

Sounds like a lot of other vendors.

Well, yeah. Putting that aside, or trying to. Yeah, they want to make money. I think they're entitled to do that.

One of the significant things about it being a big part of how we do business and how we code our apps, our products and services is that what we make money out of the fact that we can seamlessly run containers on people's laptops.

So actually much as I like to bang the drum for free and open source software, I can see a justification for charging for this. And that's fine.

Yeah.

Yeah.

But I think there are alternatives too. So in case if you want to move away from Docker and, and it must be said, Docker is not the only container technology out there, I mean, there are alternatives, there is [inaudible 00:25:04], there's Rocket, there's Containerd, but people always... Just like when you talk about container orchestration, you talk about Kubernetes, when you talk about containers, you automatically talk about Docker.

So that's still going to be the case, I think, for a few more years, at least, I believe so. And, the good thing is Docker Desktop is free for personal use, so you might still be able to get to play with it.

And for a company though, you have to find out the math in your organization and get a license for it, I guess.

Yeah, it's interesting. So, when I first saw this news, my first reaction was, "Oh, Docker! Right, I'm going to go and just do the alternative. I'll take the alternative technologies, Containerd, NerdCTL, Lima, I can brew install this. I can get this working, Rancher Desktop, et cetera. Yeah, we use that. That's just as good, and it frees us from the tyranny of having to pay money to these money-grabbing people."

And then I realized that I managed to create something that only someone who is a container expert could run confidently.

And most of our devs are experts at doing dev work. And in order to save the money to pay Docker, we would have to make them into container experts. That doesn't scale in a company of our size.

And quite apart from the fact that the Docker Desktop thing is something we want to just stay out of the way and just do the job that it does really, really well.

And actually combining a whole load of tech together in a nice graphical user interface that you can use without having to worry about it sounds like a classic thing that actually I can justify paying.

That is a very good point, yeah.

Yeah.

Yeah. You don't have to worry about your new developer who has joined the team then picking up on this new technology, which he probably hasn't worked on before. Yeah. I can live with that. Yeah. I buy your argument on that.

I'm flipping... I'm see-sawing between the two or penduluming or whatever it is, in that there's been... So one of the key things I've always believed in for a long time is that a company with a monopoly on technology is a bad thing and Docker have actually done quite well at getting that.

And to my mind, it's good that actually there are open and free alternatives to all this to solve these problems. And there's been a flurry of activity in building those things up since the announcements.

So yeah, there's things like competitors from... There's a Rancher Desktop, for example, from SUSE which operates in much similar way. It's not as mature. Reason I mention it, I'm running an M1 Mac, is that it doesn't run on that.

So as the world transitions to... Or as Adaptivist transitions to getting people in M1 Macs, we won't be able to use it, so that's not good.

But there's activity out there, and there's competition. And I think that is good. So yeah, it's been an interesting one.

Well, in terms of cost, I mean, it is $7 per user, per month, for the team licensing. And it includes Docker Desktop. I mean, I think we're paying more for our Netflix subscriptions per month.

Not just Netflix, right? Almost all the streaming services in the world.

Exactly, if we tally that up.

It's another truth bomb, yeah.

So we are going to add Docker to the mix. Okay. I can do that.

You must quit one streaming service to replace a Docker.

Sounds good to me.

Well, I guess onto the next topic, Adaptivist is hiring!

Are you a creative, tenacious, passionate individual who would love to come and work for one of the greatest companies in the world?

We have a global presence. We are partnered with the likes of GitLab, CloudBees, AWS, and many, many more partners coming in the pipeline. We would love to have your innovation and your passion as part of our team.

So please jump onto our website, check adaptivist.com, check out the open postings. Not only do we have DevOps openings, but we have openings across our entire organization for different job types at different levels of skill.

So please come and join us, come join the fun.

You must like Kubernetes.

Otherwise Matt will kill you.

I'm not sure this is the image we're trying to project for, but yes I will come and kill you.

Oh and you need a good sense of humor.

Yeah. That is a must. Yeah.

Right. So now that we've tried to sell, Adaptavist so well to the general public, do we have time to talk about the Adaptavist hackathon? Because I feel like that's a good sales point.

That is the actual sales point. I love our hackathon scene, that's where the ideas bloom. Yeah. But Matt had been here way longer than I was. So I don't know, Matt, you might want to talk about it first.

Yeah. So yeah, we do a hackathon about once a year, once every 18 months?

Yeah. We try to do it once a year.

Yeah. And so yeah, the whole idea is 24 hours of yeah, basically, hacking. It's progressed quite a lot in the last few years in that we've been able to get more than just devs involved.

So a traditional hackathon has been getting some devs who want to add a feature or do something a little bit left field that they don't ordinarily get the opportunity to and get some focused time on it. Brilliant.

But yeah, we've had other initiatives, like things like the marketing team coming together. And for example, redoing a lot of how our internal content. That's been a hackathon project in the past.

And yeah, I hope this year will be no exception. We got that coming up next week, I think.

Yeah. I'm pretty sure last year it was a non-engineering based team that won overall as well, voted for by the company. There wasn't a single piece of code in the entire project.

That, that is the beauty of it, right? I mean, it's not just the engineering solutions, it's everybody in the company thinking about what I could do different to make this company a better place. What is the new product that I have in mind, right?

And it doesn't come from devs heads, It comes from the people who are actually working on this since the engineers, anybody within Adaptivist, or the company, right?

And also they actually pull together people from different teams. It's not just the engineering team, right? It is also the product team, the sales team, the marketing team CS, customer service teams. They all come together.

They come with their own ideas and perceptions. And finally, when it comes together and an idea is born, I think people worked on it and out of the 10 ideas, I'm pretty sure there will be three or four, which are really good and we take it forward, right?

Who were the winners last year? Because I was annoyed because it wasn't my team that won. Came second.

What was your idea, Romy?

Why don't you remember?

Yeah. Yeah. I've chosen to wipe that from my memory. That's why I can't remember their name. Our one, we made a Jira Cloud API notifier.

So every time the Jira rest APIs changed for Cloud, it would update us in a Slack channel because sometimes we'd found that they would just update things and not tell anyone in advance and it just broke everything.

I remember that, I remember that. It was a good project.

Managed to do it. It's still in an active channel if you want to find it.

Well, that, that's great. I think one of the key things around hackathons is when you're planning them, you work out what you're going to do and the logistics and the timings. But part of it is also well, "What do I actually do with these things that are deemed to have been good?"

And it's a terrible waste, I find, when my hackathon things just get finished and then just left them aside. So that's great that it's still there and it's still doing good work.

Yeah. I think... Didn't Scriptrunner come from a hackathon as well? Or is that a lie that I've made up?

No... Yeah, I do not think so. I mean, as much as I would've loved to have Scriptrunner coming through a hackathon. I think it was by Adaptivist, Jamie, who came up with that idea. Maybe he thought about it during a hackathon, I don't know.

Yeah. We'll have to get him on an interview and ask him.

And I don't think it's just an idea that is only in Adaptivist, I mean, it's been something that's going on in different companies. I'm right now in the US, but I was in the UK before working for British Telecom. We had something called the hot houses.

So what we do is, again, three days, not the 24 hour window that we have at Adaptivist, but we start... I think it went all the way from 8:00 to 8:00. Start at 8:00 AM in the morning, 8:00 PM, goes all the way.

And you know, three days the business on the engineering teams come together, build upon an idea. And most of the time it eventually becomes a product, at least the winner, right?

So I have seen this happening in other companies, too. So all the big companies might have some version of the hackathon, I believe.