Transcript: DevOps Decrypted Ep. 2 - Pipeline

Romy Greenfield:

Hello everyone, and welcome to DevOps Decrypted. This is our second episode. And today we're going to be discussing what a DevOps pipeline is and how to build one. I'm your host, Romy Greenfield. And today I've got Matt, Lisa and Jobin with me again. Say hi everyone.

Matt Saunders:

Hi everyone.

Lisa Schaffer:

Hi everyone.

Romy Greenfield:

Excellent. I've got them all trained. So I'm going to start off the conversation with, what exactly is a pipeline?

Matt Saunders:

What's the pipeline? Well, I can start on that.

Romy Greenfield:

Yeah.

Matt Saunders:

Well, traditionally, it's something that carries oil from one side of the country to another. Oh no, sorry, that's the wrong podcast. Back in the olden days, when we used to hack on our massive computers with our horrible green screens, we used to type lots of commands to make the computers do things. And eventually, that became code. And then we typed lots of other commands to make sure it worked to deploy it. And to test it, all those sorts of things. The pipeline is basically the modern distillation of that. So it's a load of jobs, that take some source code from a source code repository, maybe go and build it, maybe run some tests on it, maybe deploy it if we're doing some Git Ops, or continuous delivery, that are high level is what I think a pipeline is. Discuss.

Jobin Kuruvilla:

So basically, what you're saying is, it's nothing but it's the point you're taking software from source code to your production environment, or chocolates. In our example last time.

Matt Saunders:

Chocolates, I forgotten about the chocolates analogy.

Romy Greenfield:

Yeah, we started making the ingredients and then putting them in the shop.

Matt Saunders:

Yeah.

Jobin Kuruvilla:

Exactly.

Matt Saunders:

Pretty much that. Yeah. With everything that's involved in between. I guess, the pipeline analogy slightly less itself down in that, or maybe I'm just using the wrong one, that you put some stuff in the top, it goes through a pipe. So when it comes out the bottom, and it's finished, and doesn't need any further messing around with. So that's probably what a pipeline is. A very high level. Any other dissenters on it?

Jobin Kuruvilla:

No, I think I completely agree. Basically, I think I had this definition of a set of automated processes and tools, which is nothing but what you said. I mean, it's a pipeline that you have laid down, which takes the source code to production. A set of processes and tools, I guess.

Romy Greenfield:

Yeah.

Jobin Kuruvilla:

In my view.

Romy Greenfield:

Just the way that I would describe it is, I just remember the first time I ever saw it, and not knowing what it was when I first became an engineer. And it's just all the green blobs that I need to turn green, not red in order to get my code from my laptop into production.

Jobin Kuruvilla:

Building blocks. Yeah.

Romy Greenfield:

And I love the fact that you could take things out, put things in. I have the control to disable a test if I really wanted to, because it kept failing. Don't tell anyone.

Lisa Schaffer:

Did you guys have the hats?

Romy Greenfield:

Wait, hats?

Lisa Schaffer:

So if you broke the ball you had to wear a hat for the day, the dunce hat.

Romy Greenfield:

No, we were not advanced. But I have heard about people having a UFO in the office, a physical UFO that would light up and start making noises if someone breaks the pipeline.

Jobin Kuruvilla:

And then silence, going off. Everybody panics.

Matt Saunders:

The first time I saw some lights reflecting the build process was when I started work at a company just after Christmas. And I went into their office and I thought, on my first day, it was on January the fourth, something like that. And I thought, oh that's really nice. So what the Christmas lights on this light, red lights all around the ceiling, all of them lovely and red. And over in a corner, there are a few little green ones. And it was only a few weeks later that I saw that they hadn't taken the Christmas lights down. The center is not going to be pleased with them next year because their lights were up after 12 night. The colors. I think a lot of people's first introduction to these things are the red and the green and the terrible aesthetics. They're in. And maybe it's less of a pipeline or more of like, you're going down a street through lots and lots of traffic lights.

Jobin Kuruvilla:

It's interesting, because when you typically start off you always start with the red lights. Be it test driven development. We always talk about test driven development. You have your test written but then you don't have the code yet. And so you'll be seeing red lights allover, then you start seeing each one of them just slowly turning into green. It's an interesting process. I mean, you panic at first. But once you get into TDD, you know that you're going to see red lights first, and then slowly becomes green one by one. And whenever they become green, it's Christmas.

Matt Saunders:

Yeah, it's Christmas. Sounds like you're good at this, because I think you're describing test driven development and infrastructure, which is this kind of hello to state that I find it very difficult to get to, it's mostly just like, oh, this doesn't work. It still doesn't work. Or it still doesn't work.

Jobin Kuruvilla:

Everything green is still in my brains. I mean, we're still getting there. But that's marijuana.

Romy Greenfield:

It's the ideal. That's what we're aiming for.

Jobin Kuruvilla:

It's the ideal.

Romy Greenfield:

So where do people start when they want to build a pipeline? What are the typical stages that you'd want to put in there?

Jobin Kuruvilla:

That's an interesting one. Because whether you start simple, or you got to the typical idea of a pipeline, which goes through all these different stages. I think when you're talking about DevOps pipelines, you're probably talking about developing a code, putting it in the source controller, then start building it, doing some tests on it, which Matt mentioned that could be your integration testing, unit testing, probably when some security analysis, static and dynamic, there are a lot of different things that you can do. Dynamic obviously comes at a later stage, but thinking about testing, different kinds of testing, and then package your code, you deployed somewhere.

Jobin Kuruvilla:

So there are different stages. But why do you typically want to start? That's an interesting one, because we would always say first, if you're a beginner, probably start simple, right? I mean, probably go for CI first, maybe aim for CD later time. It all depends on how proficient you are. And I guess where you are on the journey.

Romy Greenfield:

Yeah, I suppose the pipeline can be any length that you want it to be, you can just start with just having an audit check, some linting. Then build up from there. You get that one stage done, and then you can move on to the next.

Lisa Schaffer:

Jobin mentioned start with CI and do CD later on. Do we want to talk a little bit about, what are the differences between CI and CD? I think people can get those mixed up?

Romy Greenfield:

Yeah, I think it'd be really good to cover those at this point, then we can expand the conversation.

Jobin Kuruvilla:

Yeah. CI means continuous integration. Rewind a few years back, you have developers working on software, your central depository, people probably start working on some code, they probably check-in once in a month. It wasn't actually late in the process, that they integrate the code together. And then you had this problems with merging the code, you've added a lot of features once you deploy it into your integration environment. So I think the solution that people realize was, integrate faster, continuously. So that way, if multiple developers are working on the source code, you make sure that you're working on a feature, you integrate it with the mainline, start testing as early as you can. So that is where the concept of continuous integration came by. So basically pulling to the code, testing it together. That's what continuous integration is.

Matt Saunders:

Again, a bit more context around that. When we first start doing pipelines, and we start using CI tools, like Jenkins, there we go. We got quite a few minutes in before I mentioned the J word.

Romy Greenfield:

Never actually used Jenkins as well. Never once. Jenkins' so old.

Jobin Kuruvilla:

We need to talk.

Matt Saunders:

That's poking a hornet's nest. Isn't it? CI, continuous integration. I like to make sure we've got the basics covered first, which is like you've got to build. You got some source code, does it build into the thing that you expected it to. Before we've even got tests or anything like that, get to that stage. And then you start bringing in things like, well, we've got other contributors. It's not just me doing this, other people are doing it, are writing code as well. And the more time or the more effort you can spend on making your build nice and seamless, making sure that all works, the easier it is for other people to run the build. Once you've got a build, you've got your software that's compiling and it's being tested perhaps, it's being deployed perhaps then that's great.

Matt Saunders:

You don't want somebody else to come along and break it or you don't want to break it yourself in the future. You make a feature branch in your source code system. Or maybe not a feature branch, but a branch that's not the main branch. And then the CI system or the build system goes and builds that over there. The main builds to the right, but your feature branch is over there, it's out of the way. So you can make some little changes, commit them, and then CI server, build your branch, make some more changes. CI server builds that for you. Oh, you made a mistake. Cool. We can fix that on the branch. That's the fundamental CI, the continuous thing, do it over and over and over again, over there on the feature branch. So just out of the way if your main branch in case you need to deploy that. So yeah, just adding some color around what Jobin said.

Jobin Kuruvilla:

I think it's interesting. You mentioned about the feature branches, because there are different branching workflows. One thing folks probably won't realize is how important the branching strategy is. With the introduction of Git, obviously, there are different branching strategies that you can employ. Like when we were with CVS or subversion in the past, it was all check out your file, you do something on it, a second back, and boom, it goes into the test environment. But with the introduction of Git, you have Git flow there. Then you have one for GitHub flow, you have the feature branching workflow. Then there are other custom workflows that we as consultants go and implement with our customers, basically looking at their requirements, how frequently you need to manage your card, which environments you are going to deploy? How will you incorporate the Hotfix process, bug fix process? How will you do a roll back? A lot of these elements has to come together. And it all starts with debranching workflows. So when you're thinking of the pipelines, you typically forget that one. But that's where it all starts, isn't it?

Romy Greenfield:

Yeah.

Matt Saunders:

Yeah. Massive area of feature branch thing. Sorry. The whole branching strategy thing. I said feature branch a few minutes ago. I mean, the branching strategy, because even just saying the word feature branch can invoke some fairly emotional responses from some people. Then you started branches, CI. Off you go, Lisa.

Lisa Schaffer:

Without getting too deep into, I know we don't want to mention tools. What does one look for? In a good CD tool? I mean, what are some of the non-negotiables?

Jobin Kuruvilla:

They're not all those different tools? Obviously, the J one, the Jenkins one.

Lisa Schaffer:

The J one.

Jobin Kuruvilla:

If you're talking about planning, then there is J-like tool. So that's two of the J tools. But it's important to see where you're going at the end of the day. You need to start simple. That's true. But at the same time, you're not focusing on today. You're also focusing on tomorrow.

Lisa Schaffer:

Yeah.

Jobin Kuruvilla:

And how much of the functionality that you need going forward? That's very important. I believe in terms of the different stages that we mentioned about earlier, does your tools support all of that. That's probably one aspect. The second aspect would be, are you going to be on-prem, not in terms of the software that you're delivering, for some customers for example, there's a requirement that your CI-CD tool should be on prem? Because they cannot rely on taking a code into the cloud for a sample. So you probably won't be able to use something like the AWS code deploy. So that could be another consideration. So there could be different considerations based on which you will take one tool or the other. I believe.

Matt Saunders:

I used to have a bit of a Luddite answer to that, which is that basically, all these CI tools, what they do is they're just running scripts. Running a build script, or they're running a test script. But I'm modernizing my view on that now, which is in ways that means that is just running scripts, but actually, in writing those scripts and setting up those scripts. There's a whole load of functionality. I mean, the first time I wrote a CI pipeline, I had a script that did Git checkout, blardy blardy blar, to get the files. And it was only a little while later that I realized that when actually there are plugins that can do that thing for you. So you don't have to worry about getting it into the right directory or making sure you get the right branch. The software will do that for you.

Matt Saunders:

So you link your CI system to your source code repository in a more intelligent way than putting in a bash script. And I think that reflects what you'd be looking for. And also why the J word is so popular why Jenkins is so popular because it's a plugin for everything that you could have to do for Jenkins involving multiple different ways of storing your source code, multiple ways of building things. Whether you have docker integration or not, multiple ways of deploying software out with Jenkins. Some of the other tools out there maybe don't have such a wide range of tools. But again, that's not necessarily the right answer either. You don't necessarily take your car to the mechanic who's got the most tools, you take your car to the mechanic who's got the tools that, the ones that you need. So here the big things for me are going to be the extent to which it hooks into your source code, and does the right thing there. Whether you can use it to also deploy out, if you're deploying your software out to containers on AWS perhaps, or onto a server in your data center, etc, those sorts of things.

Matt Saunders:

And also other enterprise things like, can we authenticate against this tool? Can we use our single sign for our enterprise login through Docker or whatever, to make sure that Fred can't trigger the build, because she's not in the right group, but Alice can, and Alice can deploy those granular access control type things, those are the things that tend to be important. And the last one I'd mention off the top of my head is, can you get a good view of what's working, what's not? That can be consolidated with lots of other tools. So integrations with other tools.

Jobin Kuruvilla:

The integrations with other tools, that's very important by them, especially considering that you are looking at the all the different stages, including planning whether your tool integrates with JIRA, for example. That's a big consideration. Of all things, you mentioned about seeing it just as a scripting tool. A lot of the people did that with Jenkins, for example. But one of the reason a customer actually chose Jenkins over other tools is because they wanted to cost the tool in AWS. But at the same time, they wanted some of the agents to run on on-prem, because they had specific requirements where the agent needs to run some specific software on the on-prem servers. So they were that they come in, and just because of that they went for that architecture, where the agents are running on-prem where the control [inaudible 00:17:25] was running on AWS. But yeah, different requirements I guess.

Matt Saunders:

That's a key concept actually, that we should probably introduce here Romy. When you talk about CI servers, there's generally two parts to that. There's the server bit, which knows what your jobs are, what is your build, etc. But then you have agents. So the way you scale out CI is by having lots of agents, and the agents go off and do the builds. So if you've got 100 builds, then maybe you can spin up 182 boxes to do those builds on. So those are the agents. So yeah, having something like Jenkins, most of the other tools that you do this to varying degrees as well. More or less granularity of what agents use, be they instances or containerization. I'm always flirted with containerization. The ability, the different tools to do those things is probably going to be important to you.

Jobin Kuruvilla:

Yeah, speaking of agents, I still remember when I'd lost in Bumble came up with elastic agents and how you can spin up agents as you need dynamically on AWS, that also interesting feature which a lot of our customers love.

Romy Greenfield:

So, circling back to what you were saying earlier Jobin, about start simple with your CI, at what point do you would you move to CD and and what can we tell our listeners to help them with that journey.

Jobin Kuruvilla:

I think it's all about monitoring the impact that it has on the team. Once you CI is working, it's giving you the desired outcome, basically you're able to deploy it fast into one of the environments, then you look at continuous deployment. Well, CD actually, there are two different terms. One is continuous delivery, there is continuous deployment. So continuous delivery means, making sure you have an artifact that is ready to be deployed into production, whereas continuous deployment means you are continuously deploying it into the production environment. That doesn't happen often. I mean, you look at the big companies like Amazon, Netflix, sure it happens multiple times a day. I had probably over 100 or 1000 times a day that deployment happens in the production, but most customers are not ready for that.

Jobin Kuruvilla:

They are probably looking at continuous delivery. So once you have your continuous integration working there obviously you need to figure out your continuous testing, which means in your automating as much as test as you can, and once that is ready, then you can start doing continuous delivery. So you have your code, you have to then automate the deployment into your environments, whether it is cloud or on-prem, it doesn't matter. Slow but incremental steps, I guess.

Romy Greenfield:

You can always deploy into a dev environment or an environment that's not actually relied upon automatically and see what happens and what breaks there and test it there. Before you have to continuously delivering to production.

Jobin Kuruvilla:

Absolutely, it's good that you brought that up, because you have to then figure out what are the different types of environments that you need? You definitely need a dev environment? Are you doing the integration testing in that same environment? Or are you going have a separate test environment, then you might have user acceptance testing happening in UET, you might have a scaling environment, and then it gets pushed to production environment. There are a lot of other detailed concepts that you can talk about, like blue-green deployments. Do you have a blue environment and a green environment where you push it and then you'll switch to production. So there are a lot of different concepts that we can talk about. I don't want to go that detail. But figuring out that environment and release strategy, that's very important.

Romy Greenfield:

I think is you can have your pipeline go all the way up to production, and then have that manual if you're too scared to actually have that deploy automatically for you. There's a lot of companies that do have that manual switch, even though they probably could automate it, they still want that safety net.

Matt Saunders:

I say in a nutshell.

Jobin Kuruvilla:

That's really it

Matt Saunders:

Yeah, that's what people do.And is one of the key differences between continuous delivery, which is that continuous deployment, which is just automatically deployed to production. And again, it's a confidence thing. Picking up on what you said a minute ago Romy, about you want to be able to deploy your stuff to somewhere that doesn't matter. Reality is it doesn't matter. And it matters to us that software developers have an environment that that can matter to them. But that doesn't matter to anything else. So when you start getting into CD, it becomes another fairly nebulous thing around environments. Disposable or otherwise. Because you're absolutely entitled to have an environment where you can go off and break things in. Not because you're intending to break things, but that's generally how innovation happens. That's how you come up with good stuff is by doing it wrong, and then figuring out the way of doing it right.

Matt Saunders:

Or, at least I get it wrong more often than I get it Right. And I think most people are like that in the software delivery world. Environments. That's how you start to tie in the concepts behind CI with the ones of CD. The two things are often bracketed together. And it's important to spread them apart a little bit. Was all about CI-CD. CI is very much about testing your stuff. CD is about delivering. Two sides of one coin.

Jobin Kuruvilla:

Yeah. And Matt you mentioned about the confidence. That doesn't come until you have proper automated testing happening. So with manual testing, there's always that checkpoint where you have to stop your pipeline, wait for people to do the manual testing, do the confirmation and do the sign off. But once you have automated testing going on, you're more ready to do continuous deployment, right?

Romy Greenfield:

Yeah, definitely. If you're confident in your tests, and you believe that you've covered all of the things that are going to take down your app, or really anger your customers, then why not just have it deploy straight to production.

Matt Saunders:

Politics.

Romy Greenfield:

I mean, that is a big problem.

Matt Saunders:

Yeah, it's politics. And it's confidence. I'm sure we've all been in this position where you say, if you get all your tests right, then you don't need any manual tests. You don't need that manual oversight. And people look at you a little bit funny. And it's a big step for a lot of people. But the reality is, it's a big step and also it is inevitably political. I mean, it's not so prevalent these days. But you have change boards, where senior or even I've seen is doing it ourselves within a database. It's not that we got an official change board. But we've got some of those patterns going on. It's like, oh, we need to deploy this, or let's have someone senior just take a look at this just to make sure, and so on. And in you're like, I can see why you want to do that. But actually, this is technical stuff. The technology is there. Years ago, we used to say things like, well, you can run all your unit tests and your integration tests. But I actually want someone to have a look at this website in a browser to make sure it looks the same as it did before the release. We can test for that automatically. You can do headless browser tests. Just being able to do it doesn't necessarily mean that the organization is confident in it.

Matt Saunders:

Everyone's afraid of that runaway deployment that deploys some bad code, and suddenly it's on 100 servers, and it takes 17 hours to back it out. You see in the press, I mean, you see things like what happened recently, firstly internet outage. Firstly in front of half of the internet. And they broke off of the internet, in a slightly aggressive wording on my part, but not far off the truth through some bad configuration, it took them a while to figure it out and fix it. And everyone's scared of that. Gets you in the press, etc. Customers are, but if you're sensible enough if you pull it out to where you can actually get there. And remove those sorts of objections and do continuous delivery.

Matt Saunders:

State of DevOps report. I think I mentioned it last time. There is a definite link between people deploying stuff quickly and automatically, with high performance. Whatever high performance means. It's fairly well defined in the state of DevOps report. It's a mountain. It's political. There's even issues like, if you're going around the change board, then why do we need a change board? Why do these people have jobs? And that's not a good place to be in, you get all sorts of subversion going on.

Jobin Kuruvilla:

It's interesting you talk about that. Because at least in the last couple of months, we have come across two customers who actually wanted to automate that change advisory board activities. So basically, they had JIRA, in one case, GitLab, and another case Jenkins, where you have a workflow for care process within JIRA, but at the same time, it triggers bills or the bills in Jenkins, or GitLab, triggers the workflow changes within JIRA once the build is successful or not. So they do evidence starting from once you're ready to cut the release plants, i creates the ticket in the JIRA with a certain status, and then the release plans goes, it creates an ephemeral environment for the particular release. It does automated testing within that release environment. And then if the testing all passed, at the end of the pipeline, it goes and moves the ticket in JIRA to a different state.

Jobin Kuruvilla:

And finally, there is an approval, that can be manual that can be automated, depending upon your confidence level, you'll finally approve it. Let's say manual, somebody comes in, let's put the blame on him. If everything goes wrong, but that person approves it, it then triggers the deployment into production from the release branch. It's possible, we are already doing it for a couple of customers. But interesting that came up. Cap integrating care process into the pipelines. That's a big thing right now.

Lisa Schaffer:

It's also important to note that, like Matt was saying that there's politics and it's a cultural shift. Too often, when we think of DevOps, we go, "Well, what tools are we going to use for our DevOps stuff?" Instead of saying, "Well, how are we as an organization going to change our mindset to get to the point of being a DevOps organization." And that is breaking down those fears and the traditions that we've put in place for so long, that we now have to move to a different way of working?

Romy Greenfield:

Yeah. And if you give, if you give someone lower down the food chain autonomy, they're going to have more pride in their work, and they're going to be more panicked if they've just done something really quickly and not really checked it properly. So in giving them that responsibility, rather than having this change board or these senior people that have to approve it, actually, you're probably going to get a better quality of work coming from those people anyway.

Lisa Schaffer:

Yeah, you're right.

Matt Saunders:

It's the feedback. That's what it is. Romy, what you touched on there is, it's almost like the olden days where you'd, maybe you're at school, you write an essay and you give it to your teacher, and you wait for him or her to look up and either would be the horrible Pharaoh on their brow, or their eyes light up. It's a bit like that, when you go to change off the change board, you don't know what they're going to ask. Even if they're trying to be entirely consistent, it's probably going to be fairly random. And so the likelihood of your change coming through isn't something that you yourself have any control over, or you have some control over it. But there's still such a large element of surprise there. Whereas with tests, you can see what the tests are going to do in your pipeline.

Matt Saunders:

Pipeline is there, you know exactly what it's going to do. To the extent that you can just fry any old rubbish code that you know is go going to fail, just to see how far it will get. And then you can see where it does fall, then you know what to fix. If you've got a high degree of confidence in your tests. And you could do that so quickly. If you're doing it in CI, until you come back to one of Jenkins founding DevOps three ways principles, which is that instant feedback thing.

Romy Greenfield:

Yeah.

Matt Saunders:

But it's so powerful. It's so undeniably powerful and empowering for dev to be able to do that, because you're just iterating, you're building your confidence all the time. And that's how people do good work.

Jobin Kuruvilla:

Yeah. And we keep talking about being just a cultural change, all the time focusing on the negative side of it, but looking at the positives, if something goes wrong in production, and if you have implemented your pipeline, with the right checks and balances, who to blame, I mean, you're not going to blame the developers. I mean, you should be blaming the pipeline rather. So making sure that you have enough checkpoints in your pipeline is very important. And it's a good cultural shift, because then you don't have to worry about being accountable for something going wrong in production, because you expect the pipeline to handle that. Do you have enough tests? Do you have enough checkpoints? That's important.

Romy Greenfield:

Yeah. And it will highlight where you're lacking very quickly.

Jobin Kuruvilla:

Exactly.

Matt Saunders:

You could say it's like an experience thing. It's like, "Oh yeah, of course Romy wrote the bill, she's new, she doesn't know about this and that, this thing that we did in 2016." That doesn't make any sense. But we'll start with when you start encoding all of that knowledge into your tests. So Romy can come along on day one. Sorry, I'm inadvertently labeling you as a junior engineer, which I'm sorry, but you're a good example. Or good enough for this talk. You can go along and you've encoded, you've used the system to encode all those years of knowledge. So you don't have to go to a senior dev who says, "Well, let me tell you a story about why that's like that." You can encode it, you can write code that saves you from it.

Matt Saunders:

That means that you're protected from that thing. It also means it's right out there and visible. One of the biggest best things about automating things is it means that previously invisible or guarded or cloudy nebulous procedures, processes that you use are not anymore. They're there, they're in code. And you can start engineering out those weird and wonderful foibles, so that the next time a junior engineer does her first build. There's no likely defeat in those problems. I think I've ever thought that point now, but there you go.

Romy Greenfield:

Just trying to look at what questions we haven't covered. [crosstalk 00:33:51]. We were talking about pipelines. What is the feedback loop? We haven't asked that one? Or what is [crosstalk 00:34:00] feedback loop? We've discussed a bit of it already.

Matt Saunders:

I think we talked about it a bit.

Lisa Schaffer:

Yeah. Fail fast.

Jobin Kuruvilla:

Yeah, I think that's the problem that I have seen with feedback loops because people always focusing on the feedback. They're not focusing on the dope part of it. So if you see the DevOps infinity loop, I mean, you would actually see it in our logo. Ryan would happily agree with us. They had ones that really make a nice feedback loop there. But speaking about the loop, you actually say that failures are often projected, but that's the whole point. And then you take that feedback, start working on the code again, fix the issues that you have, deploy it again, check those same problems and see if you are getting the same or not. If not, you start working on new features that look continuous. Focusing on that loop is very important because obviously, you're going to get failures. Irrespective of the text that you put in place, you might actually have some bugs appearing in production. But make sure that you take them, go through the same process that's very important, not just focus on the feedback.

Matt Saunders:

This is a psychological safety element here, which has a particular interest in mind, which is that it does change culturally somewhat, when this sort of information, these loops are available, and you can see them. Almost in every situation, you find that just making things visible. People are sometimes afraid of doing this, because they're like, "Oh, they're going to find out about that little hack I did, like three years ago."

Matt Saunders:

As a consideration, I found the more that, the more feedback you are listing, the more you're making yourself open to it. You find that people start to swarm on your problems, or help you to fix things. Even outside of your direct team, maybe you're sending pull requests on code. And they go to your direct team, your peers, but are also visible across the wider organization. It promotes its great opportunities to learn, its great opportunity to bring the overall standard of things up. All through this visible feedback loop.

Romy Greenfield:

Yeah. I don't think any developer has ever not had the experience of looking back on code they wrote a few years ago, going, "Who the? Oh, it was me." No matter how much experience you have, you'll always find something.

Jobin Kuruvilla:

In the past, when nobody would have known it, it was done by Romy. But now with source control management in place and Git tracking every commit that you're making, we're definitely going to find out Romy.

Romy Greenfield:

My Hands up, it was me.

Matt Saunders:

Everyone's going to find out it was you Romy. But there's more to it than that, in that, everyone also knows that they themselves could be found out for doing something. And because of that, a little negative language disappears, because we're on the same boat.

Romy Greenfield:

Yeah.

Lisa Schaffer:

Like Romy was saying earlier, just being a bit more cautious about what you write. And how many tests you put in, etc.

Jobin Kuruvilla:

And it is not just that. Nobody is asking the question, who wrote it? They're also asking the question, who approved it?

Romy Greenfield:

Yeah.

Lisa Schaffer:

Oh yeah.

Jobin Kuruvilla:

Where is this code? And who approved that code? That's an important question that people ask these days.

Matt Saunders:

Who approved it? Again, I'm going to subvert that point as well, to repurpose it and take it down the avenue of, well, everybody could see these. So actually, everybody could see these changes. So everyone's approved it. Nobody's actively disproved it. Therefore, it's everyone's fault.

Jobin Kuruvilla:

It is.

Matt Saunders:

And once you realize that, then all the negative language that we've accidentally started using around this. It's because I've been in organizations as well. People realize that you can see it in a different light. Yes, it was someone's fault. But actually, nobody spotted it. We've got all these skills that can fix it. And we've got all these skills that can preemptively stop it happening again, and we start to get on a more positive tack which is good.

Jobin Kuruvilla:

I believe it was Amazon's purpose, and that you're convinced and when once AWS went down, and the person who was responsible for it, it was still a manual error. Is that person who was responsible for it fired? And the answer was no, not really, because it wasn't that person's fault. It was a fault of the system, that somebody could actually go in and make that manual error and bring the whole of AWS down. Who is to blame? The pipeline.

Matt Saunders:

Yeah. The organization.

Romy Greenfield:

The team.

Matt Saunders:

Without which to get academic on this. There's some work by a guy called Ron Westrum, who has done a lot of research, and you'll find it in the book called Accelerate by Nicole Forsgren. If you want to look that one up. He's done a lot of work on analyzing organizations, and I can't remember the three classifications. Damn you, Matt, you're terribly prepared for this. But it classifies organizations, one of them is pathological. And you get some of these pathological organizations where if something does go wrong, then the organization will seek someone to blame and it will seek someone to fire or discipline. I've worked in a few. This is one of them. And that does happen. And one of the challenges in doing, or one of the things you can achieve with DevOps, believe it or not, is to change an organization from being pathological to being a nice one. The other nice things in restaurant's definitions.

Matt Saunders:

That's what we're looking for. It's the science. It's the sensible human approach, like from Amazon, I still see organizations demonstrating pathology in public. I can't think of a good example right now. But things on the register where people are saying, Yeah, we had an outage, but we've done something bad to the person who did it. Wrong answer. It's the organization. You can't go blaming individuals, unless the organization has just one employee. There's always more you can do.

Romy Greenfield:

Cool. So thanks everyone, for joining us to discuss what a DevOps pipeline is and how to build it on DevOps Decrypted. We hope you enjoyed the show. Let us know what you think on social media @Adaptavist. We look forward to keeping this conversation going there. And so for me, Romy Greenfield, your host, thank you for listening and Matt, Jobin, Lisa, do you want to say goodbye to everyone? [crosstalk 00:41:22]. So well trained.

Matt Saunders:

See you next time.