Devex

...

"One click" deploy (*) Recently our CEO Raj Kumar spent a week in our Barcelona office where the majority of our engineering department is located. I was working on some ideas for him to meet the team and get some insights into our daily work. While most of the company is aware of UI/UX and front-end changes we are doing to our site, some of the back-end and infrastructure work can seem like black magic. I think it is important to take every opportunity to bring this output closer to the rest of the company and especially the leadership. By chance I watched the very good presentation “Getting Real about Managing up” by Kellan Elliott-McCrea, which contains as one example the idea of letting your CEO deploy to production. One of the goals of our engineering team is continuous deployment. I set this out when I joined Devex to give us a far goal to aim for. I was inspired by Etsy’s Code as Craft blog and the book “Web Operations”. For me the important part was not continuous deployment itself, but all the changes in engineering culture required to achieve it. You need a good technical base from unit to integration tests, infrastructure as a code, continuous integration and a infrastructure team that is working side by side with the developers. At the beginning this seemed to be an impossible task, QA and deployment were completely manual, there was no unit testing, no code reviews, a clear separation between developers and operations. The code itself was a mess too, with lots of moving parts, outdated libraries and no easy way to introduce testing. But we slowly made progress, simplified the system and slowly worked our way up from deploying once in a blue moon, to once, then twice a week. Currently we are at one deploy a day, with some manual involvement of QA. The deployment gets kicked off by a chatbot and is well documented in our engineering handbook. The short version looks like this: Check QA status Tell the chatbot to deploy Check the metrics Everybody in the engineering team is already in the rotation of deploys and it is easy enough for everyone in the company to do it. Our CEO Raj Kumar was happy to do it and sat together with our two infrastructure engineers to help him along. Because there are some permission requirements it was also easier to do it from the workstation of our lead engineer. After some hiccups in QA the progress went smoothly and we had a new release in production. I guess in the end he was surprised how boring it turned out to be. Which a deploy should be. btw: we are hiring: Check out our current open positions *) the hat is part of the deploy protocol and not a fashion statement

When I joined Devex a bit more than six years ago most of the code base was in one large monolith. This monolith was based on Ruby On Rails and was responsible for delivering devex.com. urn This part was called "neo". I guess because it was a rewrite of a previous Java version. I am not sure if it was a reference to Matrix or the Greek meaning. There were already some initiatives in place rewriting parts of it in different services, but the monolith was always looming in the background. Any changes to it required increasing amount of work. It was like someone tried to put the definition of technical code into code. I generally against rewrites, they tend to last for years and you loose a lot of knowledge that is baked into the original. I also have nothing against Monoliths. So I tried refreshing the code base by upgrading the dependencies, getting the few tests to run and making local development easier with Vagrant. I realized fairly early that this was not going to work. So we went with a gradual rewrite. Whenever we worked on part of the site we migrated it to our new architecture (Also a Monolith or at least a Duolith). Now six years later we finally switched off the remainders of "neo" and I thought we should celebrate in style. Legacy code is after all something that successfully supported you for a long time and enabled you to start with something fresh. Devex wouldn't be were we are without "neo". Cremation - ignore the dog So we staged a proper funeral, where everybody who wanted said some words. We also collected printouts of the most annoying bits of code, some very dated looking screenshots and some email and chat conversations from the very beginning. This was my eulogy: Today we say goodbye to maybe the best known system at Devex.+ Neo - from Greek meaning young or new, which he was neither. Or from the Matrix character, handsome, powerful and agile, which he was also not. We all have memories of him that will stay with us for a long time. We laughed about his quirks. He had amazing depths, which brought us sometimes close to tears trying to understand him. His heritage will live forever in our data structures. Finally we burned the printouts, collected them in an urn (aka hummus glass) and cheered with Cava while chatting about the "good" old days. I hope you can give your legacy code also the sendoff they deserve. The End Update (March 2020) We got a present from our CEO while he was visiting the Barcelona Devex office this week.... RIP Neo Mug

Original is here: https://github.com/christofdamian/manager-readme#manager-readme Update December 2020: I decided to make the repository private for now. I have some doubt if a manager README makes sense and mine was also not up to date. I might pick this up again in the future. Manager README? A few people (not only managers) now share information in a README or how-to style document on the web. The first one I came across was How to Rands. I thought it is a good idea for new team members, but also for people in the company who don't have constant contact with me. What I do I am the Technical Director at Devex. I am mostly a manager, but as we grow progressing into a manager of managers. My job is to make the product development run as smoothly and productive as possible. This includes slowly growing the team to adapt to a growing company. It also means making everyone in my team successful in his job by providing everything they need. I am also usually the first contact point when there any questions regarding technology from other teams or outside. I do make decisions about a lot of things and prefer to make an early decision to waiting for the perfect decision. What I am about I am an introvert, this means I get exhausted if I have a lot of meetings and no time in between to recover. Social events with a lot of people I don't know are especially taxing. I am very patient and prefer evolution over revolutions. I am the person who says "no" to most people, this goes for my team as well as the rest of the company. It is easier to say "yes" because it is preferred by most people and you always to want to help everyone. I tend to think a lot before I talk, this can be awkward in conversations. I think meetings with more than four people are usually a waste, unless they are highly structured or ritualised like a retrospective or an all hands meeting. All meetings should result in action items (having another meeting with the same group is not considered an action item). I prefer nudging over forcing people into one direction. I have strong opinions about how do to things, but I can be convinced with the right arguments. I strongly believe in being agile, from the project management side to programming. I try to be as transparent as possible, please nudge me if I am not I am pragmatic and prefer to have something good enough today, then something perfect in some distance future. I take the blame, but share the kudos. My schedule I am usually in the office from 9:00 to 18:00. I don't like to stay longer and will cancel meetings set up after 18:00 unless I agree that it is very important and can't be scheduled to another time. I don't read slack or email outside of working times. I live in my calendar and it is always really packed, but it is fully public and you can just schedule a meeting with me whenever you like. If you are on my team we will have regular 1:1s and also retrospectives. If something important comes up I am always available to talk. Communication My preferred communication medium for actionable items is email. Jira is on par with email. Slack also works, but I am treating it as asynchronous and will not respond as soon as possible, depending on the urgency. I might also not scroll back in a lot of channels when catching up. I will also post updates to the internal blog, maybe once a month. Finally there are the meetings, either 1:1s, team meetings or impromptu meetings. 1:1s I currently have my 1:1s every four weeks. I am trying to increase frequency by sharing the load with others. With indirect reports it will be more like once a quarter. The 1:1 is the place where you can bring up any topics that you are not comfortable sharing in retrospectives or other meetings. I am asking some questions to get the conversation started. These have evolved a bit over time. How is life? What are you worried about at the moment? What are you excited about at the moment? How can I help you learn and grow? Do you have any feedback for me? What I expect from you (if you are on my team) I expect you to try to do the best job you can. If there are any road blocks or problems I expect you to bring these up with me so we can make sure you don't get stuck. Come to me early when there is a problem to avoid any frustration. There will never be blame, we will just fix it together. Books I recommend The Manager's Path: A Guide for Tech Leaders Navigating Growth and Change by Camille Fournier Web Operations: Keeping the Data on Time by John Allspaw, Jesse Robbins Extreme Programming Explained: Embrace Change (The XP Series) by Kent Beck, Cynthia Andres The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity by Kim Malone Scott Getting Things Done: The Art of Stress-Free Productivity by David Allen What I am doing aside from work Most of my free time I try to spend on my bicycle, this is where I decompress and my way of meditation. I am interested in urbanism and life sized cities and have strong opinions about this. I also like to follow the Linux and Open Source world. I am a contributor to the Fedora Linux distribution, but nowadays I have nearly no time to work on it. Me on the Internet If you like to keep up to date with my random rants on the internet / https://twitter.com/cdamian https://github.com/christofdamian/ https://www.instagram.com/christofdamian/

Originally posted on the Devex Tech Blog. Frankenscrumcyle presentation slide At Devex we try to optimise every aspect of our work and at times try radical new things. In this post I describe one of these journeys, that led us from Scrum, to Six Week Cycles, to Frankenscrumcyle and a possible future. Scrum In the product team at Devex we used a pretty standard Scrum process for about four years. In the beginning with one team in Manila and one in Barcelona and later with one or two teams just in Barcelona. As it is the case for most companies this wasn’t a 100% out of the book Scrum, but slightly adapted to our needs. We always had one week sprints, with Demo and Retrospective on Fridays. Grooming on Thursday and a daily stand-up meeting. Everything managed with Jira. It was going pretty well, but there were certainly some aspects we were not quite happy with. Pros Very flexible in adjusting work to requirements Easier team / resource allocation, we can pull the things from the backlog which fit the available team Communication is encouraged through various meetings It is widely used and very easy to find help and information Transparent by relying on Jira and communication We had some tech backlog management, even if it wasn’t perfect Cons Too many meeting with everyone, even if they are not relevant to all Jumping between tasks as there tended to be a lot of unrelated issues in each sprint Bigger teams, which also didn’t always work on the same project Micro management through detailed Jira issues and easy access to the work in progress No fixed scope for a project, it is easy to add yet another sprint to a project as you can always find another feature to add Not dotting the i’s for a project, even though the project took forever we never had a chance to give it a final polish and to fix small bugs we had left over Difficult to fit in tech tasks as features always seem to trump tech backlog No space between projects to regroup, you always feel rushed Jumping between projects for some people, as you would be assigned depending on requirements Six Week Cycle At some point some people suggested trying something created, used and advocated by Basecamp called here the Six Week Cycle. You can read up on it their posts, but the basic idea is that you have a focused work of six weeks for one project (or a collection of smaller ones). The six weeks are chosen to limit the scope (“there’s a great six week version of nearly everything”) and to allow a time without distractions from other projects. Between cycles there is a time to regroup, clean up and plan the next cycle. Teams are relatively small with up to four members improving in-team communication. We gave this a go for nearly six months and saw some advantages, but already run into problems relatively early on. Pros Scope limit, because there is a fixed amount of time we have to reduce the scope to release something Buffer week, a time to regroup and plan for the future in between projects Enough time to also work on the technical side of projects and not just user facing features Deeper understanding of product by developers, because they are bound to one project for the cycle and often had direct contact with stakeholders Smaller teams, compared to the multifunctional scrum teams Autonomy / self organization to be flexible in planning order of tasks Fast in team communication, due to the smaller team Less context switching as there is always just one project per person Less irrelevant meetings, this is mostly about the grooming and whole team kickoffs More direct contact with relevant stakeholders Time to consider in between projects Less reactive as the scope should be predefined and the cycles are too long to quickly change Cons Fixed six week length of cycles doesn’t always fit the projects Short three weeks cycles, which er also tried, had a very high ramping up/down overhead, compared to working time The buffer week was not always used for the intended purpose, the project continued into it and planning moved into the cycle. A problem with reducing the scope. Team composition dictated the solution, instead of the opposite Not flexible enough to adjust to projects, team or changes Lack of communication and transparency between teams and to the outside (infrastructure, QA, planning, … ) Not widely used and difficult to find documentation or experiences from others Some confusion about where to fit tech tasks into the cycles Progress during the cycle was not very transparent Frankenscrumcycle I have to confess that I was not happy with the Six Week Cycle idea from the start. But I was willing to give it a go, because I saw problems with our Scrum and was hoping to at least learn from the experience. One problem was the lack of information about the whole idea. In the beginning there were only two blog posts by Basecamp themselves and then a few others who also gave it a go. Beginning of this year Basecamp released a series of YouTube videos with more insight. One interesting insight is that they just use it for two teams themselves, all other teams are organised in a different way. In the end I decided that we had to change again and I had to convince the team that this new way is better, even if it won’t be perfect either. It needed a name to sell it, so I made up Frankescrumcycle, because it takes ideas from the Six Week Cycles and Scrum. In reality it is mostly Scrum, but with some hard rules to avoid some of the problems we previously had. Fully epic based Each team is going to work only on one epic per sprint, to allow focusing on the work and context switching. This is a takeaway from the cycles and also allows deep diving into each project and involvement with stakeholders. Team size will depend on the projects, but should be two to four developers. Epics from one to three sprints Depending on a rough estimation and how much value an epic brings we can decide on the length of each epic. This takes the idea of limiting scope from cycles and avoids the never ending epics of Scrum. This also includes some planning and ramping down at the end of the epic, closing all the remaining issues and tech tasks for this epic. The different lengths are allowing a bit more flexibility for different sized projects or value. Optional epic extension Each epic can be extended by one sprint and only one sprint. If the team, together with product owner and stakeholders decided that an extension is necessary an epic can be extended by one sprint. Sprints are now two weeks and this makes extending expensive option, so it shouldn’t be the standard option for an epic. Bug / tech tasks / stakeholder request sprints As with the current small batch team, we are going to have some sprints which are just issue based. We can work on our smaller tasks and bugs that pile up when we are focusing on the epics. These should also be scheduled between product owners and stakeholders (including technology stakeholders). One epic = one slack channel For each epic we are going to have one slack channel, which will be closed at the end of the epic. No issue estimation and grooming At least for now. We are going to watch the number of stories instead. I checked historical sprint data and story points and issue number burndown charts behave pretty much the same. Each epic team has to figure out how to best organize their stories to fit them in the time decided on. Something we are picking up from the #NoEstimates movement, which really is a no time/story point movement. You are still estimating the size of stories and split them up in reasonable chunks. One board with epic filters For now we are just going to use one Jira board and project and will add or remove filters for epics, whenever we start a new one. This also means that everyone would use stories to make it possible for others to see what is going on and for QA and infrastructure to see what is being launched or needs testing. During the Six Week Cycle experiment some teams decided to use Google Docs to organise their work, which made it really hard to find out about the status of the different projects. #Ping channel Instead of daily stand-ups we are going to have a slack channel with a daily reminder. (spoiler: this is not really used) This is just to post blockers or information that might be relevant to other teams. For status updates we have the Jira board. The Future We have been running with this for a while now and it improved some of the problems we had seen previously. But some things didn’t work as planned. We are having too many Small Batch sprints to accommodate the many requests from different stakeholders. This kind of brings us back to jumping between tasks and random backlog of Scrum. The #Ping channel doesn’t work. It is no replacement for an in person or even hangout stand-up. This won’t scale in the future. Our team is growing and managing four epic teams at the same time is borderline, five or six is going to be impractical. Managing this all on one Jira board is also cumbersome. Constantly forming new teams is also confusing and teams don’t really gel. The often cited “forming, storming, norming, performing” doesn’t happen in a meaningful way and will get harder the larger the team becomes. One of the problems we have at Devex is the number of different products. It makes it difficult to form product teams with the number of developers we have. Until now we always had project teams that reformed after projects, with all the problems this brings. It would be nice though to move in the direction of product teams to align the teams with the company goals and avoid a lot of the context switching not only from the developers, but also the product owners and even stakeholders. More on this in the future …