Mob Programming
Programming performance isn't about the speed at which we can type out code, it's about the quality of ideas; 3 minds produce higher quality ideas and would result it more effective coding, provided the team has good values around efficiency and timeliness; or it can be a detriment, if someone doesn't share the values and takes the team into costly rabbit holes because they are focused more on code and less on delivering value.
There are plenty of studies out there on the costs of context switching, usually shown as ~15-25 minutes of lost productivity for each interruption (more mistakes, lower quality, lower quantity). Assuming a team of 3 engineers, each working independently but requiring cross-reviews by the other two for each PR, if each produces a PR on Monday, on Tuesday each is now juggling 4 tasks (their own next PR, the two PRs from their peers, and their own PR from Monday that came back with comments). Even in an ideal-case scenario, this adds 3 additional context switches to them, or a loss of about an hour of productivity.
From planning perspective, the context switches are never accounted for in each engineer's sprint plan, and they are likely to prioritize their own coding over code reviews (seen than over and over again), meaning on Wednesday the situation is likely even worse - there's now 9 work items in flight (3 from Monday, now in 2nd review cycle, 3 from Tuesday in first review cycle, 3 new ones being coded). At this point you need to hire a project manager to keep track of all these items. You also likely have merge conflicts as they need to be rebased on top of each other or on a regular basis. Things only get worse by Friday, at which point there may be a couple of PRs merged, with a dozen "work in progress"; no engineer achieved their weekly sprint goals, there's half-complete code everywhere, and, oh, there's 10+ context switches per person per day, meaning every person loses ~3 hours of productivity per day on context switches alone.
Mob programming benefits arise from reduced “work-in-progress”, faster cycle time from ticket assignment to production, higher quality of ideas due to multiple perspectives, higher quality of code due to deeper code review since everyone’s already in context, higher on-going learning and redundancies in case people are out on PTO. Mob programming brings many of the Lean Software Development benefits to practice.
To help us in our search for waste, it helps to visit the work of another of TPS's forefathers, Shigeo Shingo. Shingo identified seven major types of manufacturing waste [2]:
Inventory
Overproduction
Extra Processing
Transportation
Waiting
Motion
Defects
Mary and Tom Poppendieck later translated these seven wastes into "The Seven Wastes of Software Development" [3]:
Partially Done Work
Extra Features
Relearning
Handoffs
Delays
Task Switching
Defects
The principles of Lean dictate solving for Flow, rather than resource utilization, to improve the overall output. Goldratt's "The Goal" [Goldratt] is a good primer for understanding those principles and why they work. After Toyota demonstrated the efficiencies at scale, it launched the whole "Lean Startup" movement (Eric Ries), which has spread now to “Lead Product Playbook” [Dan Olsen], “Lean UX” [Heff Gothelf], “Lean Software Development” [Poppendieck] and, obviously, mob programming. Peter Drucker wrote about it even before that, to optimize for solving "idle work" (tasks sitting in the queue somewhere, e.g. Inventory in Lean vocabulary) vs "idle workers" (the tendency for managers to try to ensure every employee or work center is always 100% utilized) - because inventory (partially done work sitting in a queue somewhere) is more expensive than having a worker idle for some time.
Standard Operating Procedure for Mobs
The mob consists of 3 engineers with diverse range of experience, ideally with a senior, mid-level and early-career, with equally balanced skills in frontend, backend and database.
All 3 members of the mob are each 100% accountable for all code produced by the mob
The mob has negotiated and agreed upon a minimum of 6 hour overlapping block of time each business day during which they are on the zoom call performing programming as a mob.
The mob has agreed upon a lunch/dinner break time where the entire mob breaks for food. Additional short 5-15 minute breaks for the entire mob may be agreed upon also to handle domestic affairs or clear the head to ensure maximum productivity. The mob starts together and breaks together.
Webcams should always be turned on. Being able to see non-verbal signals is hugely helpful, both for reducing friction as well as noticing when someone may have a different opinion or wants to chime in.
A “day in SOCi” consists of at least 6 hours of programming as a mob, a lunch break of up to a hour, and 2 hours of individual research, studies, 1:1s, meetings and other such activities. The mob is expected to do a full 8 hours of programming if there are no auxiliary activities needed or planned for the day. These are the minimum expectations - when the mob is behind on commitments, dealing with fires, or on mission-critical solutions, additional time is expected to be put in to complete the tasks and ensure the success and health of our customers and the business.
Daily Kickoff (30 minutes or less)
Mob assembles at pre-agreed-upon time. The mob commits to each other to be on time so that the day can start without interruption.
The mob members brief each other on things that they did or learned during their solo time.
The mob members review their current project and set a plan for the day of what they desire to achieve. They identify areas in which they might need to contact tech lead, design or product and proactively send out requests to those parties so that they can have their answers in a timely manner.
Programming time (at least 6 hours)
There are three roles, Typist, Navigator and Support.
The Typist is a smart input device, the only thing they input into the computer is what the Navigator told them to, nothing more, nothing less. Watch out for Typist doing things they were not told to - “run-away typist antipattern” leaves the other two just watching, unengaged and loses all the benefits of the mob.
The Navigator tells the Typist what to do. They are doing the majority of the thinking, and executing those plans through the fingers of the Typist.
The Support observes the process as a whole, is responsible for keeping track of the timer, and assists the Navigator as needed; any orders given directly to the Typist by Support are to be ignored by the Typist, as that would lead to “run-away seniors” antipattern, in case the Navigator is less experienced.
After 10 minutes, Support is responsible for enforcing the switch - Support becomes the next Typist, Typist becomes the next Navigator, and Navigator becomes Support. PHPStorm has a simple Pomodoro timer at the bottom status bar that works for keeping the time.
The switch happens at the time interval, not at logical code / ticket boundaries. This ensures that all team members are fully engaged and participating in each logical section (as usually a section requires more than 10 minutes to complete). In case of more complex test setups, this also ensures that each has it set up, or forces the mob to automate the test setup through TDD. Switching happens mid-sentence, which makes picking it up for the next person easier as the code is likely in a syntax error state. For fast switching, use https://github.com/remotemobprogramming/mob command-line tools.
Stop the timer if the mob goes into research / discovery mode. Resume the timer once coding starts again.
Lunch break may be held if the team desires (doesn’t count towards the 6h). Some teams may prefer to do one long session, others may split into two 3+ hour sessions.
Additional 2 working hours are to be used for:
Meetings with product, tech leads, customers or other departments
Set up for success in current tactical tasks for next day:
individual research on upcoming tasks
writing additional unit tests
improving your local setup / fixing any configuration issues
“kicking the tires” / ad-hoc testing of any recently released functionality on production / client accounts
Broaden domain knowledge of all things SOCi
checking other recently merged PRs from other teams to maintain broader awareness of other teams’ work
attending or listening to provided recordings of customer interviews with product managers to understand customer problems better
attending or listening to recordings of product, marketing and sales demos of other SOCi products
Personal development plans
1:1s with management (should always occur outside the mobing hours)
improving your general technical skills in areas identified by your manager through the reading of books and online reference materials
improving your understanding of related technical areas, such as DevOps, machine learning, non-relational database engines, systems design, and others
improving your interpersonal skills through the reading of recommended materials
improving your understanding of areas adjacent to engineering,such as product design, product management and product marketing through the consumption of recommended books, online resources, such that you can collaborate with, and understand their perspectives better
Some time during the day, the deployment process reaches a stage where the mob needs to stop their work and verify their tickets on production. Stop the timer and ensure everything works well on production.
Wrapping up the day
Upon reaching a suitable stopping point for the day, the mob stops and begins reflecting on the day.
The mob reviews their plans from the morning and discusses what was completed, what went well and what they could have done better, both from collaboration, process and project perspective.
The mob writes up an EOD report that summarizes the day and publishes it to the corresponding channel.
The mob disbands, either to sleep (in case of eastern timezones), or to their individual work (in case of western timezones).
Every two weeks or so, the mob meets for a retrospective and reflects on their team values, agreements and commitments (including the product manager).
Antipatterns to watch out for
Multiple Navigators - confused Typist due to conflicting orders; often causes Runaway Duo; assign 1 person to be the navigator in each rotation
Runaway Typist - happens when the typist is allowed to think and code; leads to two engineers watching one engineer work alone; typist should only do what the navigator instructs them to do; Support should call this out whenever observed.
Runaway Duo - two experienced engineers leaving the 3rd behind due to insufficient Single Navigator role enforcement
Sleepy Wheel - Support person zones out; happens when switching is done at context boundaries (every few hours) instead of fixed time (10 min) boundaries; same issue as with Runaway Duo, allows the third to disengage, reducing quality and depth of ideas; caused by inadequate timer management; assign the Support role person to always enforce timer to solve this
In-and-Out Members - 1 out of 3 out separately for a hour (lunch, errand, 1:1, meeting) leads to others needing to catch up the 3rd when they return; code only vetted by 2 people at the time of writing, wasteful due to having to bring another person up to speed; instead, align your breaks for the entire mob
Rabbit-Holing - complexity increases during solutioning; unchecked, this rapidly leads to schedule overrun. Good solutions are simple. Call in the tech lead early to assist.
Code Complexity Overrun - if code becomes overly complex, there’s a problem with the data architecture; review data structures or call in tech lead.
Coding Without Architecture - leads to complex and difficult code; good architecture leads to simple code; always visualize the data structures and flows upfront, before diving into code
Split to Parallelize - mob breaks task up to individual tickets to work in parallel, code reviewing together; leads (after 3-4 weeks) to too many work items partially done and mob requesting we hire a project manager; never has this worked to achieve the desired timeline. Splitting the mob to work in parallel always makes the timeline longer, even less work is fully complete and project management / coordination overheads skyrocket.