The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

William Mason, the director of QA, reveals that more issues are cropping up with Phoenix than being fixed. Moulton and Fingle accuse Davis and Palmer of mismanaging their teams. Palmer decides to personally investigate Brent’s role in IT Operations and watches him take multiple calls, some of which lead him to fix outside problems instead of working on Phoenix. Palmer addresses the issue with Brent, who explains that no one seems to know how to do anything, so they ask him. Palmer tells Brent to focus on Phoenix. Brent should set his voicemail and email to alert anyone who asks for help to contact Davis or Palmer, and he should reach out to Palmer if anything threatens to distract from Phoenix. Brent tentatively agrees.

Palmer meets with McKee and Davis and tells them the plan. Davis explains that Brent’s approach is almost magical, preventing any documentation of his actions. Palmer outlines his plan to have a team of senior engineers take over the requests that were going to Brent. He notes that only these engineers should ask for Brent’s help, their work must be documented, and Brent should never work on the same issue twice. This method, McKee agrees, will help the senior engineers catch up with Brent’s level of knowledge and leave Brent available to work on Phoenix nonstop. They agree that they will need to give Brent time off after Phoenix launches.

McKee calls Palmer to look at an issue in the change process. She reports that only 40% of the changes are being completed and the remainder are incomplete because of conflicts, insufficient time or resources, or needing Brent for implementation. Upset that Brent has been integrated as a critical component in every element of IT Operations, Palmer calls Davis. With McKee, they debate whether to retain the new process of reporting and requesting changes, and Palmer points out that many changes require Brent would have had to wait anyway, since he cannot complete all of them himself. Palmer realizes that Brent is like a factory machine on which all other processes depend. Instead of reversing the change process, he tells McKee to track which changes need Brent, what they need him for, and their priority level. The team will try to distribute work to the senior engineers designated to protect Brent, consulting Brent only when necessary. Palmer starts to see what Reid meant when he compared IT Operations to manufacturing, and he wonders what other insights Reid might have.

At 7:30 pm on Friday, two hours past schedule, Palmer watches Development and IT Operations struggle to test Phoenix. Mason complains that Development is still sending releases and files at random intervals, preventing any meaningful testing from proceeding. The IT Operations team is stuck waiting for the files while continually trying to set up proper test environments. Palmer gathers McKee, Davis, and Mason, and they agree that the rollout is going terribly. Palmer tells Mason and McKee to start regulating the developers, keeping Davis informed of what they need to test. Palmer emails Masters, Moulton, and Allers, letting them know that a potential failure is in progress, which could impact stores’ ability to accept orders. Palmer calls Masters, who tells him to push ahead unless Moulton agrees to delay the rollout. Palmer talks to Moulton, who claims everyone has already done their part for Phoenix and blames IT Operations for holding back release. Furious, Palmer tells Moulton that Phoenix is about to become a disaster.

Davis tells Palmer that the database conversion for Phoenix is progressing too slowly. Phoenix will not deploy until Tuesday, he says, adding that they do not have the physical space for all the servers they need. Though Allers promised that virtualization would save them, the process failed, forcing the team to resort to physical servers. Palmer tells Davis to find out how the issue will affect POS system conversions.

By Saturday morning, Phoenix is operational, but the POS systems are down. Palmer asks Moulton to draft an email address for the hundreds of complaints coming from their stores, which must now process transactions manually. Maggie Lee, the Senior Director of Retail Program Management, takes over the emails. By 2 pm, stores are struggling with manual operation, and errors in Phoenix’s transactions are cropping up, including lost orders and multiple charges per transaction. Ann sets up a war room to field customer complaints.

Palmer heads home, too tired to help anyone. After a couple of hours of sleep, Palmer wakes to a call from Davis informing him that Phoenix is displaying customers’ credit card information, and an angry Pesche is trying to cover their legal liability. Palmer heads back to work.

On Monday morning, Masters criticizes the teams involved in Phoenix, demanding that everyone work to fix the issue. The media is covering the Phoenix fiasco nonstop, and customers are furious. When Masters leaves, Moulton tells everyone to get to work on improving Phoenix’s functionality, but Palmer says they need to make Phoenix function at all and to repair POS functionality. Allers and Mason agree, marking a rare moment of cooperation between Development, QA, and IT Operations.

Palmer checks on Ann’s war room and is unsurprised to see hundreds of stacks of orders and transaction documents. Ann’s people are trying to resolve order issues, but Pesche notes that they have restricted credit card information printed all over the room. Checking his black binder, Pesche tells Palmer that auditors are in the building and notes that if they see the private data printouts in the war room, the company will be in danger of fines, transaction fee increases, and potential legal trouble. Palmer tells Pesche to distract the auditors while he works out a way to destroy the illegal data.

Pesche calls to tell Palmer that he successfully arranged for the auditors to stay in a different building but reminds Palmer that the internal audit plan is due today. Palmer laughs that he has not thought about that since the Phoenix issues started, and Pesche offers to lend IT Operations a couple of engineers from IS. Palmer accepts, noting that IT Operations, IS, QA, and Development are all working together today.

Palmer waits outside Masters’s office with Fingle, Allers, Ann, and Pesche. Moulton comes out looking petrified, and she sends Palmer and Allers in. Masters complains that the Phoenix issues have cost the company money and respect. Palmer reminds Masters of his warnings about rushing Phoenix, but Masters doubles down on blaming Palmer, noting that IT Operations seems to be in the way of the other departments’ work. He reveals that the board is considering either breaking up and selling the company or outsourcing IT to a third-party firm. Palmer realizes his job and the jobs of all the IT Operations workers are in danger. Masters says if he gets solutions to the company’s problems, he will consider keeping IT Operations in-house.

At lunch, Palmer and Allers get drinks and discuss how people outside of Development and IT Operations expect miracles from their departments. Since no one outside the departments understands their work, upper management thinks they have more power than they do. Palmer sees that the need for continuous improvement in technology, coding, and features presents a critical issue, forcing the teams to constantly rework and restart projects. Allers apologizes for setting an unreasonable rollout date for Phoenix, and the two agree to work together moving forward.

Later, Allers invites the IT teams to a party in another building. Palmer forwards the invitation to Davis and McKee, encouraging them to attend, but Davis says they are still mired in Phoenix’s problems. Realizing he needs to foster cooperation between Development and IT Operations, Palmer decides to call Davis.

At breakfast, Palmer gets an email praising the newly changed management process, which prevented overlapping changes between engineers. As soon as Palmer arrives at work, McKee calls him to the CAB room, where the week’s scheduled changes are in a separate rescheduling pile. None of them are getting done due to the Phoenix cleanup. Palmer realizes that the fourth type of work is unplanned work, which disrupts everything else that needs to get done. Leaving McKee to rearrange the changes, Palmer calls Reid.

Reid confirms that the four types of work are business projects, internal IT projects, changes, and unplanned work. Reid praises Palmer, noting that he is already working through the Three Ways. By protecting Brent, Reid explains, Palmer has identified IT Operations’s constraint. Now he needs to protect Brent from unplanned work, as well. Reid references Goldratt’s The Goal, noting that, in addition to protecting the constraint, the business needs to restrict workflow to match the constraint’s allowable rate of completion. Further, workflow improvements must ensure that the constraint is never overwhelmed and never unoccupied. Reid laughs at Allers and Pesche, saying they focus too much on the wrong type of work. The next step, he says, is to identify the work that matters to the business, rather than completing irrelevant tasks.

Palmer gets an email from Landry informing him that the invoice system has been malfunctioning for three days, potentially resulting in $50 million in lost payments. Landry blames IT for the issue. Palmer meets with McKee, who compiles all the changes made in the last three days. Palmer organizes the team around investigating each change to find which might have caused the issue. By the end of the day, the team has isolated eight changes that could be responsible, and Palmer tells them to continue working on the issue. They will reconvene to compare at 10 pm.

Palmer goes home for a break, and Masters calls him, angry that the invoice issue remains unresolved. Masters reports that Landry blames IT’s slow progress and Brent blames Palmer’s leadership for the delays. Upset, Palmer calmly explains that his reorganization of the department is designed to prevent anyone from worsening the ongoing issues, as with the payroll system crash. Masters tells Palmer to bring the IT team back to the office to work on the issue overnight. He wants everyone, including Brent, to focus only on the invoice issue, and he wants updates every two hours. Palmer warns Masters that this plan will exacerbate the issue, but Masters insists that he contact the IT team. Palmer tells Masters to call them himself and expect his resignation in the morning. Palmer’s wife, Paige, panics, asking Palmer how they will pay their bills if he quits.