Using ROI to Redesign a Federal Supervisor Training Program

For 17 years, Dr. Bill Brantley worked across multiple U.S. federal agencies, including OPM, Social Security Administration, and the U.S. Patent and Trademark Office (USPTO).

In one of his most challenging assignments, he was asked to rescue a failing mandatory supervisor training program.

He chose to manage it “by the numbers” using the Phillips ROI method.

Why ROI Belongs in Training and Projects

Most project managers and training leaders have heard of return on investment (ROI), but often only as a financial afterthought.

In learning and development, many know Kirkpatrick’s evaluation model.

Phillips’ ROI approach adds a more explicit financial and business-impact layer and is designed to link training and project activities to tangible organizational outcomes.

Stakeholders are always asking the same question: “I’m giving you money—how do I know it was worth it?”

ROI becomes the bridge between project activities and strategic value, especially when you need to justify investments or redesign something that isn’t working.

The Problem: A Mandatory Training Everyone Hated

Brantley inherited the Supervisor Certificate Program, a mandatory course for all new federal supervisors.

He actually turned the assignment down three times because he knew how unpopular it was.

Eventually his boss insisted, and within minutes of taking ownership he was confronted by an angry stakeholder group (trademark training leaders) threatening to pull out and build their own program.

The original design had all the hallmarks of outdated training:

• Eight full-day sessions, spaced out over time

• 8-hour blocks of passive PowerPoint

• Minimal interaction (a few questions counted as “discussion”)

• Low perceived relevance to supervisors’ actual work

Completion rates told the story. Although every new supervisor was required to attend within one year, only about 47% actually finished on time.

People stopped showing up, made excuses, and treated it as something to avoid, not something that helped them succeed.

Metrics that Matter vs. Vanity Metrics

Earlier in his career, during the dotcom era, Brantley learned about “vanity metrics”.

These are numbers that look impressive but say little about real impact.

For example, his team counted “communication events”—every email or phone call—as a success measure, without distinguishing between positive and negative interactions.

High counts looked good on paper but said nothing about quality.

He sees the same risk in training and project environments: measuring what’s easy rather than what’s meaningful. When redesigning the supervisor program, his goal was to connect training metrics to strategic outcomes, not just to satisfaction or attendance numbers.

So he asked: what outcomes actually matter for supervisors?

• Staff performance and productivity

• Fewer grievances and HR actions

• Higher employee engagement

• Better alignment with organizational objectives

At USPTO, the answer kept coming back to a single word: productivity. The organization repeatedly reminded everyone, “We are a production environment.” Patent and trademark fees fund the agency, and better supervision should translate into better-examined patents, faster throughput, and more revenue.

Why Traditional Project Metrics Aren’t Enough

Traditional project metrics—like earned value, cost and schedule performance indices—have their place, but Brantley argues they don’t tell the whole story. A project can hit time, budget, and scope targets while leaving behind a burned-out team and no lasting behavior change.

He’s worked on projects that were “successful on paper” but so painful that team members never wanted to work together again. Those hidden costs—low trust, poor morale, high turnover—don’t show up in basic financial metrics but can undermine long-term value.

That’s where the Phillips ROI model comes in, with its structured “V-model” and five levels:

1. Reaction
2. Learning
3. Application
4. Impact
5. ROI

It encourages you to:

• Start with payoff needs and business impact (e.g., increased productivity)

• Define learning and performance objectives that support those outcomes

• Design for positive reaction, so people actually engage with the program

• Measure change in behavior (application) and business results (impact)

• Convert impact into monetary terms and compare to the cost of the program

Crucially, it also asks you to isolate the program’s effects from other factors.

Training is rarely 100% responsible for a result; incentives, leadership attention, hiring quality, and external events (like a pandemic) all play roles.

Brantley often found training might account for perhaps 40–50% of observed improvements, with the rest coming from other changes.

Redesigning the Supervisor Certificate Program

With ROI as his guiding framework, Brantley set out to make the supervisor program more engaging and more clearly tied to goals.

Key changes included:

• Decreasing the PowerPoint load: He removed about half of the slide content; no one missed it.

• Adding interaction: More role plays, discussions, and practical exercises replaced passive lecturing.

• Reordering content: The sequence of courses was reshaped to better match supervisors’ learning and work realities.

• Transforming “HR information day” into a fair: Instead of an eight-hour parade of HR presentations, participants attended a fair-style event with HR offices at booths, games, and prizes.  Brantley embedded a trivia-style quiz to measure what people learned and whether they used that knowledge back on the job.

Measuring Improvement: More Than Smile Sheets

From day one, Brantley knew he would be asked, “How do we know this is better?”

So he built measurement into the redesign.

On the surface, he collected familiar Level 1 data: reaction surveys (“smile sheets”) about satisfaction and perceived relevance. But he looked at them comparatively, across cohorts, and tied them to specific changes.

For example, he saw a roughly 50% increase in how strongly participants agreed that course content was connected to their real work—especially after replacing a strategic management class that supervisors felt didn’t match their responsibilities.

He didn’t stop at reaction.

He also looked at:

• Supervisory performance indicators (e.g., employee engagement scores, grievances, productivity measures)

• Feedback from supervisees and HR about changes in supervision quality

• Patterns across cohorts that highlighted where further improvements were needed

The data showed overall improvements, but also differences between groups—signaling where to adjust design or facilitation. The numbers became a feedback loop, not just a final report.

Beyond Financial ROI: Intangibles and Trust

Phillips’ ROI method encourages attention to intangibles. Things like teamwork, morale, and trust that are hard to quantify but clearly affect performance.

Sometimes Brantley used proxy measures or ideas from books like How to Measure Anything to approximate their value.

He also drew on concepts like Stephen Covey’s “trust tax”: the hidden costs of low trust in the form of extra approvals, duplicated effort, and “cover your back” behaviors.

While not every intangible can be cleanly converted into dollars, acknowledging and tracking them helps explain why some “numbers-driven” projects succeed long-term and others quietly falter.

Using ROI as a Lighthouse

Brantley likes the metaphor of ROI as a lighthouse: set up your strategic objectives and success metrics at the start, and then use them to navigate throughout the project—not just to judge it at the end.

For the supervisor program, that lighthouse was clear: better supervisors should lead to more engaged staff, which should lead to higher productivity and more revenue.

Every design decision and measurement choice was anchored to that chain.

He also notes that in environments like the federal government, strategies can shift when administrations change. If you understand and can articulate how your project supports current strategic priorities—and show that with ROI data—you have a much better chance of protecting or adapting your work when priorities shift.

Communicating with Different Stakeholders

Finally, Brantley emphasizes tailoring ROI messages to different audiences:

• Senior executives want to see alignment with strategy and clear outcomes. Did the project deliver the promised value?

• Finance and oversight groups care about benefits vs. cost and financial justification.

• Customers and business units want to know if their needs were met and performance improved.

• Project teams care about feasibility, workload, and whether the effort leads to meaningful results (and hopefully recognition or rewards).

Using ROI data and stories together lets you tell a richer story: not just “we finished on time and on budget,” but “here’s how this project changed behavior, improved productivity, and supported our mission.”

In the end, Brantley’s message is simple: managing by the numbers doesn’t mean ignoring people or intangibles. It means defining value up front, measuring what matters, and using that insight to design better projects, make smarter decisions, and lead with greater credibility.