Host: Welcome to CET Talks, the International Accreditors for a Continuing Education and Training’s podcast, where we convene thought leaders in the continuing education and training ecosystem to share ideas, research, best practices, and experiences that promote the creation of a world that learns better. Enjoy the episode.

Randy Bowman: Hello, and welcome to CET Talks. I am Randy Bowman, president and CEO of IACET. Joining me today is my co-host, Mike Veny.

Mike Veny: Thank you, Randy. My name is Mike Veny. I’m a certified corporate wellness specialist, global keynote speaker, and the proud CEO of an IACET-accredited provider. I’m so excited about this conversation because we are getting into a question nearly every learning organization has to wrestle with. How do we know whether what we are delivering is actually working?

Randy Bowman: Well, Mike, you’re in for a treat. We have one of the best people in the entire world here today to help us unpack that exact question.

Mike Veny: Awesome! Our guest today is Dr. Patti Phillips, co-founder and chief executive officer of the ROI Institute Inc. Patti has spent her career helping organizations strengthen how they measure and evaluate results. ROI Institute works with organizations in 70 countries, and Patti is the author or co-author of more than, get ready for this number here, 75 books. She has also helped shape professional credentialing and evaluation practices across the entire learning field. Patti, welcome to CET Talks.

Dr. Patti Phillips: Thank you, Mike. And thank you, Randy. It’s a pleasure to be here.

 Randy Bowman: Thank you; we’re so glad to have you. Patti, to get started, let’s start with the question we raised at the top of the episode. We can count attendance, we can count completion, and we can even measure satisfaction. But how do we know whether the learning actually made a difference?

Dr. Patti Phillips: It’s a great question. I guess my question back to you would be made a difference for whom? So, it gets into the definition and the specificity around it, right? What does make a difference mean? Now, measures like “completion” and “attendance”, those are what we call activity measures. They’re the inputs. They’re not the value add, they’re not results; they’re the activity. And unfortunately, we put so much emphasis on those measures, they are reminders of the cost. I always like to say, and I know you guys have heard me say it before, activity represents cost, and costs get cut. When we’re those types of metrics, as a badge of honor, you are reminding people how much you’re spending on learning, and how much work is not getting done. Those are activity measures. Now, when you get to satisfaction measures, categorized as reaction data, it depends on how you’re measuring that satisfaction.

Measures of relevance and usefulness of content, which is a measure of satisfaction or reaction, that can provide some insight into the likelihood people will use it, which then gets meaningful to those supervisors and managers and those people investing in it. You’re sort of leaping, I guess, over to those measures that matter to them. So, I start with “Make a difference to whom?” I mean, if you’re looking to cut budgets, you’ve got the measures—how much time we’re spending, how much money we spent on training—you’ve got the data to tell them, “Here’s what you can cut.” But if you’re looking at if it made a difference to the learner, “Did it inspire them?”, that’s a reaction measure. “Was it relevant?” That’s a reaction measure. “Did they actually learn something new?” That’s a learning measure. “Did they go back and apply it?” That’s application measure; that’s a difference.

But did it make a difference to the supervisor? Well, it depends what they did with what they learned. Did it make a difference to the investor? Depends if whether what they did with what they learned led to improvement in output, quality, cost and time. And more importantly, did they improve those business measures efficiently? And that gets us to ROI.

Mike Veny: One of the things that you have me thinking about right now, Patti, is our podcast as we’re recording and what difference is it making on the listener. I’m thinking through both the lens of the accredited organization that gives training and the podcast co-host. When you encourage organizations to move from activity to results, what should they be looking for? What should Randy and I be looking for in this podcast beyond attendance, completion, and satisfaction? And how do application, organizational impact, and ROI fit into this picture?

Dr. Patti Phillips: Well, what are you trying to achieve from it? I tend to answer your questions with questions. It goes back to what you are trying to achieve from it. Impact and ROI in the ideal world should begin at the beginning, right? What is the problem you’re trying to solve or the opportunity you’re trying to leverage, and what is the value of doing so? If it’s problems resolution, well, what is the problem costing you? If it’s opportunity, what is the value of that opportunity? It really starts there. Then to address that problem and that opportunity, what do you want people to do that they’re not doing now, that if they did, it could help us address the problem or opportunity? And then what is it you want them to learn from the podcast so they can go do what you want them to do to address that problem?

When we look at the value of the podcast, for example, what are people taking away from this that they can then take action on to achieve some end, to achieve some goal? What do you want them to learn? What do you want them to do? And why do you want them to do that or why should they do that?

Mike Veny: Well, obviously that’s a conversation we’re going to have to have offline, but to our listeners out there right now, what are you getting out of this podcast, and what have you been able to implement and put into action? We would love to know, so feel free to share that with us.

Randy Bowman: Patti, you just said, “Impact comes in at the beginning.” That’s such a great design principle—begin with the end in mind. That’s what we always have to start with. What does that look like in practice? We talk about it a lot. People think they do it, and then they get to the end and find out they didn’t do it. What’s the one key takeaway you would tell someone, “This is how you know you’ve started with the end in mind?”

Dr. Patti Phillips: When you’re clear on what the business objectives are, and when you’re clear on the why of what you do. The “why” is not because anyone needs more learning because they don’t. It’s not because anyone needs to do anything different because they don’t. It’s because something else is going on, and there are measures that we are responsible for that tell us it’s going on, and that’s why we do it. Then we change the behaviors or apply new processes so we can achieve that “why”, and we learn what we need to know so we can do those things, so we can address the “why”. It’s really drilling down into the “why”, and I always call it the “so what”, the “so what” of it. Okay, so what if we do that? But we’ve heard the five whys and all of those things are right. It’s drilling down.

Unfortunately, I think we are in a space today where no one wants to take the time to drill down. It’s like, “Let’s go get it; let’s go fix it. It’s out there, just slap it on there.” And it’s like, no. Why do you really want to do this? Why do you want to spend half a million dollars on this initiative and not know why you’re doing it? I mean, to me it’s just logical. Why are you doing this? What are you going to get in the end? So in practice, that’s what it is. It’s really getting clear on the KPIs. Unfortunately, what we often hear from people, from learning professionals and people in the education, training, and L&D space is we don’t have the data. You may not have the data, but you know the people who have the data. Are you having the conversation with those people to understand what their problem is rather than just reacting to the request?

What can you ask them you’re not asking them now to get to the data? We often find that a lot of people don’t understand the business of the business. They understand their business very well. They’re very good at what they do in terms of designing learning and delivering learning, but understanding the business so they can ask those questions of their clients.

Randy Bowman: It’s curious to me that some people need to be curious. They need to be inquisitive and not be content with the first answer. So, part of me wonders, how do you teach that skill to L&D professionals so they get good at asking, “And what do you mean by that?”

Dr. Patti Phillips: Well, step one is allowing them to be curious. I mean, how many times have you heard from colleagues, “Well, I tried to ask, but they cut me off.” We don’t have a culture of curiosity, right? So it’s allowing people to be curious, but it’s the desire to be curious, as well. We’re so engrossed in getting the job done: get it off my desk, let’s move it, let’s design this thing, get it done. Instead of taking a step back and saying, “Why am I getting it done, and how might I get it done better?” I think all of that’s fundamental principles of design work, whatever kind of design work it is. It’s like be curious about it. When we look at problems, before you ever get to designing the solution, you’re looking at your metrics, your KPIs, your client’s KPIs, or whatever.

Your productivity is down, your quality is down, your efficiency is down, your costs are up. All these things are happening, and your customers are no longer satisfied. And we say, “You know what? I went to a program. We’re going to slap this program on there and hope it works.” That’s not curiosity. Curiosity is, “Okay, why are these measures doing this?” And at least picking up the phone and calling someone who might have some insight. We’re often busy and want to get things done and don’t take that time. Culture of curiosity; that’s what we need. Cultures of curiosity.

Randy Bowman: I like that. It’s a great turn of the phrase, culture of curiosity. Because I think everybody understands the value of measuring. You talk to any business professional, “Yes, we need to measure”, but so many still struggle to do it well. I think maybe you hit on a few of the things as we’ve been talking, but why is that? Is it a barrier of time? Is it a barrier of resources, lack of expertise, fear of results, or is it something else?

Dr. Patti Phillips: We hear all of that. Fear of results is a big one we hear. I don’t want anyone to know. Well, what they’re not realizing is that if your programs and processes aren’t working, they probably already know anyway. So when the boss says, “Show me the money,” she or he has it figured out already. It’s like, “I’ve been watching what you’re spending. I know what’s going on and nothing is happening that needs to happen as a result, but I’m going to give you a chance to see.” So we hear that a lot. We hear that difficulty is one of the reasons people give. I call it excuses because something is only difficult if we don’t know how to do it. We hear that it’s too expensive. No, it’s not too expensive. It’s only expensive because you don’t know how, because you’re looking at evaluation and measurement as an afterthought, because you’ve reacted to requests, you’ve thrown something at it, and you didn’t build it into your process.

But if you build it in upfront and look at value creation—not just demonstrating value after the fact—but if you look at true value creation, your measures are there. You know what the measures are. You design for results, building your data collection in, you just go collect a little data. It’s not that expensive to do that. A person could do an ROI study for 5% – 10% of their program cost, and we know that. We’ve done it for decades, and so we know that. It’s not expensive. And then the other thing is time-consuming. It’s too time-consuming only if you don’t know how, only if you don’t build it in. But the big one we hear, Randy, Mike, the big one we hear is “No one is asking.” No one is asking; therefore we’re not going to do it.

Yet there’s research all over. We did research in 2009 with ATD. We went to 96 CEOs of Fortune 500 companies. We got data from them, and they told us, “The number one most important data point to us is impact. Number two is ROI.” Yet they weren’t getting it. Big gap in who was getting that type of data and who wasn’t. In 2025, ATD did the learning evaluation study. 2025, 66% of those people responding to that survey say, “We’ve got to start showing it. ROI is coming. We need to show it.” I don’t quite remember that number, but I think it was only 38% or so who could do it well and only 18% who were showing impact. I mean, it was really low numbers. Now that’s better than it was, but you still have a gap. And one of the reasons that we see it is they’re not being asked to do it, so why do it?

And here’s the problem. We’ve all been there because some of us on this podcast have been around for a while, and things change. Leadership changes. People change their mind. How many times has a leader changed their mind?

And when they do, they go to Mike and they say, “Hey, Mike, show me the ROI.” Well, guess what? That question just made it difficult for Mike. That question just made it costly for Mike, just made it time-consuming for Mike. And more than that, Mike now has something to fear because the program the boss is now asking to see the ROI for probably isn’t delivering value because they already know. So we have a saying around the office, “Hope is not a strategy. Like is not a factor. Doing nothing is not an option. Change is inevitable. Progress is optional.” And we believe that. People change their minds. So why are you waiting to be asked for that? So that’s the number one. We see there’s a lot of reasons that people give for not doing this kind of work, but it doesn’t have to be difficult, costly, time-consuming, and you sure don’t want to wait until somebody asks for it because now the pressure is on.

You bring a consultant and spend all this money on a consultant to do an ROI study when you could have done it, you could have learned how, built the skill, built it in. It’s not a problem.

Mike Veny: Speaking of culture of curiosity, thinking of the person listening to this podcast, the provider, training provider, learning provider, based on what you just said, I love what you said about the “why”, and I’m going to simplify it like this. If I am building courses, let’s say, you’ve got your title, description, learning objectives. In order to be more effective at measuring what learners achieve and giving them impact, do you think it’s important that we make sure we put the “why” into the description more, into the learning objectives, repeat it more? Can you just elaborate on that, how that would look for a provider?

Dr. Patti Phillips: Absolutely. The “why” should be front and center. The “why” begins the development or the communication of the business case for doing it. Here’s why you need to be doing this. So you’re talking about putting the “why “in front of the learner who’s going to the program? Absolutely. Now, do you want to put your target ROI in front of the learner? Not necessarily. I mean, that’s not what they’re there for, but they need to know why they’re there. We are doing this because productivity is down. Guess what? You will get so much buy-in because people know now why they’re there. Where people get unhappy with training is when they don’t know why they’re there. Now they’ll go, they’ll have a great time, and you’ll get positive reaction data, but it’s useless, and there’s no accountability around the provider. There’s no accountability around the learner.

Supervisors aren’t going to support it because they don’t know why people are going either. So yes, I think the “why” should be there. I think you need to be smart about how you position programs and processes and the “why”. You want to make sure you’re communicating it clearly. And you always have to think about what are the consequences of my communication or the unintended consequences, positive and negative. You always have to manage that, but yeah, people need to know why. I think they would value it more if they knew why. They may disagree, but at least they know why they’re there, right?

Mike Veny: Right. Going back to this podcast, would it even help the listener to know why they’re here, why we do what we do for them? So that’s something to think about, but I want to bring up everyone’s favorite topic these days: AI. It’s making it possible to create learning content faster and on a scale we have not seen before. If producing more learning becomes easier through AI and less expensive, does that make demonstrating effectiveness even more important? And what should these learning leaders that are listening pay attention to right now?

Dr. Patti Phillips: Oh my gosh, yes!

Mike Veny: I struck a chord. Yay!

Dr. Patti Phillips: Oh my gosh, yes. There’s already, forget AI, just human beings. There’s so much junk out there now; it’s just nauseating. I was asked to be the subject matter expert for a program a university was designing. It was business analytics program, and then there was another course that they were doing, but it’s this one, business analytics. I’m subject matter expert; I’ve done that kind of work before, it’s not a problem. You provide insight into what they really need to focus on. So they took the objectives, the purpose of the program and the objectives in the outline, threw it in this AI tool, and then turned it over to me to provide my input. I have never been so appalled and mortified at the garbage that they were okay handing off. It was great. I’m looking at this going, “Are you kidding me? This is an institution of higher learning. Are you kidding me?” I could not believe it.

I’m a fixer, which is a big problem around the house because it’s like, okay, I’m going to spend way too much time on something, but it’s like you cannot do this to these adult learners who are in this graduate program. You can’t do this. So I’m digging through it. They had cited as research a blog post from the random Joe’s blog post. They cited as research vendor data. Now, I’m not saying surveys vendors develop are not credible, but there’s a potential bias in it, and sometimes they’re quick and easy. I’m not saying that’s all bad, but that was that. There was no peer review. I think there was a little bit from maybe Deloitte or PWC, which they do great research and there’s a lot of good stuff out there, but still.

What they had produced for adult learners; I was appalled, just appalled. So when I think what AI is doing for us—I’m an AI person. I love it; I think it is amazing, but it is not an excuse for crap. It is not an excuse for poor design. It is not a reason for the individuals responsible for that program to stop using their brain, and that’s where I think we are. People do not want to think. They do not want to use the brain that they have been given, and that brain of ours is so powerful. Now you’re starting to see research where that brain is just shrinking up because Claude can think faster and better for us. I love it; I’m a huge proponent of AI. Yes, you can design programs quick and easy, but we’ve got to assess the value, credibility, and reliability of what AI is doing for us. We cannot give up our brains to AI.

Mike Veny: Some listeners may hear the terms “impact” or “ROI” and immediately think they do not have the staff, budget, or data to do this well. I know when I first heard it from IACET, when I reached out a while ago, I just froze. I was intimidated by it. What does a realistic approach to effectiveness and measurement look like for a smaller association, nonprofit, or continuing education provider?

Dr. Patti Phillips: Another great question. First, I’d like to say, if a listener happens to be nonprofit, something we often hear is that “I’m a nonprofit, so I can’t do ROI.” That is wrong. Yes, you can. That is a misconception of ROI. We are all responsible for the resources bestowed upon us, and as a donor to nonprofits, I resent that comment because yes, you can. You can be more efficient, you can deliver greater value within the organization and outside of the organization, just like for-profit organizations. That being said, for smaller organizations with limited resources, you still do it. It’s just how you go about doing it. The interesting thing about smaller organizations: smaller organizations are often closer to the measures that matter. When you’re designing solutions, you can actually see those metrics, those KPIs. In the larger organizations, a lot of times the learning provider within the organization, the L&D team, they are a little more farther removed from some of the frontline metrics that matter most, but smaller organizations are closer to it.

With smaller organizations, we can often see the impact more clearly and quickly because we’re so focused on where the money goes. But to do the evaluation in a smaller organization or a large organization where the L&D budget is really tight, think about what you’re offering to your people. What are the programs, and which of the programs are really designed to move the needle on the measures that need to be moved, whether it’s donations or whether it’s efficiency or cost savings within the organization. What are those investments you’re making that are truly touching those? Those become the candidates for the higher levels of evaluation. And then the other things that you’re doing is if you need to do something good for your team, do it; just monitor the cost. So you have to think about what are the things that you’re offering your people? It’s the same for large organization.

We say, and this is another data point that’s misinterpreted quite a lot, you can show impact and ROI on any program, any process, for any type of organization. You can do it. The question is should you, given the intent of that program project or initiative? For most organizations, if we follow the 80 / 20 rule, about 20% of what the organization is investing in is really targeting strategy and operational efficiency. They’re the expensive initiatives; they’re really targeting those things. Those are your candidates for that impact ROI analysis. Other programs, you want to see if people are doing something with it. Like your podcast, for example. This is a gift to people. This is something that you all want to do. It’s a gift. What do you want them to do with it? And then stop the evaluation there. You don’t necessarily want to do an ROI study on all this.

You could do it. You could reach out to all of your people and see, but really it’s what, and we’re here for an hour, less than an hour. You want to give them insights that they can take away and take one action on. And that’s where you stop to see if that’s actually happening. So, I think for smaller organizations, look at what you’re investing in. Look at those inputs, those activities that you have. Where are you investing in them and then why? Sort it out and then decide, “Do we really need to build some mechanism in here to start collecting data so we can show ROI?” Sometimes it’s just a matter of changing the questions on the front end because that’s where value’s created—on the front end. The back end is showing it. The front is what are you asking up front? Why are we doing it?

Randy Bowman: So many great insights here today. Patti, we always like to close every CET Talk with one final question. What does a world that learns better look like to you?

Dr. Patti Phillips: A world that learns better to me is a world where learning is accountable, where the suppliers, the developers of learning, those learning providers, are held accountable for what they are delivering. That what they are delivering meets a certain standard, that it is content and learning that is needed and that is designed to move the needle on that need. That’s one. Accountability from a learner perspective. We’ve got to hold learners accountable for learning. And I don’t mean test scores, that’s not what I’m talking about. It’s going into these programs and these projects and these opportunities with clear purpose and intent. What am I going to learn? What am I going to do with it? And why am I going to do it? What problem am I going to solve for the organization or the community or whatever it is? So we need some accountability for the learner perspective, but we also need accountability for the system that supports the learner because you can’t just put someone in learning and hope it sticks.

Now, there are some learners that make it stick because that is who they are, but sticking someone in a program and hoping it just works without any accountability for the system, what a missed opportunity. We need accountability around measurement evaluation. Assessment measurement evaluation is not an activity, but we treat it as one, right? That’s why we have these rote end-of-course evaluations: bad questions, bad surveys, but we do it anyway. We need accountability around the assessment measurement evaluation. Using the data, not to kill programs. Negative ROIs do not mean that a program should go. It means we need to take a closer look at what worked, what didn’t, how can we improve it? If we’re not using the data we’re collecting through our assessments, through our evaluations, then stop doing it because all you have done is create one more activity and one more cost to the organization.

And then the last is accountability around the executives who are just spending the money on training and hope it all sticks. As again, a donor nonprofits, as shareholders in public trade organizations, I resent it when I look at a board presentation and it tells me all the money that’s been spent on development with no results tied to it. You are spending all of this money on your leadership, what are we getting for it? Why are the margins what they are when you’re spending all this money on leadership development? The executives need to be held accountable by their board for the learning investment. We talk about it, and we talk about human capital reporting, and the board wants to see it, but what is it they’re really asking? So accountability across the board. So again, that is the world that I would love to see. If we can do all of that, we can hold everyone accountable for the value that we’re creating and delivering, then we will have a learning that delivers value that matters most.

Randy Bowman: That is so insightful; thank you so much! Thank you for helping us move this conversation beyond simply counting activity and toward understanding what learning achieves. I have written down so many notes for today. Activities are costs and costs get cut. Culture of curiosity and hope is not a strategy. And what I’m going to be taking away from today’s conversation is this understanding of intention matters and set that out at the beginning. Collect your data at the beginning so that you know you’re making an impact. Mike, we’ve talked about measurement lots of times and what accreditation impact it had on changing your mind about measurement. What were your takeaways today?

Mike Veny: So many. One of the big ones is that when it comes to this subject of measuring anything— ROI, weight, your own personal financial situation—it’s uncomfortable. It makes us uncomfortable because it’s honest. It’s really, really honest. I think the big thing I got from all this is that we need to dig deeper to get more honest with ourselves and ask the uncomfortable questions. It’s not going to be pretty, but the end result will be something better for the learner. So, as we wrap up today’s discussion on effectiveness measurement, we would love to hear from you. What is one learning program in your organization where you know what people completed, but you still need to understand what changed?

Randy Bowman: And don’t forget, you can submit topic ideas, suggestions for guests, review our archives, and much more at the CET Talks website, available at CET-talks.org. Subscribe on your favorite podcast platform so you don’t miss any episodes. And thank you so much for joining us today.

Host: You’ve been listening to CET Talks, the official podcast of IACET. Don’t forget to subscribe to the podcast on Spotify, Apple podcasts, or wherever you listen to podcasts. To learn more about IACET, visit iacet.org. That’s I-A-C-E-T.org. Thanks for listening, and we’ll be back soon with a new episode.