Dan Frisk and Paula Randall on bringing innovation to government
Cameron Boozarjomehri (Left), Paula Randall (Center), Dan Frisk (Right)
Interviewer: Cameron Boozarjomehri
Welcome to the latest installment of the Knowledge-Driven Podcast. In this series, Software Systems Engineer Cameron Boozarjomehri interviews technical leaders at MITRE who have made knowledge sharing and collaboration an integral part of their practice.
When you’re a large government organization, it can be hard to be innovative. You have a lot of moving parts and not a lot of time or resources. You can always lean on an innovation organization to help you along the way, but the question remains: “How do you know you’re really innovating?” Fortunately, Paula and Dan are on the case. Listen in as they break down what these organizations do, and how they measure innovation.
Click below to listen to podcast:
| Cameron: | 00:15 | Hello, everyone, and welcome to MITRE’s Knowledge-Driven Podcast. I’m your host, Cameron Boozarjomehri, and I bring you to another great discussion with brilliant minds across MITRE. For this episode, we are speaking with Dan Frisk and Paula Randall. Would you both like to introduce yourselves? |
| Dan: | 00:29 | Hi, my name is Dan. |
| Paula: | 00:30 | And I am Paula. |
| Cameron: | 00:31 | Excellent introduction. |
| Cameron: | 00:32 | Dan, Paula, as I understand, you’ve come to talk to us about something I legitimately did not know exist until you mentioned it to me. And this is going to sound wild at first. We’re going to discuss metrics for government innovation organizations. So, I have the great question of asking, what is an innovation organization? |
| Dan: | 00:50 | That’s a good question and there’s no clean, clear-cut definition of it. We think of them as organizations that take new processes, new ideas…they may just be new to the government and not new in general…and apply them to problems they have to come up with new solutions. |
| Cameron: | 01:06 | And does MITRE qualify as a, I guess, innovation organization? |
| Dan: | 01:09 | Under that definition, I would say MITRE does qualify. However, in the work we did, we intentionally excluded FFRDCs, federally funded research and development centers, and the National Laboratories. Because those, we felt, were traditional, more are research and development-oriented organizations, and the ones we were looking at tend to be newer, smaller, focused on specific aspects of innovation. |
| Cameron: | 01:33 | And so I guess the big question there then is, well, how do they help government? What is an innovation organization driving force that turns into actual tangible results for government? |
| Dan: | 01:43 | That’s, again, a good question. The way we would describe the organizations is they help advance innovation into the government. So they may not be providing the new technique or product themselves, but they help the people or organizations that develop that thing apply it to government problems. Not to… I heard the description, have you ever heard their commercial from, I think it was BASF: “We don’t make the things that you use, we make the things you use better.” |
| Cameron: | 02:13 | Yes, yes. I remember that commercial. |
| Dan: | 02:14 | Yeah. So I would, and Paula may have a different take on this, but I think the innovation organizations that we looked at take new ideas and help facilitate their adoption in government. |
| Paula: | 02:26 | The government innovation organizations help the government at any point in the innovation process be more innovative or apply innovative techniques and tools. They’re not necessarily the innovators, but they’re helping facilitate the process. |
| Cameron: | 02:42 | Yeah. I think this is a story of, I have a really cool idea. You have a really cool organization. My idea can help your organization, but you as the organization, you don’t either have, maybe you don’t have the time, maybe you don’t have the resources to understand how this thing maps into my enterprise. |
| Cameron: | 02:58 | So, an innovation organization comes and says, “All right, well, this thing clearly has some sort of value to you, and we’re going to help you get there. We’re going to say, let’s say you need a new database to help you better track customers or to give healthcare to people and, obviously, we want to meet that goal as quickly as possible. We want to make sure that if you need help, if you need resources, we get you there as soon as we can. But you are a large organization, you have lot to do. So it’s not easy for you to figure out how this is going to plug in.” So, you take the time to reach out to an innovation organization or, as I understand sometimes, they’ll even start their own innovation organizations that are tasked with figuring out how these pieces fit together. |
| Dan: | 03:40 | I think that’s a good summary. And one other thing I would add is that the government innovation organizations, they really have two sets of customers. There’s the end-users that are adopting and using the technology and there’s also the, what we’ve described as the innovators who approach the government organizations. So, they’re the go-between between those two parties. |
| Paula: | 04:02 | There are many stakeholders involved with the government innovation organizations. And you have the end-users, who I think of as the people or organizations who ultimately benefit from whatever service or product came out of the innovation process. |
| Cameron: | 04:21 | So like me, as an individual, I could be the end-user. |
| Paula: | 04:24 | It could be you, it could be a soldier. It could be an organization who applies a new technology and now they can… |
| Dan: | 04:31 | Work better, work faster. |
| Paula: | 04:33 | Faster, exactly. |
| Dan: | 04:33 | Work more connected. |
| Paula: | 04:35 | Yeah. There’s also the, I think of it more as the customer, who is saying that they want something. So that it’s not always the innovator who has a new idea that’s coming to the innovation organization. Sometimes it’s a customer, a PM, something like that, that says, “Hey, I know I have a problem and I don’t know how to solve it. So, how can you help me?” |
| Cameron: | 04:55 | I think from there, the important question then becomes, why are we interested in innovation organizations? What about this work? What about these organizations drew your attention? |
| Paula: | 05:05 | We became interested in the innovation organizations because they’re getting a lot of attention in the news and things like that lately. You hear a lot about AFWERX and Softworks and they’re doing all these cool things. Then, we realized that there were more organizations like this or who wanted to be like that popping up. And we wanted to figure out if they were actually either doing what they said they were going to do, intended to do or if they were facilitating the innovation process, were they making it better, were they making it faster? |
| Cameron: | 05:35 | And I think now that we’ve had this discussion about the setup, I think now we can really get into the nuts and bolts of it. So, what did you guys find? What kind of metrics do we now know that your work was able to uncover? |
| Paula: | 05:46 | We just started with the types of organizations that we found and we kind of piggybacked on some other work that had been done at MITRE. Dan Ward and Pete Modigliani, we know worked in this space before, and had categorized innovation organizations into a few different types. Then when we were surveying the participants, we asked them to identify themselves as one or more of these types. Or they could provide information about if there was a different type that they thought they were, they could write that in. |
| Dan: | 06:18 | The types of organizations we found, there’s seven of them. And we felt that these categories encompass virtually all the activities that innovation organizations would do. And those types are investor, which obviously provides funding; an incubator, which helps early-stage organizations figure out their product or service that they’re going to provide; an accelerator, which expands a product across a larger base of customers. An acquisition facilitator, which is somewhat unique to government, but that helps… |
| Cameron: | 06:50 | I think an acquisition facilitator is the first thing I thought of, and I guess the less jargony version of that would be, the people who help you find the thing you’re looking for. |
| Dan: | 06:58 | Yes. And get on contract. |
| Paula: | 07:00 | Right. So it’s not just helping them find it, it’s helping them get it. |
| Dan: | 07:03 | The other types are networker, which is fairly self-explanatory. That’s just bringing different organizations and individuals from private and public and nonprofit together. We also identified developer organizations, which tend to be an internal team that helps solve a problem that’s by and identified by their organization. Think of it as almost a tiger team to solve a problem. |
| Cameron: | 07:26 | So I’m not familiar with that term, tiger team? |
| Dan: | 07:28 | Neither am I. |
| Dan: | 07:31 | I think of, developers are one of the more difficult ones to explain. But the example I like to give was when in the Apollo 13 movie, where they have a problem on the space capsule, and they dump a bucket of parts in front of the engineers in Houston, I believe, and say, “Please solve this problem. We know these are the materials they have on the spacecraft.” Those are government personnel, maybe with a couple of contractors mixed in, but tasked to come up with an innovative way to solve a defined problem. |
| Dan: | 08:04 | And I believe the last, the seventh type of organization we identified was an educator or advisor, which help promulgate innovative activity and best practices and also provide advice to organizations that are trying to be more innovative. |
| Paula: | 08:20 | In general, they encourage innovation. So, and they would typically span the entire innovation process as opposed to some of these others that might work with just within specific sections. |
| Cameron: | 08:30 | So you’re saying that we have organizations that they’re, like you said, networkers, just trying to get different groups to talk. You have the ones who are trying to help us get what we want, the actual acquisitions people. We have people who are trying to find new work or have people who are trying to make sure that as we incorporate that work, the people who need to know how it works, actually know. And that last one, I believe is the educator. What I want to understand is, so those seven types you guys determined, right? |
| Paula: | 08:57 | Yeah, they’re the categories that we used throughout our research, but we essentially borrowed them from other MITRE work. |
| Cameron: | 09:04 | So from these categorizations, what kind of insights did you glean? |
| Paula: | 09:08 | We asked the participants to give us examples of the types of activities that they do, examples of the metrics that they collect. Then, we essentially mapped those to the categories, to the types, basically to see if they match up. And we found that, as you would expect. So first of all, we found that many of the innovation organizations identify as multiple types. |
| Cameron: | 09:34 | So, there weren’t clean buckets. It’s more like a bunch of Venn diagrams, and you just have to find where that specific dot set on any one cluster. |
| Paula: | 09:42 | Yep. |
| Cameron: | 09:43 | Were there any that found their way into all seven? |
| Dan: | 09:44 | No. |
| Paula: | 09:45 | Yeah, I don’t think so. |
| Cameron: | 09:46 | That’s a shame. I think at that point, once you have all seven, you just become your own government organization. |
| Dan: | 09:50 | We don’t know exactly why they’re in multiple categories, but we have some theories, one of which is that when you’re fairly new as an organization, and you’re not exactly sure what you’re going to do best, that you want to try different activities. So that’s one theory. The second one is that if you can provide kind of a suite of services as an innovation organization, then you can help provide networking, for example, but then also acquisition, facilitation, those kind of naturally dovetail together. |
| Paula: | 10:20 | Right. So some of them do naturally fit together. So, it may not be that you want to provide more services. It’s just that if you are an acquisition facilitator, you probably know a lot of people on both ends, and then you naturally become a networker as well. So… |
| Dan: | 10:39 | In terms of frequency, networker was the most frequent type of organization we encountered in our dataset. There are definitely networking needs, but it’s also, not to be flip about it, but a fairly easy type of organizations that stand up. |
| Paula: | 10:53 | Right. You don’t need a lot of resources, and there’s not demanding activities that you need to perform. It’s essentially connecting different people. |
| Cameron: | 11:03 | So I think an important question to ask is, how big can these organizations be? What’s the, I guess, headcount? |
| Paula: | 11:08 | So, we did collect data on the characteristics of the organizations, and the average size is 19 FTE. But the– |
| Cameron: | 11:19 | And the FTE stands for? |
| Paula: | 11:20 | Full-time equivalent. So those would be the people that are actually working for that organization. And the median was 12. So there were a couple outliers that had large organizations but for the most part, we found that they’re relatively small. |
| Cameron: | 11:34 | You mentioned earlier, these can be stood up by a government agency or they can be somewhere, someone in the private sector or somewhere else building these for a specific need that they see that an agency might benefit from? |
| Dan: | 11:45 | So we did. Some of them are government organizations that have been stood up internally by a parent organization and its government employees. Others are for, excuse me, not-for-profits that their role is largely to support government organizations. |
| Paula: | 12:03 | The majority are government organizations that have been established because their parent organization had a need for them. |
| Dan: | 12:12 | This is the government trying to be agile, exactly the area where you guys were doing your research, agile, connected government. I know a lot of people think government is this big monolithic thing that will take its time getting wherever it goes. But people in government, they really want to work to make things faster. They know that people, their time is money. They know that if you’re a person who needs services, you’re expecting the services to be delivered as soon as possible. And it’s nice to see that they have this outlet, this opportunity by which they can, if they themselves cannot achieve it, they can find someone who can, or they can try and set up a group who can solve these problems or help them get there. |
| Paula: | 12:50 | Yeah, it’s… Yeah, exactly. And I’d say that the government largely knows that their processes are slow and that they have these gaps and they are doing exactly what you said. They’re trying to figure out how they can make things happen more efficiently and faster. And– |
| Cameron: | 13:08 | I think that comes back to the question of the study, which is, as you stand up these organizations, how do you know they’re working in the way you intend them to work? And that ties right to the metrics and how your metrics align with that. The mission of your user base. Maybe we can walk through kind of how you did this metrics evaluation. How did we go from, I showed up day one at the door and said, “All right, this is what you do. Let’s see how well you do it”? |
| Dan: | 13:34 | So, once we developed the thesis and the problem statement, we decided to engage with these organizations using a survey. So we collected information from them with a two-page survey. We tried to keep it short and to the point. And we reached out to these organizations based on existing contacts from MITRE, which helped a lot. Then also organizations that recommended that we reach out to organizations. So in other words, peer-to-peer, our main way of engaging with them was a survey, which had a mixed response because surveys are typically difficult to capture data with because people just don’t have time to do them. |
| Cameron: | 14:14 | So based on those outcomes, based on how you did these surveys, what recommendations would you make? |
| Dan: | 14:19 | So, understanding that collecting good metrics is typically not easy. But we recommend that these organizations try to collect outcome metrics whenever possible. For example, if you have… take a health care organization, a government health care organization. One of their important outcome metrics is patient health. So if you’re an innovation organization supporting that healthcare organization, you want your activities, the results of your activities to show that you are positively impacting patient health as well. |
| Paula: | 14:48 | So we found that many organizations collect, what we call output metrics, and that those are typically things that you can count. |
| Cameron: | 14:57 | How many cookies are we cranking out this year? |
| Paula: | 14:59 | Right. Exactly. But those don’t really tell you what the benefit of this innovation is. And we tried to provide recommendations that we’re focused on the outcome, because that’s hard to do. As these new organizations are being stood up and they’re trying to figure out what they are, we think it’s important that they’re collecting metrics from the beginning, which is, we understand it’s very difficult. So, we wanted to be able to provide at least the first step in that and some ideas and things like that. |
| Cameron: | 15:34 | Yeah. I imagine a common problem you’d see is when you’re a big enough organization, you think that just because you’re able to output enough means that enough people are actually getting their needs taken care of. But in reality, it’s less about how many widgets can we turn out this year and more about how many people are actually able to get those widgets and benefit from the actual application. |
| Paula: | 15:55 | Right, exactly. And we think that a lot of the output metrics or other kind of metrics that are being collected are being pushed down from the parent organization. So, give me an easy metric, probably account metric, that I can say this is what we did with that money, but that’s not really measuring the benefit of the innovation. |
| Cameron: | 16:17 | And I imagine that’s always a big difficulty as if you’re a large organization, you think you know how all these different, very complicated and intricate processes are supposed to yield a final result. But as the innovation organization helping them innovate and also as the one who’s trying to take actions on their behalf, it can be very hard to translate those high-level objectives into the actual on-the-ground results you’re looking for. |
| Paula: | 16:40 | Right. And we often call them the mission outcomes, too. That’s exactly right. |
| Dan: | 16:45 | And that’s not a problem exclusive to innovation. But the innovation organizations we talked to had the added complication that when they pass along a technology or solution to a parent organization, they may not know the outcome right away. So, they need to go back to the parent some point in time in the future, could be months, could be years down the road and say, “Okay, we gave you technology X, you used it. How did that help you?” |
| Paula: | 17:11 | The other thing that makes it difficult for the government organizations compared to industry is that a lot of times, industry is looking at profit. So somehow profit determines their measure of success. That’s not always the case in government. Like Dan said, it’s often backwards-looking. You have to, in the future, that you get set up in the beginning to measure the… So you know what you’re going to measure in the beginning so that you’re prepared to engage with the stakeholders and say, “I would like to come back to you in a year and then collect it, as opposed to just losing that connection.” |
| Cameron: | 17:47 | And so this is actually where I think it’s appropriate to ask the question, why MITRE? |
| Paula: | 17:48 | As we are wrapping up our report and thinking about where we can go, what we can do next with it, I think that MITRE’s in a great position as an FFRDC to help the government figure out how to work in this new innovation space. There are, as we said, we’ve worked with a lot of, there’s a lot of new government innovation organizations. We think that many of them are still trying to figure out exactly what they should be doing or what they want to be doing, kind of what their purpose is. And we think that MITRE is in a perfect position to help. |
| Dan: | 18:23 | And I’ll add to that, the innovation organizations in the government are not exclusive to national security or intelligence. They’re across all departments and agencies we’re seeing, and they seem to be proliferating at a rapid pace. So there’s a need across healthcare, across Veterans Administration, then other areas of government, civilian and national security areas that need help with building and operating these organizations, so they’re as successful as possible. One other thing is, yeah, I think it was important for us to note to these organizations as we talk to them that we were not grading them. We were just looking for information on the current state of metrics within the innovation area, and looking for best practices that we can then pass along to other organizations in this ecosystem. |
| Cameron: | 19:12 | I know we’re kind of out of time. Before we wrap up, is there anything else that you think is important we know about these innovation organizations? |
| Dan: | 19:20 | One other thing we learned was that the innovation community, for government organizations, appears to be fairly small still. Despite the large number of organizations, they do communicate with one another, even across departments and certainly within departments. So it is important that they continue to maintain that communication as they work out their best practices, not just for activities and metrics, but maybe for hiring, for how they engage with vendors and things of that nature. And we’re hoping that MITRE will also stay involved in that community as much as possible to help grow government innovation as a whole. |
| Cameron: | 19:53 | I think that’s an excellent point to go out on. I’d first like to say thank you to MITRE and the Knowledge-Driven Enterprise for making this show possible. And most importantly, thank you to you, Dan and Paula, for the incredible work you’ve done, helping shine a light on the importance of not just having innovation organizations, but making sure they’re doing the work they’re setting out to accomplish. |
| Dan: | 20:11 | Thank you, Cam. |
| Cameron: | 20:12 | Yeah, thank you. |
Cameron Boozarjomehri is a Software Engineer and a member of MITRE’s Privacy Capability. His passion is exploring the applications and implications of emerging technologies and finding new ways to make those technologies accessible to the public.
© 2019 The MITRE Corporation. All rights reserved. Approved for public release. Distribution unlimited. Case number 19-3386.
MITRE’s mission-driven team is dedicated to solving problems for a safer world. Learn more about MITRE.
See also:
Interview with Julie McEwen on why privacy is key
Interview with awais sheikh on deciphering business process innovation
Interview with Jackie Morin on her journey from intern to senior engineer
Interview with Jay Crossler on why passion is the key to success
Interview with Dan Ward, Debra Zides, and Lorna Tedder on streamlining acquisitions
Interview with Dr. Philip Barry on blending AI and education
Interview with Dan Ward, Rachel Gregorio, and Jessica Yu on MITRE’s Innovation Toolkit
Interview with Tammy Freeman on redefining innovation
Interview with Jesse Buonanno on blockchain