Marcie Zaharee and MITRE’s Open Innovation Challenge
Cameron Boozarjomehir (Left), Marcie Zaharee (Right). Photo: Travis Lu
Interviewer: Cameron Boozarjomehri
Welcome to the latest installment of the Knowledge-Driven Podcast. In this series, Software Systems Engineer Cameron Boozarjomehri interviews technical leaders at MITRE who have made knowledge sharing and collaboration an integral part of their practice.
Innovation doesn’t just happen, and it cannot be forced. It takes time, effort, and commitment to find a new path forward. Still, sometimes asking the right question can set us on that path. That is the goal of MITRE’s Open Innovation Challenge. Join us as Marcie Zaharee walks us through what these challenges are, and how MITRE has made this framework available to the public so that anyone can start on the path to inspiring their own innovation.
Click below to listen to podcast:
| Cameron: | 00:13 | Hello everyone, and welcome to MITRE’s Knowledge-Driven Podcast, a show where I, your host, Cameron Boozarjomehri, have the great fortune of interviewing brilliant minds across MITRE. Today I’m joined by Marcie Zaharee, a project lead for the Chief Technology Officer, who’s going to talk to us today about open innovation challenges and their place here at MITRE. |
| Cameron: | 00:32 | Marcie, would you like to introduce yourself? |
| Marcie: | 00:33 | Absolutely. Thanks, Cameron. I’m Marcie Zaharee, I’m the project lead for MITRE’s Innovation Program. I’m responsible for the overall process, infrastructure, information management, and knowledge sharing for MITRE’s internal R&D program. |
| Cameron: | 00:46 | I want to know more, so R&D, all of these points. What does that translate to for your day-to-day? |
| Marcie: | 00:52 | Day-to-day is leading MITRE’s Innovation Program, which is the internal innovation program that we have here at MITRE. And it’s just, it’s managing the overall process and infrastructure, to make sure that our… Any particular potential proposers at MITRE can submit ideas. Working with our innovation area leads, who manage the overall portfolios within our internal innovation program, and just make sure everything runs smoothly. |
| Cameron: | 01:21 | And I can safely say I am a veteran of the program. My own MIP is coming to its conclusion for its first year. And yeah, it’s a very fluid process, where if you’re at MITRE and you think you have a good idea, you go, you pitch that idea, and then Marcie and her team are able to make sure that those ideas end up in a pool, where they can be evaluated and determine what can we do now, and what should we put on the back burner and maybe look up later. |
| Marcie: | 01:48 | Absolutely. You’ve got it. |
| Cameron: | 01:49 | But I think there’s a bigger, more interesting facet of MITRE that you’ve brought to us today, which is this open innovation challenge. And I was hoping maybe you’d like to get into a little bit of what exactly that is and how you got involved with them? |
| Marcie: | 02:02 | Certainly. As being part of the Office of Chief Technology Officer, it does not roll off the tongue, does it? One of my colleagues is Duane Blackburn, and Duane has been leading the PMO team for open challenges at MITRE. |
| Cameron: | 02:16 | And the PMO team stands for? |
| Marcie: | 02:17 | The Program Management Office. |
| Cameron: | 02:19 | Gotcha. |
| Marcie: | 02:20 | Just for challenges at MITRE. And just a little bit of background on that, is five years ago, MITRE made a decision to embrace open innovation as a specific strategy to help some of our sponsor’s critical needs. And MITRE, as a not-for-profit corporation that operates seven federally funded research and development centers for government agencies, MITRE was in an ideal position to host these types of challenges. |
| Marcie: | 02:43 | And challenges in general, I bet you’re going to ask me, what are they? Help bring the world to bear to solve complex problems through open competitions, so that a broader set of ideas and solutions can be identified to our government sponsors. |
| Cameron: | 02:58 | I love how bringing MITRE to bear, seems to be… I know it’s our unofficial motto, but it’s just, it’s so evocative in my mind. I appreciate that the challenge itself is exactly that. As I understand it, it’s this way for a specific problem or technology or solution to really be evaluated in a space where MITRE is saying, “Come to us, come bring these ideas to us.” |
| Marcie: | 03:18 | Absolutely. Because MITRE is not always the smartest person in the room. As a challenge organizer, we want to be able to reach outside of MITRE to individual entrepreneurs, students from universities, experts, small firms, to invite them to submit interesting solutions for challenging problems that satisfy certain criteria within a specific timeframe. |
| Cameron: | 03:43 | Just to be clear, I think a lot of people when they think of a challenge, they might think something more along the lines of a hackathon. |
| Marcie: | 03:46 | That is one type of challenge. There are many types of challenges. |
| Cameron: | 03:49 | Okay. |
| Marcie: | 03:49 | The ones that MITRE usually gets more involved in are more scientific and technology challenges or white papers or potentially demonstrations. |
| Cameron: | 04:00 | And so how did you become involved with this team? |
| Marcie: | 04:01 | I was fortunate enough to work with Duane, and he had asked me to help him with this, so I had the opportunity to shadow him for a year working on our latest challenge, which finished up earlier in the year. In fact, well I’d like to share a little bit, is MITRE’s actually run four challenges. Before I do that though, I’d like to mention that having open innovation challenges is a specific strategic decision that the company made. And now we’re shifting away from that a little bit. |
| Marcie: | 04:26 | But the four original goals in 2011, in terms of why MITRE, why does MITRE do these open innovation challenges? One is to create a variety of solutions for some of our sponsor’s perplexing issues with a lower than typical cost for a single solution project. The second is providing tangible evidence of MITRE’s added value to sponsors. |
| Cameron: | 04:48 | Okay. |
| Marcie: | 04:49 | The third, enhance the recognition of MITRE inside and outside our federal government. And lastly, gain experiences with challenges which will be useful for MITRE and our sponsors, and to document and share those lessons learned, and that’s what I’m here today is to tell you about that and the document that I wrote so that we can share with others. |
| Cameron: | 05:08 | I think you touched on something important there, which is I think something we always ask ourselves here, which is why MITRE? Why specifically is any problem or any challenge ours? Because I know industry, I know government, I know every place where a problem could exist, there are already people trying to chase that solution, but there’s something special about when MITRE comes and brings our own, not unique, but our own specific perspective that can help see things from an angle that maybe those with a specific interest might not be able to see. |
| Marcie: | 05:41 | Yes. Absolutely and I think that’s where MITRE is in a good space, and that since we own and operate seven federally funded research and development centers, we can see what those problems are. Cross-cutting problems more important across our sponsors, so that when we look challenges, we are very specific in terms of what type of challenges that could benefit a lot of our sponsors, not just one. |
| Marcie: | 06:05 | And I can tell you about the four that we’ve done, and the first one, the Multicultural Name Matching Challenge, and the background here was that this was one, MITRE’s first external competition, and it was actually inspired and modeled after the Netflix Prize. |
| Cameron: | 06:19 | The Netflix Prize? |
| Marcie: | 06:22 | Yes, in which participants were asked to improve the recommender movie algorithm, if you remember that? Netflix gave participants test datasets which they ran through their algorithms and submitted to Netflix for scoring. If you can think about that way, is that we focused on evaluating matching systems for multicultural names. |
| Cameron: | 06:41 | And just to be clear, the idea there is that you have people who have… This is specifically like people’s proper names? Or is there… |
| Marcie: | 06:49 | Yes, yes. |
| Cameron: | 06:49 | Was there anything especially that you’d like to share with us that came out of that one? |
| Marcie: | 06:52 | Keith Miller here at MITRE has actually written a couple of papers on that, and he’s a great resource for a follow-up. |
| Cameron: | 06:58 | Excellent. From there, what other opportunities have we had so far? |
| Marcie: | 07:02 | The next one is the Countering Unauthorized Unmanned Aircraft Systems Challenge. This one was in response to our government sponsors needs to assess the capabilities of commercial unauthorized unmanned aircraft systems, small UASs, so you’re talking five pounds or less. |
| Cameron: | 07:18 | This challenge is saying: we have a problem; quad rotors are a thing now, and we need to make sure that, first of all, we know where they are, because they can carry cameras or they can carry all sorts of things on them. |
| Marcie: | 07:27 | Absolutely. |
| Cameron: | 07:28 | Some of them we don’t even want to think about. And then we have the second problem, which is not only do we have to identify them, we need a way to take something out of the sky without hitting whoever’s underneath it. |
| Marcie: | 07:37 | You got it. Absolutely. To get them without collateral damage. We wanted to be able to detect them during flight and determine which were threats based on a geographic location and flight trajectory. And then secondly, as I mentioned, interdict them, the ones that were perceived of threats, and recovering them in a safe area without damage. |
| Cameron: | 07:56 | And that could be anything from taking control of them and forcing them down programmatically, to maybe some sort of catch and release system that would be able to disable them while they’re airborne, but not make them drop out of the sky. |
| Marcie: | 08:07 | Exactly. And our two leads there, one of which you’ve already interviewed, Michael Balazs and Jonathan Rotner, they were the technical leads for that. And it was actually an actual physical test fly off in Quantico, where we partnered with our government sponsor. |
| Cameron: | 08:21 | This is just the second challenge. What else do you have for us? |
| Marcie: | 08:23 | I’ve got two more for you. Another one was the unique identification of the internet of things, IoT. |
| Cameron: | 08:29 | We’re talking smart light bulbs, smart plugs, Amazon Alexa’s, all sorts of doodads that don’t have screens, but do have internet connectivity for reasons that perplex me. |
| Marcie: | 08:38 | Absolutely. You got it. And we wanted to look at that because it’s important to our sponsors. With this explosive growth of the IoT devices, as you mentioned, their highly diverse and potentially modifiable characteristics, very hard to accurately identify what devices either rogue or planned are on various networks, and to determine when a device joins, leaves or is replaced. |
| Cameron: | 09:01 | If I understand this correctly, a big problem here is the fact that because these devices have so few screens and interfaces, it’s actually hard to tell how they’re behaving, and when they’re connected to what network. This is a sensitive network, we obviously don’t want Amazon Alexa’s on it because god knows what they could be able to record or share. And then vice versa, there is opportunities for hackers to exploit these devices and use it to gain access to the larger network. |
| Marcie: | 09:26 | You got it. In our challenge, we sought to find a simple affordable solution that can fingerprint devices within an IoT network of interconnected devices. |
| Cameron: | 09:36 | These problems you mentioned, you’ve only gotten through the first three, very real problems for everyone. It doesn’t matter if you are government or a civilian or an enterprise, you have to deal with IoT, you have to deal with people using drones. You might have to deal with all sorts of things that they feel trivial on their face, but when you take even a moment to appreciate all the implications, all the little unintended consequences that come from any solution, it shows the breadth of knowledge needed to achieve something meaningful and lasting. |
| Marcie: | 10:05 | Yes, so we have MITRE subject matter experts in this area, and as I mentioned earlier, picking these topics. We just don’t pick them overnight. It takes a long time to look into our whole sponsor base to see who could benefit the most with a particular problem set. |
| Cameron: | 10:21 | All right, and this last one you have for us? |
| Marcie: | 10:23 | This last one, and this is the one that I was able to sit in on both as a PMO participant as well as part of the project team, is strengthening eligibility verification for federal benefit programs. You can imagine all of the payments that are made to people that should not be having them. |
| Marcie: | 10:42 | In 2019, this year we executed the Eligibility Verification Challenge to motivate creative inventors to find groundbreaking approaches to solving the problem of improper enrollment and payments in the federal benefit programs. |
| Cameron: | 10:57 | This is I think one of the most sensitive because there’s so many different ways that people have been approaching this problem. I know at some point someone somewhere listening to this probably thought blockchain. The long and short of it is you don’t need to go to the newest technology right away, sometimes there is an existing technology that makes it easier to make sure that the right person gets the right money. And I imagine that was a big part of what made this research very, not just interesting to pursue, but also necessary. |
| Cameron: | 11:42 | As I understand this, unfortunately MITRE is discontinuing the challenges. Not in that they don’t want to do them anymore, but in that… When we always ask ourself “why MITRE”, we’re slowly appreciating that we got some very valuable insights, but it may be someone else’s time to take the mantle. |
| Marcie: | 11:42 | Absolutely. And these challenges have been around for a while now and people are starting to figure out how to manage and execute them. And at some point, every organization has to go back and look at their strategy, and the strategic direction that they’re going. And we recently did that. And as I mentioned, I had the opportunity to shadow Duane Blackburn, the lead for MITRE’s challenge PMO. And he told me when we started this program, we were a much different organization. We were very focused on internally performing directly sponsored activities and not really considering how to leverage the nation’s non-federal innovation ecosystem. |
| Marcie: | 12:19 | And now that’s not the case anymore. And Duane believes that the MITRE Challenge Program played a significant role in transforming your corporate thinking and actually opened up some newer opportunities, where MITRE serves as a bridge between non-government innovation and government needs. And I also want to mention that we have this… Our government sponsors now have Challenge.gov, that helps them with their challenges. Always having to relook what is the best role for MITRE. |
| Cameron: | 12:47 | And this is a pretty invested group effort on MITRE’s part. It was you, Duane, and who else was on the team? |
| Marcie: | 12:54 | For the MITRE Challenge PMO, as you can imagine, risk management would be really important one, and also legal for IP. That is a huge challenge that organizations who want to manage and execute challenge really need to think about is IP. And lastly, I don’t want to forget finance. And another important role in this as part of the PMO office is corporate communications. So important to consistently communicate with potential participants. |
| Cameron: | 13:25 | This was a pretty invested effort. This is countless people across countless parts of the organization working together to not just put on these challenges, but also make sure that their findings are meaningfully received by sponsors and anyone else who’s interested? |
| Marcie: | 13:39 | Yes. And then not only do you have the PMO team that helps from a strategic perspective, you actually have the project teams. Each one of those challenges I talked to you about also had its own project team to be able to manage and execute the day-to-day activities in the challenge. |
| Cameron: | 13:54 | From my understanding, this goes beyond just all the teams we mentioned because you also took the time to… As you were shadowing, as you were becoming involved, start putting together a document, a living document in some ways, that is how can someone outside of MITRE benefit from these challenges and make sure that challenges aren’t something that just lives and dies with when MITRE wants to do them. |
| Marcie: | 14:18 | Absolutely. And that’s what I’m here to talk about. And a fun part of my job is being able to sit in on things like this, and be the knowledge management person and capture what’s going on. I had the opportunity to not only shadow the PMO office, but to sit in on this last challenge as part of the project team and listen in on what are the steps of the challenges, what are those key questions that they were asking themselves along the way. And as a result of this year long effort, I was able to put together a document entitled Managing Open Innovation Challenges: Key Questions for Successes. |
| Marcie: | 14:57 | As a part of the document I explained the four MITRE challenges that I talked to you about. I also have a diagram in of a logic model, challenge framework for challenges, that you need to think through from beginning to end. |
| Cameron: | 15:11 | I think the most important part of the document is you’re not really telling people how to run a challenge, you’re telling people how to ask questions. |
| Marcie: | 15:18 | Absolutely, so for each one of those sections of the framework, it’s broken down into phases and steps and what are the key questions you need to think about, and then the suggested guidance. As part of it there are five key lessons learned. The key lessons came from interviews I did with the PMO, as well as some of the other project teams. |
| Cameron: | 15:38 | And they are? |
| Marcie: | 15:38 | As Stephen Covey would say, “Begin with the end in mind.” You see so many organizations that say, “Oh, this is so cool. I want to run a challenge,” but not really thinking through, well what does that mean? And what are you going to get at the end? And there’s really two things to think about in terms of what the outcome is. |
| Marcie: | 15:55 | One is you’re either going to get ideas, technologies, or products, or you’re going to come up with wanting to make a change in the community. Community engagement, raising awareness, mobilizing action, or inspiring transformation. Think about that first, what do you want to get out of it? |
| Marcie: | 16:12 | The second one is, as we’ve talked about many times, having that dedicated PMO, really important to have that subject matter expertise from a strategic perspective, the communications, the legal, the finance and actually managing. The third is risk management, really important. There’s risk about executing a challenge, but then there’s the day-to-day risk of once you get in and you just start dealing with participants, having a risk management plan to mitigate things. |
| Cameron: | 16:39 | And this is kind of a 50/50, you want to give them enough information about what you’re trying to solve, so that they can help you, but not so much that they can walk away with some sensitive piece of information about your organization? |
| Marcie: | 16:51 | Correct. And the last two, one is managing participating teams well, and that is having a consistent communication plan so that you’re telling all of your participants the same information at the same time. |
| Marcie: | 17:03 | When I interviewed some of the project team members for the past challenges, one of the big concerns was they wanted to make sure that we were fair and MITRE was perceived as being fair. Something really important to think about in terms of how you disseminate information. |
| Marcie: | 17:18 | And lastly, as you hit upon a little bit ago is the IP. In NASA’s case study on grand challenges, one of their quotes was, “Involve someone from legal early and keep them on the team.” Because one of the big questions you’re going to have to answer is what are you going to do with intellectual property? And it’s just really important to know that upfront, so that both the organization knows as well as the participants, who’s going to own that IP. |
| Cameron: | 17:45 | Thank you for this incredible insight into MITRE’s Innovation Challenges. And if people want to find this incredible work, where can they go? |
| Marcie: | 17:51 | They’d want to Google, MITRE.org, and look under our publication’s tab. |
| Cameron: | 17:55 | Thank you. |
| Marcie: | 17:56 | Thank you, Cameron, for the time. |
| Cameron: | 17:57 | This has been my conversation with Marcie Zaharee, on MITRE’s Open Innovation Challenges. |
Cameron Boozarjomehri is a Software Engineer and a member of MITRE’s Privacy Capability. His passion is exploring the applications and implications of emerging technologies and finding new ways to make those technologies accessible to the public.
© 2019 The MITRE Corporation. All rights reserved. Approved for public release. Distribution unlimited. Case number 19-3664
MITRE’s mission-driven team is dedicated to solving problems for a safer world. Learn more about MITRE.
See also:
Dan Frisk and Paula Randall on bringing innovation to government
Interview with Awais Sheikh on Deciphering Business Process Innovation
Interview with Jackie Morin on her journey from intern to senior engineer
Interview with Jay Crossler on why passion is the key to success
Interview with Dan Ward, Debra Zides, and Lorna Tedder on streamlining acquisitions
Interview with Dr. Philip Barry on blending AI and education
Interview with Dan Ward, Rachel Gregorio, and Jessica Yu on MITRE’s Innovation Toolkit
Interview with Tammy Freeman on Redefining Innovation
Interview with Jesse Buonanno on Blockchain
Interview with Dr. Michael Balazs on Generation AI Nexus
Interview with Dr. Sanith Wijesinghe on Agile Connected Government