Authors: Marilyn Kupetz and Angela O’Hanlon

MITRE’s Analysis Tool Shed Lab (ATS) program is one of MITRE’s largest R&D Labs, focusing on analysis capability and analytic activity. Lab users acquire hands-on experience with 100+ tools and datasets, MITRE prototypes, and government-off-the-shelf tools. Lab activity includes integration and development work. Through strong partnerships, the ATS Lab keeps pace with industry while focusing on the needs and requirements of MITRE’s sponsors. The ATS initiative was originally established as a wiki initiative by a community of MITRE analysts. The ATS wiki includes a mature collection that serves as a repository for MITRE analysts, developers, and staff – staff document their experience with analytic tools. The ATS Lab makes it possible for analysts, developers, and technologists—MITRE staff and sponsors alike—to do more than share war stories, they can gain access to analytic tools of interest. The ATS Lab is virtualized and easily accessible to staff over MITRE’s corporate intranet.—Editor

Angela O’Hanlon, the director of the Lab, spoke with Marilyn Kupetz about the Lab’s successes and activities.

MK: I remember when the ATS wiki first came online. The initiative has traveled a very long way. You’ve moved the original idea past a simple set of linked wiki pages to a full-service environment with a staff, newsletter, trackable metrics, an industry roundtable, and direct sponsor support. How did you lead the Lab to this point? What has enabled the success?

AO: Clear vision! We knew that we needed to build a Lab and analytic capability for MITRE and MITRE’s sponsors—and we’ve been lucky to have the full support of MITRE leadership. As we got the Lab off the ground and began seriously working with our industry partners, we listened. A lot! We listened to our user base and sponsors and industry. We listened to MITRE senior leaders, who offered guidance and lessons learned, and best practices that they wanted to see in MITRE Labs. We knew that we should be building and scoping the capability (the Lab) to meet the needs of our user base, or we would miss the mark in enabling, serving, and supporting missions. The ATS Lab team is highly motivated and responds rapidly to what users ask for. We try to emphasize customer service, which means good response time, a good Lab experience, and good service. I love working with industry, and have a great peer group that’s developed over many years. People move around, of course, but the same group of folks still tend to gather around the table. It feels good to get so much done together. I have had other positions that included outreach, engagement, partnerships. I really enjoy this type of work.

MK: What are your criteria for hosting a tool in the Lab? Do you ever turn things off?

AO: We host analysis tools, data, and big data analytic tools in the Lab. Analytic tools are now leveraging big data analytic technology. This is a good thing. In terms of our customer base, we have a good mix of defense, intel, and civilian activity in the Lab. Our customer base is pretty evenly distributed across MITRE, and this is great. We have many different projects and sponsor organizations leveraging the Lab : VA, Social Security, Army, Navy, USMC Intel, and so on. The tools that we host in the Lab and our new installations, too, are guided by our Lab user base, and specific sponsor requests to evaluate a tool. Our hosting is also guided and recommended by industry— our industry partners may send beta tools for MITRE to test and evaluate. Cloudera, MongoDB, IBM, BASIS Technologies, for example, all provided beta tools in 2016. We ran formal evaluations for some vendors, and provided ad-hoc and casual feedback to others. We also host a handful of legacy tools, as some sponsors are still using these older tools. It’s important to have these (legacy) resources available to our staff, as well as cutting edge tools, beta tools. That said, we do retire tools that are not being used at all. Resources are then recycled.

MK: About how many staffers take advantage of the Lab? And for what? Solving problems for sponsors?

AO: We’ve had several hundred staffers and well over one hundred sponsors taking advantage of the Lab. Projects are usually supporting a sponsor’s R&D evaluation, prototype work, or light weight testing. Or staff may be working on internal MITRE research projects.

MK: Can you describe the Lab’s biggest success to date?

AO: Our biggest successes are helping sponsors with analytic solutions – and also connecting and engaging sponsors with each other, in order to leverage each other’s analytic solutions, lessons learned, products, methods, approaches, architectures, best practices, etc. Another success for ATS is resource sharing —the Lab model is so appealing because it saves resources for sponsors, industry, and MITRE. Vendors and government benefit greatly from a shared-resource model. Vendors have high-visibility in a Lab such as ATS, with generally low overhead or investment costs. That being said, our real interest is in long-term relationships and engagement with our industry partners.

MK: One of the ways that you share information within the company is via newsletter—good KM practice, given the wealth of metrics you share and the opportunities you publicize. I saw in the most recent issue that you’re now hosting tools on Amazon. Is this to enable easier access for MITRE’s sponsors? What kind of traffic is the site getting? Feedback from sponsors?

AO: We are installing SAS now—it will be the first collection of ATS Lab tools available on AWS (Amazon Web Services). We very much want to expand the options we offer our user base. Some sponsors are transitioning some of their operations to the cloud, but not all operations. So it’s important to understand what works, how things work, and jump in! ATS Lab should offer solutions on new architectures so that our sponsors can explore cloud services and analytic tools available to them. We’ll be installing roughly two dozen analytic SAS tools on AWS, and will make them available to users and then mirror that image, and make it available on our regular servers. This is also SAS’s first foray into this type of image release and work on AWS—it’s a learning experience for both companies.

MK: I’ve asked you a lot of detail-oriented questions, but I’m also interested in broad cultural changes. Has the Lab had an effect on MITRE’s focus on analytics? Does the centralized access make it easier to learn and share? Solve real problems?

AO: Yes, let me build on what I said earlier about shared access. In addition to making Lab resources available, we are also building an analytic capability. Project leads ask if we (ATS Lab) can help with a product evaluation—but then also want us to build and author an analytic product and evaluation on link analysis products. Social Security, for example, asked us to help with an evaluation in 2015. We are constantly working with MITRE’s technical teams to build our bench, leveraging MITRE analysts, technologists, developers, and SMEs, to fill analytic requests to the Lab. The Lab is a good place for training, too—staff can use our tools to learn a new skill. And we can even ask for more additional gratis training for our staff, from most vendors. This is good for customers, of course, and good for the company at large in terms of collaboration, learning, and developing awareness of tools and how to use them. I mentioned previously the great support we have from leadership. This is one of the reasons we collect such good metrics—leadership wants to show the value and impact of MITRE Labs. In addition to metrics (which are not always easy to collect!), we also survey users as they move into the Lab. And we require exit surveys, to collect more feedback. There are some things that are really hard to measure. It’s hard to measure Lab impact to a program, for example, but we know we need Labs at places like MITRE, and that they are critical to our work. ATS has a deliberate marketing campaign; we advertise and promote Lab resources across the company. It is part of our responsibility to educate MITRE staff on the R&D software and resources available for them to use in our Lab. We must share this good news with staff, and MITRE sponsors. We use the company intranet to do everything from announcing new R&D agreements, new tool installations in the Lab. We run vendor banners on the corporate webpage, host vendor demos, host wiki pages, and we send out newsletters across the corporation, as you noticed. We also host industry events centered on analysis tools and analytics, which include our sponsors, and industry partners. And, most recently, we kicked off an Intelligence Analysis Tools Industry Roundtable, with a dozen of our industry partners.

MK: How many staff work with you to run the Lab? What does the company expect of your team?

AO: Ten to twelve people work on the ATS Lab team. We have dedicated support from MITRE Legal and Contracts; several staff that manage and administer our systems, software and data; a customer service and engagement group; and a few developers that are essential to Lab health and performance. This does not include the analysts, technologists, and SMEs that help with products. The whole ATS Lab team is essential to our health and performance; each layer that we’ve developed is important.

MK: This is a blog, so now’s your chance to ask the world a question. What’ll it be, Angela?

AO: I’d love to hear how other companies build similar capabilities. We are as thorough as we can be, but I’ll bet there are things we could learn. Please send us your own practices and questions—I would love to engage!

© 2016 The MITRE Corporation. All rights reserved. Approved for public release; Distribution unlimited.