Axon - Training the Next Generation of Pilots - Ep 1 Published Jan. 31, 2023 Transcript for EVTOL Podcast Dr. Megan Hennessey Good morning. Welcome to the Air University Teaching and Learning Center podcast. It's my pleasure this morning to speak with Colonel Don “Stryker” Haley and Dr. Steve Ellis. Colonel Don “Stryker” Haley is a career F15E pilot and military strategist. In his current role as Commander ATC Detachment 62, he supports AFWerx Agility Prime by accelerating the commercial and military adoption of electric vehicle, excuse me, electric vertical takeoff and landing vehicles, EVTOL, that's a mouthful, through curriculum development and operational training. Welcome “Stryker” and we also have Dr. Steve Ellis, a 25-year Air Force veteran, with over 40 years of experience as an instructor, curriculum developer training program evaluator, and learning systems innovator. He is a scholar practitioner, serving as a learning coordinator for the AFWerx Agility Prime program, helping bring electric aviation to the forefront, and I am Dr. Megan Hennessey, the Director of the Teaching and Learning Center. Welcome to you both. Thank you for joining me this morning. Col Don "Stryker" Haley It's good to be here. Dr. Hennessey. Thank you for having us and Merry Christmas, Happy New Year, Happy Holidays and all the things in between, as we're in the middle of our holiday season. Dr. Megan Hennessey Thanks Stryker. Dr. Steve Ellis Yes, thanks for inviting us to this. This is a a great opportunity for our small detachment. Speaker 1 Thank you, yes, I'm so excited to dive into what Detachment 62 does and how it's connected to teaching and learning. So please describe your current project and why it's important from a national security perspective. Col Don "Stryker" Haley You bet. I'll start off there, Megan, so I think it may be helpful to start out with kind of a broader perspective. So, we, our team, works directly with AFWerx. So, AF Works is the innovation arm of the Air Force. And the AFWerx mission is to connect, basically your innovative technology developers with airmen and guardians. And then, we work with a program called Agility Prime. Agility Prime is a program under AFWerx that's focused on this EVTOL industry. Electric vertical takeoff and landing, a lot of people call EVTOL. It's been referred to, colloquially, as the Third Revolution in aerospace. If you think of the Wright brothers, to the Jet age, to the electrification of aviation, it's kind of, it's a really exciting time in aerospace, and specifically working with these vehicles that are being developed. Electric vertical takeoff and then you said that's it's a mouthful. There's an Anderson Cooper 60 Minutes special, and he actually talked about how it's kind of an awkward term, but something an awkward term to describe a really exciting technology. And you know what we're looking at here is as this, as we look at what EVTOL can be. A lot of people have projected that this is going to be a multi trillion-dollar industry. More specifically, I think the big AHA is electrification is deemed to be something as the democratization of air travel. So, hopefully making it a lot more affordable to do shorter, you know, regional kind of operations. It's going to change cargo operations. It's going to make your Amazon packages show up even more in time for Christmas. So, it's a really exciting technology. The project that we're doing is specifically looking at it around learning and because EVTOLs are being designed with distributed electric propulsion, and they're being designed with all new technologies and composites. You're kind of building a little bit off what happened in the drone industry. Small UAVs, you know, a lot of these vehicles are going to be a lot more simple from an operator perspective to operate than your traditional aircraft like your Cessnas and things like that. And the other thing that's different is that they're not airplanes and they're not helicopters, they're kind of a combination of the two. So, when you take these trends of, you know, technology, making vehicles more simple to operate and more accessible to the average public. When you take the fact that it's not a helicopter, but it's not an airplane either. So, you kind of get some very interesting things from a learning perspective. So, our project has been focused on trying to understand how learnable these aircraft are. We're looking at some of the leading manufacturers. We want to try to understand. You know, compared to what our history has been in learning how to fly helicopters and airplanes. You know, what do we imagine the future looking like with this new feature? So, the experiment has all been around trying to understand what that looks like and that obviously is going to form what kind of policy we have, what kind of rule-making the FAA does and other international bodies, and also, how we train future operators of these aircraft. Dr. Megan Hennessey Thank you so much. Yes, I can speak from experience, when I was visiting Steve on site there in San Antonio a few months back, I got to try one of the simulators and it is incredibly difficult. So, I'm glad to hear that you're putting significant time and thought into the best way for people to learn how to do this. Steve, I'm wondering if you could go into a little bit more on the research design. Stryker mentioned that you're doing a study around the learnability behind EVTOL. Tell us more about that. Dr. Steve Ellis OK, we didn't actually start out with a research plan in mind when we stood up the debt. It was really to bring to train the first generation of EVTOL pilots and what we quickly realized is there were a lot of claims and assumptions that these would be easier to learn than traditional aircraft, but we didn't have any data to back that up and some of the estimates were wildly varying. So, we decided that since nobody else has studied it, we would, and we would research this in an operational environment. So, it’s OPS research, it’s conducted in a hangar, not in a lab, with our best available resources and the people that we have available. So, what we built was a quasi-experimental 2 by 2 matrix study where we're looking at pilots and non-pilots in two different vehicles, one highly automated and one less automated vehicle, because the vehicles have various levels of autonomy in this fly by wire design. Our goal is to capture as much data as possible. So, we are using a a research model that collects data from instructor pilots and the simulators. We have some psychometric tests. We have some knowledge tests and surveys. Overall, with 80 subjects, we have somewhere north of 12 million data points. Speaker 1 That is insane and amazing. As a researcher, I'm jealous. Dr. Steve Ellis Well thanks. Col Don "Stryker" Haley I hang out with all these PHD's that get really excited about data points and I have a hard time completely comprehending the excitement, but it's been explained to me by our statisticians and now I have an appreciation for it, at least. Dr. Steve Ellis But, uh, yeah, it's a rich data set. Speaker 1 Yes. Steve, did you want to say anything more about the research design? Dr. Steve Ellis UM? It has turned out to be a very robust research design. Like I said, 12 million data points gathered. It afforded us a lot of opportunities to look at some of the nuances and the data that we've collected. So, we've got objective data, we've got subjective data, and we've got quantitative data and qualitative data. So now we're living in a data rich environment. Speaker 1 Yes, I think that's putting it lightly. So, what do you do with the data and who's looking at it? Who's helping you make sense of it? How do you make sense of it? Tell us a little bit about the analysis phase. Dr. Steve Ellis The analysis phase has been quite interesting. We did partner with a research company that helped us with a lot of the data analysis and they're helping us pour through all this data and they had some very useful data collection tools that helped us to standardize this research. And we're also partnering with Embry-Riddle. Right now, we just got a a partnership started with them. We're also going to be working with, I think, Florida Institute of Technology to let some of their researchers and grad students have access to this data. The data analysis is ongoing. We've done some to begin with, and when you have that much data, it's kind of like eating an elephant, one bite at a time. Col Don "Stryker" Haley Yeah, I think we started with, you know, with all the data and you're like, you know how do you direct analysis as the commander? I've tried to take the handle there as best as I can, with a lot of help of course, but I've started with what I would consider to be operationally relevant questions that I think we can answer. Questions that are going to matter for our audience, I think, are the manufacturers. What feedback can we give to them as they're designing these aircraft, also, policymakers. We're interested in influencing policy and helping folks in the FAA, and larger international bodies, understand how we're going to regulate the industry. We've asked questions. For instance, like we asked, how do automation features in EVTOL and specifically when you look at distributed electric propulsion? How does that impact learning curves and operate your proficiency? And then, we just looked at specifically, because it's not a helicopter because it's not an airplane, we wanted to understand how does aviation experience impact your learning curve? So, if you're a rotorcraft pilot, how quickly do you adapt to this new type of aircraft? If you're a fixed wing pilot, how well do you adapt? Especially when you get into the takeoff and landing phases, which are more vertical oriented. We teamed up with the test pilot school as well and we looked at things from a human factors point of view. We looked at psychomotor resource availability and how does that affect performance? You know what parts of your psychomotor capabilities are being taxed the most and what does that mean for design? We can kind of give some human factor feedback and so with this data set, we go with those questions and then we run the analysis to try to answer those specific questions. We have a list of probably about 15 or 20 different questions and we've worked through about half of them at this point where we've come up with some reasonably good answers from the analysis that we have. And then we try to find environments where we can then brief them. For instance, we were just at IITSEC and we took a subset of those questions and answered those, submitted a paper, and then we were able to present on it. So, we're slowly going through the analysis and finding places where we can share what we're learning with the larger crowd of interest in EVTOLs. Speaker 1 Yes, and count the TLC in, in that interest. Can you share any initial findings? Col Don "Stryker" Haley You bet. We’ve got a couple of things. I'm pulling up some stuff that we hear. So, we first started with this. Some of the questions that I was asking, so we wanted to try to understand automation features. A couple of the different aircrafts that we looked at, they’re representative of different, I would call it, maturity levels and production because different companies are going after different markets, and they've been around for different amounts of time. So, you have differing levels of maturity and prototypes that we're working with and with the Agility Prime program we are able to, kind of compare apples and oranges and learn from them. So, for instance, one of the manufacturers we're working with, they're at this stage of development where they haven't input a lot of automation features into their control walls yet, so it's pretty much like it's still fly by wire, but it's pretty much the operator is directly, more or less, controlling the aircraft. There's very little automated hover features or automated altitude hold or anything like that. So, what we're able to do is look at that aircraft, which, I think, would be representative of distributed electric propulsion. When you have multiple motors, you're going to, you can't individually control 6 engines, like, you would maybe in a multi-engine or a two engine Cessna type aircraft. So, there's going to have to be control laws around that, just like when you have a drone. I can't control every little electric motor. So, I would say, that's the most simple, the basic starting level. Then we have a manufacturer who's been around a little bit longer than some of the others and has a lot more automation features, so we can compare. And what I think we learned is just distributed electric propulsion. Just the fact that you're going to have to have some level of augmentation and just kind of your entry level, very little automation. We find that it is a very learnable aircraft that folks that don't fly helicopters can, in our experiment, just in a couple of hours, they're able to gain high levels of proficiency and learn very difficult tasks like takeoff and landing, so that's really interesting. And of course, as you might expect, as you add automation features in, that as you automate your altitude and airspeed control, and as you automate your hover functions, that the learning, the learning curves are the same, but your baseline is a lot better and what that means is you get more consistent performance. Even people who don't have, we tested their psychomotor capabilities. So, people that maybe don't have as good quote-unquote hands like, there are some folks that maybe have better pilot hands, if you will, those folks with that, the more automation features. The automation features help folks that maybe aren't quite as or don't have the same eye hand coordination. And so, you get a better, consistent level of performance. What that means ultimately, and for the industry, is it's going to be safer to operate. You won't have to train as much, or the currency requirements will be such that even people that don't fly it very often, they will be able to consistently perform. And so, we're able to show you know what feature sets, when you put these automation features together, it increases safety, increases and and improves overall performance, and so we're able to measure that and have real data to show all that which is really exciting. Speaker 1 That is really incredible. Congratulations for everything you and your team have done thus far. You know in AETC or Air Education and Training Command, and here at Air University, we are always talking about force production. How does your project relate to that? Dr. Steve Ellis So, one of the things that we did. In our study, is we included people with no piloting experience. And this is a long way to get to force production. Our initial assumption was anybody who goes into one of these vehicles is going to be a highly qualified pilot and we debated this amongst ourselves. Like, do we put people with no piloting experience in there, and so we decided, you know, we're going to do that for the study. And we put people with no piloting experience in there and found out that yes, they can learn how to fly these things, and they learned how to fly it in a relatively short amount of time. They're not as proficient as the experienced pilots, but they have a similar learning. And when we look at the data from the industry, they're looking at some estimates. 10,000 pilots a year needed in these aircraft in addition to the pilots that we already are short in the in the US, and so this is going to be a large global market that's not going to be able to. That's going to require a lot of new pilots, and so now that we're starting to see that it is possible to put somebody who's not flown a Cessna, who's not gone through a flight school, and put them directly into an EVTOL, it opens up the aperture for what a training program for a manufacturing company would look like or for a training company in the civilian market. How do you train a large number of pilots? And then what does that training look like? How quickly do they pick up these skills? So, we were able to do some projections based on our data of how long it takes for ab initio students, the students with zero piloting experience, to safely and effectively perform simple aircraft maneuvers. And we were able to project how long that training for takeoff or enroute flight would take. And then this isn't the airmanship piece of it and communicating with the air traffic control, but just the stick and rudder skills, so to speak. So, we're able to give a a projection of what that looks like. And that was some of the data that we provided to the G35 Committee or SAE. This is that International Committee on advising rule making bodies around the world. Col Don "Stryker" Haley I think just to add on to that, Megan. I think some of the more exciting work that we've been able to do is really to form competency-based training. I think there's been a lot of talk around competencies, and I know from an international perspective and from the kind of the civilian perspective, there's a desire to move towards that. Certainly the military, we're interested in that. I would say that our understanding, And how you take and apply competencies, is still developing. That still requires better levels of understanding. Kind of move away from task-based training only. So, I think a lot of our work is really helpful in informing and helping us to understand competencies, you know. For wanting to accelerate training there, there's some things you can't you know I can't. I just you got to get air I call it air under your butt, you know, just you have to have aviation experience there. Speaker 1 I love that. Col Don "Stryker" Haley There's just some part of that you can't get away from. To teach airmanship, you have to be in the air, but there's some pieces that we can accelerate. You know that we, I think we're learning where certain training devices or certain approaches are. We're able to accelerate our development of certain competencies more quickly. And then there's some things you just need those aviation experiences, and I think that when you kind of start with a completely new industry and you're able to more or less clean. Training, design and start with competencies. Instead of taking a very task-based system and then trying to convert it to competencies. It helps us to be a part of that conversation and a lot of the research that we're doing is helping us to advance our understanding and the progress with how we move forward with competency-based training. Speaker 1 Thanks Stryker. You are speaking my language now in terms of competency-based education, outcomes-based military education and I know that both of you have a significant training and development background. Can you share a little bit, perhaps about how that has influenced your work on this project? Col Don "Stryker" Haley Steve, you're the guru, I'm gonna let you go first. Dr. Steve Ellis So, I've been interested in student-centered learning and competency-based training for several years now and have gotten an opportunity to work with some of that and do some research and publish some. So, I think, like Stryker said, being able to go walk into a brand-new project with a clean sheet, and uh, no expectations of what training is supposed to look like and start with a clean sheet approach has been huge in being able to apply some of the things that were, for me, largely theoretical in the past and say alright, now we're applying these things and how does this look in application and does it really work? Do the theories really hold true? I think that's been a huge benefit for me in being able to apply some of the, you know, the experiences that we've seen and some of the things that we picked up over the years and say, alright, this is what training should look like. Or we want to test that model and see if our assumptions hold true. And so, we did actually build our training from a competency-based approach using a situated learning environment. And for me that was a unique way to be able to see that actually put into action and realize that, yeah, this really does work better than we had anticipated, that we didn't run the ship aground, so to speak. Stryker. Col Don "Stryker" Haley Yeah, so my background is in, not necessarily looking at things from a competency, this just being a T38 instructor pilot to start out my career and then an F15E instructor evaluator. So flying training just from a military perspective, so talking about competencies has been new for me. You know, we've now spent two years being a part of the competency movement, if you will. I feel like it is a bit of a movement and being involved in a lot of the discussion. So, I think I bring a little bit more of the operational flavor of, OK, how does this actually work, where I think Steve brings a lot of the expertise and the academic piece behind it. And we have that experience, not only in our team, but through the network that we've created and partnerships. What's mighty about our team is not necessarily the amount of people that are on the core team, with the partnerships that we've built over the last two years, with academia, and with industry. And so, we're a part of like Steve mentioned, SEG 35, we worked with a number of universities and we're involved in other research, and then we kind of bring the, I bring the, so what out of it. So, what does this actually look like for military pilot training? I think that has helped us with our research project in specific, specifically is not to be just completely academic, but to be operationally focused. So how do we take our research? And how do we inform things that really matter for day-to-day activities in the Air Force and beyond? That's, I think, what's made our research a little bit more. It's not just research in a vacuum, it's not just a dissertation that is, but it's connected to a lot of realities, not just with our team, but with the partnerships that we have. Speaker 1 It seems like you both have very complementary skill sets along with the rest of your team, but I would not be doing my due diligence as a podcast host if I didn't ask you this. What are some of the critiques that you've run up against and how do you counter those? Are there people who disagree with the way that you've approached this project? Col Don "Stryker" Haley That's a really good question. Steve, do you want to go first or do you want me to go first here? Dr. Steve Ellis Well, I'll take it from the academic side. The critiques. Anytime you do research, there's nothing that researchers love more than to poke holes in other people's research and when we built the research proposal, the research plan, the idea was to plug as many leaks as possible. How do we make this as defensible as possible? Because when you present your research to a bunch of other researchers, they're gonna look for things like, well, that's the wrong statistical test. Or your statistical tests, did you do a power analysis? You know what? How did you come to that conclusion based on these statistics? Or how do you know that that was the right statistic? A lot of thought goes into that to make sure that our research is defensible and the other is to scope it properly that you know we're not trying to solve all the world's problems, but we can present within these guardrails that, or we can say authoritatively within these guardrails that this bit of data is as accurate as we could make it. And the research findings are as solid and concise as available. In an operational research environment, we're not in a laboratory. When you're doing human factors and human social science research in an operational environment, it can get messy. Try to put in as many controls as possible to eliminate that from an academic standpoint. Stryker, over to you. Col Don "Stryker" Haley Yeah, I would say that most of our resistance that we had to this particular project was from my perspective. I'll talk from an operational perspective. It was probably in the very beginning when we came up with the research proposal. We were talking with some of our OEM partners. Of course, in this industry, Agility Prime, we work with a number of OEM's. They're all competitors in some way or another, so there's always concern about intellectual property and then, also managing public perceptions. And if you find one thing, and if you have the wrong context, if, you know, we have a finding, but it's absent of the context, say hey, this is a prototype vehicle, that it's like, you know, they're now four prototype evolutions down the road, and they fixed all these problems, and so they, you know, worry about investors and things like that. So, there's a lot about controlling messaging and making sure that we're being a good partner with our OEM's. And so, we had to work a lot with the relationships at the beginning when we were pitching some of our research ideas and then initially there was this, I don't think people quite understood the value or the potential value at first. There was some resistance of investing the money that it's taking to get the contractors and the Sims and everything. But what we found now, in retrospect, is that the data that we produce has been very helpful and it continues to be helpful and it's whatever. We've shown up at a number of venues, be it a large training conference like IITSEC or in a regulatory meeting with SAE or the FAA, where we're the only folks in the room who have hands on experience, and actual data to support some of our opinions and thoughts about how rulemaking should go or how we should approach or have certain curriculum approaches. And so, we have found that the data that we produced has become very helpful. In the beginning, though, there was a lot of resistance, and it was mainly from the OEMs, but I think we've been really careful to try to understand what the OEM's concerns are and to be a good partner in the way that we message things, the way we share the data, and to whom we share the data. And so, I think part of it is just relationship management and I think that we've done a good job of that. So that was, most of it was around relationships and how you handle IP. I think we've navigated that really well through the last year. Speaker 1 This sounds like such a complex project. As we're winding down, is there anything else you want to share about the future of this research initiative and what's next for EVTOL? Col Don "Stryker" Haley You bet I'll jump in on that one, so. At Det 62, we don't consider ourselves a research organization. We saw an opportunity and a kind of a hole that needed to be plugged. So, we're, I think our ears are open and we're looking for how we can help further the conversation, and so I think there's a lot of work that we can do to try to advance our understanding of competencies as we apply this kind of clean sheet design to this industry and and help inform others. So, I would say over the next year or so, we're going to be taking some of the work that we've already done as an Air Force and then try to apply that to the industry with some of the things that we're starting to draw out, and that certainly in the new year we're going to put some meat on the bones of our thoughts over what's next for us. So, I think that there's where we're going. To continue to help fund and direct some of the research activities we're going to focus a little bit more on competencies over the next year. And a lot of that has to do with the overall maturity of the OEM's as we get closer to transitioning these technologies, we're actually buying our first EVTOL aircraft next year and getting them down at Edwards and Eglin. And as we start putting actual Airmen and Guardians in these aircraft, we're going to take our research from kind of more of the basic research and understanding some basic questions into a little bit more operationally flavored competency-based curriculum development, kind of activities. And that's what you'll see out of us over the next year. Speaker 1 Wonderful, thank you so much. Dr. Ellis, Colonel Haley, thank you for joining us today. How can we stand by for more information? How can we reach out to you to learn more? Dr. Steve Ellis That's going to be around for a while, and we will continue to look at this, like Stryker said, from a clean sheet approach. You can reach out to us through the AFWerx community and through the new AFWerx Director, Colonel Lee. We will keep our, like Stryker says, keep our ear to the ground and see what's coming up that we can put some of our muscle behind and help advance. We're, like I said, thinking that a couple of things are on the horizon, the competency-based training, of course, is going to be a big thing. Another gap in the research kind of so to speak. The other may be helping with emergency procedures and the EVTOLs, because nobody knows what those look like yet and so how do we continue to further that industry? But if people need to get in touch with us, the easiest thing to do would be good through the AFWerx directors or reach out directly to us. Col Don "Stryker" Haley For AU, this certainly with our partnerships that we already have with academia, like I definitely have AU be involved with those activities. I'm looking forward to, you know, growing that partnership, but a lot of this has been. I think a lot of our success, that we've had over the last couple of years, has been not necessarily the work that we're doing, but the way that we're able to connect to different organizations just because of the relationships that we have. With that, we're able to have through the Agility Prime program, with the OEM partners, and the creators and the relationships that we have with industry and with academia, allow us to kind of be the kind of the team that brings everyone together, especially around the questions around learning and curriculum development for EVTOL, so I think we're kind of right in the middle of that space, and we're a good connector and a good partner with this industry, and in the way it's growing. Speaker 1 And I'm going to try my best to get you to present at the Military Scholarship of Teaching and Learning forum, MSOTL Forum, in December of 2023. So, our listeners hopefully can look forward to learning more from you then. Thank you both so much, Steve and Stryker, and we look forward to hearing about the other good things that come out of this program. Thanks for joining today. Dr. Steve Ellis You're welcome. Col Don "Stryker" Haley You guys thank you so much, Megan. Return to Axon Podcast Return to Axon Podcast: /TLC/Podcast