Axon - Mixed Virtual Reality and Leadership Development - Ep13 Published July 25, 2023 The opinions, conclusions, and recommendations expressed or implied in this podcast are solely those of the speakers and do not necessarily represent the views of Air University, the United States Air Force, the Department of Defense, or any other U.S. government agency. Dr. Megan Hennessey Welcome to the Axon podcast. This is the official podcast of the Air University Teaching and Learning Center, and I'm your host, Doctor Megan Hennessey, the Director of the AU TLC, together with Doctor Andy Clayton and Lieutenant Colonel Lisa Stokey today. Doctor Andy Clayton is a Lieutenant Colonel in the United States Air Force Reserve. As a civilian, he is an Assistant Professor of Leadership and Director of the Augmented and Virtual Reality Research Task Force at Air University. He earned his Ed.D from Grand Canyon University in Organizational Leadership with an emphasis in Higher Education. We're also joined by Lieutenant Colonel Lisa Stokey, the Division Chief for the Leading Inclusively Virtual Experience, which we are going to focus on in our discussion today. She is an Aircraft Maintenance Officer and a graduate of Air Command and Staff College. Lieutenant Colonel Stokey also holds a master’s degree in Aeronautical Science from Embry-Riddle Aeronautical University. Welcome to you both. Dr. Andy Clayton Well, thank you. Dr. Megan Hennessey So, Lisa, start us off. What exactly is the Leading Inclusively Virtual Experience or LIVE? Lt. Col Lisa Stokey Yes. Thank you for having me on the podcast Dr. Hennessey. This is really exciting. So, the Leading Inclusively Virtual Experience, otherwise known as LIVE, it's a mixed virtual reality experience that enables leaders in the Department of the Air Force to practice crucial conversations regarding diversity, equity, inclusion, and accessibility. It's authentic, it’s real time dialogue and it gives leaders a repetition to navigate difficult topics before having to do it as a supervisor. Dr. Megan Hennessey So, in terms of leaders, like what levels and ranks are we talking about here or does it not even matter? Lt. Col Lisa Stokey It doesn't matter. All levels and ranks, anyone that's ever going to find themselves in a supervisory position as a member of a team within the Department of the Air Force, Leading Inclusively Virtual Experience is essentially for everyone in the Department of the Air Force. It's fully funded by SAF/DI, so this is a program that's accessible to Airmen and Guardians across the force. Dr. Megan Hennessey Including civilians? Lt. Col Lisa Stokey Including civilians. Our civilians are Airmen too. Dr. Megan Hennessey Alright. Andy, how does LIVE make effective use of immersive learning technology? Dr. Andy Clayton Thank you, Megan. So that's an interesting question because we're actually probably using more, I would call a low-end use of technology. So, the immersiveness aspect of it comes from the fact that this LIVE program is using what we call a human in the loop simulation. So we're all comfortable we all probably know about what a simulation is and what it does, using some form of stimuli to have somebody respond to practice a certain skill set of behavior, and normally in simulations we see technology being used and we call those more the process based simulations flying a plane or shooting a gun doing something in healthcare. Those types of things you would think of using higher-end technology to get after immersive learning. But in our case for LIVE, the immersive learning is not so much on the technology aspect of it because what we're actually using in LIVE, really is just Zoom and then what's happening is there's a human in the loop controlling an avatar on screen that's being shared to the participants through Zoom. So, we're using low-end technology as zoom, but the immersiveness comes from this avatar that's being controlled by a human in the loop, we call these immersive learning specialists. The technical term for this is actually called an Avatar Mediated Interactive Training and Individual Experience System called AMITES and AMITES has been. Dr. Megan Hennessey That's a mouthful. Dr. Andy Clayton That is a mouthful and that's been around for quite some time. But even back when that when this initially started probably way back all way back to some universities back in the early 90s or more late 90s bringing this ability for someone to control an avatar that would be displayed on screen and in our case we use zoom and the immersiveness comes from the fact that an individual talking to this avatar can't believe that this avatar is real. They're not sure how is this avatar talking to me in live conversation? And so it's because obviously there's an individual, a simulation specialist, a trained actor, if you will, controlling that avatar and because of that, what happens to the participant on screen on Zoom, talking to this avatar, they create presence and agency and those are the two things that get after what we call those cognitive affective, immersive learning and so when a student or participant feels that they're immersed or present, with this avatar and they have agency control over what they can say and how they say it just as well as the avatar has control over what it says and how it says it. You have this human-to-human interaction and the participants trying to figure out how is this even possible? How is this technology making me feel like I'm present with somebody? Well, it's because of what we call, suspension of disbelief, that participant believes that this avatar is real. So, they suspend belief that, wait a minute an avatar can be a real person, and that's because it's being controlled by a real person, but that individual doesn't know that. They just see this avatar in front of them on screen through Zoom, very simple technology, but because it's responding in live form, it looks like a real person and its eyes are blinking and his mouth is moving. The voice that's coming from the avatar has rate, pitch, and tone, just like a real conversation. They suspend disbelief and the term is called anthropomorphization. That's a big word too, right? We take an inanimate object like a digital avatar or virtual avatar we make it look like it's real, like it's life-like, and that's anthropomorphizing, bringing something that's inanimate and making it animated. And so, because of that, on screen, the student believes it's a real person. And that's what makes it immersive. And so, because of that, the student feels present, and they have agency and those are the two main factors that you need to get after something to make it immersive or immersive learning and technology again is the side part of it. The low end, the Zoom. It's the fact that we create suspense of disbelief because I can talk to this real avatar. Dr. Megan Hennessey So, here's the big question that I've had ever since I first observed you model the program, Lisa, and that's if the people on the other end are live actors, why not just see the actor’s face? Why the avatar? Dr. Andy Clayton Megan, that's a great question. And it comes up often from the program standing up and throughout its history, if you will, we get that question quite often and what the term that we use to respond to this question is real, it's called the uncanny valley. And so, this uncanny valley, it's actually rooted in literature and poetry if you've ever read a poem right, you think about this idea of like, where am I going with this poem, or if I'm watching a play, if I'm watching a movie that's using computer generated avatars. You look at those avatars and you're wondering why they look real, or they look too real and so what happens is if we put a real person in front of you, you’ll start to believe that person that you're talking to may not have those real issues that they're playing a role. And so, if they're playing a role and you don't believe that they're actually this person has this issue then you don't have suspension of disbelief and if I don't have suspension of disbelief then I won't demonstrate authentic behaviors and so we don't want to put a real person in front of someone because role-playing is just not as effective for adults. Role-playing works quite well for K12 education, but for adults they need that a little bit more suspension of disbelief. Children can suspend disbelief when another child or a teacher is going to play a role and they're going to interact in these roles. Hey, I am a salesperson and you're going to buy something from me so that the child can play that role and they go into that belief that this is the environment that we're in. For adults, it doesn't work when you put a real person in front of them. And so, we need something else to get that, that student, that participant, to suspend disbelief and so we put an avatar in front of them, but we also still need that interaction, that human to human interaction, because of the topic that the LIVE goes after diversity inclusion. I need that authentic like dialogue between two people, so I can't make the avatar, just have some computer system behind it giving words. It has to have that human connection. And so, when we have this avatar, this uncanny valley. We can find the sweet spot, the avatars look a little cartoonish and they don't look too real, and so it's this uncanny valley, where we find this nice, sweet spot. If you ever watch some of, maybe some movies and that movie has zombies in it, one movie may have zombies in it and zombies, they look kind of cartoonish or like over the top kind of things like, OK, it's just a zombie, it's part of the movie. Other zombie movies make the zombies, look real like that could actually be like a dead person that came alive and it's trying to eat me or something right, and so, this is the uncanny valley where that's too real or potentially too possible for that to occur. So that makes you feel unsafe, you don't want to be in that space, but if I make it cartoonish or too over the top I get it, it's not real, it's just it's a zombie movie kind of thing I have it. So, that's more of the sweet spot. The same thing happens with LIVE, we have avatars on screen that look a little cartoonish and they don't look too real, and so, we are in the right sweet spot. If they look too real, then it's right back to our response about why do we have, why do we put a person in front of you? Cause now it looks like a real person, it feels unsafe, like I'm playing a role, but it's an avatar and I also now put the suspense of disbelief on there, and I also I'm talking to an avatar. It's responding to me in live form and my brain is connecting to suspension of disbelief because this avatar is animated now like a real person, your brain gets tricked and it goes, oh, this is a real thing. This is a real problem. I need to demonstrate authentic behaviors and start communicating like I would normally. But there's also this space because the avatar is an avatar, there's a safe space, so it's a connection between these two things. Suspension disbelief, but also a safe space that allows us to have a conversation about a very complex problem like diversity inclusion. Dr. Megan Hennessey Yeah, this is a fascinating paradox because if I'm hearing you correctly, you're saying that in order for the experience to be authentic, it's better to have something that's inauthentic, the avatar, in front of you, right? And I've seen the scenarios a few times and I will say the avatars are super impressive. They're more impressive than the Cyber Awareness challenge avatars, and you can actually see what looks to be, you know, the avatar thinking, showing nonverbal behaviors like movement of the eyes and the eyebrows, and looking to different directions, and it makes it feel very real. Dr. Andy Clayton Sorry, really quick, you're exactly right. We even talk about the nonverbals aspect of it, which really kind of settles into that suspension of disbelief. So, it's not just the voice in that live conversation, but it is also because that simulation specialist is also creating nonverbals right, which is like a normal human being would. And this avatar was able to do that. It takes it again to that next level of helping to create that suspension of disbelief. So, we didn't even mention the nonverbal aspect of it. But you're absolutely right, with that, that helps lead into and create even stronger suspension of disbelief. Dr. Megan Hennessey It's super cool. Lisa, I was going to ask you, how did the students feel about this? What has the student reaction been or the learner reaction? Lt. Col Lisa Stokey Specifically, to the avatars and the suspension of disbelief? Dr. Megan Hennessey Yeah, or just the whole experience. Lt. Col Lisa Stokey Yeah, and while Doctor Clayton was talking about the suspension of disbelief, I also wanted to bring up why the avatar and not being able to see the actor’s real appearance behind the avatar lends itself so well to the learning experience, because you know, when we look at a program like LIVE and the objective is to foster inclusion and think about how we're doing that as leaders into, you know, for biased literacy and really thinking about what are our own blind spots bringing some you know, self-recognition to our own whatever bias we've brought to that conversation, another reason why it's great to use the avatar and to have that suspension disbelief is because it is customizable in a way. So based on your participant or your audience in general, you can customize the gender or the rank, and that can often impact the behavior and the cognition of the participant, and so that suspension of disbelief is really, really great for that, because without giving away too much of the magic, you know you could have the same actor running through a couple of different scenarios. In one session and it be a completely different experience for each conversation. So, I always like to bring that up without giving away too much magic, because I think it's a really great example of why the avatar works so well for learners and a lot of times people, they ask questions immediately following a session and they start referring to the avatar as AI. Well, that AI is so believable that it was, it was really convincing, not realizing how real time that interaction was in that conversation, and we don't like to keep a lot of focus on that, we like to focus back to the diversity, equity, inclusion topic and steering that debrief on you know others perceptions and experiences regarding the actual DEIA topic. Dr. Megan Hennessey Are the students on board? Even if they sometimes think it's AI? Lt. Col Lisa Stokey They are very on board. It's really, it's such a great tool because you have the ability as a facilitator, based on your participant and your audience, when I talk about that customization for the suspension disbelief, you can even communicate in real time with the actor on how intense you would like that crucial conversation to be. So, for instance, if I were to facilitate a scenario, for as an example, an ALS classroom full of, you know, brand new NCO's, first time supervisors. I likely would not want the intensity of that conversation to be the same as if I went and facilitated for the first Sergeants’ Academy cadre, where you have a room full of experienced senior NCO's that are going to be, you know, the commander’s liaison to their to their enlisted core and advising them on a lot of things regarding their Airmen or legal matters. I have an ability as a facilitator in real time just sort of engage the actor to crank it up or if a conversation is going way, way off the rails, because maybe perhaps the participant that has been selected randomly, their emotional intelligence isn't very good. And you want to bring that conversation back to a point where you can get it back on track and facilitate more of that discussion during the debrief, you can communicate in real time and so, and being able to do that, it's great for a facilitator and then the students respond to it in kind, right? The learners respond to it in kind and it helps you, as a facilitator, really lead that conversation, that dialogue following the session. Hopefully, that answered your question. Dr. Megan Hennessey It does, and now I'm just personally very curious what sort of things would you, if you were the actor, change to ratchet up the intensity level? Lt. Col Lisa Stokey So, I can't say that I'm super involved with all the prep that goes behind the scenes for the actors, but what I can say is that they speak DoD. So, when we build a scenario with them, just like any really skilled actor they really get into character, so there's a whole background story that they've sort of adopted for themselves that you as the facilitator or as the learner, might not ever need to you know, and they've really committed to that. They read whatever that scenario is, they read the AFI's, the guidance, the policy, everything that pertains to it, so that if anything comes up in the course of that conversation, they can reference, what we know to be true in the DoD, as you know, Guardians and Airmen, they can reference that. So, I think a lot of times they're just very professional and very skilled and very good at what they do. If a participant’s emotional intelligence is not that great, or if they're coming across pretty aggressive they can respond to that with sort of the an equal defiance, I guess, is the best way to say that, and it makes the conversation more challenging and it engages the observational participants a lot as well, so they can be, there's one actor in particular, and I don't want to give away too much of the magic. There's one actor in particular, that I really like working with because some of the scenarios we have, he plays the “very close minded, very aggressive, well because I'm in charge and that's the decision I made sort of stance” very well and there's really no way to navigate around it because anything a participant might throw out there to try and engage that avatar to a better understanding that actor is very skilled at shutting it down at every corner and it's just fascinating to watch and it's equally fascinating to watch how the participants and even the observational participants in the room react to that. Dr. Megan Hennessey Yeah, I've observed a couple times now and I will say even with the use of the avatar, Andy, I was like deep in the uncanny valley of discomfort watching the interaction happening between the person who's in the hot seat and the actor was even second hand just so powerful. Dr. Andy Clayton Megan, and that's why the Air Force, I think one of the main reasons why this program is effective is because how do you scale something like this when you're using technology, right, to get after something like diversity inclusion and saying at the DAF level we need to get after some training and education and diversity inclusion and we're going to use this program and it uses technology. The question is, well, then how do you scale that for everyone in the Air Force? How do we get every single person, every Airman, Guardian, right in Air and Space Force to actually use this simulation, this LIVE program. Well, when they asked us that question we told them like not every Airman and Guardian actually has to be the one in the seat because you can have others observing, watching a fellow classmate, another Airman or Guardian execute the simulation through LIVE and be that observation participant and like you mentioned, Megan, right, you feel those same feelings, you hit that same suspension, disbelieve the uncanny valley hits you just as we as it does as the participant actually in the seat talking to the avatar. Others that are watching, and again because we're using Zoom, whether we're online Zoom or in a classroom, looking at it on-screen, you have the same cognitive and affective processes occurring in your mind. So, you have what we call a very similar experience that the participants going through. And so, because of that, we can scale, and we can tell the Department of the Air Force that everybody is getting that experience at a larger level instead of having every individual in the Air and Space Force have to go through it individually. And that's what's made it so scalable and effective for us. Dr. Megan Hennessey How do you know, and this is the loaded question, that it's working, that it's helping the learners to achieve the outcomes they need to achieve? Dr. Andy Clayton Yes, that is the $1,000,000 question that the DAF and everyone from there down has asked us those questions about how we get an accurate assessment of this. There's a lot happening obviously within diversity inclusion. There's a lot of things happening within the human domain and these skill sets, emotional intelligence, and empathy and understanding those things through the lens of diversity inclusion can be very challenging to assess and so we're not there yet on the quantitative side and at least I can explain some of the qualitative side of it. I'll give us some of the future thought process of how we're looking at this. If we really want to get after and get that study and get those quantitative kind of measures to see, if someone's competencies I guess in diverse inclusion, understanding has improved right experiential learning, right? Here's their behavior. They've had an experience, they've reflected on that experience and then there's behavior change in the future to see that difference in scale, what we probably have to do, the student would have to do it twice or can we just assess the way it's occurring, once when I see it's going through that. That means we need to probably have more technology added onto the LIVE program and when that occurs, then the program becomes a little more challenging, it's going to be hard to scale that type of technology, it would require us to record sessions, right, and then use different assessment tools. One would be called CAMIL, right Cognitive Affective Model for Immersive Learning which is going to be able to get after the connection between the affective and the cognitive. So really what’s that looking at is the method that's being applied and is the student learning through that method. And so, it's going to look at some different things if we can record the student, then we can use webcam tracking of different things that are the biometrics of that student heart rate, eye dilation, eye pupil location, those types of thing right in pitch of the voice. Those can be all analyzed through some higher end uses of technology such as AI, which would be needed to be used and so we're just not there yet, but we know that those things are out there, we've seen demos of other similar programs where somebody is interacting on camera, which is what we're using, just Zoom so, we're on camera and then through AI and some other technologies you can track how immersed that individual is and is learning occurring? We're just not there yet, so we're still on the qualitative side right now and I think Lisa could give us some antidotes on the qualitative side of things, but that's where we are on the quantitative side. We're trying to get after AU, but. We're just not. Yet that's the future focus. Lt. Col Lisa Stokey So, I get really excited when Doctor Clayton talks about the possibilities for how we could capture some real time data on how learners are responding and is it effective and is there, you know, a cognitive shift. Technology is out there, we just need to find a way to incorporate it into the various different programs that uses this technology, like LIVE, especially for things regarding topics like diversity, equity, inclusion. Since you know, fostering a culture of belonging is such an important tenant. I can say in terms of qualitative data. The responses we get in participant surveys, the surveys from our facilitator courses is very, very positive. The program, the facilitator course itself was in beta up until this past January of ‘twenty-three, and so all the facilitator courses we've executed since then are, as we're collecting this data and our facilitators are going out into the Department of the Air Force and you know, requesting that the participants, participate in the surveys. It's been overwhelmingly positive, and I can say 94% of participants either agree or strongly agree that a LIVE session or participating in the LIVE session has improved their leadership abilities and cultural competencies. And that's just a quick survey snapshot for a very brief period of time as the program is continuing to mature and the facilitator course itself is continuing to improve. Dr. Megan Hennessey That's huge and very exciting. I think you both should be very proud of your involvement with this program. Going to ask one more question. Just having worked in specifically PME for a while and understanding a bit about the faculty cadre and how we have some crusty, salty, choose your other synonym, faculty on board who may have doubts about this tool. How do you share the value with them? How do you sell them on at least trying it? Dr. Andy Clayton Yeah, I can start a little bit with that. So that's another one of those. I think $1,000,000 questions at Air University is how do we get to the application of some of the things that we teach across Air University. I think it may be a little bit easier to look at Air Power and Joint Planning and things like that where we can do War Gaming simulations. We could write a paper to see how well the student understand international politics and those types of things. But in the human domain and leadership, when we have those courses across Air University, those courses, we ask the question how often have you had your students be able to demonstrate the application of leadership, that's always hard because I can give you an essay question and say if you're in a situation, you know, write a leadership essay that uses some leadership model about how you would address the problem. Well, it's easy to write a paper and say this is what you do in a passive form. But how often do we put you into that situation? How often do you get that muscle memory? Do we ask our pilots or security forces or our firefighters to write an essay on how to fly a plane, put out a fire, or shoot a gun? We actually put them through simulation, and so would it be nice if we were able to do the same for leadership in the human domain. And the way in which the program works all I need is an hour and about fifteen students, which is the average size of a classroom across Air University. Your students could have an hour experience to look at three or four different scenarios to get to the application of the human domain, whether it's diverse inclusion, emotional intelligence, empathy, accountability, command decision making, and put you in that situation on some of the topics that we're teaching across the Air University and within an hour, I can see the application being used, which you've never been able to do before. So, if you can spare an hour and get to an experiential moment to get to some muscle memory at the application level within PME, I think we can sell that to most faculty. Dr. Megan Hennessey You've convinced me, I’m in, I was already. Anything else you all want to add as we wind down our time? Lt. Col Lisa Stokey Yes, ma'am, if I may? I will just say that I'm incredibly proud to be part of the program, like LIVE, you know my previous assignment I was a commander for four years and I see what this type of learning and what a program like this can do for the Department of the Air Force, I think it's incredible whether it's my shameless plug that as we come to the close of an academic year, with faculty developments coming up, there's value in a program like this to understand your diverse faculty. You know, we're made-up of civilians coming from all types of university backgrounds and active-duty military, that's so diverse understanding the differences. That we've all experienced, either in a university setting or military setting, and how we bring that to teaching or even how our students here at Air University are bringing that to the table as learners, or if it's just looking at it as a tool that's available for leaders, whether it's here at Air University or out in the Department of the Air Force to really sort of engage their blind spots and how they think and what their Airmen are thinking. And how their Airmen are learning or what their experiences are, that they're going to bring to the plate to innovate and lead differently, it's just a really great tool and it gives all of us an opportunity to really engage in that repetition in that conversation and understand those things about ourselves before we do it in real time and mess it up and perhaps lose their credibility. Dr. Megan Hennessey Absolutely, Doctor Clayton, anything to add? Dr. Andy Clayton That’s really what Lisa just hit spot on is can we give people that opportunity to have that muscle memory before they experience it for that, real time and don't have that experience. And I think the LIVE program, especially in diversity inclusion allows people to have this shared experience of what diverse inclusion conversations look like, and then to be able to talk about the shared experience that they just all saw through the LIVE program so they can have an honest and open and brave conversation about diverse inclusion and so they can take that with them back out into the workforce. Dr. Megan Hennessey Thank you both, Lieutenant Colonel Stokey, Doctor Clayton, we'll make sure to put information in the episode notes if you are interested in learning more about LIVE. Thank you for joining us today.