Augmented/Mixed Reality for Manufacturing Applications George N. Bullen, FSME, CPIM Technical Fellow, Northrop Grumman Corporation

Augmented/Mixed Reality for Manufacturing Applications George N. Bullen, FSME, CPIM Technical Fellow, Northrop Grumman Corporation

Who am I? Automated/mechanized Drilling and Countersinking of Airframes Book by George N. Bullen, SAE International, 2013. Successful Composites Technology Transfer: Applying NASA Innovations to Industry from the Max Launch Abort System Program Book by George N. Bullen, SME, 2015 Economics of Composites Book by George N Bullen, Alan Hiken, Carroll Grant, Dan Day, and David Champa, SAE International, 2015 Aluminum Auto Body Joining Book by George N Bullen, SAE International, 2015 Successful Composites Technology Transfer: Applying NASA Innovations to Industry from the Max Launch Abort System Program Video by George N Bullen, SME, 2015 Defining the Future State of Airframe and Automotive Assembly Book by George N Bullen, SAE International, (In Production for release in 2017) The Future of the Airplane Factory: Digitally Optimized Intelligent Airplane Assembly Book by George N Bullen, SAE International, (On contract with 30% complete. To be released first quarter 2018).

Who are you?

PURPOSE Not a detailed description (Not enough time) General Overview

Why is this important or relevant? Virtual prototypes have shown value for their ability to present both small scale and large scale three dimensional spatial information. Low cost compared to traditional design reviews, especially physical mockups. Tool Proof First Article

Why is this important or relevant?

Why is this important or relevant?

Why is this important or relevant? Army Case Example Changing the Norm

What Is the Difference Between Augmented Reality (AR) and (VR)? A Lesson in Altered Realities Augmented reality and virtual reality. Related, but different. The difference is major, and worth explaining. Augmented and virtual reality have one big thing in common. They both have the remarkable ability to alter our perception of the world. Where they differ, is the perception of our presence. Virtual reality is able to transpose the user. Through closed visors or goggles, VR blocks out the room and puts our presence elsewhere. Oculus Rift, Samsung Gear VR, Google Cardboard, are names you may have heard about by now.

Putting a VR headset over your eyes will leave you blind to the current world, But will expand your senses with experiences within. The immersion is dramatic, with some users reporting feelings of movement as they ascend a staircase or ride a rollercoaster within the virtual environment.

Augmented reality however, takes our current reality and adds something to it. It does not move us elsewhere. It simply "augments" our current state of presence, often with clear visors. Samsung is near ready to introduce its Monitorless AR glasses, which would connect to phones or PCs via WIFI and replace the screen on those devices

What is the real difference? Think scuba diving vs. going to the aquarium. With virtual reality, you can swim with sharks. With augmented reality, you can watch a shark pop out of your business card. VR is more immersive, AR provides more freedom for the user because it does not need to be a head mounted display.

Microsoft is injecting interactive holograms into our world to bridge the gap between your PC and your living room. Using HoloLens, you can literally surround yourself with your Windows apps. In 2016, the world witnessed augmented reality take center stage in the form of Pokemon Go. This was the first major example of AR finding mass market acceptance and infiltrating our daily lives. Virtual and augmented realities in 2017 are already making dramatic leaps forward as startups find ways to introduce smell and touch to expand your sensory experiences. Technology company Immersion has introduced TouchSense Force, using haptic feedback to bring player's hands into VR worlds, and researchers at Stanford University s Virtual Human Interaction Lab are having to resist eating foam doughnuts as they experiment with adding scent to VR. Beyond the obvious media and entertainment applications for AR/VR technologies, design and engineering companies the likes of Solidworks are demonstrating their commitment to immersive design with AR and VR related partnerships, including NVIDIA, Microsoft, Lenovo, and HTC Vive.

The State of the AR/VR Adoption Rate Both augmented reality and virtual reality are gaining speed, and are more relevant in our current marketplace than ever before. Both are hindered by our ability to render 3D environments in real time. AR less so, but the problem of creating high resolution, life like objects, still persists.

Mixed reality (MR), sometimes referred to as hybrid reality is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co exist and interact in real time. Mixed reality takes place in the physical world and the virtual world. It is a mix of reality and virtual reality, encompassing both augmented reality and augmented virtuality via immersive technology.

Augmented virtuality (AV), is a subcategory of mixed reality which refers to the merging of real world objects into virtual worlds. As an intermediate case in the virtuality continuum, it refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with, the virtual world in real time. This integration is achieved with the use of various techniques. Often streaming video from physical spaces (e.g., via webcam) or using 3 dimensional digitalization of physical objects. The use of real world sensor information (e.g., gyroscopes) to control a virtual environment is an additional form of augmented virtuality. External inputs provide context for the virtual view.

A topic of much research, MR has found its way into the manufacturing and education worlds with systems including IPCM Interactive Product Content Management. Moving from e learning to s learning state of the art in knowledge transfer for education. Simulation/VR based training, interactive experiential learning. Software and display solutions with scalable licensed curriculum development model.

Real Asset Virtualization Environment (RAVE) 3D Models of Manufacturing Assets (for example process manufacturing machinery) are incorporated into a virtual environment and then linked to real time data associated with that asset. Avatars allow for multidisciplinary collaboration and decision making based on the data presented in the virtual environment. Virtual models are used to allowed scientists and engineers to interact with a possible future creation before it touches the factory floor. Models provide the opportunity to gain an intuitive understanding of the exact product, including real size and constructions details that allow a closer inspection of interior parts. Virtual models are also used to find hidden problems and reduce time and money. Artificial Intelligence: Calculates the best possible decision to be executed by a device based on algorithms supported by previous data record and previous decisions registered in the system.

If you would like to make a predictions for the future, you actually need to find the trajectory of events.

Emulation for Logic Validation also referred to as virtual commissioning. The process involves replicating the behavior of one or more pieces of hardware with a software environment (typically for a system under design).

The goal of the emulation engineer is to create an environment that mimics the real automation hardware. The ultimate goal of emulation is to provide an environment for the manufacturing automation engineers to validate the type, placement, motion, and throughput of machines integrated into the assembly process prior to system debug in the plant environment. Improving quality and enabling a seamless transition from the virtual to physical environment. Another benefit is to deliver plant maintenance operators and machine conductors with realistic virtual environments for training themselves in safe and optimum conditions.

Virtual Commissioning of an Existing Manufacturing Cell at Volvo Car Corporation Using DELMIA V6 Validating that equipment meets the required cycle time Enabling collaboration between engineering disciplines Early operator training before startup Rapid development of standards Evaluation of the PLC code and the robot programs

TRAINING DERIVED FROM MIXED REALITY

One of the pioneers of this technology is a company based in suburban Atlanta. AGCO has factories all over the world where it makes large tractors, chemical sprayers and other farm equipment. At one of AGCO's factories, Heather Erickson is putting together a tractor engine before it goes on to the assembly line. She's wearing a red and black uniform over her blue jeans at a facility in Jackson, Minn. And she's wearing something else: Google Glass on her head. "It took a little getting used to. But once I got used to it, it's just been awesome," Erickson says. Google Glass tells her what to do should she forget, for example, which part goes where. "I don't have to leave my area to go look at the computer every time I need to look up something," she says. With Google Glass, she scans the serial number on the part she's working on. This brings up manuals, photos or videos she may need. She can tap the side of headset or say "OK Glass" and use voice commands to leave notes for the next shift worker.

They're the headsets that look like regular glasses but have a small computer on the side to speak to and access the Internet. If that's not ringing a bell, it could be because Google Glass fizzled out and was discontinued in the consumer market. But now, it's getting a second life in the manufacturing industry.

Peggy Gullick, business process improvement director with AGCO, says the addition of Google Glass has been "a total game changer." Quality checks are now 20 percent faster, she says, and it's also helpful for on the job training of new employees. Before this, workers used tablets. "We had a lot of tablets on our floor, and the tablets were being broken just by being dropped. And tractors are very tall machines when you're climbing on and off," Gullick says. "So we were looking for a solution that offered them more information in a more timely manner. AGCO has about 100 employees using the custom Google Glass, which is attached to them and harder to lose. Each costs about $2,000.

QUESTIONS