VR Headset for Endoscopy and Microsurgery Client: Dr. Azam Ahmed Advisor: Mr. Willis Tompkins, Dr. John Puccinelli Team Members: Tom Geissler geissler2@wisc.edu Team Co-Leader Sam Peters speters9@wisc.edu Team Co-Leader Sam ssimon8@wisc.edu Communicator Sam Schini schini@wisc.edu BSAC Jake Cohn jmcohn2@wisc.edu BPAG Josh Niesen jsniesen@wisc.edu BWIG Reporting Period: October 19th through October 25th, 2018 Problem Statement: Endoscopic surgeries have become increasingly prevalent in the operating room along with the visualization techniques used to perform them. Traditionally, large monitors have been used to display the images from the surgical tools, however, these are bulky and don t provide as immersive of an experience as other methods do. This problem has been partially addressed by using virtual reality headsets. One major limitation of these however, is that they do not allow the surgeon to see anything outside of the operative view. This is a problem as the surgeon will have to remove the headset everytime he or she has to change instruments, or perform macroscopically. The goal of this project is to design a VR headset that would allow the surgeon to transition from the operative view to normal view without the use of the surgeon s hands. Summary of Team Role Accomplishments: - Team: Collectively met at the Makerspace to test out the HTC Vive and split into teams focusing on the various aspects of the Vive that we will be altering. - Leaders: This week we split up the group into two subteams, displaying video and switching between endoscopic view and environmental video. - Communicator: Distributed the weekly progress report to our client and advisor. - BWIG: The project page was updated with team s progress report and the meeting notes from the past week were posted in lab archives. - BPAG: As we work on our software solutions, I will not be purchasing anything, but I am maintaining a running estimate of what the cost will be for our project so our client is aware. - BSAC: Prepared to attend and contribute opinion to this week s meeting with faculty. Chem 344 lab may be discussed again. Design Accomplishments: Our group met two times this week to focus on two key aspects of the project: displaying the endoscopic view within the headset (Peters, Shini, Cohn) and transitioning between the two views desired by the client (Geissler,, Neisen). We met at the Makerspace early in the week so a few members
could test out the headset, get advice from our tech guru (Taylor), and split into 2 teams, each addressing one of the two aforementioned aspects. Throughout the week, groups delegated work (i.e. reaching out to Vive communities and researching information received in the responses) and shared their findings when we reconvened Thursday (tonight, 10/25/2018). Activities: Date Name Task Time (hours) Weekly Total Semester Total 10/19/2018 Team Team meeting at Makerspace to work with HTC Vive Pro. Designation of major tasks and formed groups. 10/25/2018 Team Team meeting to discuss research findings amongst the two groups (video display and camera transition) 2 2 25.5 2 2 27.5 10/22/2018 Sam Schini Research (Live Video Feed in HTC Vive) 2 2 20 10/23/2018 Sam Schini 10/20/2018 Sam Reviewed notes from our meeting with Kevin Ponto and looked into his suggestion of using a webcam Submitted a post to the HTC Vive Community discussion board asking how to approach a hands-free switch between camera and screen views 2 2 7.5.5 21.5 10/21/2018,10/22/201 8 Sam Research (IVRTrackedCamera, OpenVR Pluggin, HTC Vive Camera Access) 4 4 25.5 10/23/2018 Sam Emailed Kevin Ponto in order to get his opinion of the action suggested by the online HTC community.5.5 26 10/24/2018 Sam Peters Researched video output format. 1 1 14.5 10/22/2018 Josh Accessing gyroscope values research and accessing front cameras research. 2 2 26.5 10/25/2018 Josh Update progress report and website. 0.5 2.5 27 10/22/18 Tom Addition Practice on Unity 3 3 19.5
10/23/18 Tom Messaged Reddit VR Board Looking for Feedback on How to Perform the Switch Function 1 1 20.5 10/22/18 Jake Research live video feed to Vive 1.5 1.5 14.5 Statement of Team Goals: Since our group has spent significant time conducting research on camera transition and screen display mechanisms, we would like to set a concrete goal for each team to construct a flowchart and/or raw code that would allow the HTC Vive to achieve the that group s intended functionality. We plan to update and reference Kevin Ponto and Taylor of the Makerspace along the way to make sure these outlines are feasible and theoretically functional. Each group will have done appropriate testing on the HTC Vive to act as primary feedback on their progress. Individual Goals: - Sam : In my research, it is hard to know what code will or will not work without having the headset or an experienced user immediately available. I d like to spend a significant amount of my time this next week going to the Makerspace during Taylor s shifts to test the camera transition suggested to us both by the HTC Vive community and by Kevin Ponto. - Sam Schini: Next week, we will further progress our project through the programming and assembly of endoscopic communication with our chosen headset. We will also continue to work closely with our client following up on patents/electrochromic glass. - Sam Peters: Next week I would like to combine the two aspects of our software application. We currently have a video aspect and a switch aspect, and I would like to work on integrating them together. - Tom Geissler: I will continue scrubbing online VR discussion boards looking for bits of code to help further our understanding of how to code the view transition. We have already looked extensively into Reddit the the HTC Vive communities, but it might be wise to see if there is overlap between the Vive and other similar hreadsets. I will join the rest of my group (, Neisen) in testing out code in the Makerspace when we get to that point. - Jake Cohn: Next week, I will be working in my sub group on getting the live feed from the endoscope to the HTC Vive seamlessly so that the client can actually see what he s doing in real time. This is the most fundamental step of our project and needs to be accomplished soon. - Josh Niesen: Now that I have a background in the Unity software and possible means of accessing the gyroscope in the headset, we will work in the makerspace to create an app that accesses these values. Displaying these values will then give rise to experimenting and trying to use these values to control the headset. Therefore, this will be focused toward writing code to access the gyroscope position values. Difficulties:
Having a hands-free transition between views seemed to be a unique dilemma on the discussion boards we searched. Many of the responses we got referenced the double-tapping mechanism for switching views, which did not further our progress. A few responses offered broad suggestions for plugins or methods we could use to achieve the transition, but recommended we do our own personal research to affirm whether these methods would work. While these responses did not give us the immediate, full-circle procedure which would have been ideal, we feel as though they have put us on the right track moving forward. Project Schedule/Timeline: Task September October November December 7 14 21 28 5 12 19 26 2 9 16 23 30 7 14 Project R & D Current Products X X X Scope Based Surgery X X X X Virtual Reality X X X X Mechanical Solutions X X X X X Materials X X X Cost Estimation X Prototyping Documentation Progress Reports X X X X X X Preliminary Presentation X X X Preliminary Deliverables X X X Final Poster Final Deliverables Meetings Team X X X X X X X
Client X X X Advisor X X X X X X Website Update X X X X X X X Filled boxes = Projected timeline X = Task was worked on or completed Expenses: We have not incurred any expenses yet.