VR Headset for Endoscopy and Microsurgery

Size: px

Start display at page:

Download "VR Headset for Endoscopy and Microsurgery"

Dortha Shields
5 years ago
Views:

1 VR Headset for Endoscopy and Microsurgery Client: Dr. Azam Ahmed Advisor: Mr. Willis Tompkins, Dr. John Puccinelli Team Members: Tom Geissler Team Co-Leader Sam Peters Team Co-Leader Sam Communicator Sam Schini BSAC Jake Cohn BPAG Josh Niesen BWIG Reporting Period: October 25th through November 1st, 2018 Problem Statement: Endoscopic surgeries have become increasingly prevalent in the operating room along with the visualization techniques used to perform them. Traditionally, large monitors have been used to display the images from the surgical tools, however, these are bulky and don t provide as immersive of an experience as other methods do. This problem has been partially addressed by using virtual reality headsets. One major limitation of these however, is that they do not allow the surgeon to see anything outside of the operative view. This is a problem as the surgeon will have to remove the headset everytime he or she has to change instruments, or perform macroscopically. The goal of this project is to design a VR headset that would allow the surgeon to transition from the operative view to normal view without the use of the surgeon s hands. Summary of Team Role Accomplishments: - Team: The group divided into two subgroups: one focusing on the video transfer from the endoscope to Unity, and the other on the view transition from VR to environmental. The subgroups each met at the Makerspace to walkthrough the basics of Unity and get video recommendations for learning resources so each member could gain some familiarity with the program. - Leaders: Continued to lead subteams in accomplishing our task in the project. - Communicator: Distributed the weekly progress report to our client and advisor. - BWIG: The project page was updated with team s progress report and the meeting notes were posted in lab archives. - BPAG: Still waiting on our proof of concept before we purchase thousands of dollars of equipment for our project. - BSAC: Voiced opinion on shop fee at meeting with faculty and prepared to attend and contribute opinion to next week s meeting. Design Accomplishments:

2 With the majority of the group still unfamiliar with Unity and the headset, our group took one night to meet with Taylor and get his recommendation for materials (primarily Youtube videos) to bring our knowledge up to a serviceable level. In addition, both subgroups were tasked with conducting research to accumulate code snippets or walkthroughs on how to reach their separate goals. These snippets will provide a framework which we can modify in the upcoming weeks to meet our client s specific needs. Activities: Date Name Task Time (hours) Weekly Total Semester Total 10/31/2018 Team Meet with Taylor in the Makerspace for a Unity Introduction and HTC Vive Tinkering Schini Schini Worked with Unity and researched video conversion. Contacted potential experts to assist with video streaming obstacles /31/2018 Jake Worked with Unity trying to find way to get video into the program 11/1/2018 Jake Contacting people for assistance with the video transfer 11/1/2018 Jake Watched videos on VR that Taylor recommended /28/2018 Sam Research (IVRTrackedCamera, OpenVR) /1/2018 Sam Watched and took notes on the Unity/Steam VR Basics Videos Recommended by Taylor Research (Camera Stabilization and Accessing HMD Rotation Values) /30/2018 Sam Peters Researched video IO for VR applications /31/2018 Josh Watched the basic unity coding videos that Taylor from the Makerspace suggested we watch and took notes 11/1/2018 Josh Worked on progress report updates and updating website

3 10/28/18 Tom Unity Syntax research /29/18 Tom Steam VR to Unity research Statement of Team Goals: The video displaying subgroup will use the next week to reach out to experienced members of UW Madison s computer science department and VR headset research groups. These resources will primarily be helpful in decoding the information we received from Miles Green on the endoscope s output specs and determining the best means to display a still frame in Unity. The view transitioning subgroup will continue to gather code segments from online VR blogs that have attempted to perform a similar function. This group will test the code on the HTC Vive Pro in the Makerspace, making adjustments as necessary. Every team member is expected to watch all of the videos recommended by Taylor (our Makerspace resource) to become familiar with Unity. We would like to set aside some additional time to prepare a video demonstration of Unity/the HTC Vive for the show-and-tell coming up in BME 200/300 on November 9th. Individual Goals: - Sam : Taylor provided us with a value blog for Unity this past week which included code geared towards the view transition (VR to environmental) that we have been looking for. In this upcoming week I am anticipating spending significant time at the makerspace to test and tweak this code while also surfing the internet for additional snippets. Tom proposed a specific method for coding the headset to transition views when a coding - Sam Schini: Next week, I will follow up with potential Unity/video conversion experts to solve our issues regarding transferring endoscopic video to a VR headset. I will continue to familiarize myself with Unity and research different video formats and compatibility. - Sam Peters: Next week I will reach out to the resources that my subteam has identified that could be helpful in displaying a live video feed. We were able to get a prerecorded video to display in Unity, however we are currently unsure of how we can interface with the camera to get it to stream live on the VR headset. - Tom Geissler: I spent this past week looking for specific unity syntax we might need for our project. This includes obtaining headset rotation values and being able to modify these values. Additionally, I looked into how to interface between Steam VR and Unity, as the function to switch the camera views is within Steam VR and we are running a unity program. - Jake Cohn: This past week, we struggled to find a way of conversion of the video. Next week we will focus on trying to get help with trying to get the video converted to a form viewable by the Unity software so that it can ultimately be transferred to the Vive for surgical use. - Josh Niesen: Based on my collaboration with Tom, Sam, and Taylor, it seems present that we will be able to access the gyroscope values and control the feed being displayed on the headset based on gyroscope orientation. Next I will conduct more research about quaternary coordinates on the headset along with attempting to find already written snips of code that will allow us to switch the feed displayed.

4 Difficulties: This week presented many difficulties. The subgroup focusing on video transfer hit a roadblock in the process of getting the video from the endoscope into the Unity platform. The endoscope s specifications which were given to our group by Dr. Ahmed present a lot of information, most of which we are unsure how to use. This subgroup plans on asking for help from experts in virtual reality and computer science. The second subgroup, focusing on the handsfree view transition from VR to environmental, made some strides this week by accumulating chucks of useful code from the internet and stumbling on a theoretical process for performing this function on Unity. However, the group struggled to navigate through the HTC Vive software in order to test the code. Taylor seemed hesitant to help our group until we are more knowledgeable about the technology we are using, so it is our hope that he will volunteer more of his time after every team member finishes watching the tutorial videos Taylor recommended. Project Schedule/Timeline: Task September October November December Project R & D Current Products X X X Scope Based Surgery X X X X Virtual Reality X X X X Mechanical Solutions X X X X X Materials X X X X Cost Estimation X X Prototyping X Documentation Progress Reports X X X X X X Preliminary Presentation X X X Preliminary Deliverables X X X

5 Final Poster Final Deliverables Meetings Team X X X X X X X X Client X X X Advisor X X X X X X X Website Update X X X X X X X X Filled boxes = Projected timeline X = Task was worked on or completed Expenses: We have not incurred any expenses yet.

VR Headset for Endoscopy and Microsurgery

VR Headset for Endoscopy and Microsurgery Client: Dr. Azam Ahmed Advisor: Mr. Willis Tompkins, Dr. John Puccinelli Team Members: Tom Geissler geissler2@wisc.edu Team Co-Leader Sam Peters speters9@wisc.edu