The goals of "General Surgery Training and Simulation with Augmented Reality" project are to develop a laparoscopy simulator equipment along with its software, and meet with the training and application needs of General Surgery operations.

Uludağ University Faculty of Medicine, Department of General Surgery was our partner during this project. First, an artificial organ structure is created with holograms, then virtual and real platforms are synced via a holographic glass application. Both training and examinations are included in the simulation program to help the user improve their skills in diagnosis, planning and operation stages.

Various studies were evaluated during the researches made as part of the project such as training in medical applications, planning solutions visible patients etc. After the research stage was complete, the details of the project were planned according to our infrastructure and resources, and the benchmarks, the topics of applied scenarios and hardware (mechanical, electronic, informatics etc.) were determined as well. Upon the collaborations done with specialists from General Surgery department, procedures of diagnosis, planning, alternative work flow structures of surgical operations, training and exam were complete.

In studies carried out with Prof. Dr. Ekrem Kaya and Researcher Dr. Murat Şen from Uludağ University, Department of General Surgery, the data from patients who underwent/needed a liver lesion were used. Custom surgical plans were made using various diagnosis data such as MR, EKG, EMG, BT etc. taken from the patients who were treated in general surgery. As the complications and illnesses were customized due to custom patient data, the variety of application scenarios increased accordingly. 2D data of the tissues/organs were listed using MR, EKG, EMG, BT etc. while 3D model designs (holograms) were made using Unity 3D. Recreating these models not only helped to build 3D visuals of the tissues/organs, but it also allowed holograms to sense real tissue structures and imitate them.

After detecting the movements of the simulator equipment with a gyroscope integration, the reality perception was enhanced due to the feedback from software applications. The position, distance and multiple axis perceptions of the used surgical tools were enhanced with sensors and electro mechanical solutions. Operator controls were created by confirming between the perceived movements and the scenarios. Delivering the visuals and reality structures to personal use was achieved with Hololens Augmented Reality program. These software developments allowed the simulation to be used during the training as well as surgical planning. The training and the exam options in the simulation allowed the track of possible complications and illnesses in a human body while a system checking the margin of error was developed with the gyroscope sensors integrated into the system.

We utilized from 3D design softwares and 3D industrial scanning as well as radiology devices to develop the holograms. 3DMAX, Unity 3D, Visual Studio and Maya softwares were used for holograms and augmented reality while the hardware product developments were supported with Ardunio/Raspberry Pi module and various sensors. Communications systems utilized Bluetooth and Wireless. The interfaces, mechanical systems, electronic systems etc. developed during the prototype stage of designs are integrated both as hardware and software. Tests were carried out to determine the system integrity. Then, type tests and controls were concluded to guarantee the norm suitability of the developed prototype.