Research ArticleOpen AccessGoogle Scholar indexed
A Prototype AI Surgical Assistant for Real-Time Consultation during Laparoscopic Surgery
Medical School, University of Nicosia, Nicosia, Cyprus
IASO Group of Hospitals, Athens, Greece
IASO Group of Hospitals, Athens, Greece
IASO Group of Hospitals, Athens, Greece
Athens Medical Center, Athens, Greece
- 1 Medical School, University of Nicosia, Nicosia, Cyprus
- 2 IASO Group of Hospitals, Athens, Greece
- 3 IASO Group of Hospitals, Athens, Greece
- 4 IASO Group of Hospitals, Athens, Greece
- 5 Athens Medical Center, Athens, Greece
Copy link · social · email
Abstract
Recent advancements in generative AI and large language models (LLMs) have sparked new opportunities in surgical innovation. We present our prototype AI Surgical Assistant Prototype System, integrating real-time vision support for streaming video of the operative field, advanced speech recognition, multilingual natural voice interaction, both long- and short-term memory simulation, and customizable behavioral profiles. Testing showed that the system exhibited contextual awareness from visual feedback in 38% of instances, ability for verbal interaction, and dynamic memory logging throughout the procedure. This feasibility study suggests that real-time, context-aware AI support is technically viable in the OR and may serve as a basis for future clinical models.
KeywordsAI Surgical AssistantReal-Time Video AnalysisGPT-4o in SurgeryAI in the ORVoice-Activated Intraoperative Support
- Hashimoto, D.A., Rosman, G., Rus, D. and Meireles, O.R. (2018) Artificial Intelligence in Surgery: Promises and Perils. Annals of Surgery , 268, 70-76. https://doi.org/10.1097/sla.0000000000002693
- Maier-Hein, L., Vedula, S.S., Speidel, S., Navab, N., Kikinis, R., Park, A., et al . (2017) Surgical Data Science for Next-Generation Interventions. Nature Biomedical Engineering , 1, 691-696. https://doi.org/10.1038/s41551-017-0132-7
- OpenAI (2023) GPT-4 Technical Report. arXiv:2303.08774. https://arxiv.org/abs/2303.08774
- Radford, A., Kim, J., Xu, T., et al . (2022) Robust Speech Recognition via Large-Scale Weak Supervision. Whisper by OpenAI. https://openai.com/research/whisper
- Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y. and Elhoseiny, M. (2023) Minigpt-v2: Large Language Model as a Unified Interface for Vision-Language Multi-Task Learning. arXiv:2310.09478.
- Chen, Z., Guo, Q., Yeung, L.K.T., Chan, D.T.M., Lei, Z., Liu, H., et al . (2023) Surgical Video Captioning with Mutual-Modal Concept Alignment. In: Greenspan, H., et al ., Eds., Medical Image Computing and Computer Assisted Intervention — MICCAI 2023, Springer, 24-34. https://doi.org/10.1007/978-3-031-43996-4_3
- Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I. and Xing, E.P. (2023) Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/
- Das, A., Khan, D.Z., Williams, S.C., Hanrahan, J.G., Borg, A., Dorward, N.L., et al . (2023) A Multi-Task Network for Anatomy Identification in Endoscopic Pituitary Surgery. In: Greenspan, H., et al ., Eds., Medical Image Computing and Computer Assisted Intervention — MICCAI 2023, Springer, 472-482. https://doi.org/10.1007/978-3-031-43996-4_45
- Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J. and Maier-Hein, K.H. (2020) NNU-Net: A Self-Configuring Method for Deep Learning-Based Biomedical Image Segmentation. Nature Methods , 18, 203-211. https://doi.org/10.1038/s41592-020-01008-z
- Szeliski, R. (2010) Computer Vision: Algorithms and Applications. Springer Science & Business Media. https://doi.org/10.1007/978-1-84882-935-0
- Chen, Z. and Luo, X. (2024) VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons. arXiv: 2405.08272
- Hirides, S., Hirides, P., Kalliopi, K. and Hirides, C. (2024) Artificial Intelligence and Computer Vision during Surgery: Discussing Laparoscopic Images with ChatGPT4—Preliminary Results. Surgical Science , 15, 169-181. https://doi.org/10.4236/ss.2024.153017