Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision

David Z. Li; Masaru Ishii; Russell H. Taylor; Gregory D. Hager; Ayushi Sinha

doi:10.1007/978-3-030-60946-7_6

Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision

David Z. Li, Masaru Ishii, Russell H. Taylor, Gregory D. Hager, Ayushi Sinha

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. We use three different methods to manipulate these latent representations in order to predict tool presence in each frame. Our fully unsupervised methods can identify whether endoscopic video frames contain tools with average precision of 71.56, 73.93, and 76.18, respectively, comparable to supervised methods. Our code is available at https://github.com/zdavidli/tool-presence/.

Original language	English (US)
Title of host publication	Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings
Editors	Tanveer Syeda-Mahmood, Klaus Drechsler, Hayit Greenspan, Anant Madabhushi, Alexandros Karargyris, Cristina Oyarzun Laura, Stefan Wesarg, Marius George Linguraru, Raj Shekhar, Marius Erdt, Miguel Ángel González Ballester
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	54-63
Number of pages	10
ISBN (Print)	9783030609450
DOIs	https://doi.org/10.1007/978-3-030-60946-7_6
State	Published - 2020
Event	10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 - Lima, Peru Duration: Oct 4 2020 → Oct 8 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12445 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020
Country/Territory	Peru
City	Lima
Period	10/4/20 → 10/8/20

Keywords

Endoscopic video
Maximum mean discrepancy
Representation learning
Tool presence
Variational autoencoder

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-030-60946-7_6

Cite this

Li, D. Z., Ishii, M., Taylor, R. H., Hager, G. D., & Sinha, A. (2020). Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision. In T. Syeda-Mahmood, K. Drechsler, H. Greenspan, A. Madabhushi, A. Karargyris, C. Oyarzun Laura, S. Wesarg, M. G. Linguraru, R. Shekhar, M. Erdt, & M. Á. González Ballester (Eds.), Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings (pp. 54-63). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12445 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-60946-7_6

Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision. / Li, David Z.; Ishii, Masaru ; Taylor, Russell H. et al.
Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. ed. / Tanveer Syeda-Mahmood; Klaus Drechsler; Hayit Greenspan; Anant Madabhushi; Alexandros Karargyris; Cristina Oyarzun Laura; Stefan Wesarg; Marius George Linguraru; Raj Shekhar; Marius Erdt; Miguel Ángel González Ballester. Springer Science and Business Media Deutschland GmbH, 2020. p. 54-63 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12445 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Li, DZ, Ishii, M , Taylor, RH , Hager, GD & Sinha, A 2020, Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision. in T Syeda-Mahmood, K Drechsler, H Greenspan, A Madabhushi, A Karargyris, C Oyarzun Laura, S Wesarg, MG Linguraru, R Shekhar, M Erdt & MÁ González Ballester (eds), Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12445 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 54-63, 10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020, Lima, Peru, 10/4/20. https://doi.org/10.1007/978-3-030-60946-7_6

Li DZ, Ishii M , Taylor RH , Hager GD, Sinha A. Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision. In Syeda-Mahmood T, Drechsler K, Greenspan H, Madabhushi A, Karargyris A, Oyarzun Laura C, Wesarg S, Linguraru MG, Shekhar R, Erdt M, González Ballester MÁ, editors, Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 54-63. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-60946-7_6

Li, David Z. ; Ishii, Masaru ; Taylor, Russell H. et al. / Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision. Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. editor / Tanveer Syeda-Mahmood ; Klaus Drechsler ; Hayit Greenspan ; Anant Madabhushi ; Alexandros Karargyris ; Cristina Oyarzun Laura ; Stefan Wesarg ; Marius George Linguraru ; Raj Shekhar ; Marius Erdt ; Miguel Ángel González Ballester. Springer Science and Business Media Deutschland GmbH, 2020. pp. 54-63 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{2c4c51cc864e4455ac5034843b05c7be,

title = "Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision",

abstract = "In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. We use three different methods to manipulate these latent representations in order to predict tool presence in each frame. Our fully unsupervised methods can identify whether endoscopic video frames contain tools with average precision of 71.56, 73.93, and 76.18, respectively, comparable to supervised methods. Our code is available at https://github.com/zdavidli/tool-presence/.",

keywords = "Endoscopic video, Maximum mean discrepancy, Representation learning, Tool presence, Variational autoencoder",

author = "Li, {David Z.} and Masaru Ishii and Taylor, {Russell H.} and Hager, {Gregory D.} and Ayushi Sinha",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 ; Conference date: 04-10-2020 Through 08-10-2020",

year = "2020",

doi = "10.1007/978-3-030-60946-7_6",

language = "English (US)",

isbn = "9783030609450",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "54--63",

editor = "Tanveer Syeda-Mahmood and Klaus Drechsler and Hayit Greenspan and Anant Madabhushi and Alexandros Karargyris and {Oyarzun Laura}, Cristina and Stefan Wesarg and Linguraru, {Marius George} and Raj Shekhar and Marius Erdt and {Gonz{\'a}lez Ballester}, {Miguel {\'A}ngel}",

booktitle = "Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings",

address = "Germany",

}

TY - GEN

T1 - Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision

AU - Li, David Z.

AU - Ishii, Masaru

AU - Taylor, Russell H.

AU - Hager, Gregory D.

AU - Sinha, Ayushi

PY - 2020

Y1 - 2020

N2 - In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. We use three different methods to manipulate these latent representations in order to predict tool presence in each frame. Our fully unsupervised methods can identify whether endoscopic video frames contain tools with average precision of 71.56, 73.93, and 76.18, respectively, comparable to supervised methods. Our code is available at https://github.com/zdavidli/tool-presence/.

AB - In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. We use three different methods to manipulate these latent representations in order to predict tool presence in each frame. Our fully unsupervised methods can identify whether endoscopic video frames contain tools with average precision of 71.56, 73.93, and 76.18, respectively, comparable to supervised methods. Our code is available at https://github.com/zdavidli/tool-presence/.

KW - Endoscopic video

KW - Maximum mean discrepancy

KW - Representation learning

KW - Tool presence

KW - Variational autoencoder

UR - http://www.scopus.com/inward/record.url?scp=85092616710&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85092616710&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-60946-7_6

DO - 10.1007/978-3-030-60946-7_6

M3 - Conference contribution

AN - SCOPUS:85092616710

SN - 9783030609450

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 54

EP - 63

BT - Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings

A2 - Syeda-Mahmood, Tanveer

A2 - Drechsler, Klaus

A2 - Greenspan, Hayit

A2 - Madabhushi, Anant

A2 - Karargyris, Alexandros

A2 - Oyarzun Laura, Cristina

A2 - Wesarg, Stefan

A2 - Linguraru, Marius George

A2 - Shekhar, Raj

A2 - Erdt, Marius

A2 - González Ballester, Miguel Ángel

PB - Springer Science and Business Media Deutschland GmbH

T2 - 10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020

Y2 - 4 October 2020 through 8 October 2020

ER -

Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this