OldMeetingMinutes

Chair for Computer Aided Medical Procedures & Augmented Reality
Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality

THIS WEBPAGE IS DEPRECATED - please visit our new website

Meeting Minutes Archive

Back to Meeting Minutes.

August, 10 2006

Amir Tadayon

  • explain the evaluation problem and its solutions,i.e
    • Direct solution
    • Forward and Backward algorithm
  • define our HMM model and its elements
  • test our Forward algorithm code by simple examples
  • in next week write some code for Viterbi algorithm

Ahmad

  • Further evaluation of Mussak surgeries --> develop mapping between feussner and mussak surgeries (results expected for next meeting)
  • Evaluate a concept on improvement of mapping by combining DTW with HMMs (--> dynamic adjustment of the distance function)
  • Paper presentation: prepare a presentation on HMMs (probably together with Amir)

Nicolas & Uli

  • Meet Nassir in a private meeting for the discussion on the usage of Computer Vision for instrument/tool classification
  • Paper presentation: prepare a presentation on "Dynamic Bayes Nets" (DBNs) including a comparison to HMMs (advantages/disadvantages)

Michael & Michael

  • Simulation of online synchronisation by creating LCSS-calculated warppaths for our GUI (results expected for next meeting)

Martin

  • Goal for July: record new surgeries with Feussner
  • Preparation work for new recordings:
    • Create a complete list of all possible recordable OR signals, maybe together with Armin (e.g. sensor belt for movement, anaesthetic data etc.)
    • Arrange a meeting with Armin Schneider (and maybe Prof. Feussner) for discussions on data acquisition possibilities
    • Possible meeting dates: 22. or 23. June (after Nassir's return to Germany


May, 09 2006: Distribution of tasks

Ahmad

  • Read out instrument state vectors from Mussak and synchronize it vs. surgeries from Feussner
  • Transfer all synchronization code into C++ (hopefully together with Tobias Blum)
    • get DTW class from Tobias and Michael/Tobias
  • Machine learning (long-term):
    • learn how instruments are used in different phases
    • find an automatic correlation between the instruments of Feussner and Mussak
  • No more acquisition of surgery data in near future until Feussner agrees and is ready to give more signals in collaboration with Armin

Amir

  • Will be working on workflow phase recognition with Hidden Markov Models (HMMs)
  • Should be provided with documentation and papers about the workflow project, about the DTW and about HMMs (--> Ahmad)

Michael & Michael

  • Integrate online synchronization into Workflow GUI

Tobias Blum

  • Help to transfer Ahmad’s synchronization code into C++
  • Confidential project at Siemens: extremely accurate and fast C-Arm for which an anti-collision software is planned for the OR
    • C-Arm equipped with 6 cameras (including digitization)
    • a proof of concept will probably be provided in cooperation of CAMP chair
    • Integration of knowledge about workflow will be extremely useful in this scenario and might be a first commercial application for our project

Uli & Nicolas

  • Prepare a set of video sequences for testing
  • Some easy and some more difficult cases of video sequences for testing and evaluation

Publication tasks

  • Monitor the work of Lemke
    • He has full visualization and simulation of OR, simulation of OR
    • Publications on MMVR conference (Medicine Meets Virtual Reality)
  • If MICCAI work paper is accepted, we have to get in contact with Lemke and create a workflow work-group
  • Journal Publication tasks:
    • Take up work after paper was accepted at MICCAI
    • Feussner: Journal of Laparoscopy
      • Basically, we have to write the paper but Feussner also has to contribute 1-2 sections.
      • A lot of medical background is required! (--> Feussner)
    • Our desired Journal Publication : IEEE Journal of Transactions


Feb, 01 2006:

Hardware (Ralf)

PROGRESS

  • positions in the OR top view can be displayed, but the data from different cameras is not synchronised yet

NEXT STEPS

  • documentation of GUI software
  • fusion of position data from 3 cameras in the GUI

Offline synchronization (Ahmad)

NEXT STEPS

  • create a better average surgery:
    • it should change incrementally (if phases/different workflows are introduced or dropped)
    • it should contain all possible workflow steps, represented by respective probabilities
  • implement Tobias idea of fusing different warp paths in order to obtain an overall average

Online synchronization (Michael)

PROGRESS

  • online synchronization performs better than offline synchronization when it comes to recognition of phases --> online DTW is performing quite well

NEXT STEPS

  • evaluate the performance of workflow phase recognition when an arbitrary surgery is synchronized with the average


Jan, 20 2006:

Hardware (Ralf)

PROGRESS

  • OR top view has been set up in the Worklow GUI, so far no positions are displayed

NEXT STEPS

  • ERCP
  • record other surgeon (Dr. Mussak, Prof. Prinz)
  • User Interface -> Tracking --> position drawing next week
  • displaying data by clicking on an icon in the GUI (e.g. receiving the instrument track by clicking on the instrument icon with the future idea of clicking on pulse monitor for anaesthetic data)
  • define list of signals for future recordings
  • support Armin in wrapping up his thesis --> official help by Nassir and involvement in Workflow project

Software (Ahmad)

PROGRESS

  • attempt of combining the warp paths of surgeries in a 3D cube and relating them onto each other by projection

NEXT STEPS

  • investigate in MD-DTW (multidimensional DTW) and MD Dynamic Programming ( Bellman & Dreyfus, "Applied dynamic programming" )
  • implement 3D DTW in a cube with a distance function covering all three test surgeries
  • medium-term goal: evaluate synchronization quality through data simulation - prerequisite for this: create theoretic test data set

Next Meeting

  • Friday, 27 Jan 2006, 12am, CAMP conference room


Jan, 16 2006:

Hardware (Ralf)

PROGRESS

  • Progress so far: see Bachelor thesis (PDF)

ROADMAP

  • Visualize a top view of the OR in the Workflow GUI
  • Derive a method of using the OR lamp as a common reference point (marker) in all external videos. This would help to determine the position of the cameras in the OR. In addition to that, the camera positions for the external videos recorded so far could be calculated.
  • In future (1 year+), an OR will probably be designed in Rechts der Isar (according to Feussner and Nassir) which will feature several cameras mounted to the ceiling.
  • Objectives for future surgery analyses are:
    • The ability to do any laparoscopic surgery
    • The realization of an adaptive user interface (offline first)
    • The long-term goal: an online adaptive user interface (--> online instrument detectiona and online surgery synchronization necessary!)

Software (Ahmad)

PROGRESS

  • Progress so far: see Bachelor thesis (PDF)

FUTURE OBJECTIVES

  • Create an average curve which is independent of a reference surgery
  • Therefore: an average warp path has to be created which contains all the information from all surgery warp paths

NEXT STEPS

  • Create three average surgeries out of the 3 test surgeries, where each surgery is taken as a reference once
  • Try to visualize correspondences between the 3 test surgery warp paths in a 3D cube

Next Meeting

  • Friday, 20 Jan 2006, 12am, CAMP conference room


Sep, 22 2005:

Hardware (Ralf)

PROGRESS

  • Two surgeries were recorded on Friday, 16.9.; all signals were obtained except the sound of the HF generator - reason: the microphone was attached to the camcorder recording the laparoscopic image, but the camcorder does not record sound when a video track is recorded through the S-Video interface
  • Importing and re-coding of surgery video tracks has started, but will take significant time.
  • (work on the ISMAR project)

NEXT STEPS

  • Ensure HF sound recording for the next surgeries
  • Further import and re-code the video tracks
  • (Support Tobias with the ISMAR demo)

Software (Ahmad)

PROGRESS

  • laparoscopic videos of both surgeries have been evaluated
  • synchronization with the DTW gave good results
  • The workflow data track of surgery 2 was synchronized with video tracks using the time offsets from the stopwatch.
  • Possible method for recovering the workflow: surgery phases can be assigned to the average surgery; other surgeries inherit this "global surgery phase" from the average; more detailed phases can be named for each single surgery using their respective instrument tracks; certain instructions (switching on/off the lights, calling the next patient) can also be assigned to the average

NEXT STEPS

  • Program a simplified GUI mockup in Matlab, since Ralf is assigned to support Tobias with the ISMAR demo. The GUI should display the workflow phase, progress bar, "speed-meter" and the instruments in use.


Sep, 15 2005:

Hardware (Ralf)

PROGRESS

  • Real surgeries were delayed - two surgeries scheduled for Friday, 16.9.
  • Test-Run on Tuesday, 13.9.: rough layout for camera setup was determined, S-Video interface for laparoscopic video was tested, microphone was attached to the laparoscopic camera
  • Introduction in QT started

NEXT STEPS

  • Continue learning QT and working on the GUI

Software (Ahmad)

PROGRESS

  • A structural average for the three test surgeries was created showing similarities between the three surgeries and common workflow phases.
  • An automated weighting mechanism for the importance of certain instruments in the DTW synchronization was implemented. The iterative method is described by Kassidas et al. (PDF).

NEXT STEPS

  • Optimizing the code
  • Start working on the recovery of workflow in order to display surgery phases in the final GUI


Sep, 08 2005:

Hardware (Ralf)

PROGRESS

  • A concept for the final GUI was created. The mockup can be found here.
  • A test run was performed in the CAMP AR lab. Two videos and a laparoscopic video were recorded. The laminated markers were attached with double-sided tape.

NEXT STEPS

  • Prepare the recording of a real surgery on Monday next week.
  • Start creating the GUI with QT

Software (Ahmad)

PROGRESS

  • Average warppath was created in two ways: 1) transforming the warppath into a 45° coordinate system and calculating an average there; 2) taking the arithmetic average of infinite segments and calculating an average
  • All 3 test surgeries were synchronized onto the reference surgery; method 2) for the creation of an average warppath yielded better results

NEXT STEPS

  • Create a structural average
  • Assist in preparing the real surgery next week


Sep, 01 2005:

Hardware (Ralf)

PROGRESS

  • Discovered and corrected a bug in the tracking software which results in a faulty scaling of dimensions
  • Started to implement a Kalman filter from Martin's Pose Tool in order to smooth the rotational data obtained during the marker tracking
  • Ordered 2 further progressive cameras

NEXT STEPS

  • Laminate markers with a matt finish
  • Test different modes of the camera (different shutter times and focal lengths) and their effects on image and tracking quality
  • Preform a complete test run in the CAMP AR lab next week
  • Draft a concept for the final GUI together with the team

Software (Ahmad)

PROGRESS

  • Discovered and corrected a bug in the DTW algorithm: the warp path now does not lose data points from the original function
  • Implementation attempts of the multi-surgery synchronization using DTW failed; reasons:
    • different number of measurement points for different surgeries prohibit a straight calculation of the time warp average
    • interpolation impossible due to infinite slope portions in the warping path
  • Paper by Kassidas(98) suggests an implementation of subsequent symmetric and asymmetric DTW in order to synchronize multiple trajectories of unequal lengths

NEXT STEPS

  • Discuss different methods for multiple-surgery synchronization using DTW and implement the best solution
  • Support the test run in the CAMP AR lab
  • Draft a concept for the GUI together with the team


Aug, 25 2005:

Hardware (Ralf)

PROGRESS

  • Position data interpolated
  • Tracking algorithm de-bugged and further adjusted --> Accuracy and robustness of offline computation sufficient for the moment
  • Progressive camera arrived

NEXT STEPS

  • Compare performance of tracking algorithm when applied to videos from progressive vs. interlaced camcorders
  • Merge position data from different video tracks

Ideas / Future Prospects

  • Implement further robustness in the tracking algorithm by allowing deformation of the marker; proposed method: randomized trees of marker deformation samples according to the work of Vincent Lepetit and Lagger --> no cardboard would be necessary for the markers
  • Use the CAMP logo for tracking

Software (Ahmad)

PROGRESS

  • Multi-dimensional instrument vectors implemented; arbitrary distance functions and instrument weightings can now be used for the calculation of the DTW matrix; usage of XOR and Euclidian distance functions so far deliver the same warping paths; Different weightings of the instruments so far do not have any effect on the warping path
  • Proposal for an optimized backtracking in the DTW: try to cross valleys in the DTW matrix as diagonally as possible
  • A two-step method for the synchronization of multiple surgeries by Wang and Gasser: create a time scale average for all the surgeries and then create a structural average; combine both in order to obtain an average surgery
  • Kickoff presentation for Friday, Aug, 26 2005 finished

NEXT STEPS

  • Try to implement the multi-surgery synchronization with DTW average
  • Adapt backtracking algorithm for a diagonal crossing of valleys in the DTW matrix


Aug, 18 2005:

Hardware (Ralf)

PROGRESS

  • Tracking under real-life conditions successful: different filter algorithms ensure a registration in worse-than-real lighting conditions
  • Extracted position and rotation data from marker identification and exported into txt-file
  • 3 progressive color sensor cameras ordered

NEXT STEPS

  • Synchronize different video streams and data streams
  • Merge position data from different cameras for a tracking that is as complete as possible
  • Interpolate position data

Software (Ahmad)

PROGRESS

  • Synchronization attempts with multi-dimensional instrument tracks
  • Automatic ranking of instrument importance due to number and duration of usage phases
  • Synchronizations with automatic weighting of instrument importance
  • Import of position data into MATLAB

NEXT STEPS

  • Synchronization with multi-dimensional instrument vectors; experiments with different DTW distance functions and weighting vectors
  • Displaying of "surgery speeds"
  • Synchronization of other data than instruments


Aug, 11 2005:

Hardware (Ralf)

PROGRESS

  • Software tracking of markers working
  • Tracking algorithm is so far only working offline
  • Synchronisation of multiple video streams with a clapper board
  • Two cameras facing each other can be synchronized with a double-sided marker

NEXT STEPS

  • Testing of tracking system under real-life conditions (esp. lighting with and without a real surgery lamp)
  • Extraction of position and rotation data from marker identification

REQUIRED HARDWARE

  • 3 progressive cameras with color sensors

Software (Ahmad)

PROGRESS

  • Test data generated from three manual protocols during laparoscopic cholecystectomy
  • DTW algorithm adjusted to underlying problem
  • Mockup of surgery synchronization is working even with a simple coding of the laproscopic instruments

NEXT STEPS

  • Consider a more complex coding of laparoscopic instruments: multi-dimensional signal space with binary signals (e.g. "Intrument in use/not in use")
  • Research into synchronization of more than two surgeries
  • Research into automatic feature detection of surgical workflow


Aug, 04 2005:

Definition of signals which will be recorded during surgery:

Signal-# Signal Signal Type Recording device Purpose
01 Video of operation 2-3 video streams 2-3 DV cameras * Tracking of number and position of operation staff
* Information about ambience lighting
02 Instrument used Discrete states Manual detection (offline) from signal (4) * Identificaiton of operation phase
* Progress of operation
* Detection of operation complexity
03 HF-Mode Audiostream Microphone attached to HF-generator * Detection of cutting and coagulation phases
* support with identification of operation phases
* Determination of operation complexity
04 Laparoscopic image Videostream Video switch at Laparoscope device * Illustration of laparoscopic surgery
* Manual identification of laparoscopic instruments
05 Ambience Lighting Manual Identification from (1) Will be (hopefully) controlled by software in a later stage
06 Voices and voice commandos Audiostream Directional microphone attached to one of the DV cameras * Integration into GUI
* Correlation of commandos and events can be observed


Jul, 27 2005:

Setting of objectives and schedule


Edit | Attach | Refresh | Diffs | More | Revision r1.2 - 29 Jul 2007 - 23:07 - AhmadAhmadi

Lehrstuhl für Computer Aided Medical Procedures & Augmented Reality    rss.gif