Automatic Feature Extraction in Wound Healing Videos A. Rodriguez D.M. Shotton O. Trelles N. Guil
April 2000 Technical Report No: UMA-DAC-00/06
Published in: The 6th RIAO Conf. on Content-Based Multimedia Information Access (RIAO’2000) Paris, France, April 12-14, 2000
University of Malaga Department of Computer Architecture C. Tecnologico • PO Box 4114 • E-29080 Malaga • Spain
Automatic Feature Extraction in Wound Healing Videos 1
Andrés Rodríguez1, David M. Shotton2, Oswaldo Trelles1and Nicolás Guil1 Computer Architecture Department, University of Malaga, 29017 Malaga, Spain 2 Department of Zoology, University of Oxford, Oxford OX1 3PS, UK {andresr, nico, ots}@ac.uma.es
[email protected]
Abstract We present an automatic feature extraction procedure for the analysis of time-lapse light microscopy wound healing videos, which are used to record the closure of in vitro wounds made in confluent monolayers of cultured epithelial cells under a variety of experimental conditions. Because of the size and complexity of the moving image data type, and hence the length of time required to undertake detailed manual analysis and annotation of the content of such videos, enormous efficiency gains may be achieved by the development of automated systems for the tasks of video content analysis and subsequent content-based querying.
Introduccion Extracting useful information from massive data files, such as those produced in the biological and biomedical area in the form of 3D microscopy images and videos, is becoming an increasingly important activity in many scientific and commercial domains. However, while the amount of image data being produced is exponentially increasing, our ability to absorb and process this information remains nearly constant. Manual analysis of biological multidimensional image data allows only a small part of its visual information to be abstracted, sometimes leading to an incomplete understanding of the biological processes being observed. Vast quantities of valuable information contained in the images may be lost because of time constraints and lack of objectivity in the human interpretation of content. On the other hand, automatic procedures for querying and recovering information from this type of video often fails to detect critical events. In a move to close this technological gap, we present in this paper an automatic feature extraction procedure for the analysis of wound healing videos. These are created using time-lapse light microscopy to record the closure of linear in vitro wounds scraped in confluent monolayers of cultured epithelial cells growing on glass coverslips. This experimental paradigm permits the study of directed cell migration under controlled experimental conditions, for example transient perfusion exposure of the cells to drugs that bring about cytoskeletal depolymerization (Lewis, 1993; Lewis and Shotton, 1994 and 1995), and has been used to investigate the normal cytoskeletal remodelling and the reorientation of the Golgi apparatus and the microtubule organizing centre (MTOC) that accompany cell migration (Kupfer et al., 1982; Eutereuer and Schliwa, 1992) Because of the size and complexity of the moving image data type (Boudier and Shotton, 1999a and 1999b), and hence the length of time required to undertake detailed manual analysis and annotation of the content of such videos, a successful automated approach would be highly desirable. In this context, the objective of this work has been to develop an automatic procedure for the generation of information (specific intrinsic metadata; Shotton, 2000) related to the area of the wound and the rate of wound closure. This information, properly stored and combined with traditional manually entered ancillary metadata detailing the specific conditions of the experiment, allows subsequent queries to be performed. We show that enormous efficiency gains may be achieved by the development of this automated system for the tasks of video content analysis and subsequent content-based querying.
System and Methods The three wound healing videos used in this study were recorded by Jonathan Lewis in David Shotton's lab using time-lapse light microscopy (Lewis, 1993; Lewis and Shotton, 1994). In these videos, each image (frame) represents the instantaneous state of the wounded epithelial monolayer along the space-time axis (see Figure 1). The original analogue videotape recordings were first digitized to give Motion-JPEG encoded digital videos in QuickTime format, with a screen resolution of 768 x 576 pixels, playing at an interlaced frame rate of 25Hz. Prior to our analyses, the spatial resolution of the videos was reduced to 384 x 288 pixels by 2 x 2 binning, to reduce computational time.
Figure 1: Frame gallery from one of the wound healing videos at different times during the experiment. Times are given as hh:mm:ss in the lower left corner of each frame, below the original date of the recording. The width of each video frame is approx. 300 :m. A clear and distinguishable difference in texture can be observed between the uniform character of the cellfree open wound and the regions of cellular monolayer on either side. The process of automated video analysis involves four stages. First, image processing techniques are employed to identify the wound boundaries in the digitized video. Next, the wound lengths and the wound area per unit length are computed for each video frame analysed. From these data, the rate of wound healing is calculated (measured as the rate of loss of wound area per unit length of wound). Finally, an automatic annotation procedure is activated to store this information in a searchable database where it may be used subsequently for the video query and retrieval system. Identification of wound boundaries In the first step of our analysis, the wound boundary has to be recognised. To this end, we use computer image processing techniques to distinguish the uniform character of the cell-free open wound from the regions of cellular monolayer on either side, where the image texture is quite different, being characterised by high frequency modulations of the image intensity (Figure 1). This was done on every tenth frame of the digital videos analysed, which were of varying lengths, containing from approx. 2800 to approx. 4500 frames. Segmentation of the uniform wound region from the rest of the image in these frames is undertaken by using a differential operator that calculates the changes in the first spatial derivative of the image (Guil et al. 1999). Further histogram equalization and image thresholding allow us to exclude high contrast zones, and result in a threshold that closely follows the margins of the ruffling wound edge cells.
This segmentation procedure is undertaken in the following stages: First, a differential operator (Sobel, 3x3 kernel size) is applied to calculate the value of the first derivative at each point. The resulting image is enhanced by scaling the histogram to improve the image contrast. This resulting image is first processed row by row, looking for long horizontal runs of adjacent pixels in which the intensity value is below an intensity threshold computed in the previous histogram scaling operation. Pixel runs below the intensity threshold that exceed a length threshold minLen are retained in a temporary output image entitled “Horizontal analysis of smooth areas”. The same strategy is then applied to each original enhanced video frame vertical column by vertical column, generating a temporary output image entitled “Vertical analysis of smooth areas”. Finally, both results are combined by an AND operation to filter out further non-suitable points, yielding an output image “Smooth areas”. After this initial step, an dilation process is applied to restore open wound areas accidentally deleted in previous steps (for instance, those produced when the wound edge is adjacent to the image of the central cross of the microscope’s fiducial graticule, that is positioned in a plane conjugate to the specimen plane). The sizes of the uniform regions that have been detected are then tested to eliminate those that fall below a minArea threshold, thus eliminating noisy regions. For this, uniform region extraction is carried out using a region growing algorithm based on the use of seeds. This process identifies and counts all the pixels belonging to each region (island), and then, by comparison with the minArea threshold, decides which regions must be eliminated. The image resulting from this processing is then subjected to a closing routine to eliminate holes within uniform regions. Determination of wound lengths and the wound area per unit length Once the image processing has been completed, we address the second step, namely the measurement of the wound lengths and the wound area per unit length. To do this, we first establish the dimensions of an internal bounding box within which to make our measurements. This internal bounding box, which is used as a safeguard to exclude end effects, is defined as a rectangle oriented parallel with the axis of the linear wound (computed by principal components), which is wider than the maximum wound width, and which lies fully within the video frame. The size and position of this bounding box is kept constant for all the frames of the video which are to be analysed (see Figure 2).
Figure 2: Bounding box definition used to measure the wound area. (A) Once the wound-object has been segmented within the video frame, the main orientation axes are determined by using principal components calculation. (B) In a second step, the outer-left and outer-right points of the object are detected (those with maximal distance to the main axis), and from these points two parallel lines to the main axis are drawn. (C) From the intersection point of these parallel lines with the border of the image frame, two new lines are drawn at right angles. In this way, an internal bounding box is computed automatically and is used to eliminate edge artefacts in the border of the image.
Determination of the rate of wound closure The rate of loss of wound area, measured in pixels lost per unit length of wound within this bounding box every tenth video frame, permits the rate of wound closure, and hence the effectiveness of the healing process, to be calculated from the slope of the data, with units of :m2 wound area lost per :m wound length per minute. Significant changes in this slope might reflect variations in the healing rate due to changes in the external environment of the cells, for example drug treatment. It should be mentioned that the particular videos analysed in this study were unusual in that, periodically during the original recording of these videos, the microscope optics were briefly switched under computer control from the normal transmission phase contrast mode in which the wound edges are visible, to the epifluorescence mode. This was done in order to observe the positions and reorientation of the fluorescently labelled Golgi apparatus in each of the migrating epithelial cells, which usually moves to a position in front of the nucleus (Kupfer et al., 1982). Since at that time David Shotton did not have facilities for separately recording the transmission and fluorescence images as separate digital files, both data types were recorded alternately on the single analogue time lapse videotape. In the digital analysis of these videos, all the video frames were treated equally. However, since the wound edges were not visible in those video frames recorded in fluorescence mode, the wound area measurements obtained from them using the above procedures were invalid. This led to artefactual values for the apparent wound areas computed for the fluorescence video frames. Normal videos recorded with continuous uniform illumination do not suffer from this artefactual noise. Therefore, in the case of these particular videos, an additional post-process was required to suppress this periodic noise in the data, before the true rate of wound healing could be determined. Abrupt apparent changes in the instantaneous wound closure rate were used to trigger a filtering procedure which eliminated the noisy data points from the record, changes greater than 20% in the rate being considered to be abrupt. The filtering out of such noisy measurements produced by such an instantaneous abrupt changes in the apparent wound closure rate involved the activation of a window counter. Wound area values giving rise to an apparent wound healing rate change of more that 20% were only allowed to remain in the data set if the new value for the healing rate persisted for at least 20 frames. In practice this never occurred because all the abrupt rate changes have been produced by transient noise. This procedure was successful in smoothing the very noisy raw wound area data from the automated analysis of these videos (see Figure 4, Results), demonstrating the power of this simple filtering method. Data entry into a searchable database The final step of the automatic procedure for feature extraction from the wound healing videos involves recording the acquired information in a searchable database. At this point we need to stress that the data we are interested in recording are of the type specific intrinsic metadata (for definition, see Shotton, 2000, and also the companion paper in these proceedings by the same authors entitled Automatic tracking of moving bacterial cells in scientific videos). Specific intrinsic metadata relate to visual features within the image data, describing the spatial positions of specific objects within images and the spatio-temporal locations of objects and events within videos (see Table 1). Obviously, additional text-based information concerning the video (title, experiment, date, format, etc) are also stored, and can be employed to locate videos using the traditional keyword-based paradigm.
Results In this section we present an evaluation of the proposed strategy for the automatic feature extraction in wound healing videos. Three separate time-lapse video microscopy experiments have been
analyzed, all recording the healing of NRK epithelial cell wounds under the same conditions in the absence of drugs. The results obtained in 1993 by manual analysis of the same original video data (Lewis, 1993; Lewis and Shotton, 1994 and 1995) have been used as a basis for comparison and validation of the methodology. Identification of wound boundaries, and determination of wound lengths and areas In the initial step of each analysis, the open wound area was successfully automatically segmented in each frame of the video, and the bounding box determined, as described in System and Methods (see Figure 3).
Figure 3: Feature extraction in a wound healing video. The wound healing process after manual wounding of a confluent epithelial cell culture are recorded in videos of the kind illustrated here. Two frames from the analysis of Video 4 are shown, the first at time 21:48:34 and the second at time 23:16:22. To measure the progressive loss of open wound area in this video, we used computer image processing techniques to distinguish the uniform character of the cell-free open wound in each frame from the regions of cellular monolayer on either side, where the image texture is quite different. The boundary line between the cells and the cell-free wound is shown in each panel (see text for details). Determination of the rate of wound closure Once the wound boundaries and the bounding boxes had been determined, the wound open areas and hence the rates of wound closure were automatically calculated. Figure 4a (left) shows the raw experimental data from every tenth frame of the three videos analysed, in which noisy measurements can be observed due to the unsuccessful automated attempts to locate the wound margins in the periodic fluorescence images. Using the filtering method described above, these spurious measurements were successfully eliminated from the data, generating the smooth filtered wound healing curves shown in Figure 4b. Comparison of the wound healing curves obtained by our automated procedure with the original data points derived by manual analysis of selected video frames at eleven time points during the healing of each wound shows an excellent correspondence, with an almost perfect overlap of the data.
Figure 4: Graphs showing the change in automatically computed wound areas (expressed on the vertical axis in µm2 per :m of wound length within the defined bounding box) against the elapsed time since wounding (minutes, horizontal axis), for every tenth frame of three videos of similar control wound healing experiments in the absence of drug treatment. (a) Raw data, before filtering to remove the spurious measurements derived from the periodic fluorescence images (see text). (b) Filtered data after the elimination of the noisy measurements that gave abrupt but non-permanent changes in the apparent wound closure rate.
Figure 5 : Graphs showing filtered computed wound area curves from Fig 4b with, superimposed, the original calculated wound areas (expressed on the vertical axis in µm2 per µm of wound length within the defined bounding box) against the time from wounding (minutes, horizontal axis) for the same three videos determined manually at eleven time points for each experiment (Lewis, 1993). (The last two data points for Wound 1 are not shown). Data entry into a searchable database The specific intrinsic wound area metadata obtained from these analyses have been organised in a searchable database, using the Informix Dynamic Server with the Universal Data Option to built the prototype (see Informix DataBlade Development Kit, User’s Guide, 1998), and employing three special data tables, the Identity Table, the Spatio-Temporal Position Table, and the Events Table (Shotton, 2000), as shown in Table 1.
IDENTITY TABLE
Video_ID Video_descrip Hz n_frames Timelapse time_min res_pixel ratio 1007 WH_Lewis1 25 4505 161.2 484 (384, 288)
pixels/µm 1.28
SPATIO-TEMPORAL POSITION TABLE
Video_ID 1007 1007 1007 etc . . .
Frame 09 19 29
w_length 315 315 315
w_area 1218 1213 1207
w_h_rate 5 5 6
EVENTS TABLE
Video_ID 1007
Frame_ini 372
Frame_end Ev_Type 465 17
Drug A2
Dosage_mM 0.05
Table 1 : Data are organised using a hierarchical model, with a first level Identity Table for the particular video sequence analysed (filename, length in frames, resolution, etc). The specific intrinsic metadata describing the image content of each video is stored using two tables: the Spatio-Temporal Position Table containing data about the wound area, the wound healing rate, etc., and the Event Table that can contain information about instantaneous or temporally-extended happenings (e.g. perfusion of a drug). Since Experiment WH_Lewis1 did not involve drug treatment, the entries in the Events Table are for illustrative purposes only, showing a fictitious 10 minute exposure to 0.05 mM of Drug A2 starting 40 minutes after wounding. In addition to the data obtained by automated analysis of the visual information in the videos, the database can accommodate a range of additional information. This includes customary ancillary metadata concerning the ownership of the video, the data of the analysis, etc., and information that might be relevant to other experiments, for example details of the times of application and concentrations of drugs applied to the cells by perfusion of the microscope observation chamber during time-lapse video recording, which is stored in the Events Table. The database allows the following types of query to be made concerning such wound healing videos: What is the rate of wound healing at particular times? What is the rate of change of wound healing in response to treatment by a particular drug? What is the rate of cell motility at wound edges in comparison with the rate of motility of the same type of cells at the free margins of small unwounded colonies? The first query can be answered by directly looking at the appropriate register in the spatio-temporal position table, where we can find the wound area for each frame in the video, with its associated derived metadata, the instantaneous wound closure rate. Changes in this rate due to drug administration can be found by searching the database for differences in the healing rate before and after administration of the drug, the drug type, and the timing and dosage of the drug administration being recovered from the Event Table. The third query involves healing rate comparisons between different videos.
As result of a successful query, a list of pointers to video files is returned by the system, together with a set or ranges of frame numbers, allowing the video clips matching the query to be recovered.
Discussion The procedure described allows the automatic generation from moving image data that are initially recorded digitally or on videotape of information of high intrinsic scientific value which is specific to the image content of the video being analysed (i.e. specific intrinsic metadata). This is done by intelligent analysis of their visual content, and produces information that it is difficult to obtain manually. For the three videos studied, we observed a significant increase in both accuracy and efficiency when undertaking such analyses using the algorithms developed, rather than by hand, since the analysis generated many more data points in a much shorter time. The specific intrinsic metadata is stored in a database that supports subsequent queries. It allows not only the use of query methods that rely upon the traditionally recorded ancillary textual metadata, but also permits queries to be performed on the specific intrinsic metadata generated by the automatic analysis, allowing important factual and analytical information to be obtained, and selected video sequences matching the query criteria to be retrieved. Although the prototype system is being used to extract knowledge in an automated manner from research video recordings arising from cell biological microscopy, it is clear that this system is easily extendable and that it could have widespread usefulness for the analysis of other types of moving image data.
Acknowledgments This work has been partially supported by grant 1FD97-0372 from the EU-FEDER Programme (Fondo Europeo de Desarrollo Regional), and by a collaborative Joint Research Venture between the Department of Zoology at the University of Oxford, UK and the Computer Architecture Department, at the University of Malaga, Spain.
References Boudier, T. and Shotton, D. M. (1999a), "Biological videos for an Internet image database", Innovation et Technologie en Biologie et en Medicine 20: 51-57. Boudier, T. and Shotton, D. M. (1999b). Video on the Internet: an introduction to the digital encoding, compression and transmission of moving image data. J. Structural Biology 125: 133-155. Eutereuer, U. and Schliwa, M. (1992) Mechanism of centrosome positioning during the wound response in BSC-1 cells. J. Cell Biol. 116: 1157-1166. Guil, N., Gonzalez, J.M. and Zapata, E.L. (1999). Bidimensional shape detection using an invariant approach. J. Pattern Recognition, 32: 1025-1038. Informix DataBlade Development Kit, User’s Guide (1998). Version 3.6 (Part No 000-5052). (Available on-line at http://www.informix.com.answers) Kupfer, A., Louvard, D. and Singer, S. J. (1982) Polarization of the Golgi apparatus and the microtubule-organising center in cultured fibroblasts at the edge of an experimental wound. Proc. Natl. Acad. Sci. USA 79: 2603-2607. Lewis, J. W. (1993) The effects of colchicine and brefeldin A on the rate of experimental in vitro epithelial wound healing. Undergraduate research project report, Department of Zoology, University of Oxford. Lewis, J. W. and Shotton, D. M. (1994). Effects of colchicine and brefeldin A on the rate of experimental in vitro epithelial wound healing. Proc. 4th Annual Meeting European Tissue Repair Soc., Oxford. p. 230. Lewis, J. W. and Shotton, D. M. (1995). Time-lapse video microscopy of wound healing in epithelial cell monolayers: effects of drugs that induce microtubule depolymerization and Golgi disruption. Proc. Roy. Microscop. Soc. 30: 134-135. Shotton, D. M. (2000). Classifying video metadata for query by content in an image database. Computer Vision and Image Understanding (submitted for publication).
