Closed-Loop Optogenetic Brain Interface - IEEE Xplore

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 62, NO. 10, OCTOBER 2015

2327

Closed-Loop Optogenetic Brain Interface Ramin Pashaie∗ , Member, IEEE, Ryan Baumgartner, Thomas J. Richner, Sarah K. Brodnick, Mehdi Azimipour, Kevin W. Eliceiri, and Justin C. Williams

Abstract—This paper presents a new approach for implementation of closed-loop brain–machine interface algorithms by combining optogenetic neural stimulation with electrocorticography and fluorescence microscopy. We used a new generation of microfabricated electrocorticography (micro-ECoG) devices in which electrode arrays are embedded within an optically transparent biocompatible substrate that provides optical access to the brain tissue during electrophysiology recording. An optical setup was designed capable of projecting arbitrary patterns of light for optogenetic stimulation and performing fluorescence microscopy through the implant. For realization of a closed-loop system using this platform, the feedback can be taken from electrophysiology data or fluorescence imaging. In the closed-loop systems discussed in this paper, the feedback signal was taken from the micro-ECoG. In these algorithms, the electrophysiology data are continuously transferred to a computer and compared with some predefined spatial-temporal patterns of neural activity. The computer which processes the data also readjusts the duration and distribution of optogenetic stimulating pulses to minimize the difference between the recorded activity and the predefined set points so that after a limited period of transient response the recorded activity follows the set points. Details of the system design and implementation of typical closedloop paradigms are discussed in this paper. Index Terms—Brain interface, closed loop, fluorescence imaging, hemodynamic signals, optogenetics, spatial light modulator (SLM).

I. INTRODUCTION Brain–machine interface (BMI) mechanisms were developed to implement direct data communication links between the brain and artificial computers to bypass or compensate possible damages in data processing pathways within the brain or to exchange data between the brain and the peripheral nervous system [1], [2]. BMI systems are usually made in the form of prosthetic devices that record neural activity and transfer this data to a computer which decodes the information and translates results to a sequence of commands, for example, to move a robotic limb. In closed-loop interface paradigms, the system also generates feedback signals to pass sensory information, such as the precision of the generated movement, back to the brain by stimulating the neural circuitry [3]. Manuscript received June 5, 2014; revised May 4, 2015; accepted May 14, 2015. Date of publication May 22, 2015; date of current version September 16, 2015. The work was supported by the Defense Advanced Research Projects Agency MTO under the auspices of Dr. J. Judy through the Space and Naval Warfare Systems Center, under Pacific Grant/Contract N66001-12-C-4025. Asterisk indicates corresponding author. ∗ R. Pashaie is with the Electrical Engineering Department, University of Wisconsin-Milwaukee, Milwaukee, WI 53211 USA (e-mail: [email protected]). R. Baumgartner, T. J. Richner, S. K. Brodnick, M. Azimipour, K. W. Eliceiri, and J. C. Williams are with the University of Wisconsin-Milwaukee. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2015.2436817

The most common method in recording brain activity for BMI applications, due to the preference for minimum invasiveness, is electroencephalography where electrical activity is recorded by a set of electrodes along the scalp. Nevertheless, the more invasive technique of epidural electrocorticography (ECoG), in which electrode arrays are implanted directly on the surface of cortex without penetrating into the tissue, provides more robust and reliable signals, due to the close proximity to the neural tissue [4]–[10]. For realization of closed-loop interfaces, ECoGs are occasionally combined with penetrating microstimulating electrodes. However, simultaneous stimulation and recording has proven to be difficult due to potential electric artifacts that cannot be eliminated completely [11]. In addition, targeting specific cell-types is not feasible since electrodes stimulate all cells in the region with no specificity. Also, electrodes predominately stimulate cells to increase neural activity and they offer no direct mechanism for silencing. In recent years, by combining optics and molecular genetics, a new methodology for optical modulation of neural activity was introduced. In this approach, known as optogenetics, cell-types of interest are genetically targeted to produce light-gated cation channels or anion pumps that are sensitive to a specific range of wavelengths in the visible portion of the spectrum [12]–[16]. Once these proteins are expressed, the activity of the targeted cells can be increased or suppressed just by exposing the cells to appropriate wavelengths. Optogenetics provides distinctive mechanisms for cell-type targeting and bidirectional control of neural activity [17]–[21]. In addition, the inherent parallelism of optics helps to target relatively large areas of the cortex noninvasively simply by patterning light on the surface of the tissue using spatial light modulators (SLMs). In this paper, we present a new multimodal optoelectronic brain interface platform which was designed to combine microfabricated electrocorticography (micro-ECoG) technology with optogenetic stimulation and fluorescence microscopy. This versatile design can be used for different applications such as source localization in ECoG recordings [10] or the study of neurovascular/metabolic coupling in the cortex [22]. One major application of this design is the implementation of optoelectronic closedloop brain interface protocols, which is discussed in this paper in further detail. II. SYSTEM ARCHITECTURE A. Optical Design Block diagram of the optical setup is shown in Fig. 1. In this setup, a high-power wide-spectrum Arc lamp (300 W Xenon, Perkin Elmer, MA, USA) functions as the major source of the optical power for the system. The produced light passes through a fast switchable optical path, controlled by a couple of

0018-9294 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

2328


Fig. 1. Block diagram of the optoelectronic projection and imaging device. The system is a combination of two separate optical paths, one for imaging, including bright field or fluorescence microscopy, and one for projection. A DMD is used as a SLM which modulates the beam of light produced by a high power Arc lamp or a laser source for optogenetic stimulation and/or fluorescence excitation. In the imaging branch, a highly sensitive EMCCD camera is used to capture images through the installed epifluorescence optics. The system is connected to a computer for automatic data acquisition and data processing.

programmable parabolic mirrors to pass the beam of light through appropriate optical filters (DG-4, Sutter Instruments, CA, USA). Before each filter, a hot mirror is placed to reduce the thermal tension caused by the intense light on the filters. For this application, sputter-coated high transmission excitation filters are used since these filters can resist high temperatures generated by ultrapowerful light sources. When higher optical power is required for optogenetic stimulation or inhibition, laser sources (e.g., 445 nm, 2 W, DPSS Laser Ultralasers, ON, Canada) are also installed on the setup and the beam is coupled to the beam of the Arc lamp after passing through a speckle remover optics (an optical diffuser rotating at speeds 1000 r/min) and an aspheric condenser lens. The filtered beam from the Arc lamp or the laser is then homogenized by an integrating rod and collimated by a telecentric illumination lens mechanism to expose the active area of a digital micromirror device (DMD) through a total internal reflection (TIR) prism. The DMD is

a SLM that is fabricated as a microelectromechanical system which contains a large array of bistable micromirrors [23]. The state of each mirror is programmable and each mirror reflects light toward a lens system that projects the image of the mirror on the screen or toward an absorber so that the corresponding pixel on the screen appears dark. It is also possible to use the pulse width modulation method to control the exposure and produce high precision gray-scale patterns. The TIR prism, which is placed in front of the DMD, guides the beam of light to illuminate the DMD surface at the appropriate angle (θ = 24◦ , φ = 45◦ in spherical coordination) considering the specific design of the micromirrors in a DMD chip. This prism also separates the illuminating beam from the modulated reflected light. Next, the DMD modulates the beam and a telecentric lens (Schneider, Xenoplan Telecentric Lens, PMAG 0.5X, FOV 12.8 mm, Resolution 40 l p/mm, and f /# 3.6) scales down the beam size by a factor of 2 and projects the

PASHAIE et al.: CLOSED-LOOP OPTOGENETIC BRAIN INTERFACE

pattern on the entrance pupil of an image combiner. The image combiner is an assembly of mirrors and a replaceable dichroic or polarizing beam splitter through which we can simultaneously project the pattern on a sample and perform imaging. A Photometrics dual camera system (DC-2) is used as an image combiner in this platform. Telecentric illumination design is utilized for applications that require low distortion and uniform irradiance. In telecentric systems, the image of the systems optical pupil is at infinity, which means that the chief rays for each point in the illuminated image field are parallel and each DMD mirror is illuminated with a cone of light incidenting at the same angle. This allows the use of a TIR prism to separate the light incidenting on the DMD from the light reflected by the DMD. All rays obliquely impinging the TIR surface with angles below the critical incident angle are deflected toward the DMD surface. The modulated beam reflects back toward the TIR prism, transmits through the prism, and couples into the projection optics. The projection optics transmits and bends the rays to produce an image of the DMD array onto a remote screen or at a suitable intermediate image plane to be captured by the next optical assembly, which is the image combiner. The f-number (f /#) and the pupil position of the illumination and projection systems are matched in the design of this setup. In the imaging arm of the system, a sensitive electron multiplication CCD (EMCCD) camera is installed (Photometrics Evolve 512, Photometrics, AZ, USA). A variable magnification lens can be placed in front of the EMCCD to project the image on the camera sensor and to compensate the difference between the pitch factor of the DMD (13.5 μm) chip and the EMCCD (16.0 μm). This is required in case a close to one-to-one registration of the camera pixels and the pixels of the projection chip is needed. A high-speed filter-wheel (Thorlabs FW103H) holds emission filters in place between the image combiner and the EMCCD imaging lens within a collimating emission adaptor. This collimating adaptor is required for two reasons. Insertion of any flat slab of glass, e.g., an emission filter, causes some shift in the position of the focal point. The amount of shift depends on the thickness and refraction index of the slab. This variable focal-shift error causes problems in retaining the parfocality of the projection and imaging arms. The second reason is the change in the spectral response of the filter when the incident light obliquely hits the filter. Particularly, when a filter is placed within a focusing beam, the light at the edge of the beam hits the filter at a different angle compared with the light at the center of the beam. As a result, there is a variation in the spectral response of the filter for different parts of the beam. The collimating emission adaptor (Photometrics XT-2, Photometrics, Inc., AZ, USA) creates collimated space, thus, allowing the addition of other optical devices, such as the filter wheel in the setup, without sacrificing the performance of the system or introducing aberrations. Focusing is achieved by a separate lens mechanism (a microscope objective) that is mounted at the output of the image combiner to bring both the projected pattern and image into focus simultaneously. Since Xenon lamps cannot produce UV light which is required for excitation of some intrinsic fluorophores,

2329

Fig. 2. Spatial resolution of the imaging system measured by a 1951 USAF fluorescence resolution target. The group 7 of the target is displayed for three objective lenses. System’s resolution for fluorescence imaging with 10X objective is 2 μm and better than 1.0 μm for 20X and 40X objectives.

a beam-splitter was mounted before the objective lens to couple the UV light produced by a light-emitting diode (LED) into the optical path of the system. Nevertheless, for fine tuning of the system’s parfocality, adjustable relay lens mechanisms are used at the entrance pupils of the image combiner so that the imaging optics focuses at the same plane as the projection optics. Spatial resolution of the fluorescence imaging system was measured experimentally using a positive US Air Force fluorescence target (Edmund Optics product number 57-855, NJ, USA) (see Fig. 2). The target was illuminated by blue excitation light (center wavelength 445 nm) and green emission (center wavelength 532 nm) was imaged through three microscope objectives: 10X, 20X, and 40X. Based on our experimental data, the resolution for fluorescence imaging using the 10X objective was 2.0 μm and it was better than 1.0 μm for the 20X and 40X objectives. This device is designed for high precision imaging of fluorescence indicators. Therefore, any systematic temporal or spatial variation of the excitation light within the field of view should be removed. To monitor the temporal fluctuation of the source, a photodetector was integrated before the homogenizer to continuously monitor the intensity of the light and compensate temporal fluctuations of the source online or offline. To monitor the spatial distribution of the DMD illuminating beam and measure its uniformity, a beam profiling CCD camera can be integrated into the system to record possible nonuniformities and compensate then by using the DMD chip. It is important that the projection system be fully synchronized with the imaging EMCCD camera. Any lack of synchronization can cause fluctuations in the captured images by the EMCCD which might be misinterpreted as fluorescence changes produced by cellular activity, for example during calcium imaging. To synchronize all the elements of this setup, the signal from a central clock was distributed in the system. B. Pattern Preservation When a beam of light enters a turbid medium, such as biological tissue, randomly distributed particles scatter or absorb photons and the radiation profile of the beam alters dramatically [24]. Few mathematical models have been introduced to estimate the distribution and the penetration depth of light inside a random medium [19], [20], [25], [26]. In this research, the tissue is stimulated by light patterns that are projected on the

2330


Fig. 3. Preservation of the projected patterns in the tissue is studied by three-dimensional Monte Carlo simulations: (a) Distribution of light at multiple depths ranging from 200 to 1000 μm are shown when the tissue is stimulated by a 1-D spatial frequency of 1.5 lp/mm, (b) normalized curves of light distribution at different depths. The dynamic range of fluctuations drops significantly as light penetrates deeper inside the tissue, (c) and (d) maximum intensity of light and the dynamic range of the fluctuations are plotted for different spatial frequencies, 1.0 lp/mm, 1.5 lp/mm, and 2.0 lp/mm for two different wavelengths, 445 nm blue light and 635 nm red light.

tissue surface and the question is: to what extent this structured distribution of light is preserved at different depths in the brain tissue? To answer this question, we modeled light–tissue interaction using a three-dimensional (3-D) Monte Carlo software [27], [28] and typical results are displayed in Fig. 3. In these tests, tissue was stimulated by patterns of light that resemble 1-D spatial frequencies: 1.0 lp/mm to 2.0 lp/mm. Fig. 3(a) shows the distribution of light at different depths when the projected pattern resembles a square wave of frequency 1.5 lp/mm. Because of the scattering, the dynamic range of fluctuations drops significantly and the distribution of light becomes relatively uniform at the depths 600–800 μm in the cortical tissue. This effect is more visible if we plot light distribution curves for different depths as shown in Fig. 3(b). In this figure, we see some fluctuations around a plateau; however, the dynamic range of these fluctuations, which is a measure of the pattern preservation, drops as depth increases. The rate of drop for the dynamic range and the maximum light intensity at different depths for two wavelengths (445 and 635 nm) are shown in Fig. 3(c) and (d). Based on these simulations, structured patterns such as the tested 1-D spatial frequencies are reasonably preserved to about 500 μm inside the tissue which covers considerable portion of the cortex in small rodents including the optogenetic mice that are used in this study. In these simulations, the cortical tissue was modeled by the reduced scattering coefficient, μs = 32 cm−1 and absorption

coefficient μa = 9.0 cm−1 for 445 nm blue wavelength and μs = 19 cm−1 and μa = 1.0 cm−1 for the 635 nm red light [25]. Monte Carlo simulations were performed by tracing 50 million photons in the tissue [24]. C. Optogenetic Micro-ECoG To create a two-way nonpenetrating chronic interface, we have micro-ECoG devices in which electrode arrays were embedded within transparent polymer parleyne C substrate [10]. In this design, the electrode sites were patterned in 4 × 4 grids (500 μm site-to-site spacing) via a biocompatible photolithography process. The platinum electrode sites, which were 150 μm in diameter, have nominal impedance around 50 kΩ at 1-kHz frequency. These prosthetic devices were then implanted epidurally to record LFPs (see Figs. 4 and 5). Next, the bone was replaced by a 3-mm diameter 150-μm thick cranial window to provide optical access to the brain tissue for stimulation and imaging. Parylene C was chosen as the insulative substrate for its flexibility, biological inertness, and optical transparency over the visible and near-infrared spectrum [29]–[31]. The electrode sites and connecting traces leave ample optical access to the cortex, occluding only 8.3% of the array. The weight of the entire implant was less than 400 mg and based on our experience, the implant can be used for data acquisition in a mouse for more than a month after the surgery.


2331

Fig. 4. Micro-ECoG electrode array assembly and cranial window implantation. (a) Cross-sectional diagram of an electrode array implanted under a cranial window. The array sits over Dura matter under a clear glass-made cranial window. The entire craniotomy is enclosed with UV curable dental acrylic. (b) Assembled electrode array with the connector. The parylene micro-ECoG array is assembled with a printed circuit board to route the connections to the lownoise amplifier. (c) Picture of an array implanted under a cranial window in a transgenic ChR2+ mouse. The electrode sites and traces are visible. The scale bar is 500 μm.

A new approach was recently introduced for similar applications where arrays of micro-LEDs were integrated in ECoGs to combine optogenetic stimulation with electrophysiology recordings [32]. Nonetheless, in most cases, micro-LEDs could not generate enough optical power to target a large number of neurons in deeper layers of cortex. A DMD system together with a laser as the main source of the optical power can provide enough light for neural stimulation and certainly offer superior spatial resolution when required. III. SYSTEM PERFORMANCE A sequence of experiments were conducted to evaluate the performance of the developed platform for optogenetic stimulation, micro-ECoG recording of optically induced activity and monitoring of metabolic and hemodynamic signals via fluorescence microscopy. All animal protocols were approved by the Institutional Animal Care and Use Committee at the University of Wisconsin and procedures were done in accordance with the National Institutes of Health guidelines for laboratory animal use. A. ECoG Recording To record optogenetically evoked LFPs across the cortical surface, transparent micro-ECoG arrays were implanted in transgenic Thy1::ChR2/H134R mice that have relatively uniform expression of Channelrhodopsin-2 (ChR2) in pyramidal neurons. The H134R variant of ChR2 opsin was specifically chosen

Fig. 5. Micro-ECoG array was implanted epidurally in a Thy1::ChR2-H134R mouse to map optogenetically induced cortical potentials. (a) and (b) Optogenetically evoked cortical potentials driven with a SLM and recorded by a micro-ECoG array (4 × 4 grid of platinum sites, 500 μm site-to-site spacing, 150 μm site diameter, 50 kΩ at 1 kHz) implanted under a cranial window. In this example, two areas were illuminated (8 ms pulses of blue light, 445 nm, 4.5 mW/mm 2 ). Potentials nearest each photostimulus region had the greatest amplitude. Potentials were recorded at 3kHz using a high-impedance amplifier (Tucker-Davis Technologies). The photostimulus was imaged through a 400–500 nm emission filter with the EMCCD camera. (c) Magnitude of the LFPs recorded by a sample electrode as a function of light pulse duration. (d) Control experiment, in which the LFP recorded after one epoch of blue light stimulation, is compared in transgenic versus wild type animal. The photoelectric artifact is also tested in a micro-ECoG, which was submerged in saline. The amplitude of the artifact is by two orders of magnitude smaller than the induced LFP in the transgenic animal.

since it has slower off-kinetics and as a result provides more light sensitivity, which is required for the experiments. All surgical tools were autoclaved before surgery and electrode arrays and connectors were sterilized with low-temperature ethylene oxide exposure. Surgical procedure was followed closely from previously published methods [10]. To implant electrode arrays, animals were anesthetized with isoflurane (2.0% in oxygen) and a 4-mm-diameter area of the skull was thinned with high-speed surgical drill (FM3545, Foredom Electric, Bethel, CT, USA) for imaging. A micro-ECoG device was placed over sensorimotor cortex and a 3 mm cranial window with 150-μm thickness

2332


was glued over the thinned area. These animals also received buprenorphine preoperatively and 8–12 h postoperatively for analgesic control. Water circulating heat pads were used to maintain healthy body temperature. Animals were monitored throughout the procedure for heart rate and blood oxygenation levels. To normalize heart rate and decrease respiratory secretions, the mice received 20 μg/kg glycopyrrolate. Supplemental oxygen and heat were provided during and after the surgery. At the end of the procedure, 0.5 mg/kg atipamezole was given to reverse the sedative effect of dexmedetomidine. For the recording session, mice were sedated with 75 mg/kg boluses of Ketamine and 25 μg/kg dexmedetomidine. Fig. 5 displays typical results when a micro-ECoG array was recording localized optically induced cortical potentials. The electrode array was connected to a high-impedance low-noise amplifier and voltages were recorded referenced to a coil wire implanted in the contralateral cortical hemisphere and averaged over 100 trials. In these experiments, the cortex was stimulated by 8 ms pulses of blue light produced by a 445-nm laser module providing 4.5-mW/mm2 optical power density and 2.0-Hz repetition rate. As shown in Fig. 5(a) and (b), photostimulating a small cortical patch was causing small localized potentials which were picked by the array, while photostimulating a larger cortical area was causing potentials of larger amplitudes and spatial spread. To better visualize the data, the site of photostimulation can be localized with cubic interpolation to obtain the color-coded maps shown in the right panels of Fig. 5(a) and (b). Increasing the duration of the optically stimulating light pulses also increases the magnitude of the induced LFP. A typical correlation between the amplitude of the LFP as a function of pulse duration is shown in Fig. 5(c). This data were used later in the realization of the closed-loop system where the level of optically induced activities were adjusted by controlling the duration of photostimulating pulses. To prove that the LFP signals are produced by optogenetic stimulation and not thermal or photoelectric artifacts, the same in vivo experiments were conducted in a wild-type mice [see Fig. 5(d)]. Based on our observations, no noticeable activity was detected by electrodes as a result of blue light exposure in wildtype animals. The chance of observing photoelectric artifact was minimum partly due to the choice of platinum as the conductor for electrode fabrication. It is known that the photoelectric effect occurs when the maximum kinetic energy of the ejected electron, defined as the difference between the photon energy and the metal work function, is positive. Platinum’s work function is about 5.1 eV, which is quite larger than the energy of a typical blue photon (2.8 eV), and therefore, the maximum kinetic energy is negative. Nonetheless, when we submerge the electrodes in saline and exposed them to 473 nm laser pulses of the same intensity, we observed some artifacts which were about two orders of magnitude smaller than the signals recorded in transgenic animals following optogenetic stimulation [see Fig. 5(d)]. This negligible artifact is most likely caused by the Becquerel photovoltaic effect which occurs at lower stimulation energy levels. One practical solution to completely avoid such artifacts is using the SLM to pattern light around electrode sites during electrophysiology recordings. To implement this procedure, the camera simply takes a picture of the implanted electrodes and an

image processing code detects and marks the electrode sites in the image and produces a mask for the projection system which helps to avoid exposing electrodes to intense light pulses [22]. B. Fluorescence Microscopy Fluorescence microscopy has been widely utilized in the study of biological systems. For example, by using chemical or genetically-encoded fluorescence biomarkers (e.g., calcium indicators or voltage sensitive dyes), researchers measure the spatial distribution and magnitude of neural activity in largescale networks via fluorescence imaging [33]. In this section, the performance of the developed system in fluorescence imaging in cortical tissue is demonstrated by two examples: hemodynamic signal recording and metabolic imaging. The hemodynamic and metabolic markers, usually imaged by fMRI or PET, are used in the study of neurological diseases. The basic assumption in this approach is that the neural activity and the cerebral metabolism and the consumption rate of oxygen and glucose are directly correlated. Consequently, any sudden increase in neural activity is logically followed by an increase in cerebral blood flow or blood volume [34] since the brain should perfuse active neurons with oxygenated hemoglobin. In addition to monitoring hemodynamic signals, which are specifically helpful in the mapping of neurovascular coupling or disease studies, it is possible to monitor neurometabolic coupling via fluorescence imaging. For example, the intrinsic fluorophore Nicotinamide Adenine Dinucleotide (NADH) is a coenzyme involved in cellular metabolism [35]–[38]. Under increased metabolic load, NADH oxidizes to NAD+, which does not fluoresce and the reduction in the level of the intrinsic fluorescent signal is used to determine regional neural activity. For fluorescence recordings, animal’s head was fixed in a stereotaxic apparatus to prevent motion artifact. Rhodamine B isothiocyanate-dextran (Sigma, 70 kDa) solution was injected into the tail vein to fluorescently label the cerebral vasculature (see Fig. 6). We applied photostimulus light pulses (1.0 s pulse train at 25 Hz, 50% duty cycle, 445 nm, 4.5 mW/mm2 ) and measured the diameter of cerebral arteries (labeled a1–a3) and veins (labeled v1–v2) before and after the stimulus. Branches of the middle cerebral artery (MCA), marked as (a1–a3) in the figure, dilated rapidly in response to optical stimulation while we hardly observed any change in the diameter of veins. For these curves, seven trails were averaged and the standard deviation is shown in gray. Changes in the fluorescence signal produced by NADH can also be monitored by the highly sensitive EMCCD camera mounted on the setup. We imaged this fluorescent signal before and after ChR2 stimulation by a 2.0-s pulse train, 20 ms ON, 20 ms OFF, 4.5 mW/mm2 . To improve the precision of the measurements, blood vessels were removed from the region of interest using a segmentation algorithm introduced by Longair [39]. For the curve shown in Fig. 6(f), ten trials were conducted at each location. As expected, the NADH fluorescent signal decreased transiently by about 4% following photostimulation. Considering the importance of neurovascular and neurometabolic responses following optogenetic stimulation, they were studied in further detail and results were published in [22] and [40].


2333

Fig. 6. Recording of hemodynamic and metabolic signals post optogenetic stimulation via fluorescence microscopy. (a) For hemodynamic recording, rhodamineB conjugated dextran was injected intravenously to label the vasculature of the cortex. (b) We applied a photostimulus (1-s pulse train at 25 Hz, 50% duty cycle, 445 nm, 4.5 mW/mm 2 ) and measured the diameter of cerebral arteries (labeled a1–a3) and veins (labeled v1–v2) before and after the stimulus. (c) Branches of the MCA (a1–a3) dilated rapidly in response while veins in the region showed less of a response. Seven trails were averaged and the standard deviation is shown in gray. (d) For metabolic recording, a patch of cortex defined by the SLM was stimulated, (e) fluorescence signal is measured in the same area after omitting blood vessels from the region of interest. (f) NADH signal decreased transiently by about 4% following photostimulation (1 -s pulse train at 66 Hz, 33% duty cycle, 445 nm, 4.5 mW/mm 2 ), indicating increased oxidation.

IV. CLOSED-LOOP BRAIN INTERFACE For realization of a closed-loop brain interface protocol, we need mechanisms to both manipulate and monitor neural activity. In the developed setup, the activity of cells in the cortex is controlled by patterning light over the brain tissue in transgenic animals using the SLM. The feedback can be taken from fluorescence microscopy data, (e.g., hemodynamic or metabolic signals as explained earlier) or electrophysiology recordings using the micro-ECoG device. To demonstrate the capability of the developed system for realization of closed-loop algorithms, we implemented a system in which the feedback signal was taken from the micro-ECoG and sent to a computer for real-time processing. The computer’s task is to match the level of the brain activity at all recording sites to temporal patterns or set points for each site. To achieve this goal, the system monitors the level of neural activity on each site and compares this data to the set points to generate the error signal. Next, the computer controls the DMD chip and adjusts the duration of optical stimulation on each site to minimize the error. A. Algorithm The closed-loop protocol that was discussed in the previous section is summarized in Fig. 7. This algorithm was implemented in five sequential steps as follows: Step 1 (Reading the signals from micro-ECoG electrodes and processing the data in real time): Data processing includes passing the recorded signals through three serially connected digital

Fig. 7. (a) and (b) Micro-ECoG optogenetic closed-loop protocol. The system records the activity using an implanted micro-ECoG device. The recorded data are processed in real time and compared to predefined patterns (set points) that are stored in the computer. Based on the difference between the recorded levels of activity on each electrode site and the set points, the computer updates the illumination pattern on the SLM to adjust the level of induced activity and minimum the error. (c) Sequence of data processing algorithms employed to analyze the recorded electrophysiology signals.

2334


Fig. 8. Calibration process for the closed-loop algorithm which is achieved by adjusting the coefficient K p . A sequence of set points are defined in the computer in the form of a square pulse that fluctuates between three discernable levels. We start the learning process by assigning a reasonable value to K p and we periodically change the duration of the photostimulating pulses through the same negative feedback algorithm until the magnitude of the optogenetically evoked potential converges to the set point. Trace of change for the photostimulus pulse duration and loop constant are shown. When the value of K p is slightly above the optimal value the system is overdamped and the signal oscillates around the set point whenever the level changes. When K p takes values smaller than the optimal value, the system is underdamped and convergence occurs slowly. When K p is adjusted to the optimal value, the system is critically damped with the best possible performance.

filters. First, a bandpass filter is applied to select frequency components within the interval 10–150 Hz. Next, the signal is rectified and integral of the signal is calculated over a 100 ms period following optogenetic stimulation. When needed, a 60 Hz notch filter is also applied to minimize the power-line interference. The output of this process is a variable that represents the area under the optically evoked potential curve for each site which is a robust feedback signal for the closed-loop algorithm. Similar signal processing algorithms have been used before, for example, to estimate the power in different frequency bands in real-time ECoG brain interface applications [41]. Step 2 (Generating the error signal): Error is computed by taking the difference between the actual measured signal after processing and the corresponding temporal desired activity for each electrode site. Step 3 (Adjusting the duration of the exposure for the next round of optical stimulation): In this step, the computer adjusts the exposure time for the next round of stimulation to minimize the error signals for all electrode sites. Obviously, the error signals are voltage signals which are compensated by adjusting the pulse widths. In Fig. 7(b), the constant that translates the voltage signal to pulse duration is marked as the proportionality constant Kp . Finding the appropriate value for this constant is a part of the calibration process, which is discussed in the next section. Step 4 (Exposure): Make the sequence of photostimulus patterns according to the photostimulus durations calculated in step 3 and expose the tissue to the laser pulses. Step 5 (Iteration): Go to step 1 and repeat the algorithm.

constant varies slightly from one electrode site to the next and should be calculated prior to the experiments. In our approach, tuning the value of the loop constant is casted like a learning algorithm, as shown in Fig. 8 for a typical electrode site. In the first step, a sequence of set points are defined in the computer in the form of a square pulse that fluctuates between three discernible levels. We start the learning process by assigning a reasonable value to Kp and we periodically change the duration of the photostimulating pulses through the same negative feedback algorithm until the magnitude of the optogenetically evoked potential converges to the set points. As displayed in Fig. 8, the performance of the system depends on the value of Kp . When the assigned value to Kp is not close to the optimal value, the difference between the set point and the recorded signal is significant. For these cases, the direction of change for the loop constant was determined by the sign of the error signal. Close to the optimal value, the feedback system can be underdamped (overdamped) when Kp is less (more) than the optimal value. In an underdamped system, the convergence to the asymptotic value occurs slowly. On the other hand, in an overdamped system, the output reaches the asymptotic value rapidly but oscillates around this value before converging. Similar behavior was observed in the closed-loop algorithm (see Fig. 8). When the value of Kp is close to the optimal value, the system is critically-damped where the recorded activity converges to the corresponding set points rapidly and minimum or no oscillation occurs around the settled values. Ultimately, the optimal value of Kp for each site is obtained and stored in the computer to be used for the realization of the closed-loop BMI algorithm.

B. Calibration

C. Single-Site Closed-Loop System

The proportionality constant Kp is the parameter that translates the micro-ECoG signal (voltage dimension) to the pulse duration (time dimension) for the DMD system. The value of this

After adjusting the proportionality constant for electrode sites, the closed-loop system was ready for the actual test. In the first round of experiments, we marked two separate sites


2335

Fig. 9. Implementation of a single-site closed-loop brain interface. The loop constants are measured and set for two separate sites marked as site-1 and site-2. Then, the closed-loop algorithm is conducted so that the signal from site-1 (blue signal) clamps to the user-predefined set points (black curve), while the signal from site-2 (green signal) is clamped to the signal from site-1. In other words, these two sites are connected by an optoelectronic bridge.

Fig. 10. Implementation of a multiple-site closed-loop brain interface. The timing of the exposures for these four sites and results of the closed-loop control are displayed in this figure. In this experiment, three separate photostimulus masks are used. The projection system patterned light on the brain tissue iteratively starting with mask #1, then mask #2 and #3. The duration of photostimulus pulse for each site (sites 1, 2, 3, and 4) changes according to the feedback loop to keep the activity level of each site close to the corresponding set point dynamically. After all three masks are displayed and the collected data is processed, the system repeats this algorithm.

as site-1 and site-2 and measured the value of the loop constants for both sites (see Fig. 9). In this test, we tried to force the activity in site-1 to converge to the set points defined by a square wave while lock the activity at site-2 to the activity at site-1. In other words, the goal was to define a bridge between these two sites via electrophysiology recording and optogenetic stimulation. As it is shown in the figure, the signal from site1 followed the set points with reasonable accuracy when the proportionality constant for this site was well tuned. Simultaneously, the signal from site-2 was tracing the activity of site-1 with some expected lag. This method has potential application in compensating injuries that disrupt the horizontal connectivity among cortical circuitries. In such applications, mathematical models can be developed to estimate the level of activity on one side of the injury by measuring the activity on the other side. Then, a similar closed-loop platform can be used to bypass the injury by adjusting the activity level at the estimated values in the disconnected region via optogenetics. D. Multiple-Site Closed-Loop System Using this platform, the activity at multiple sites can be controlled simultaneously. For the multiple-site experiment, photostimulating pulses were delivered independently to four sites

and each site had an independent feedback loop (see Fig. 10). The selected set points were intentionally designed to be out of phase so that while the activities in some sites were increasing, activities of other sites were decreasing. Once again, before starting the experiments, the optimal values for the proportionality constants for each site was measured and stored in the computer. In this closed-loop experiment, we used three separate photostimulus masks as displayed in Fig. 10. The projection system patterned light on the brain tissue sequentially starting at mask 1, then mask 2 and 3. In general, the display time for each mask was 100 ms. However, The duration of photostimulating pulse for each site (site 1 to site 4) was adjusted according to the feedback loop to keep the activity level of each site close to the corresponding temporary set points. The timing of the events is better illustrated in Fig. 11. When all three masks are displayed, there was a 700-ms wait period before starting the next round. As displayed in Fig. 10, the developed system could control the activity level of multiple sites independently to keep them close to the predefined set points. For specific applications, these set points, for example, can present some external variables such as the locations of few objects in space where each object is independent of others and free to follow any trajectory in the space. In other words, we

2336


Fig. 11. (a) Intensity diagram of recorded activity for all electrodes in the multi-site closed-loop experiment. In this image, the level of activity is color mapped so that dark represents little to no activity while higher levels of activity is marked by brighter colors over the whole duration of the experiment. The electrodes which correspond to a given photostimulus are identified by rectangles. (b) Projection sequence and exact timing of events.

can translate any external data collected from the environment to a sequence of dynamically changing activity patterns in the brain. A system of this nature with adequate spatial and temporal resolution has the potential to generate advanced brain interface mechanisms, for example, to induce artificial perception. In these examples, the feedback is taken from the electric signals of the micro-ECoG which are directly proportional to the regional neural activity. While these examples demonstrate the potential of the platform in realization of closed-loop algorithms, it is certainly possible to use other resources to generate the feedback signal. For instance, in the study of stroke, the system can monitor hemodynamics via fluorescence imaging, and whenever the system detects substantial reduction of blood flow in major cerebral vessels caused possibly by colloquially brain attack, vasodilation can be induced through neurovascular coupling and optogenetic stimulation [22], [40] to potentially readjust the blood flow. In other words, different configurations of closed-loop systems can be readily realized using this versatile platform. REFERENCES [1] B. Graimann et al., Brain-Computer Interfaces: Revolutionizing HumanComputer Interaction. Berlin, Germany: Springer, 2010.

[2] J. Wolpaw and E. W. Wolpaw, Brain-Computer Interfaces: Principles and Practice. London, U.K.: Oxford Univ. Press, 2012. [3] M. A. L. Nicolelis, J. K. Chapin, and J. Wessberg, “Closed loop brain machine interface,” Patent US7 209 788 B2, Apr. 24, 2007. [4] G. Schalk and E. C. Leuthardt, “Brain-Computer interfaces using electrocorticographic signals,” IEEE Rev. Biomed. Eng., vol. 4, pp. 140–154, Oct. 2011. [5] B. Rubelhn et al., “A MEMS-based flexible multichannel ECoG-electrode array,” J. Neural Eng., vol. 6, pp. 036003-1–036003-10, 2009. [6] J. Viventi et al., “Flexible, foldable, actively multiplexed, high-density electrode array for mapping brain activity in-vivo,” Nature Neurosci., vol. 14, pp. 1599–1605, 2011. [7] J. Wilson et al., “ECoG factors underlying multimodal control of a braincomputer interface,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, pp. 246–250, 2006. [8] E. Leuthardt et al., “A brain-computer interface using electrocorticographic signals in humans,” J. Neural Eng., vol. 1, pp. 63–71, 2004. [9] A. Rouse et al., “Cortical adaptation to a chronic microelectrocorticographic brain computer interface,” J. Neurosci., vol. 33, pp. 1326–1330, 2013. [10] T. Richner et al., “Optogenetic microelectrocorticography for modulating and localizing cerebral cortex activity,” J. Neural Eng., vol. 11, pp. 016010-1–016010-12, 2014. [11] A. Yazdan-Shahmorad et al., “High gamma power in ECoG reflects cortical electrical stimulation effects on unit activity in layers V/VI,” J. Neural Eng., vol. 10, pp. 066002-1–066002-12, 2013. [12] F. Zhang et al., “Circuit-breakers: Optical technologies for probing neural signals and systems,” Nature Rev. Neurosci., vol. 8, pp. 577–581, 2007. [13] F. Zhang et al., “Multimodal fast optical interrogation of neural circuitry,” Nature, vol. 446, pp. 633–639, 2007. [14] E. S. Boyden et al., “Millisecond-timescale, genetically targeted optical control of neural activity,” Nature Neurosci., vol. 8, pp. 1263–1268, 2005. [15] K. Deisseroth, “Optogenetics,” Nature Methods, vol. 8, no. 1, pp. 26–29, 2011. [16] J. Mattis et al., “Principles for applying optogenetic tools derived from direct comparative analysis of microbial opsins,” Nature Methods, vol. 9, no. 2, pp. 159–172, 2011. [17] R. Pashaie et al., “Optogenetic brain interfaces,” IEEE Rev. Biomed. Eng., vol. 7, pp. 3–30, Apr. 2014. [18] L. Fenno et al., “The development and application of optogenetics,” Annu. Rev. Neurosci., vol. 34, pp. 389–412, 2011. [19] O. Yizhar et al., “Optogenetics in neural systems,” Neuron, vol. 71, no. 1, pp. 9–34, 2011. [20] A. Aravanis et al., “An optical neural interface: In vivo control of rodent motor cortex with integrated fiberoptic and optogenetic technology,” J. Neural Eng., vol. 4, pp. S143–S156, 2007. [21] L. A. Gunaydin et al., “Ultrafast optogenetic control,” Nature Neurosci., vol. 13, no. 3, pp. 387–392, 2010. [22] T. Richner et al., “Optogenetic assessment of neurovascular and neurometabolic coupling,” Nature Publishing Group, J. Cerebral Blood Flow Metabolism, vol. 35, pp. 140–147, 2015. [23] D. Dudley et al., “Emerging digital micromirror device (DMD) applications,” Proc. SPIE, vol. 4985, pp. 14–25, 2003. [24] T. Vo-Dinh, Biomedical Photonics Handbook. New York, NY, USA: CRC Press, 2003. [25] M. Azimipour et al., “Extraction of optical properties and prediction of light distribution in rat brain tissue,” J. Biomed. Opt., vol. 19, no. 7, p. 075001, Jul. 2014. [26] A.-J. Si et al., “Light scattering properties vary across different regions of the adult mouse brain,” PLoS ONE, vol. 8, no. 7, p. e67626, 2013. [27] [Online]. Available: http://omlc.ogi.edu/software/mc/mcxyz/index.html [28] L. Wang et al., “MCML-Monte Carlo modeling of light transport in multilayered tissues,” Comput. Methods Prog. Biomed., vol. 47, pp. 131–146, 1995. [29] G. Loeb et al., “Parylene as a chronically stable, reproducible microelectrode insulator,” IEEE Trans. Biomed. Eng., vol. BME-24, no. 2, pp. 121–128, Mar. 1977. [30] E. Schmidt et al., “Long-term implants of Parylene- C coated microelectrodes,” Med. Biol. Eng. Comput., vol. 26, pp. 96–101, 1988. [31] D. Rodger et al., “Flexible parylene-based multielectrode array technology for high-density neural stimulation and recording,” Sensor Actuat B, Chem, vol. 132, pp. 449–460, 2008. [32] K. Kwon et al., “Opto-ECoG array: A Hybrid neural interface with transparent ECoG electrode array and integrated LEDs for optogenetics,” IEEE Trans. Biomed. Circuits Syst., vol. 7, no. 5, pp. 593–600, Oct. 2013.


[33] B. A. Wilt et al., “Advances in light microscopy for neuroscience,” Annu. Rev. Neurosci., vol. 32, pp. 435–506, 2009. [34] T. Schwartz, “Neurovascular coupling and epilepsy: Hemodynamic markers for localizing and predicting seizure onset,” Epilepsy Curr., vol. 7, no. 4, pp. 91–94, 2007. [35] B. Chance et al., “Oxidation- Reduction ratio studies of mitochondia in freeze-trapped samples,” J. Biol. Chem., vol. 254, pp. 4764–4771, 1978. [36] B. Chance et al., “Basic principles of tissue oxygen determination from mitochondrial signals,” Adv. Exp. Med. Biol., vol. 37, pp. 277–292, 1973. [37] B. Quistorff et al., “High spatial resolution readout of 3-D metabolic organ structure: an automated, low-temperature redox ratio-scanning instrument,” Anal. Biochem., vol. 148, pp. 389–400, 1985. [38] M. Ranji et al., “Fluorescent images of mitochondrial redox states in in situ mouse hypoxic ischemic intestines,” J. Innovative Opt. Health Sci., vol. 4, pp. 365–374, 2009. [39] J. Schindelin et al., “Fiji: An open-source platform for biological-image analysis,” Nature Methods, vol. 9, pp. 676–682, 2012. [40] F. Atry et al., “Hemodynamic response of cortical tissue to optogenetic stimulation in transgenic mice,” IEEE Trans. Biomed. Eng., vol. 62, no. 2, pp. 766–773, Feb. 2015. [41] J. J. Williams et al., “Differentiating closed-loop cortical intention from rest: Building an asynchronous electrocorticographic BCI,” J. Neural Eng., vol. 10, pp. 046001-1–046001-15, 2013. Ramin Pashaie (M’07) received the Ph.D. degree in electrical and systems engineering from the University of Pennsylvania, Philadelphia, PA, USA, in December 2007, under the supervision of Prof. N. H. Farhat. After Ph.D. degree, he joined Karl A. Deisseroth lab as a Postdoctoral Scholar in the Bioengineering Department at Stanford University. During his postdoctoral training, he focused on technology development for optical modulation of neural activities using the tools of photonics and molecular genetics. In September 2009, he joined the Electrical Engineering Department at the University of Wisconsin-Milwaukee, Milwaukee, WI, USA, as an Assistant Professor and the Director of the Bioinspired Sciences and Technology Laboratory where the research is about optical interrogation of the dynamics of large-scale neural networks mostly in the brain cortical regions. In particular, he is currently interested in the implementation of neuroprosthetic devices to extract details of information processing in cortical networks and the nonlinear dynamics of cortical columns. This information can be used for reverse engineering and realization of brain–machine interface mechanisms. Dr. Pashaie received the NARSAD (Brain and Behavior Research Foundation) Young Investigator Award in 2013 and the National Science Foundation Career Award in Biophotonics in 2015. Ryan Baumgartner received the Bachelor’s degree in electrical engineering and the Masters’ degree in the same department where he joined the research team at the Bio-Inspired Sciences and Technologies Laboratory to study the dynamics of the brain in live lab animals and defended his thesis in December 2014 from the University of Wisconsin-Milwaukee, Milwaukee, WI, USA. After his defense, he joined the Research and Development Branch of Johnsons Control, Milwaukee, as a Research Engineer. Thomas J. Richner received the B.S. degree in mathematics from Northland College, Ashland, WI, USA, in 2006, the B.S. degree in biomedical engineering from Washington University, St. Louis, WI, in 2008, and the Ph.D. degree in biomedical engineering from the University of Wisconsin-Madison, Madison, WI, in 2014. He is currently a Washington Research Foundation Innovation Postdoctoral Fellow in Neuroengineering at the University of Washington, Seattle, WA, USA. He is working with Prof. C. Moritz and Prof. A. Fairhall on motor decoding and sensory encoding. His research interests include optogenetic sensorimotor neural interfaces, theoretical/computational neuroscience, and neurovascular coupling.

2337

Sarah K. Brodnick was born in Elkhorn, WI, USA, in 1983. She received the B.A. degree in zoology from the University of Wisconsin-Madison, Madison, WI, in 2005. In 2005 she joined the Wisconsin National Primate Research Center as an Animal Research Technician; and in 2007, she became an Associate Research Specialist in the Anatomy Department for the University of Wisconsin-Madison. From 2008 to 2009, she shared joint Research Specialist positions in the Medical Physics and Biomedical Engineering Departments also at the University of Wisconsin-Madison. Since 2009, she has been a full time Lab Manager for Justin Williams lab at the University of Wisconsin-Madison. Her current research interests include micro-ECoG and other neural interface device fabrication, glial responses to implanted neural devices, neural prosthetics, in vivo imaging, and device signal interpretation. She has been a Member of the Society for Neuroscience since 2009. Ms. Brodnick received the 2012–2013 Critical Compensation Fund Award at the University of Wisconsin Madison for her work in the William’s Lab.

Mehdi Azimipour received the B.Sc. degree in biomedical engineering from the Amirkabir University of Technology, Tehran, Iran, and the Masters’ degree from Shahid Beheshti University, Tehran, in 2005 and 2008, respectively. He is currently working toward the Ph.D. degree at the Electrical Engineering Department, University of Wisconsin-Milwaukee, Milwaukee, WI, USA. His research interests include optical imaging, optogenetic neurostimulation, and the development of advanced in vivo fluorescent tomography instrumentation.

Kevin W. Eliceiri received a Bachelor Degree in Bacteriology and a Master Degree in Microbiology and Biotechnology from University of WisconsinMadison in May 1996 and December 1998, respectively. He is an internationally known Expert in advanced light microscopy at the Laboratory for Optical and Computational Instrumentation (LOCI), Graduate School and College of Engineering. The mission of LOCI is to develop advanced optical and computational techniques for imaging and experimentally manipulating living specimens. As the Director of LOCI, he oversees the day-to-day operations of the lab and has initiated research collaborations with faculty across the university. Several collaborations have resulted in high-impact publications and grant awards. He is also the Principal Investigator of a National Institutes of Health-funded project to develop software to annotate and archive microscopy data.

Justin C. Williams received the B.Sc. degrees in mechanical engineering in 1995 and in engineering physics in 1996 from South Dakota State University, Brookings, SD, USA, the M.Sc. and Ph.D. degrees in bioengineering in 2001 from Arizona State University, Tempe, AZ, USA. He is currently the Vilas Distinguished Achievement Professor of biomedical engineering and neurological surgery at the University of WisconsinMadison, Madison, WI, USA. From 2002 to 2003, he was a jointly appointed Research Fellow at the Department of Biomedical Engineering, University of Michigan, Ann Arbor, and a Research Scientist at the Department of Neurological Surgery, University of Wisconsin, where his work was focused on neurosurgical applications of neural engineering devices. He joined the Department of Biomedical Engineering at the University of Wisconsin in 2003. His research interests include BioMEMS, neuroprostheses, and functional neurosurgery.