7 degree-of-freedom neuroprosthetic control by an individual with tetraplegia

Background

We use our arms to transport and orient the hand which is used to grasp and manipulate objects. Upper limb paralysis or amputation limits a person’s ability to interact with their environment to accomplish activities of daily living. Brain-machine interfaces (BMIs) may provide a solution to restoring much of this function.

Methods

Two 96-channel intracortical microelectrodes were implanted in the motor cortex of an individual with tetraplegia. Thirteen weeks of BMI training were conducted with the goal of controlling an anthropomorphic prosthetic limb with 7 degrees-of-freedom (3D translation, 3D orientation, 1D grasping). Clinical measures of upper-limb function were used to assess the participant’s ability to use the prosthetic limb.

Findings

The participant demonstrated the ability to move the device freely in the three-dimensional (3D) workspace on the second day of training. After 13 weeks, robust 7 degree-of-freedom movements were performed routinely. Over time, performance on target-based reaching tasks improved in terms of success rate, completion time, and path efficiency. The participant was also able to use the prosthetic limb to perform skillful and coordinated reach and grasp movements that resulted in clinically significant gains in tests of upper-limb function.

Interpretation

This study demonstrates that a person with chronic tetraplegia can perform consistent, natural, and complex movements with an anthropomorphic robotic arm to regain clinically significant function.

Funding

Defense Advanced Research Projects Agency, National Institutes of Health, Department of Veterans Affairs, and UPMC Rehabilitation Institute

INTRODUCTION

Brain-machine interfaces (BMIs) transform neural activity into control signals for an external device. Functional electrical stimulators, exoskeletons, and sophisticated prosthetic limbs are all being developed with the goal of restoring or replacing natural function to a paralyzed or lost limb. In order to perform activities of daily living, one needs to be able to position the hand in space, orient the palm, and grasp an object. Hand placement, orientation, and grasping are normally smoothly coordinated and follow general principles of natural movement.¹–3 Ideally, a BMI will capture these features directly from neuronal activity and translate them to control of an external device with the capability of producing natural movements that follow these principles.

Animal studies using intracortical microelectrodes have shown that these natural features of movement can be captured in recordings of populations of motor cortical neurons⁴. These studies have demonstrated control of a robot arm in 4 dimensions to perform self feeding tasks⁵, 7-dimensional (7D) control for orientation and grasping⁶, and control over functional electrical stimulators of the upper limb.⁷, 8 To date, human studies have demonstrated 3D translational control⁹–11, plus control over a discrete (on/off) grasping dimension.¹² Our results build on previously established methods⁵, 6 to show that a person with tetraplegia was able to rapidly achieve control of a state-of-the-art anthropomorphic prosthetic limb (MPL, Johns Hopkins University, Applied Physics Laboratory). After 13 weeks of training, robust 7D (3D translation, 3D orientation, 1D grasping) movements were performed routinely. These movements included reaching and grasping tasks, similar to many activities of daily living, which were carried out with coordination, skill, and speed approaching that of an able-bodied person.

METHODS

This study was conducted under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and with approval from the Institutional Review Boards at the University of Pittsburgh and the Space and Naval Warfare Systems Center Pacific. This trial is registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/{"type":"clinical-trial","attrs":{"text":"NCT01364480","term_id":"NCT01364480"}}NCT01364480). No adverse events have been observed to date.

Array implantation

Two intracortical microelectrode arrays (4x4 mm, Blackrock Microsystems Salt Lake City, UT, USA), each with 96 electrode shanks (1·5 mm length) were implanted in the left motor cortex on February 10, 2012. Stereotactic image guidance (Brainlab, Inc., Westchester, IL, USA) with co-registered structural (MRI) and functional (fMRI) imaging was used to guide placement of the arrays (figure 1A).

Figure 1

Array location and experiment setup

(A) Pre-operative functional magnetic resonance imaging (fMRI) activation maps on a subject-specific brain model during video-guided attempted movement. The colored activation maps represent blood-oxygenation level dependent (BOLD) activity during video-guided attempted movements: yellow=sequential finger flexion, red=hand grasping, blue=shoulder shrug, green=lip pursing. Approximate array locations are shown as black squares on the inset figure. The arrays were implanted over motor cortex anterior to the central sulcus (CS) approximately 14 mm apart. (B) MPL and 7D Sequence Task setup. The subject was not presented with physical targets. Instead, LEDs (indicated by the white arrow) were used to instruct the participant to hit the “near” (0·35 m from the shoulder) or “far” (0·52 m from the shoulder) translation target corresponding to one of the white circles on the board in front of the MPL. Orientation and grasp targets were presented by a computer-generated verbal command. (C) Diagram of the MPL and translation targets for the 7D Sequence Task. The MPL coordinate system is shown centered at the shoulder. Translation targets had an 8 cm radius and the MPL endpoint (center of the palm) had to be within this region for a successful trial. The MPL endpoint also had to be within the translation target success region in order to successfully achieve the orientation (± 15 degrees) and grasp the targets that were given as audio cues. The time-out period was set to 10 seconds.

Target-based brain-control task

After completing the calibration, the 7D Sequence Task was used for BMI testing. Targets were selected randomly by the computer from a possible combination of 10 translation targets, 7 orientation targets, and 2 grasp targets. Endpoint velocity derived from the recorded firing rates were sent to the MPL controller which converted them to joint motor commands every 20 ms. As in calibration, a translation target was specified by an LED and then an orientation or grasp target was announced by computer-generated voice. The participant had control of all 7 dimensions (translation, orientation, and grasp) throughout the task.

In the early phases of the experiment, an adjustable amount of ortho-impedance was used in the testing phase. In addition, computer assistance was also provided to stabilize the hand’s position or orientation to facilitate learning.⁵, 6 Stabilizing control blended an ideal endpoint velocity control signal determined by the proportional MPL controller with the control signal derived from the participant’s brain activity. The relative contributions of each signal were specified by the experimenter. Stabilizing control was only applied to domains that were not actively being controlled. In other words, position was stabilized at a specified percentage (10–100%) during the orientation phase and/or orientation and grasp posture were stabilized during translation. Similarly, the experimenter could also specify the percentage of ortho-impedance. Both forms of computer assistance were adjusted subjectively to keep the participant engaged and motivated with the aim of maintaining a 60–80% success rate. It should be emphasized that after week 10 (Day 66), the subject performed all tasks with no computer assistance. A 10-second maximum movement time was enforced for each trial phase (translation or orientation/grasp).

Performance metrics including success rate, completion time, and path efficiency were calculated during these testing sessions. Success was determined based on MPL kinematics recorded in real-time. All 7 degrees-of-freedom were controlled continually and had to be correct during all three phases of movement in order for a trial to be considered a success. The experimenter could also manually mark successful (or failed) trials that might occur if the subject rotated too far in the specified direction or if mechanical limits of the MPL prevented movement along a single dimension. To calculate performance, chance levels were determined using random walk-neural signals as input to a simulation while using the assist parameters (if any) and target sizes used during the experiment. Two hundred simulations were run per trial and the median daily chance levels from these simulations are reported (figure 3A). For each trial, path efficiency (PE) was computed for each control domain using the following equation:

PE = L_OT/L_BCT

Eq. 2

where L_OT is the length of the optimal trajectory from the starting position to the end target location and L_BCT is the length of the brain-controlled trajectory for a given trial. L_BCT was determined by numerically integrating the velocity vector magnitude, V, over the duration of a single trial.

Figure 3

Summary of 7D brain-control performance

(A) The solid black dots indicate the subject’s success rate on the 7D Sequence Task for each block (20 trials) of 7D brain-control training. Blocks that the subject performed with various levels of ortho-impedance or stabilizing assist are shown as open circles and squares respectively. After Day 66 post-implant, all reported performance data were collected using full brain-control with no computer assistance. The red dots indicate the median chance level for each day with the 95% confidence interval (5 ^{and 95 percentiles from 200 simulations per trial) shown as error bars. (B) Normalized performance index for each day of 7D brain control. For each block of 20 trials, the success rate was normalized to the median chance level. Performance increased exponentially over time as described by the equation y = 1·812e· (R = 0·114, p=0·001), where x is the number of days post-implant. For subplots C-E, each dot represents the mean block time or path efficiency for one block of 20 trials of the 7D sequence task completed by the participant. Linear fits to the participant’s data are shown as a red line. The mean block time or path efficiency of the MPL under auto-control is shown as a solid horizontal black line. (C) Block completion time, not including the presentation phase time, decreased linearly over the testing period (y = −0·632x + 171·4, R = 0·142, p<0·001). (D) Average 7D path efficiency of the MPL under brain-control increased linearly over time (y = 0·001x + 0·228, R = 0·160, p<0·001). (E) The greatest improvement in path efficiency occurred in the translation dimensions. Path efficiency for 3D translation during the 7D sequence task increased linearly over time (y = 0·003x + 0·317, R = 0·165, p<0·001). Only successful trials were included in the calculation of path efficiency. For all equations, x is the number of days post-implant.}

Participant

Informed consent was obtained from a 52-year-old woman who was diagnosed with spinocerebellar degeneration 13 years prior to participation in the study. Thorough chart review as well as discussions with the participant’s neurologist revealed no indication of cerebellar involvement. Her injury was motor complete with manual muscle test scores of 0/5 for the upper limb.¹³ Physical examination revealed limited sensory deficits and some hypersensitivity.

Array implantation

Two intracortical microelectrode arrays (4x4 mm, Blackrock Microsystems Salt Lake City, UT, USA), each with 96 electrode shanks (1·5 mm length) were implanted in the left motor cortex on February 10, 2012. Stereotactic image guidance (Brainlab, Inc., Westchester, IL, USA) with co-registered structural (MRI) and functional (fMRI) imaging was used to guide placement of the arrays (figure 1A).

Figure 1

Array location and experiment setup

(A) Pre-operative functional magnetic resonance imaging (fMRI) activation maps on a subject-specific brain model during video-guided attempted movement. The colored activation maps represent blood-oxygenation level dependent (BOLD) activity during video-guided attempted movements: yellow=sequential finger flexion, red=hand grasping, blue=shoulder shrug, green=lip pursing. Approximate array locations are shown as black squares on the inset figure. The arrays were implanted over motor cortex anterior to the central sulcus (CS) approximately 14 mm apart. (B) MPL and 7D Sequence Task setup. The subject was not presented with physical targets. Instead, LEDs (indicated by the white arrow) were used to instruct the participant to hit the “near” (0·35 m from the shoulder) or “far” (0·52 m from the shoulder) translation target corresponding to one of the white circles on the board in front of the MPL. Orientation and grasp targets were presented by a computer-generated verbal command. (C) Diagram of the MPL and translation targets for the 7D Sequence Task. The MPL coordinate system is shown centered at the shoulder. Translation targets had an 8 cm radius and the MPL endpoint (center of the palm) had to be within this region for a successful trial. The MPL endpoint also had to be within the translation target success region in order to successfully achieve the orientation (± 15 degrees) and grasp the targets that were given as audio cues. The time-out period was set to 10 seconds.

Neural recording

Generally 3 sessions were conducted per week, each approximately 4 hours in duration. Neural signals were recorded using the NeuroPort data acquisition system (Blackrock Microsystems, Salt Lake City, UT). Single- and multi-unit events were classified using manually set thresholds and time-amplitude windows. Daily set-up time, including unit sorting, required approximately 30 minutes. Activity was converted to firing rate in 30 ms bins and low pass-filtered using an exponential smoothing function with a 450 ms window.

Observation-based calibration and neural decoding

BMI training progressed from 3D endpoint translation control (Weeks 2 and 3 post-implant), to 4D control of translation and grasp (Week 4), to 7D control of translation, orientation and grasp (Weeks 5–14). The joints of the fingers and thumb were combined into a single grasp dimension. The neural decoder, based on a model that linearly related neural firing rate to movement velocity (Eq. 1)⁶, 14, was calibrated daily using a two-phase calibration similar to methods used in non-human primate studies.⁵, 6

During the observation phase, the MPL moved automatically, driven by a proportional controller, to targets that spanned the reachable workspace of the arm as part of a 7D Sequence Task. In the 7D Sequence Task, a target was indicated by an LED on a board in the frontal plane (80x80 cm), 70 cm in front of the MPL shoulder (figure 1B). Once the MPL was moved to the correct 3D position, computer-generated verbal prompts described an orientation to be performed by rotating the palm in one of 6 directions, or a grasp target (open or closed hand). Possible orientations included pronation-supination (±45°), ulnar-radial deviation (±20°), and flexion-extension (±45°) of the wrist. The participant was instructed to carefully watch the limb as it moved automatically to the targets. Neural data and MPL state information including endpoint position, joint angles, and joint torque were recorded at a rate of 33 Hz. Approximately 80 trials (6 minutes) of 7D Sequence Task data were collected to train the initial observation-based 7D neural decoder. The following model was used to relate unit activity to MPL movement velocity⁶, 14:

where f was the firing rate of a unit during movement described by 7D velocity vector V (v_x,v_y,v_z,v_θx,v_θy,v_θz,v_g) and b₀, b_x, b_y, b_z, b_θx, b_θy, b_θz, b_g were the coefficients for each unit. V includes coordinates for the 3D endpoint translation (x,y,z), 3D orientation (θ_x, θ_y, θ_z) and 1D grasp (g) velocities. Units that failed to fit the model (R^{≤0.1) were excluded. Indirect optimal linear estimation (OLE)15} with ridge regression¹⁶ was used to solve for the coefficient matrix B (webappendix p1).

During a second phase of calibration, the participant controlled the MPL using the observation-based neural decoder while ortho-impedance was applied to the command signal.⁵ During this phase, ortho-impedance attenuated, by 100%, the brain-command component perpendicular to the ideal 7D trajectory, restricting movements to directions directly toward or away from the target. Data from 80 trials of the 7D Sequence Task collected during the second phase of calibration were used to build the final decoder using OLE as described above. This second round of calibration allowed the participant to actively engage in the task, while minimizing errors and corrective movements. Early in the experiment (Weeks 2–4), similar calibration procedures were used to generate lower-dimensional neural decoders. Calibration was conducted daily and took approximately 15 minutes.

Target-based brain-control task

After completing the calibration, the 7D Sequence Task was used for BMI testing. Targets were selected randomly by the computer from a possible combination of 10 translation targets, 7 orientation targets, and 2 grasp targets. Endpoint velocity derived from the recorded firing rates were sent to the MPL controller which converted them to joint motor commands every 20 ms. As in calibration, a translation target was specified by an LED and then an orientation or grasp target was announced by computer-generated voice. The participant had control of all 7 dimensions (translation, orientation, and grasp) throughout the task.

In the early phases of the experiment, an adjustable amount of ortho-impedance was used in the testing phase. In addition, computer assistance was also provided to stabilize the hand’s position or orientation to facilitate learning.⁵, 6 Stabilizing control blended an ideal endpoint velocity control signal determined by the proportional MPL controller with the control signal derived from the participant’s brain activity. The relative contributions of each signal were specified by the experimenter. Stabilizing control was only applied to domains that were not actively being controlled. In other words, position was stabilized at a specified percentage (10–100%) during the orientation phase and/or orientation and grasp posture were stabilized during translation. Similarly, the experimenter could also specify the percentage of ortho-impedance. Both forms of computer assistance were adjusted subjectively to keep the participant engaged and motivated with the aim of maintaining a 60–80% success rate. It should be emphasized that after week 10 (Day 66), the subject performed all tasks with no computer assistance. A 10-second maximum movement time was enforced for each trial phase (translation or orientation/grasp).

Performance metrics including success rate, completion time, and path efficiency were calculated during these testing sessions. Success was determined based on MPL kinematics recorded in real-time. All 7 degrees-of-freedom were controlled continually and had to be correct during all three phases of movement in order for a trial to be considered a success. The experimenter could also manually mark successful (or failed) trials that might occur if the subject rotated too far in the specified direction or if mechanical limits of the MPL prevented movement along a single dimension. To calculate performance, chance levels were determined using random walk-neural signals as input to a simulation while using the assist parameters (if any) and target sizes used during the experiment. Two hundred simulations were run per trial and the median daily chance levels from these simulations are reported (figure 3A). For each trial, path efficiency (PE) was computed for each control domain using the following equation:

where L_OT is the length of the optimal trajectory from the starting position to the end target location and L_BCT is the length of the brain-controlled trajectory for a given trial. L_BCT was determined by numerically integrating the velocity vector magnitude, V, over the duration of a single trial.

Figure 3

Summary of 7D brain-control performance

(A) The solid black dots indicate the subject’s success rate on the 7D Sequence Task for each block (20 trials) of 7D brain-control training. Blocks that the subject performed with various levels of ortho-impedance or stabilizing assist are shown as open circles and squares respectively. After Day 66 post-implant, all reported performance data were collected using full brain-control with no computer assistance. The red dots indicate the median chance level for each day with the 95% confidence interval (5 ^{and 95 percentiles from 200 simulations per trial) shown as error bars. (B) Normalized performance index for each day of 7D brain control. For each block of 20 trials, the success rate was normalized to the median chance level. Performance increased exponentially over time as described by the equation y = 1·812e· (R = 0·114, p=0·001), where x is the number of days post-implant. For subplots C-E, each dot represents the mean block time or path efficiency for one block of 20 trials of the 7D sequence task completed by the participant. Linear fits to the participant’s data are shown as a red line. The mean block time or path efficiency of the MPL under auto-control is shown as a solid horizontal black line. (C) Block completion time, not including the presentation phase time, decreased linearly over the testing period (y = −0·632x + 171·4, R = 0·142, p<0·001). (D) Average 7D path efficiency of the MPL under brain-control increased linearly over time (y = 0·001x + 0·228, R = 0·160, p<0·001). (E) The greatest improvement in path efficiency occurred in the translation dimensions. Path efficiency for 3D translation during the 7D sequence task increased linearly over time (y = 0·003x + 0·317, R = 0·165, p<0·001). Only successful trials were included in the calculation of path efficiency. For all equations, x is the number of days post-implant.}

Functional brain-control tasks

The participant used the MPL under full brain-control to perform 9 tasks (selected from 19 possible) on the Action Research Arm Test (ARAT), which is an evaluation of unilateral upper limb function used commonly in patients who have had a stroke.¹⁷ The ARAT is psychometrically sound and tracks the ability to perform activities of daily living (ADLs). The selected tests include all 6 items from the grasp subscale and 3 of the 4 items from the grip subscale (listed in table 1). Standard administration procedures¹⁸ were followed with a few modifications. One of the items was a ball, which was positioned inside a loose coil of wire to keep it from rolling off the table, although it was still possible for the subject to bump the ball hard enough to roll outside of the wire coil. The MPL hand started approximately 12 cm from the table surface, with the palm facing to the left instead of on the table. Each test item was timed and scored as 0 (no movement possible), 1 (task partially performed), 2 (task performed, but abnormally), or 3 (task performed normally). Movements that required more than 5 seconds to complete were scored as a 2. The subject attempted each evaluation 3 times and was instructed that only the best score counted. This kept her motivated and open to trying new strategies.

The participant also performed a cone-stacking task, which involved moving three plastic cones (30 cm tall, 4·1–6·4 cm diameter) from a base and stacking them sequentially on a second base located 30 cm away. The time to complete the task and the number of drops were recorded for each attempt. This test measured the subject’s ability to grasp, transport, and position objects with precision.

Role of the funding source

The sponsors of this study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all of the data and had final responsibility for the decision to submit for publication.

RESULTS

Neural recordings

The number of units, including single and multi-unit activity, recorded daily ranged from 209 to 271 (figure 2). There appeared to be an initial settling period where the number of units started off high until Day 21 post-implant and dropped in Week 4 (Day 24). In Week 4, the number of units began to increase at a rate of 0·368 units/day or 2·58 units/week. Figure 2 also shows that the number of units tuned to movement velocity with an R^{>0·1 (}Eq. 1) increased linearly over time at a rate of 0·388 units/day or 2·72 units/week.

Figure 2

Number of units over time

The top set of data points (blue dots) indicates the number of units recorded during BMI sessions conducted 10–98 days post-implant. Starting at Day 24, the number of units increased linearly (y = 0·368x + 210·0, R ^{= 0·356, p<0·001). The bottom set of data points (red squares) indicates the number of units tuned to movement velocity (}Eq. 1, R^{>0·1), which increased linearly over the duration of recording (y = 0·438x + 24·3, R = 0·155, p<0·01). For reference, 4D training began on Day 24 and 7D training began on Day 32.}

Performance on target-based reaching tasks

Figure 3A shows the participant’s success rate for both computer-assisted and full brain control trials on the 7D Sequence Task over all 22 sessions of 7D testing, as well as the chance level determined for each day. Ortho-impedance and stabilizing assist were only used until week 10 (Day 66). No testing with the MPL was conducted on Days 52–58. The participant’s performance improved over the duration of training. During the last two weeks of training, with no computer assistance, the subject achieved an average success rate of 91·6±4·4%, compared to a median chance level of 6·2% calculated using simulations with random walk neural data as an input signal. Figure 3B shows the normalized performance index (success rate/chance level) over the duration of training. The participant’s performance increased over time even as the difficulty increased.

The participant also showed improvements in MPL control as evidenced by a decrease in block completion time (figure 3C) and an increase in path efficiency (figure 3D-E). Block completion time was reduced from a mean of 148 seconds to 112 seconds over the course of 7D training and also became less variable. Average path efficiency (figure 3D) increased as the participant moved along a straighter, more coordinated path. Mechanical constraints sometimes forced the MPL to deviate from an ideal linear trajectory, causing the maximum achievable efficiency under auto-control to be less than 1 (figure 3D). The greatest improvement in control appeared to be in the translation domain which increased linearly over time (figure 3E). An MPL kinematics sample of four consecutive trials is shown in figure 4. These plots show that the participant was able to rapidly reach the target position in the specified control domain (translation, orientation, or grasp) while maintaining her position in the other two domains (webvideo 1).

Figure 4

MPL position, orientation, and grasp aperture during four 7D Sequence Task trials under full brain control

A thick black horizontal bar denotes whether a translation, orientation, or grasp target was being attempted although the participant had control of all 7 dimensions at all times. Each new translation target indicates the start of a new trial, which is also marked with an arrow along the time axis. MPL kinematics as controlled by the participant are shown as solid lines. The target position for each dimension is shown as a dotted line. Grey shaded regions indicate presentation phases in which the MPL was paused and the subject was listening to a computer-generated verbal command. A grasp aperture of 1 indicates that the hand was fully closed. The participant was successful in maintaining position in one control domain while changing position in another, as instructed. Webvideo 1 shows the subject’s performance with the MPL in these 4 trials.

Neural changes were also observed during the 7D brain control testing period. The percentage of units tuned to 7D MPL velocity (Eq. 1) showed a linear increase (y = 0·002x + 0·100, R ^{= 0·119, p=0·066) as shown in} figure 5A. The percentage of units for R ^{values >0·1 increased during the late training phase (Weeks 11–14) compared to the early phase (Weeks 5–8). The median R value was significantly larger during late training than during early training (p<0.001, Wilcoxon rank-sum test). The percentage of units with R values of 0·15–0·30 showed the greatest increase (}figure 5B). This may suggest that the participant was learning to modulate her brain activity in order to better fit the neural encoding model to improve performance.¹⁹

Figure 5

Changes in neural tuning over time

(A) Fraction of units whose firing rate predicted 7D MPL endpoint velocity (Eq. 1) with an R^{>0·10 on each day of 7D brain-control training. Each dot represents data from a single decoder. On a single day decoders were trained using observation data and brain-control data with ortho-impedance. The percentage of units tuned to 7D MPL velocity trended towards a linear increase over the training period (y = 0·002x + 0·100, R = 0·119, p=0·066). (B) Percentage of units significantly tuned to MPL kinematics for early training (Weeks 5–8, Blue) of 7D BMI training compared to late training (Weeks 11–14, Red). Each bar is centered between the upper and lower bound of R values for a given bin. All R>0·5 were combined into a single bin. Over time, the percentage of units with an R<0·1 decreased, while the percentage of units tuned to kinematics with 0·1≤R≤0·5 increased.}

Neural recordings

The number of units, including single and multi-unit activity, recorded daily ranged from 209 to 271 (figure 2). There appeared to be an initial settling period where the number of units started off high until Day 21 post-implant and dropped in Week 4 (Day 24). In Week 4, the number of units began to increase at a rate of 0·368 units/day or 2·58 units/week. Figure 2 also shows that the number of units tuned to movement velocity with an R^{>0·1 (}Eq. 1) increased linearly over time at a rate of 0·388 units/day or 2·72 units/week.

Figure 2

Number of units over time

The top set of data points (blue dots) indicates the number of units recorded during BMI sessions conducted 10–98 days post-implant. Starting at Day 24, the number of units increased linearly (y = 0·368x + 210·0, R ^{= 0·356, p<0·001). The bottom set of data points (red squares) indicates the number of units tuned to movement velocity (}Eq. 1, R^{>0·1), which increased linearly over the duration of recording (y = 0·438x + 24·3, R = 0·155, p<0·01). For reference, 4D training began on Day 24 and 7D training began on Day 32.}

Performance on target-based reaching tasks

Figure 3A shows the participant’s success rate for both computer-assisted and full brain control trials on the 7D Sequence Task over all 22 sessions of 7D testing, as well as the chance level determined for each day. Ortho-impedance and stabilizing assist were only used until week 10 (Day 66). No testing with the MPL was conducted on Days 52–58. The participant’s performance improved over the duration of training. During the last two weeks of training, with no computer assistance, the subject achieved an average success rate of 91·6±4·4%, compared to a median chance level of 6·2% calculated using simulations with random walk neural data as an input signal. Figure 3B shows the normalized performance index (success rate/chance level) over the duration of training. The participant’s performance increased over time even as the difficulty increased.

The participant also showed improvements in MPL control as evidenced by a decrease in block completion time (figure 3C) and an increase in path efficiency (figure 3D-E). Block completion time was reduced from a mean of 148 seconds to 112 seconds over the course of 7D training and also became less variable. Average path efficiency (figure 3D) increased as the participant moved along a straighter, more coordinated path. Mechanical constraints sometimes forced the MPL to deviate from an ideal linear trajectory, causing the maximum achievable efficiency under auto-control to be less than 1 (figure 3D). The greatest improvement in control appeared to be in the translation domain which increased linearly over time (figure 3E). An MPL kinematics sample of four consecutive trials is shown in figure 4. These plots show that the participant was able to rapidly reach the target position in the specified control domain (translation, orientation, or grasp) while maintaining her position in the other two domains (webvideo 1).

Figure 4

MPL position, orientation, and grasp aperture during four 7D Sequence Task trials under full brain control

A thick black horizontal bar denotes whether a translation, orientation, or grasp target was being attempted although the participant had control of all 7 dimensions at all times. Each new translation target indicates the start of a new trial, which is also marked with an arrow along the time axis. MPL kinematics as controlled by the participant are shown as solid lines. The target position for each dimension is shown as a dotted line. Grey shaded regions indicate presentation phases in which the MPL was paused and the subject was listening to a computer-generated verbal command. A grasp aperture of 1 indicates that the hand was fully closed. The participant was successful in maintaining position in one control domain while changing position in another, as instructed. Webvideo 1 shows the subject’s performance with the MPL in these 4 trials.

Neural changes were also observed during the 7D brain control testing period. The percentage of units tuned to 7D MPL velocity (Eq. 1) showed a linear increase (y = 0·002x + 0·100, R ^{= 0·119, p=0·066) as shown in} figure 5A. The percentage of units for R ^{values >0·1 increased during the late training phase (Weeks 11–14) compared to the early phase (Weeks 5–8). The median R value was significantly larger during late training than during early training (p<0.001, Wilcoxon rank-sum test). The percentage of units with R values of 0·15–0·30 showed the greatest increase (}figure 5B). This may suggest that the participant was learning to modulate her brain activity in order to better fit the neural encoding model to improve performance.¹⁹

Figure 5

Changes in neural tuning over time

(A) Fraction of units whose firing rate predicted 7D MPL endpoint velocity (Eq. 1) with an R^{>0·10 on each day of 7D brain-control training. Each dot represents data from a single decoder. On a single day decoders were trained using observation data and brain-control data with ortho-impedance. The percentage of units tuned to 7D MPL velocity trended towards a linear increase over the training period (y = 0·002x + 0·100, R = 0·119, p=0·066). (B) Percentage of units significantly tuned to MPL kinematics for early training (Weeks 5–8, Blue) of 7D BMI training compared to late training (Weeks 11–14, Red). Each bar is centered between the upper and lower bound of R values for a given bin. All R>0·5 were combined into a single bin. Over time, the percentage of units with an R<0·1 decreased, while the percentage of units tuned to kinematics with 0·1≤R≤0·5 increased.}

Clinical assessments

Table 1 shows the completion time, as well as the total score, for each test item on each day that the ARAT was completed. If no time is listed, the score for that item was a 1 since the task was partially completed. For items with completion times listed, the score was a 2 because it took longer than 5 seconds (equivalent to the approximate amount of time it would take an unimpaired adult to complete the task¹⁸). The participant demonstrated consistent performance as her total score ranged from 15–17 across 7 testing days. The subject was unable to move her own upper limbs volitionally and therefore scored a 0 without the use of the BMI or other assistive technology. Webvideo 2 shows her ARAT performance on Day 87. The movements were generally smooth, direct, and fast. The subject reported that she was thinking about the goal of the action, such as “grab the block,” rather than issuing specific kinematic commands to the MPL endpoint, suggesting that the control was intuitive.

Table 2 shows the cone-stacking task completion time and the number of drops for each day of testing (See also webvideo 3). Dropped cones were repositioned by the experimenter. The fastest performance was achieved on the last day of testing, although the participant had more drops than on Day 87. In this case, she knocked the first cone off the base 6 times before quickly moving all three cones, with only one additional drop. The participant was able to complete this task consistently on all 4 days of testing.

DISCUSSION

This study shows that an individual with tetraplegia rapidly learned to reach and grasp using coordinated 7D control of a high performance anthropomorphic prosthetic limb. Performance on reaching and grasping tasks consistently improved over the course of 34 training sessions completed over 13 weeks. The steady improvement demonstrates a robust approach and that the participant was consistently learning. In this context, learning is defined as the modification of the correspondence between neural activity and action. Although the mechanism of this modification is unknown, the clear and consistent change in neural tuning is a hallmark of the BMI paradigm¹⁹–22 and our results show that the neural population became more tuned to MPL movement velocity over time. The use of neural extraction algorithms derived from natural movement along with an anthropomorphic effector likely promoted the acquisition of the skill necessary to achieve the performance demonstrated here. This type of BMI control is model-based, in contrast to classifiers which operate on pre-defined libraries of discrete input-output categories. With our approach, continuous 3D translation of the hand was achieved with a few hours of training, and 7D control was performed routinely in less than 4 months. The movements were smooth, coordinated and skillful. The subject had the flexibility to explore a spectrum of arm/hand configurations to complete tasks with different strategies (webvideo 4).

Non-human primate studies established the observation-based calibration, computer assist methods, and the extraction algorithms used in the current study.⁵, 6 Previous human studies have demonstrated that intracortical electrodes could be used to provide rudimentary control of computer cursors²³ and robotic manipulators.¹² Our study builds upon this work by adding degrees of freedom which allow for more natural and coordinated movements, a broader workspace, and the ability to interact with a variety of objects in different orientations (webvideo 4). Across multiple days of testing, the participant was able to improve her ARAT score from 0 to 15–17 (out of 27). This achievement far exceeds the threshold of 5·7 points for a clinically significant improvement in function²¹, demonstrating that use of a neuroprosthetic device has the potential to provide clinically significant functional benefit. As shown in webvideo 2, movements were smooth and coordinated, with speeds approaching that of able-bodied adults. Since we used a velocity-based decoding model, the MPL speed was controlled directly by the participant. The level of control achieved by the participant allowed for subtle corrective movements and object manipulation (webvideo 3). ARAT testing began once the participant had achieved consistent 7D control on the target-based task and her performance on clinical measures was consistent during the last three weeks of testing.

We expect a number of further developments in BMI technology in the near future. A wider range of tasks will become possible by adding hand shape to the repertoire of volitional control. Tactile feedback will be transduced by fingertip sensors and transmitted to sensory cortex. Telemetry will eliminate the need for transcutaneous leads and connectors. Paralyzed arms will be reanimated by activating the subject’s own muscles, which we know is a high priority for individuals with tetraplegia.⁶,7,24 Our current results and the very rapid developments in this field show the potential to restore much of the function lost to individuals with tetraplegia or upper limb amputation, allowing them to again perform natural behaviors to more fully interact with the world around them.

PANEL: RESEARCH IN CONTEXT

Systematic Review

A literature review was conducted in PubMed without date or language restriction using the search terms “neuroprosthetics,” “brain-machine interface,” or “brain-computer interface.” Since our goal is to restore upper limb function, we restricted our survey to studies using robotic technology or functional electrical stimulation to achieve this same goal. In addition, the authors drew from their own substantive reference libraries. Two published reports of robotic arm¹² and hand²⁵ control in human subjects with motor impairments were identified. One non-human primate study reported 4D control of a robotic arm to perform a self-feeding task⁵, while two have investigated functional electrical stimulation of the upper limb.⁷, 8 For some individuals with amputation or residual muscle activity, alternative prosthetic control techniques including non-invasive approaches²⁶, targeted-reinnervation²⁷, or direct peripheral neural control²⁸, 29 may be appropriate.

Interpretation

Previous work has shown that it is possible to record neural activity from motor cortex and translate that activity to movement of an external device or one’s own muscles. However, to date, no human studies have shown that natural and complex movements can be performed consistently for a variety of tasks. Here, we have demonstrated that a person with chronic tetraplegia can perform complex and coordinated movements freely in 7D space consistently over multiple weeks of testing. This is novel because previous studies have limited control to translation dimensions, used staged control schemes, or limited the workspace to complete very structured tasks. Increasing control dimensionality allows our participant to fully explore the workspace by placing the hand in the desired 3D location and orienting the palm in three dimensions. This is also the first time that performance has been quantified using functional clinical evaluations. While most human studies rely on only a few days of performance data, we have shown that the participant learned to improve her performance consistently over many days using a variety of metrics. By using training methods and algorithms validated in non-human primate work, individuals with long term paralysis can recover the natural and intuitive command signals for hand placement, orientation, and reaching to move freely in space and interact with the environment.

Supplementary Material

01

02

03

04

05

Acknowledgments