Redesigned Silicon Retina Powers Computer Vision

Imagine designing a robotic arm, autonomous vehicle, or surgical tool with vision that didn’t just capture images or videos, but processed them instantly, right at the sensor. Researchers at the University of Massachusetts Amherst flipped conventional computer vision systems by redesigning the architecture of silicon retinas. This powerful hardware emulates the way the human eye processes the visual world.

Researchers merged the sensing and processing units into a single device and arranged them in a highly scalable array. Compatible with standard CMOS technology, this innovation quickly enabled energy-efficient handling of the captured data, eliminating traditional bottlenecks and unlocking new possibilities for building intelligent vision systems.

Traditional vision systems suffered from collecting far more data than necessary, overwhelming processors with redundant information. Researchers tackled this inefficiency by embedding intelligence directly into the all-silicon arrays. They assigned dynamic weights to each sensor, adjusting the amplitude and polarity of their photon-induced currents through programmable gate biases.

“We gave the sensors a certain weight in terms of how significant each sensor is to understand the visual objects,” Guangyu Xu explained, associate professor of electrical and computer engineering at UMass Amherst. 


The team structured the hardware around reconfigurable “kernels” in one of their designs: compact 3x3 arrays of gate-tunable photodetectors. These kernels performed complex light detection and computation directly on incoming signals, bypassing the need for external processors. By precisely controlling gate biases, the sensors output true zeroes in non-edge areas, allowing the system to focus exclusively on signals that define object boundaries. This hardware-first approach enabled parallelized edge detection with a compressed amount of data. It also generated high-quality, pre-processed inputs for neural networks, improving accuracy in tasks like handwritten digit recognition, while significantly reducing computational demands and preserving spatial detail.

Hardware design and scaling

Researchers fabricated the photodetectors from intrinsic amorphous-silicon (α-Si) films deposited onto SiO₂/Si wafers, enclosed within oxide and metallization layers. They chose α-Si for its compatibility with CMOS processes, enabling seamless monolithic integration of their silicon arrays with standard integrated-circuit (IC) components. This integration paved the way for fully integrated, on-chip analog in-sensor visual processors capable of sensing and recognizing visual targets in near real time.

“This innovative approach allows us to build powerful new devices by simply depositing a few layers of semiconductor materials and patterning them on top of traditional microchips, making advanced intelligence vision both mass-producible and highly cost-effective,” Xu explained .

When exploring scalability, the team virtually expected no technical hurdles in increasing pixel counts of the arrays to match those in commercial cameras. Moving forward, the team proposed strategies such as three-dimensional stacking, multilayer metallization, and boosting signal-to-noise ratios in order to shrink pitch sizes and enable tightly packed, high-resolution visual processors. 

Energy efficiency gains   

Traditional image processing demanded large amounts of energy, which transferred raw pixel data from sensors to external processors. Without the ability to prioritize, sensors collected all available information and offloaded it for resource-heavy differentiation and communication protocols. UMass Amherst’s hardware filtered and processed only essential data within the sensor itself. Ignoring irrelevant inputs reduced both data transfer and computational load, which resulted in cutting energy use.   


“By moving processing to the hardware, our arrays drastically reduce energy consumption by only recording and analyzing essential data like light changes from moving objects or critical edges in static images, eliminating the massive waste of processing redundant information,” Xu said.

Data centers already consumed 2-4 percent of U.S. electricity, with some states exceeding 10 percent, according to the International Energy Agency. As demand for image processing, AI, and other data-heavy functions surged, the environmental toll climbed. With 60 percent of U.S. electricity generated by fossil fuels, increased computing led to higher greenhouse gas emissions and strained local power grids. Cooling systems drained water in vulnerable regions, and outdated hardware rapidly added to growing e-waste. UMass Amhert’s hardware’s ability to process data at the sensor level promises to reduce the energy appetite substantially, offering a path toward more sustainable computing.

Instead of digital inter-frame differentiation or complex post-processing, the members of the team benchmarked their analog hardware for real-time classification of dynamic visual information. For human motion classification, a spiking neural network (SNN) trained with the analog readout of their arrays outperformed prior digital methods that lost details in identifying these motions. For static images such as handwritten digits, artificial neural networks (ANNs) trained with the bipolar analog output of their arrays improved classification accuracy, surpassing conventional methods that diminished important image features. This sensor-level approach reduced computational load and preserved key spatial details, raising the standard for efficient visual processing. 

 
Applications across industries 

The UMass Amherst hardware unlocked powerful possibilities across critical applications. When it comes to self-driving vehicles, this hardware could deliver real-time, low-latency motion analysis, which is essential for avoiding obstacles and navigating with precision. In bioimaging, it could enable instant identification of cellular changes or structures, accelerating diagnostics and scientific discovery. For disaster monitoring, it could rapidly detect subtle signatures, trigger earlier warnings, and reduce response times without delayed post-processing that could lead to significant losses. 

By integrating sensing and processing capabilities into CMOS-compatible silicon retinas, the UMass Amherst team redefined visual processing. This approach drastically cuts energy use by eliminating redundant data processing. Using gate-controlled photodetectors, they built reconfigurable arrays capable of performing spatial and temporal visual analysis directly on-chip. Overall, this leap in efficiency and accuracy laid the groundwork for next-generation visual AI, which runs smarter, faster, and far more sustainably.   

Nicole Imeson is an engineer and writer in Calgary, Alberta.