US-12626118-B2 - Two-dimensional photonic convolutional acceleration system and device for convolutional neural network

US12626118B2US 12626118 B2US12626118 B2US 12626118B2US-12626118-B2

Abstract

The present invention discloses a two-dimensional photonic convolutional acceleration system and device for convolutional neural network, comprising: a multi-wavelength light source, a signal source to be convolved, a modulator, a dispersion module, a 1×M power divider, an optical fiber delay array, a microring weighting array chip, a convolutional kernel matrix control unit, a trans-impedance amplifier array, and an acquisition and processing unit. The present invention realizes two-dimensional convolutional acceleration based on wavelength-time interleaving technology, a single modulator can realize the optical domain loading of the signal, and the convolutional operation speed is only limited to the speed of the modulator. The present invention can realize two-dimensional convolutional kernel convolutional acceleration of two-dimensional data in a single signal cycle based on two-level delay and microring weighting array chip, and solve the problem of data redundancy in traditional methods, the scheme is simple and efficient. The present invention realizes the control of convolutional kernel matrix coefficient based on the microring weighting array chip, can realize the fast update of convolutional kernel matrix coefficient, and is suitable for real-time data processing applications, the balanced photodetector can realize arbitrary convolutional kernel coefficient weighting.

Inventors

Qingshui GUO
Kun Yin
Chen Ji

Assignees

Zhejiang Lab

Dates

Publication Date: 20260512
Application Date: 20230304
Priority Date: 20220902

Claims (9)

1 . A two-dimensional photonic convolutional acceleration system for convolutional neural network, wherein, the system has a memory and a processor comprising: a multi-wavelength light source, generating multi-wavelength optical signals containing N wavelengths and transmitting the multi-wavelength optical signals to a modulator; a signal source to be convolved, converting an original two-dimensional data to be convolved into a one-dimensional signal to be convolved, and transmitting the signal to be convolved to the modulator; the modulator, loading the signal to be convolved onto the multi-wavelength optical signal, obtaining a multi-wavelength modulated optical signal, and transmitting the multi-wavelength modulated optical signal to a dispersion module; the dispersion module, realizing equal-interval dispersion delay for the N sub-modulated optical signals corresponding to N wavelengths in the multi-wavelength modulated optical signal, obtaining the multi-wavelength modulated optical signal after dispersion delay, and transmitting the multi-wavelength modulated optical signal after dispersion delay to a 1×M power divider; the 1×M power divider, dividing the multi-wavelength modulated optical signal after dispersion delay into M-channel multi-wavelength modulated optical signal, and transmitting the M-channel multi-wavelength modulated optical signal to an optical fiber delay array; the optical fiber delay array, which is composed of M-segment optical fibers, increasing the equal-interval second-level delay of M-channel multi-wavelength modulated optical signal in turn to obtain a M-channel multi-wavelength modulated optical signal with second-level delay, and transmitting the M-channel multi-wavelength modulated optical signal with second-level delay to a microring weighting array chip; the microring weighting array chip, comprising M microring weighting units, which are respectively used to weight and sum the N sub-modulated optical signals contained in each of the M-channel multi-wavelength modulated optical signal with second-level delay, and obtaining M first-level weighted summation electrical signals, and transmit M first-level weighted summation electrical signals to a trans-impedance amplifier array; a convolutional kernel matrix control unit, providing a convolutional kernel coefficient control signal to the microring weighting array chip; the trans-impedance amplifier array, comprising M trans-impedance amplifiers, amplifying the M first-level weighted summation electrical signals respectively, and carrying out second-level summation of the amplified M first-level weighted summation electrical signals to obtain a second-level weighted summation electrical signals, and transmit the second-level weighted summation electrical signals to an acquisition and processing unit; the acquisition and processing unit, collecting the second-level weighted summation electrical signals, and reconstructing it into a characteristic signal corresponding to the signal to be convolved.
2 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 1 , wherein, the dispersion module is a dispersion fiber, a Bragg dispersion grating or a spatial dispersion module, a time of the equal-interval dispersion delay is: Δt=1/S M , wherein, Δt is a duration of a single symbol of the signal to be convolved, and S M is a symbol rate of the signal to be convolved.
3 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 1 , wherein, the microring weighting array chip is based on a silicon-based process or III-V materials-based processes.
4 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 1 , wherein, the microring weighting unit is composed of a straight-through waveguide, a coupling waveguide, a balanced photodetector and N microring resonators, the N microring resonators are series connection with the coupling waveguide through the straight-through waveguide, an input port of the straight-through waveguide is used as an input port of the microring weighting unit, an output port of the coupling waveguide and an output port of the straight-through waveguide are respectively connected with the balanced photodetector, an output port of the balanced photodetector is used as an output port of the microring weighting unit.
5 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 4 , wherein, the N microring resonators is used to control the coupling coefficient and transmission coefficient of N adjacent microring resonators according to the convolutional kernel coefficient control signal output by the convolutional kernel matrix control unit, and successively couple the N sub-modulated optical signals corresponding to N wavelengths in the M-channel multi-wavelength modulated optical signal with second-level delay into the coupling waveguide according to different coupling coefficients, at the same time, the N sub-modulated optical signals corresponding to N wavelengths are transmitted in the through waveguide with different transmission coefficients, and a coupling waveguide weighted modulated optical signal and a through waveguide weighted modulated optical signal are obtained.
6 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 5 , wherein, the balanced photodetector is used for photoelectric conversion of the coupling waveguide weighted modulated optical signal and the through waveguide weighted modulated optical signal to obtain M first-level weighted summation electrical signals.
7 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 1 , wherein, the multi-wavelength light source is a multi-wavelength laser, a mode-locked laser, a femtosecond laser, an optical frequency comb generator, an optical soliton optical frequency comb generator or a single-frequency signal externally modulated electro-optical modulator.
8 . The two-dimensional photonic convolutional acceleration system for convolutional neural network according to claim 1 , wherein, the modulator is a Mach-Zehnder modulator or an electric absorption modulator.
9 . A two-dimensional photonic convolutional acceleration device for convolutional neural network, wherein, comprising a memory and one or more processors, the memory stores executable code, when the executable code is executed by the one or more processors, it is used to realize a two-dimensional photonic convolutional acceleration system for convolutional neural network described in claim 1 .

Description

This application claims priority of Chinese Application No. 202211070507.5, filed Sep. 2, 2022, which is hereby incorporated by reference. TECHNICAL FIELD The present invention relates to the field of photonic computing technology, in particular to a two-dimensional photonic convolutional acceleration system and device for convolutional neural network. DESCRIPTION OF RELATED ART Artificial intelligence is now widely used in machine vision, natural language processing, automatic driving and other fields. As one of the important models of artificial intelligence technology, artificial neural network is widely used because of its excellent generalization ability and stability, in the actual data processing process, convolutional operation is the pre-operation of artificial neural network, and occupies most of the computational power of artificial intelligence operation. Because the current electronic chip uses the classical computer structure that separates the program space and data space, the data workflow between the storage unit and the computing unit is unstable and the power consumption is high, which limits the efficiency of network model training. The common solution is to improve the operational efficiency by improving the integration of electronic chips or by in-memory computing, however, due to the micro-quantum characteristics and macro-high-frequency response characteristics of electronic chips, these technologies also face great challenges. Photonic technology, which uses photonic as information carriers, has the characteristics of large bandwidth, low loss and parallelizability. At present, researchers have been attracted to apply photonic technology to artificial intelligence (see [Ashtiani F, Geers A J, Aflatouni F. An on-chip photonic deep neural network for image classification[J]. Nature, 2022: 1-6.]). The combination of photonic technology and traditional neural network is expected to give full play to the advantages of the two technologies, break through the technical development bottleneck of traditional electronic neural network with high power consumption, long delay and limited speed, and solve the limitations of technical problems of traditional electronic technology (see [Huang C, Fujisawa S, de Lima T F, et al. A silicon photonic—electronic neural network for fibre nonlinearity compensation. Nature Electronics, 2021, 4(11): 837-844.]). Firstly, the photonic neural network adopts the analog computing architecture, and the storage and calculation are carried out at the same time, which can improve the computing speed and reduce the computing delay. Secondly, based on the essential characteristics of optical transmission media, optical links have low loss characteristics, which can indirectly reduce system power consumption. Finally, comparing with electronic devices, the effective working bandwidth of photonic devices has increased by several orders of magnitude, which is more suitable for the high speed real-time operation of neural networks. For example, the scheme (see Xu X, Tan M, Corcoran B, et al. “11 TOOS photonic convolutional accelerator for optical neural networks,” Nature, vol. 589, no. 7840, pp. 45-51, 2021.]), provides the convolutional operation of the signal to be convolved and the fully connect feedforward neural network based on dispersion technology, the computing speed is close to the latest chip based on electronic technology, but the power consumption of this scheme has been greatly reduced, which provides a reliable basis for the application of photonic neural network. Therefore, we propose a two-dimensional photonic convolutional acceleration system and device for convolutional neural network to solve the above technical problems. SUMMARY OF THE INVENTION The purpose of the present invention is to provide a two-dimensional photonic convolutional acceleration system and device for convolutional neural network, which realizes two-level delay based on a dispersion module and an optical fiber delay array, and realizes the signal strength weighting of different wavelengths by combining with a microring weighting array chip, so, realizes the two-dimensional convolutional kernel matrix coefficient weighting in a single signal cycle, and solves the problems of data redundancy and large volume in traditional methods, moreover, the convolutional kernel matrix can be flexibly expanded, and is applicable to the problem of multi-dimensional data convolutional operation. The technical scheme adopted by the present invention is as follows: A two-dimensional photonic convolutional acceleration system for convolutional neural network, comprising: a multi-wavelength light source, generating multi-wavelength optical signals containing N wavelengths and transmitting the multi-wavelength optical signals to a modulator;a signal source to be convolved, converting an original two-dimensional data to be convolved into a one-dimensional signal to be convolved, and transmitting the signa