CN-121997999-A - Neural network hardware acceleration method and system based on self-adaptive activation function circuit

CN121997999ACN 121997999 ACN121997999 ACN 121997999ACN-121997999-A

Abstract

The invention discloses a neural network hardware acceleration method and system based on a self-adaptive activation function circuit, belonging to the field of neural network hardware acceleration; the method comprises the steps of constructing an adaptive activation function circuit by using a curve fitting method, storing adaptive parameters through memristors, constructing a memristor cross array, converting input voltage into current output by the memristor cross array based on ohm law and kirchhoff current law, inputting the current output into the adaptive activation function circuit, carrying out nonlinear processing on the adaptive parameters to obtain network layer output after the adaptive activation function processing, calculating a loss function by the network layer output in an off-chip calculation mode, calculating gradients of the adaptive parameters, updating the adaptive parameters in the memristor according to the gradients of the adaptive parameters, and dynamically optimizing the adaptive activation function form. The adaptive activation function circuit accelerates the convergence of the model more quickly by updating parameters in real time.

Inventors

Xu Zuyu
DAI YUEHUA
TIAN JINGYU
WANG ZHITONG
ZHANG ZIYUE
LU JUNBO
NIU JIAHAO
WANG HAOJIE
ZHU YUNLAI
WU ZUHENG

Assignees

安徽大学

Dates

Publication Date: 20260508
Application Date: 20251219

Claims (10)

1. The neural network hardware acceleration method based on the self-adaptive activation function circuit is characterized by comprising the following steps of: Constructing an adaptive activation function circuit by using a curve fitting method, and storing adaptive parameters by using a memristor; the method comprises the steps of constructing a memristor cross array, converting input voltage into current output through the memristor cross array based on ohm law and kirchhoff current law, inputting the current output into an adaptive activation function circuit, performing nonlinear processing through adaptive parameters, and obtaining network layer output after the adaptive activation function processing; Training the neural network by using the adaptive activation function, outputting the network layer by an off-chip computing mode, computing a loss function, and computing the gradient of the adaptive parameter; and updating the self-adaptive parameters in the memristor according to the gradient of the self-adaptive parameters, and dynamically optimizing the self-adaptive activation function form.
2. The neural network hardware acceleration method based on the adaptive activation function circuit according to claim 1, wherein the types of the adaptive activation function circuit include a Sigmoid function circuit, a Tanh function circuit and a ReLU function circuit, the Sigmoid function circuit is implemented by curve fitting through nonlinear output characteristics of a CMOS inverter, and the Tanh function circuit is implemented by applying an inverting amplifier on the basis of the Sigmoid circuit.
3. The neural network hardware acceleration method of claim 2, wherein the activation function expression in the adaptive activation function is: Wherein, the The self-adaptive parameters are stored through a memristor; 、、 the adaptive activation functions of the Sigmoid function circuit, the Tanh function circuit and the ReLU function circuit are respectively provided, and x is the input voltage of the activation function circuit.
4. The neural network hardware acceleration method based on the adaptive activation function circuit according to claim 1, wherein the process of converting the input voltage into the current output z through the memristor cross array is: Wherein w is a weight matrix stored by the memristor cross array, x is an input of the memristor cross array, and c is an array bias term.
5. An adaptive activation function circuit is characterized by comprising any one of a Sigmoid function circuit, a Tanh function circuit and a ReLU function circuit; The Sigmoid function circuit comprises two memristors, three inverting proportional amplifiers and a CMOS inverter, wherein the two memristors, the three inverting proportional amplifiers and the CMOS inverter are used for storing adaptive parameters, the output curve of the CMOS inverter is used for fitting a Tanh curve, the output is used for obtaining the Sigmoid function curve through the inverting proportional amplifier, and when an input signal x enters the function circuit, the parameters are realized through the inverting proportional amplifiers firstly Outputting a Tanh (ax/2) product with an input x/2 through a CMOS inverter, converting the Tanh function into a Sigmoid function through an inverting proportional amplifier, and finally realizing the product of a parameter b and the Sigmoid (ax) through the inverting proportional amplifier; The Tanh function circuit comprises two memristors, two inverting proportional amplifiers and a CMOS inverter, wherein the two memristors are used for storing self-adaptive parameters, the two inverting proportional amplifiers and the CMOS inverter are used for outputting curve fitting, when an input signal x enters the function circuit, the product of a parameter a and the input signal x is realized through the inverting proportional amplifiers, then Tanh (ax) is outputted through the CMOS inverter, and finally the product of a parameter b and the Tanh (ax) is realized through the inverting proportional amplifiers; The ReLU function circuit comprises two memristors for storing adaptive parameters, two diode switches for controlling positive and negative conduction, two inverting proportional operational amplifiers for realizing multiplication and inverting proportional operational amplifiers for converting current into voltage, when an input signal x enters the circuit, the voltage on the right side of a diode is clamped to 0, when the input is positive, a lower branch diode is conducted, the input signal x realizes the product of a parameter b and the input x through the inverting proportional operational amplifiers to obtain positive output current, otherwise, when the input is negative, the upper branch diode is conducted, the input signal x realizes the product of a parameter a and the input x through the inverting proportional operational amplifiers to obtain negative output current, the positive and negative currents flow into the same node and the current is converted into voltage output through the inverting proportional operational amplifiers to obtain the negative output current A function.
6. Use of the adaptive activation function circuit of claim 5 in CNN image recognition.
7. A neural network hardware acceleration system based on an adaptive activation function circuit, for executing the neural network hardware acceleration method based on an adaptive activation function circuit according to any one of claims 1 to 4, comprising: The activation function operation module is used for constructing a self-adaptive activation function circuit by utilizing a curve fitting method and storing self-adaptive parameters through the memristor; The matrix operation module is used for constructing a memristor cross array, converting input voltage into current output through the memristor cross array based on ohm law and kirchhoff current law, inputting the current output into the self-adaptive activation function circuit, performing nonlinear processing through self-adaptive parameters, and obtaining network layer output after the self-adaptive activation function processing; The off-chip computing module is used for training the neural network hardware by using the self-adaptive activation function, outputting the network layer in an off-chip computing mode, computing a loss function and computing the gradient of the self-adaptive parameter; And the dynamic optimization module updates the self-adaptive parameters in the memristor according to the gradient of the self-adaptive parameters and dynamically optimizes the self-adaptive activation function form.
8. A computer storage medium storing a readable program, wherein the program, when executed, is capable of instructing a computing device to perform the neural network hardware acceleration method based on an adaptive activation function circuit as recited in any one of claims 1-4.
9. An electronic device is characterized by comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform operations corresponding to the neural network hardware acceleration method based on the adaptive activation function circuit according to any one of claims 1 to 4.
10. A computer program product comprising computer instructions that instruct a computing device to perform operations corresponding to the neural network hardware acceleration method based on adaptive activation function circuitry as claimed in any one of claims 1-4.

Description

Neural network hardware acceleration method and system based on self-adaptive activation function circuit Technical Field The invention belongs to the field of neural network hardware acceleration, and particularly relates to a neural network hardware acceleration method and system based on a self-adaptive activation function circuit. Background The activation function (activation function) is an integral part of the neural network training process and is primarily responsible for introducing nonlinear capabilities into the network so that it can learn and represent complex patterns. If only inter-layer linear mapping is performedNo matter how many layers are stacked on the network, the whole is still equivalent to single matrix multiplication, the model space is strictly limited in an affine transformation set, a nonlinear decision boundary cannot be expressed, and complex tasks such as image generation, language modeling, strategy control and the like cannot be met. The activation function can map each layer of output to a high-dimensional nonlinear manifold through element-by-element nonlinear processing, and if the general approximation theorem is met mathematically, the network can approximate any continuous function with any precision only by enough neurons and nonlinear activation. The activation function is used as a key component for introducing nonlinearity in the neural network, and directly influences the learning ability and generalization performance of the model. Traditional activation functions such as ReLU, sigmoid, tanh and the like have simple structure and high calculation efficiency, but often have performance bottlenecks in processing complex tasks or specific data distribution, are difficult to meet the requirements of different levels or different tasks, and cannot show universal advantages in all data sets and network architectures. Compared with the traditional activation function, the self-adaptive activation function has stronger flexibility and expression capability, is beneficial to improving the nonlinear modeling capability of a model, accelerating the convergence speed and improving the gradient propagation problem in a deep network. However, most of the adaptive activation functions still depend on the calculation implementation of a software layer, and along with the continuous expansion of the scale of a deep neural network model and the diversification of application scenes, especially in the edge calculation fields of the internet of things, intelligent terminals, unmanned operation and the like, the traditional software calculation mode based on a general processor faces great challenges in terms of speed, energy efficiency and instantaneity. Disclosure of Invention Aiming at the defects of the prior art, the invention aims to provide a neural network hardware acceleration method and a neural network hardware acceleration system based on a self-adaptive activation function circuit, which solve the problems in the prior art. The aim of the invention can be achieved by the following technical scheme: the neural network hardware acceleration method based on the adaptive activation function circuit comprises the following steps: Constructing an adaptive activation function circuit by using a curve fitting method, and storing adaptive parameters by using a memristor; the method comprises the steps of constructing a memristor cross array, converting input voltage into current output through the memristor cross array based on ohm law and kirchhoff current law, inputting the current output into an adaptive activation function circuit, performing nonlinear processing through adaptive parameters, and obtaining network layer output after the adaptive activation function processing; Training the neural network by using the adaptive activation function, outputting the network layer by an off-chip computing mode, computing a loss function, and computing the gradient of the adaptive parameter; and updating the self-adaptive parameters in the memristor according to the gradient of the self-adaptive parameters, and dynamically optimizing the self-adaptive activation function form. Further, the adaptive activation function circuit comprises a Sigmoid function circuit, a Tanh function circuit and a ReLU function circuit, wherein the Sigmoid function circuit is realized by curve fitting through the nonlinear output characteristic of a CMOS inverter, and the Tanh function circuit is realized by applying an inverting amplifier on the basis of the Sigmoid circuit. Further, the activation function expression in the adaptive activation function is: Wherein, the The self-adaptive parameters are stored through a memristor;、、 the adaptive activation functions of the Sigmoid function circuit, the Tanh function circuit and the ReLU function circuit are respectively provided, and x is the input voltage of the activation function circuit. Further, the process of converting the input voltage into the