CN-121980498-A - Memristor neural network cloud edge cooperation method and system based on multi-mode sensing

CN121980498ACN 121980498 ACN121980498 ACN 121980498ACN-121980498-A

Abstract

The application provides a memristor neural network cloud edge cooperation method and system based on multi-mode sensing. The method mainly comprises the steps of distributing multi-mode data subjected to synchronous acquisition and space-time alignment to an FPGA and a memristor array for parallel processing, splicing the processed multi-mode data in channel dimensions to generate a multi-channel fusion characteristic tensor, storing and calculating the multi-channel fusion characteristic tensor by the memristor array into a whole to calculate and output an advanced characteristic map, receiving a differential update package issued by a cloud, selectively updating weights in the memristor array according to the update package, and performing in-situ fine adjustment on part of weights in the memristor array based on locally acquired data. The method and the constructed system can effectively compensate the memristor array, improve the calculation precision, and realize the rapid self-adaption of local scenes and the continuous efficient iteration of global knowledge.

Inventors

ZHANG DAINAN
CHEN YINGBO
ZHANG ZHONGYUAN
WEN TIANLONG
JIN LICHUAN
LIU CHENG
LIAO YULONG

Assignees

电子科技大学

Dates

Publication Date: 20260505
Application Date: 20260119

Claims (10)

1. The memristor neural network cloud edge cooperation method based on multi-mode sensing is characterized by comprising the following steps of: S1, distributing multi-mode data subjected to synchronous acquisition and space-time alignment to an FPGA and a memristor array for parallel processing, wherein the FPGA is configured to execute nonlinear preprocessing operation, and the memristor array is configured to execute linear preprocessing operation based on analog vector matrix multiplication; s2, splicing the processed multi-mode data in the channel dimension to generate a multi-channel fusion characteristic tensor; s3, under a feature extraction mode, programming pre-stored feature extraction kernel weights to the memristor array, and carrying out memory calculation integrated calculation on the multi-channel fusion feature tensor by using the memristor array, outputting an advanced feature map and uploading the advanced feature map to a cloud; in an edge recognition mode, programming the complete lightweight neural network weight to the memristor array, and performing end-to-end memory integrated reasoning calculation on the multi-channel fusion characteristic tensor by utilizing the memristor array to directly output a recognition result; and S4, receiving a differential update package issued by a cloud, wherein the differential update package is generated based on global model optimization, selectively updating weights in the memristor array according to the update package, and performing in-situ fine adjustment on part of weights in the memristor array based on locally acquired data so as to adaptively compensate non-ideal characteristics and environmental changes of a device.
2. The memristor neural network cloud edge cooperative method based on multi-modal sensing, which is disclosed by claim 1, is characterized in that in the step S1, synchronous acquisition comprises triggering a plurality of target sensors through hardware synchronous signals simultaneously, acquiring time-aligned multi-modal raw data, space-time alignment comprises uniformly registering the multi-modal raw data to the same coordinate system based on pre-calibrated parameters, and generating pixel-level aligned multi-modal data, wherein the plurality of target sensors comprise a visual camera, a laser radar and an infrared sensor, and the multi-modal raw data comprises RGB images, laser radar point clouds and a thermal amplitude intensity map.
3. The memristor neural network cloud edge cooperative method based on multi-modal sensing of claim 2, wherein uniformly registering the multi-modal raw data to the same coordinate system based on pre-calibrated parameters comprises: And accurately projecting the laser radar point cloud from a radar coordinate system and the thermal amplitude intensity image from a thermal imager coordinate system to an image plane of the visual camera through perspective transformation or resampling algorithm by an external parameter matrix obtained through a checkerboard calibration method, and generating a depth confidence map and a registration heat map aligned with the pixel level of the RGB image.
4. The memristor neural network cloud edge cooperative method based on multi-mode sensing of claim 3, wherein in step S1, the parallel processing of the FPGA and the memristor array includes: Performing nonlinear preprocessing operation through the FPGA, performing median filtering on the RGB image and performing contrast limited self-adaptive histogram equalization on the registered heat map, performing linear preprocessing operation based on analog vector matrix multiplication through the memristor array, and performing convolution operation on the depth confidence map.
5. The memristor neural network cloud edge cooperative method based on multi-mode sensing of claim 4, wherein the step S2 specifically includes: and splicing the processed RGB image, the depth confidence level image, the registration heat image and the speed characteristic image provided by the laser radar in the channel dimension to generate a multi-channel fusion characteristic tensor with a fixed size, wherein the dimension is [ H, W, C ], and each spatial position (x, y) of the tensor integrates multi-mode physical attribute information.
6. The memristor neural network cloud edge cooperative method based on multi-mode sensing of claim 5, wherein the step S3 specifically includes: In a feature extraction mode, an FPGA (field programmable gate array) programs pre-stored feature extraction convolution kernel weights to a memristor array through a configuration bus according to task instructions, then step fusion feature tensors are converted into analog voltage signal streams through an on-chip high-speed interface and a data converter, the analog voltage signal streams are input into the memristor array, the array performs convolution operation in a memory calculation integrated mode, a high-level feature map with greatly compressed data volume is output, and the high-level feature map is packaged by the FPGA and uploaded to a cloud end In the edge recognition mode, when local real-time reasoning is needed, the FPGA dynamically writes the complete weight of the lightweight neural network into the memristor array through the same configuration bus, the memristor array is used as a complete reasoning engine, the full-flow calculation from convolution, activation to full connection is performed on input data, the category, the bounding box and the confidence of a target are directly output, and the end-to-end intelligent edge recognition is realized.
7. The memristor neural network cloud edge cooperative method based on multi-mode sensing of claim 6, wherein the step S4 specifically includes: The cloud end gathers the feature graphs uploaded by the multi-edge nodes to perform incremental training, optimizes the global model, calculates the weight difference of the new model and the old model to generate a differential update package when updating, wherein the differential update package only comprises a weight address, a target value and a key weight mark which need to be changed, the key weight mark is obtained according to weight sensitivity analysis, a training module of the edge end performs high-precision closed loop write verification on the key weight after receiving the key weight, performs quick batch write on the non-key weight, and Based on locally acquired data, adopting an approximate counter-propagation algorithm to perform in-situ fine adjustment on part of weights in the memristor array, and enabling a network to be self-adaptive to local hardware and environment changes by writing and verifying non-ideal effects of a closed loop compensation device by weight increment generated by fine adjustment.
8. The memristor neural network cloud edge cooperative method based on multi-mode sensing of claim 7, wherein the step of obtaining the key weight mark according to the weight sensitivity analysis specifically comprises the following steps: Weight sensitivity S (i) =is calculated for each weight wi 2L/ W2 x sigma 2 (wi), wherein S (i) is the sensitivity score of the ith weight wi, 2L/ Wi2 is the curvature of the loss function L at the weight wi, sigma 2 (wi) is the expected write variance of the memristor cell to which the weight wi is mapped; sorting the weights according to the sensitivity from large to small, and when the sensitivity of the two weights is the same, selecting the weight with larger amplitude for priority sorting, wherein the first five percent is marked as the key weight; For the key weight, calling a predicted pulse parameter issued from the cloud, applying a precise write pulse to a target unit in the memristor array, immediately performing read verification, and comparing the measured conductance value with a target value; for non-critical weight, a preset standard pulse parameter is adopted to perform quick and batch writing operation so as to improve the overall efficiency.
9. Memristor neural network cloud edge cooperative system based on multi-mode sensing is characterized by comprising: the heterogeneous collaborative preprocessing module is used for distributing the synchronously acquired and space-time aligned multi-mode data to the FPGA and the memristor array for parallel processing, wherein the FPGA is configured to execute nonlinear preprocessing operation, and the memristor array is configured to execute linear preprocessing operation based on analog vector matrix multiplication; The feature fusion module is used for splicing the processed multi-mode data in the channel dimension to generate a multi-channel fusion feature tensor; The dynamic reconstruction calculation module is used for programming pre-stored feature extraction kernel weights to the memristor array in a feature extraction mode, carrying out calculation integration calculation on the multi-channel fusion feature tensor by utilizing the memristor array, outputting an advanced feature map and uploading the advanced feature map to a cloud; in an edge recognition mode, programming the complete lightweight neural network weight to the memristor array, and performing end-to-end memory integrated reasoning calculation on the multi-channel fusion characteristic tensor by utilizing the memristor array to directly output a recognition result; The collaborative updating module is used for receiving a differential updating packet issued by the cloud, generating the differential updating packet based on global model optimization, and selectively updating the weight in the memristor array according to the updating packet; and Based on locally acquired data, in-situ fine adjustment is performed on part of weights in the memristor array so as to adaptively compensate non-ideal characteristics and environmental changes of the device.
10. An electronic device, comprising: A memory for storing one or more programs; A processor; The method of any of claims 1-8 is implemented when the one or more programs are executed by the processor.

Description

Memristor neural network cloud edge cooperation method and system based on multi-mode sensing Technical Field The application relates to the field of artificial intelligence multi-mode data processing, in particular to a memristor neural network cloud edge cooperation method and system based on multi-mode sensing. Background With the rapid development of artificial intelligence technology, the contradiction between the computing power and the energy consumption of chips is increasingly prominent. Traditional computation adopts a von neumann architecture, and a memory unit and a processing unit of the von neumann architecture are mutually separated, so that data is required to be frequently carried out when the computation is executed, thereby causing high energy consumption and low bandwidth efficiency, namely a so-called von neumann bottleneck, and severely restricting the efficient deployment of an artificial intelligent model. To break through this bottleneck, a computationally integrated architecture is proposed. The architecture integrates the storage and calculation functions into the same hardware unit, wherein the memristor is an ideal device for realizing calculation integration due to the characteristics that the resistance value of the memristor can be stored in a nonvolatile manner and can be continuously adjusted through external excitation. In the memristor array, output current is generated by multiplying input voltage and conductance, vector matrix multiplication operation can be efficiently completed based on kirchhoff current law, and the method is very suitable for matrix operation existing in a large number in a neural network, so that a hardware basis is provided for high-energy-efficiency artificial intelligence calculation. However, in a complex edge intelligent sensing scene, single-mode data is often difficult to meet the requirement of robustness, and multi-mode sensing information such as vision, laser radar, infrared and laser radar is required to be subjected to fusion analysis. The multi-mode fusion is generally divided into three types of data level, feature level and decision level, wherein the data level fusion faces the problems of data isomerism, asynchronism, mass and the like, and high requirements are put on the real-time processing capability of edge equipment. The traditional acquisition-uploading-cloud processing mode is difficult to meet the real-time requirement due to large transmission delay and high bandwidth requirement. When the memristor memory technology is applied to multi-mode edge sensing, the following challenges still exist that inherent non-ideal characteristics such as conductivity nonlinearity, volatility and the like of the memristor can cause the reduction of calculation precision, but the traditional one-time weight mapping method cannot be used for effective compensation, and is difficult to adapt to variable factors such as device aging, temperature drift and the like, the existing acceleration scheme based on the memristor is mostly used as a coprocessor with a fixed function, and lacks dynamic coordination capability with a sensing front end and a cloud system, and model updating often needs full heavy load, so that quick self-adaption for local scenes and efficient continuous iteration of global knowledge are difficult to realize. Disclosure of Invention The application aims to provide a memristor neural network cloud edge cooperative method and system based on multi-mode sensing, which can effectively compensate a memristor array to improve calculation accuracy and realize rapid self-adaption of local scenes and continuous and efficient iteration of global knowledge. The application is realized in the following way: in a first aspect, the application provides a memristor neural network cloud edge cooperation method based on multi-mode sensing, which comprises the following steps: S1, distributing multi-mode data subjected to synchronous acquisition and space-time alignment to an FPGA and a memristor array for parallel processing, wherein the FPGA is configured to execute nonlinear preprocessing operation, and the memristor array is configured to execute linear preprocessing operation based on analog vector matrix multiplication; s2, splicing the processed multi-mode data in the channel dimension to generate a multi-channel fusion characteristic tensor; S3, under a feature extraction mode, programming pre-stored feature extraction kernel weights to a memristor array, carrying out memory calculation integrated calculation on the multi-channel fusion feature tensors by using the memristor array, outputting an advanced feature map and uploading the advanced feature map to a cloud; In an edge recognition mode, programming the complete lightweight neural network weight to a memristor array, and performing end-to-end memory calculation integrated reasoning calculation on the multi-channel fusion characteristic tensor by using the memristor array to