Search

CN-121998554-A - Warehouse checking method and device based on machine vision

CN121998554ACN 121998554 ACN121998554 ACN 121998554ACN-121998554-A

Abstract

The application discloses a machine vision-based warehouse checking method and device, and relates to the technical field of warehouse checking. The method comprises the steps of generating an optimal inventory path of a warehouse through an improved group intelligent optimization algorithm, collecting multi-mode machine vision data of shelves in the warehouse, carrying out feature level fusion on the multi-mode machine vision data through a multi-mode fusion network to generate shelf digital characterization, inputting the shelf digital characterization into a commodity identification model to obtain a commodity identification result, carrying out manual rechecking on the commodity identification result, integrating all commodity identification results to generate an electronic inventory, comparing the electronic inventory with warehouse management system data, and carrying out online incremental learning on the multi-mode fusion network and/or the commodity identification model by periodically using a plurality of high-quality training samples. The method solves the problems of low warehouse checking efficiency, poor accuracy, unintelligible path planning and incapability of self-adaptive updating of the model in the prior art.

Inventors

  • WANG ENHUI
  • XIE HUA
  • TAO XUEYUN

Assignees

  • 广西建设职业技术学院

Dates

Publication Date
20260508
Application Date
20260128

Claims (10)

  1. 1. A machine vision-based warehouse inventory method, the method comprising: generating an optimal inventory path of the warehouse by using an improved group intelligent optimization algorithm based on the warehouse map information, the inventory task list and the real-time environment state; Enabling the mobile inventory robot to travel along the optimal inventory path and synchronously collecting multi-mode machine vision data of a goods shelf in a warehouse; carrying out space-time alignment on the multi-mode machine vision data, and carrying out feature level fusion through a pre-trained multi-mode fusion network to generate a shelf digital representation of a shelf in a warehouse; Inputting the goods shelf digital representation into a pre-trained commodity identification model, and carrying out hierarchical identification to obtain a commodity identification result; manually rechecking commodity classification results of commodity examples, of which the uncertainty measurement exceeds a preset measurement threshold value, in the commodity identification results to obtain commodity rechecking classification results of the commodity examples, and taking the commodity rechecking classification results as a high-quality training sample; integrating commodity classification results and/or commodity rechecking classification results of all commodity examples in all commodity identification results to generate an electronic inventory list, and comparing the electronic inventory list with warehouse management system data to generate a difference report; And periodically using a plurality of high-quality training samples to perform online incremental learning on the multi-modal fusion network and/or the commodity identification model to obtain an updated multi-modal fusion network and/or an updated commodity identification model.
  2. 2. The machine vision-based warehouse inventory method of claim 1, wherein the improved swarm intelligence optimization algorithm is a hybrid enhanced ant colony algorithm that incorporates chaotic sequence initialization, levy flight mechanism, and PSO algorithm.
  3. 3. The machine vision based warehouse inventorying method as claimed in claim 2, wherein generating an optimal inventorying path of the warehouse using the improved swarm intelligence optimization algorithm based on the warehouse map information, the inventorying task list, and the real-time environmental status, comprises: Based on a cloud server, constructing a grid map of a warehouse according to the map information of the warehouse, converting the grid map into a weighted directed graph, and giving a corresponding dynamic weight to each side in the directed graph according to the real-time environment state of the warehouse; setting key parameters and cost functions of a hybrid enhanced ant colony algorithm; acquiring an inventory task list from a warehouse management system, and distributing corresponding priority weights for each shelf node in the directed graph; Generating a chaotic sequence by using Logistic mapping, and mapping the chaotic sequence to a pheromone space to obtain initial pheromone distribution; In each iteration, starting from the starting point of the directed graph for each ant, selecting a shelf node for next access based on a mixed state transition probability formula according to initial or updated pheromone distribution until one path construction is completed, and obtaining an ant construction path; Acquiring a cost value of a construction path by using a cost function, and taking the construction path with the lowest cost value as a local optimal path; If all ants complete one-time path construction, updating the pheromones on all construction paths to obtain updated pheromones, and entering into the next iteration; and taking the construction path with the lowest cost value in all the local optimal paths as a global optimal path, namely the optimal inventory path of the warehouse, until the iteration number reaches the maximum iteration number or a plurality of continuous generations of local optimal paths are not changed.
  4. 4. A machine vision based warehouse inventory method as claimed in claim 3, in which the multi-modal machine vision data includes 2D image data, 3D point cloud data and thermal imaging data; The commodity identification result comprises commodity classification results of a plurality of commodity examples, wherein the commodity classification results comprise commodity position identifiers of the corresponding commodity examples, commodity SKUs and commodity quantity.
  5. 5. The machine vision-based warehouse inventory method of claim 4, wherein the multi-modal fusion network comprises a 2D image feature extraction module constructed based on ResNet algorithm, a 3D point cloud feature extraction module constructed based on PointNet ++ algorithm, a thermal imaging feature extraction module constructed based on lightweight CNN algorithm and a feature fusion module constructed based on a Transformer algorithm, wherein the 2D image feature extraction module, the 3D point cloud feature extraction module and the thermal imaging feature extraction module are all connected with the feature fusion module; The commodity identification model comprises a cargo space positioning module constructed based on a U-Net algorithm, a commodity target detection module constructed based on PointRCNN algorithm and a commodity fine classification module constructed based on a YOLO algorithm provided with a triple Loss function, which are connected in sequence.
  6. 6. The machine vision-based warehouse inventory method of claim 5, wherein controlling the mobile inventory device to travel along the optimal inventory path and synchronously collect multi-modal machine vision data of shelves within the warehouse comprises: The method comprises the steps of issuing an optimal inventory path of a warehouse to a mobile inventory gateway, extracting a shelf node sequence related to each mobile inventory robot in mobile inventory equipment based on the mobile inventory gateway, and sending the shelf node sequence to the corresponding mobile inventory robot; enabling the mobile inventory robot to run along an optimal inventory sub-path formed by the shelf node sequences, and sending a synchronous trigger signal to multi-mode data acquisition equipment of the same machine when each shelf node is reached; according to the synchronous trigger signal, using multi-mode data acquisition equipment to synchronously acquire the original data of the shelf nodes, and uploading the original data to a mobile inventory gateway; Based on the mobile checking gateway, data preprocessing is carried out on the original data uploaded by the mobile checking robot, so that multi-mode machine vision data are obtained, and the multi-mode machine vision data are uploaded to the cloud server.
  7. 7. The machine vision based warehouse inventory method of claim 6, wherein performing space-time alignment on the multi-modal machine vision data and feature level fusion through a pre-trained multi-modal fusion network generates a shelf digital representation of a shelf within the warehouse, comprising: Based on a cloud server, receiving multi-mode machine vision data uploaded by each mobile inventory robot; according to internal parameters of a preset multi-mode data acquisition device, carrying out space calibration and alignment on multi-mode machine vision data to obtain aligned multi-mode machine vision data, and inputting the aligned multi-mode machine vision data into a pre-trained multi-mode fusion network; Respectively extracting a 2D image feature map, a 3D point cloud feature map and a thermal imaging feature map of the aligned multi-mode machine vision data by using a 2D image feature extraction module, a 3D point cloud feature extraction module and a thermal imaging feature extraction module of the multi-mode fusion network; splicing the 2D image feature map, the 3D point cloud feature map and the thermal imaging feature map in the channel dimension to obtain a spliced feature map, and inputting the spliced feature map to a feature fusion module of the multi-mode fusion network; and fusing the spliced feature graphs according to the self-attention weight generated by the feature fusion module to obtain the shelf digital representation of the shelf in the warehouse.
  8. 8. The machine vision-based warehouse inventory method of claim 7, wherein inputting the shelf digital representation into a pre-trained commodity identification model for hierarchical identification to obtain a commodity identification result comprises: Inputting the shelf digital representation to a pre-trained commodity identification model; according to the goods shelf digital representation, carrying out goods location positioning by using a goods location positioning module of the goods identification model to obtain accurate pixel masks of each goods location area and text contents of goods location identification; Intercepting corresponding sub-shelf digital representations in the shelf digital representations according to the accurate pixel mask of each goods space area, and inputting the sub-shelf digital representations to a commodity target detection module of a commodity identification model; according to the sub-shelf digital representation, commodity target detection is carried out by using a commodity target detection module, so that the position and the size of each commodity instance are obtained, and the corresponding 3D boundary frame is positioned; extracting fusion characteristics in a 3D boundary box of each commodity instance, and inputting the fusion characteristics into a commodity fine classification module of a commodity identification model; Mapping the fusion features to a high-dimensional space by using a commodity fine classification module to obtain corresponding high-dimensional feature vectors, and carrying out nearest neighbor search with a pre-established SKU feature library to obtain commodity SKUs and commodity quantity of each commodity instance; Integrating the goods space identification, the goods SKU and the goods quantity of the same goods instance to obtain corresponding goods classification results, and integrating the goods classification results of all goods instances to obtain goods identification results.
  9. 9. The machine vision-based warehouse inventory method of claim 8, wherein the manually rechecking the commodity classification result of the commodity instance with the uncertainty metric exceeding the preset metric threshold in the commodity identification result to obtain the commodity rechecking classification result of the commodity instance, and the commodity rechecking classification result is used as a high-quality training sample, and the machine vision-based warehouse inventory method comprises the following steps: acquiring uncertainty measurement of commodity classification results of each commodity instance in commodity identification results; If the uncertainty measure is larger than a preset measure threshold, marking the multi-mode machine vision data of the commodity instance and the commodity classification result as a sample to be rechecked; And sending the sample to be rechecked to a manual rechecking terminal, receiving a correct label confirmed manually, combining the correct label with the multi-mode machine vision data of the commodity instance to form a commodity rechecking classification result, and storing the commodity rechecking classification result as a high-quality training sample in a high-quality sample pool.
  10. 10. A machine vision-based warehouse inventorying device for implementing a warehouse inventorying method as claimed in any one of claims 1-9, the device comprising: The path planning unit is used for generating an optimal inventory path of the warehouse by using an improved group intelligent optimization algorithm based on the warehouse map information, the inventory task list and the real-time environment state; the data acquisition unit is used for enabling the mobile inventory robot to travel along the optimal inventory path and synchronously acquiring multi-mode machine vision data of the goods shelves in the warehouse; The feature fusion unit is used for carrying out space-time alignment on the multi-mode machine vision data, carrying out feature level fusion through a pre-trained multi-mode fusion network and generating a shelf digital representation of a shelf in a warehouse; the commodity identification unit is used for inputting the goods shelf digital representation into a pre-trained commodity identification model, and carrying out hierarchical identification to obtain a commodity identification result; The manual rechecking unit is used for manually rechecking the commodity classification result of the commodity instance, the uncertainty measurement of which exceeds the preset measurement threshold value, in the commodity identification result to obtain the commodity rechecking classification result of the commodity instance, and the commodity rechecking classification result is used as a high-quality training sample; The inventory generation unit is used for integrating the commodity classification results and/or commodity rechecking classification results of all commodity examples in all commodity identification results, generating an electronic inventory, comparing the electronic inventory with warehouse management system data and generating a difference report; And the continuous training unit is used for regularly using a plurality of high-quality training samples to perform online incremental learning on the multi-modal fusion network and/or the commodity identification model so as to obtain an updated multi-modal fusion network and/or an updated commodity identification model.

Description

Warehouse checking method and device based on machine vision Technical Field The invention relates to the technical field of warehouse checking, in particular to a warehouse checking method and device based on machine vision. Background With the rapid development of electronic commerce and logistics industry, the efficiency and accuracy of warehouse management become critical. Warehouse inventory is used as a core link of inventory management, and the traditional mode mainly relies on manual work. The manual inventory is time-consuming, labor-consuming and low in efficiency, and the problems of wrong inventory, missing inventory and the like are easy to occur, so that inventory data are inaccurate, and normal operation of enterprises is affected. To solve these problems, some automated inventory schemes have emerged in the prior art. For example, the two-dimensional code or the RFID tag is used for checking, and although the efficiency is improved, the existing warehouse is required to be modified on a large scale, the labeling cost is high, the tag is easy to damage and fall off, and the maintenance cost is also not good. The other scheme is that a mobile robot with a single camera is used for image recognition inventory, but the method has limited recognition accuracy and insufficient robustness under the conditions of complex illumination, commodity stacking, similar appearance and the like. In addition, path planning is a key to efficient operation of the mobile inventory robot. Conventional path planning algorithms (e.g., aAlgorithms) typically consider only the shortest distance and ignore dynamic environmental factors (e.g., human traffic) and task priorities within the warehouse, resulting in poor robot performance, possibly delayed due to congestion, or failure to prioritize high value, fast turnaround commodity areas. Finally, existing machine learning models are typically static after deployment, cannot accommodate new merchandise, new packaging, or environmental changes, and model performance may degrade over time, requiring periodic offline retraining, cumbersome processes, and delayed responses. Therefore, how to provide a high-efficiency, accurate and self-adaptive warehouse checking scheme is a technical problem to be solved in the current intelligent warehouse field. Disclosure of Invention The invention provides a machine vision-based warehouse checking method and device, which solve the problems of low warehouse checking efficiency, poor accuracy, unintelligible path planning and incapability of self-adaptive updating of a model in the prior art. In a first aspect, an embodiment of the present invention provides a machine vision-based warehouse inventory method, where the method includes: generating an optimal inventory path of the warehouse by using an improved group intelligent optimization algorithm based on the warehouse map information, the inventory task list and the real-time environment state; Enabling the mobile inventory robot to travel along the optimal inventory path and synchronously collecting multi-mode machine vision data of a goods shelf in a warehouse; carrying out space-time alignment on the multi-mode machine vision data, and carrying out feature level fusion through a pre-trained multi-mode fusion network to generate a shelf digital representation of a shelf in a warehouse; Inputting the goods shelf digital representation into a pre-trained commodity identification model, and carrying out hierarchical identification to obtain a commodity identification result; manually rechecking commodity classification results of commodity examples, of which the uncertainty measurement exceeds a preset measurement threshold value, in the commodity identification results to obtain commodity rechecking classification results of the commodity examples, and taking the commodity rechecking classification results as a high-quality training sample; integrating commodity classification results and/or commodity rechecking classification results of all commodity examples in all commodity identification results to generate an electronic inventory list, and comparing the electronic inventory list with warehouse management system data to generate a difference report; And periodically using a plurality of high-quality training samples to perform online incremental learning on the multi-modal fusion network and/or the commodity identification model to obtain an updated multi-modal fusion network and/or an updated commodity identification model. The technical scheme provided by the embodiment of the application at least has the following beneficial effects: The mixed enhanced ant colony algorithm of chaotic initialization, levy flight and PSO algorithm is introduced, the path length, real-time congestion, task priority and electric quantity limitation are comprehensively considered, the generated inventory path is better, congestion can be effectively avoided, high-value tasks are preferentially pr