CN-121980285-A - Port machine shadow driving data acquisition and man-machine divergence triggering method and system

CN121980285ACN 121980285 ACN121980285 ACN 121980285ACN-121980285-A

Abstract

The application relates to a method and a system for acquiring shadow driving data and triggering human-machine bifurcation, wherein the method comprises the steps of acquiring a first instruction set of a manual control system and multi-mode auxiliary judging data associated with the first instruction set, wherein the multi-mode auxiliary judging data comprise visual information, three-dimensional point cloud information, lifting appliance state parameters, port machine posture information and environment information, inputting the multi-mode auxiliary judging data into a shadow model and obtaining a second instruction set output by the shadow model, calculating the first instruction set and the second instruction set to obtain a human-machine bifurcation index, judging bifurcation triggering events according to the human-machine bifurcation index, and synchronously recording the bifurcation triggering events. The method and the system for collecting the shadow driving data of the port machine and triggering the human-machine divergence can detect the difference between the shadow model and human operation in real time in a manual driving state and obtain retraining data for the shadow model based on the difference, so that a continuous learning and closed-loop optimization system for the auxiliary driving model of the port machine is constructed.

Inventors

LU FAN
Wu nanhai
CHEN YUMING
YAN YIMIN
Weng Yuanbin
Tian Shejin
WANG CHUANZHI

Assignees

博大视野(厦门)科技有限公司

Dates

Publication Date: 20260505
Application Date: 20260202

Claims (10)

1. The method for collecting the shadow driving data of the harbor machine and triggering the human-machine divergence is characterized by comprising the following steps of: acquiring a first instruction set of a manual control system and multi-mode auxiliary judging data associated with the first instruction set, wherein the multi-mode auxiliary judging data comprise visual information, three-dimensional point cloud information, lifting appliance state parameters, port machine attitude information and environment information; inputting the multi-mode auxiliary judging data into the shadow model and obtaining a second instruction set output by the shadow model; calculating the obtained human-computer bifurcation index of the first instruction set and the second instruction set; judging a bifurcation triggering event according to the human-computer bifurcation index and synchronously recording bifurcation type and risk level of the bifurcation triggering event; If the judgment condition for judging the divergence triggering event is met in the continuous frames, marking the event corresponding to the continuous frames as a continuous divergence triggering event; The shadow model operates in parallel with the artificial pilot control system, and the second instruction set output by the shadow model is not given to the port machine.
2. The harbor machine shadow driving data collection and man-machine bifurcation triggering method according to claim 1, wherein calculating the obtained man-machine bifurcation index of the first instruction set and the second instruction set comprises: The first instruction set of the manual control system at the sampling time t i is marked as A h （t i ), and the second instruction set of the shadow model is marked as A m （t i ); Calculating a difference vector between the first instruction set and the second instruction set: ∆A(t i ) = A h （t i ）- A m （t i ); Setting a weighting coefficient vector: ; and carrying out weighted normalization on the difference vector to obtain a human-computer bifurcation index: 。
3. the harbor machine shadow driving data collection and man-machine bifurcation triggering method according to claim 2, wherein determining bifurcation triggering event according to the man-machine bifurcation degree index comprises: Setting a divergence threshold The trigger decision function is defined as: ; In the formula, Indicating that the trigger event is true; The bifurcation triggering event further comprises a continuous bifurcation triggering event, and the judging condition of the continuous bifurcation triggering event is as follows: ; In the formula, For a continuous frame window length, Is a decision threshold.
4. The harbor machine shadow driving data collection and man-machine divergence triggering method of claim 1, wherein collecting the first instruction set of the manual control system comprises: Determining acquisition content: ; Establishing a unified time axis: ; the collected content is synchronized with the control quantity corresponding to the collected content: ; In which the sampling frequency is Sampling period 。
5. The harbor machine shadow driving data collection and man-machine bifurcation triggering method according to claim 1, wherein after the bifurcation triggering event is determined, further comprising buffering the multi-modal auxiliary determination data associated with the bifurcation triggering event, the buffering comprises: creating a cache time window: ; corresponding multi-mode auxiliary judging data are determined according to the cache time window: ; In the formula, In order to pre-cache the duration before the trigger, For a post-trigger extension period; caching multi-modal auxiliary determination data in a circular buffer In (a) And indexed by timestamp and event level.
6. The method for capturing shadow driving data and triggering human-machine divergence according to claim 5, wherein the multi-modal auxiliary judgment data is buffered in a circular buffer Generating event description tuples and generating data packets by using the cached multi-mode auxiliary judgment data and event levels; generating event description tuples from cached multi-modal auxiliary judgment data and event levels as follows: ; In the formula, The divergence type; risk level; environmental context; The generated data packet is as follows: ; In the formula, encode () represents a compression and indexing operation.
7. The method for capturing harbor machine shadow driving data and triggering man-machine divergence according to claim 6, further comprising transmitting the generated data packet to the cloud end and training a shadow model of the cloud end, wherein training the shadow model of the cloud end comprises: Cloud end uses received data packet Constructing a difficult case set: ; Identifying a high-frequency scene by using a clustering algorithm: ; and adding each class of typical difficult sample to the retraining set ; Periodically updating the shadow model: 。
8. the utility model provides a harbor machine shadow driving data collection and man-machine bifurcation trigger device which characterized in that includes: the information acquisition unit is used for acquiring a first instruction set of the manual control system and multi-mode auxiliary judging data associated with the first instruction set, wherein the multi-mode auxiliary judging data comprise visual information, three-dimensional point cloud information, lifting appliance state parameters, port machine attitude information and environment information; the information input unit is used for inputting the multi-mode auxiliary judging data into the shadow model and obtaining a second instruction set output by the shadow model; The computing processing unit is used for computing the obtained human-computer bifurcation index of the first instruction set and the second instruction set; The event processing unit is used for judging the divergence triggering event according to the human-computer divergence degree index and synchronously recording the divergence type and the risk level of the divergence triggering event; If the judgment condition for judging the divergence triggering event is met in the continuous frames, marking the event corresponding to the continuous frames as a continuous divergence triggering event; The shadow model operates in parallel with the artificial pilot control system, and the second instruction set output by the shadow model is not given to the port machine.
9. A harbor machine shadow driving data acquisition and man-machine divergence triggering system, comprising: One or more memories for storing instructions, and One or more processors to invoke and execute the instructions from the memory to perform the method of any of claims 1 to 7.
10. A computer-readable storage medium, the computer-readable storage medium comprising: Program which, when executed by a processor, performs a method according to any one of claims 1 to 7.

Description

Port machine shadow driving data acquisition and man-machine divergence triggering method and system Technical Field The application relates to the technical field of mechanical intelligence, in particular to a method and a system for collecting shadow driving data of a port machine and triggering human-machine divergence. Background With the development of port automation and intellectualization, port machines (including shore bridges, field bridges, tire cranes, front cranes, stacking cranes, etc.) are gradually introducing an auxiliary driving system based on visual perception and deep learning during loading and unloading operations. The existing intelligent auxiliary driving system of the port machine mainly relies on fusion of multiple cameras, radars, IMU and lifting appliance sensors to realize the functions of target detection, obstacle avoidance early warning, track planning, lifting appliance gesture control and the like. However, the coverage and environmental diversity of the training data set are limited, and the model still has identification errors and decision deviations under the complex working conditions of strong light, rain and fog, night, swing of a lifting appliance or reflection of a special container, and the stability and safety of the system are difficult to ensure. In the prior art, in order to improve the model precision, a large amount of video data is usually collected off-line and manually marked, but the following problems exist in the mode: (1) The data acquisition efficiency is low, and the abnormal scene and critical state of the port machine in the real operation environment are difficult to cover; (2) The data screening and labeling cost is high, and the misjudgment of the model and the human operation difference are difficult to automatically identify; (3) Model updating lacks a closed-loop mechanism, and dynamic optimization of learning in use cannot be realized. Disclosure of Invention The application provides a method and a system for collecting shadow driving data of a port machine and triggering human-machine divergence, which can detect the difference between a shadow model and human operation in real time under a manual driving state and obtain retraining data for the shadow model based on the difference, thereby constructing a continuous learning and closed-loop optimization system for an auxiliary driving model of the port machine. The above object of the present application is achieved by the following technical solutions: In a first aspect, the application provides a method for collecting shadow driving data of a port machine and triggering human-machine divergence, which comprises the following steps: acquiring a first instruction set of a manual control system and multi-mode auxiliary judging data associated with the first instruction set, wherein the multi-mode auxiliary judging data comprise visual information, three-dimensional point cloud information, lifting appliance state parameters, port machine attitude information and environment information; inputting the multi-mode auxiliary judging data into the shadow model and obtaining a second instruction set output by the shadow model; calculating the obtained human-computer bifurcation index of the first instruction set and the second instruction set; judging a bifurcation triggering event according to the human-computer bifurcation index and synchronously recording bifurcation type and risk level of the bifurcation triggering event; If the judgment condition for judging the divergence triggering event is met in the continuous frames, marking the event corresponding to the continuous frames as a continuous divergence triggering event; The shadow model operates in parallel with the artificial pilot control system, and the second instruction set output by the shadow model is not given to the port machine. In a possible implementation manner of the first aspect, calculating the derived human-machine bifurcation index of the first instruction set and the second instruction set includes: The first instruction set of the manual control system at the sampling time t i is marked as A h（ti), and the second instruction set of the shadow model is marked as A m（ti); Calculating a difference vector between the first instruction set and the second instruction set: ∆A(ti) = Ah（ti）- Am（ti); Setting a weighting coefficient vector: and carrying out weighted normalization on the difference vector to obtain a human-computer bifurcation index: 。 in a possible implementation manner of the first aspect, determining the divergence triggering event according to the human-machine divergence index includes: Setting a divergence threshold The trigger decision function is defined as: In the formula, Indicating that the trigger event is true; The bifurcation triggering event further comprises a continuous bifurcation triggering event, and the judging condition of the continuous bifurcation triggering event is as follows: In the form