CN-121438302-B - Food material information identification method, food material information identification device and refrigeration equipment

CN121438302BCN 121438302 BCN121438302 BCN 121438302BCN-121438302-B

Abstract

The application discloses a food material information identification method, a food material information identification device and refrigeration equipment, and belongs to the technical field of refrigeration equipment. The method comprises the steps of obtaining a target image sequence of the refrigeration equipment, carrying out target detection on each image frame of the target image sequence to obtain target detection results of the image frames, wherein the target detection results comprise food material detection results and hand detection results, tracking food material targets in the target image sequence based on the food material detection results to obtain a first target track, tracking hand targets in the target image sequence based on the hand detection results to obtain a second target track, determining interaction states of the food material targets and the hand targets based on the first target track and the second target track, and obtaining food material picking and placing information of the refrigeration equipment based on the interaction states of the food material targets and the hand targets, wherein the food material picking and placing information is used for food material management of the refrigeration equipment. The method can accurately identify the food materials to be taken and placed, and provide reliable data for food material management.

Inventors

WANG JIAN
BI YANHUA
LIU HANG
SHI DONGWEI
Ju Chengen
Dong Xingcheng
DOU ZHENDONG
WU GUOZHANG
LIU XIUDONG
WANG ZHIWEI
REN SHUFEI
TANG ZHIWEI
TANG HAIDONG
KONG LINGLEI
ZHANG HANG
WANG WEIQING
ZHANG JINGRUI
GAO HONGBO
SHAO DANDAN
XIE FAN

Assignees

青岛海尔电冰箱有限公司

Dates

Publication Date: 20260512
Application Date: 20251226

Claims (11)

1. A method of identifying food material information, the method being for a refrigeration appliance, the method comprising: Acquiring a target image sequence of the refrigeration equipment; Performing target detection on each image frame of the target image sequence to obtain a target detection result of the image frame, wherein the target detection result comprises a food material detection result and a hand detection result; Tracking food material targets in the target image sequence based on the food material detection result to obtain a first target track, and tracking hand targets in the target image sequence based on the hand detection result to obtain a second target track; determining an interaction state of the food material target and the hand target based on the first target track and the second target track; Based on the interaction state of the food material target and the hand target, obtaining food material taking and placing information of the refrigeration equipment, wherein the food material taking and placing information is used for food material management of the refrigeration equipment; Wherein the determining the interaction state of the food material object and the hand object based on the first object track and the second object track includes: Determining that the food material target and the hand target are in a contact state under the condition that the detection frame intersection ratio of the food material target and the hand target is larger than a first intersection ratio threshold; determining that the interaction state is that the hand target is in contact with the food target under the condition that the similarity of the motion models of the first target track and the second target track is larger than a first similarity threshold; The motion model similarity includes a similarity of motion directions and velocity vectors of the first target track and the second target track.
2. The method according to claim 1, wherein, in a case where the food material object and the hand object are in a contact state and a similarity of a motion model of the first object trajectory and the second object trajectory is greater than a first similarity threshold, determining that the interaction state is that the hand object grips the food material object includes: Counting continuous frame numbers of which the food material target and the hand target are in a contact state and the similarity of the motion model of the first target track and the second target track is greater than the first similarity threshold; And under the condition that the continuous frame number is larger than a first frame number threshold value, determining that the interaction state is that the hand target grasps the food material target.
3. The method for identifying food material information according to claim 1, wherein tracking the food material target in the target image sequence based on the food material detection result to obtain a first target track includes: And under the condition that the food material target is in a shielding state and the track complement condition is met, complementing the first target track based on the motion model of the food material target.
4. A food material information identification method according to claim 3, wherein the trajectory completion condition includes at least one of: the duration of the food material target in the shielding state is smaller than a first time threshold; The continuous frame number of the food material target in the shielding state is smaller than a second frame number threshold value.
5. The method for identifying food material information according to claim 1, wherein the refrigeration equipment comprises a drawer, the target detection result further comprises a drawer detection result, the obtaining the food material taking and placing information of the refrigeration equipment based on the interaction state of the food material target and the hand target comprises the following steps: and determining the food material taking and placing information corresponding to the drawer based on the interaction state and the drawer detection result.
6. The method of claim 5, wherein the drawer detection result includes a boundary of the drawer, and wherein determining the food material pick-and-place information corresponding to the drawer based on the interaction state and the drawer detection result includes: When it is determined that the hand target enters the boundary of the drawer at a first moment and the motion direction of the second target track corresponding to the hand target intersects with the plane where the drawer is located, and the hand target leaves the boundary of the drawer at a second moment, determining that the food material target has been placed in the drawer based on the interaction state corresponding to the first moment being that the hand target grabs the food material target and the interaction state corresponding to the second moment being that the hand target and the food material target are in a separated state, wherein the first moment is before the second moment; and determining that the food material target and the hand target are in a separated state under the condition that the detection frame intersection ratio of the food material target and the hand target is smaller than or equal to the first intersection ratio threshold value.
7. The method of any one of claims 1-6, wherein the hand detection result includes a detection frame of the hand target, a hand category label, and a confidence score, the hand category label being used to characterize whether the hand target is left-handed or right-handed.
8. The food material information identification method of claim 7, wherein the hand category label is determined based on wrist keypoints of the hand target.
9. A food material information identification apparatus for a refrigeration device, the apparatus comprising: The acquisition module is used for acquiring a target image sequence of the refrigeration equipment; The first processing module is used for carrying out target detection on each image frame of the target image sequence to obtain a target detection result of the image frame, wherein the target detection result comprises a food material detection result and a hand detection result; the second processing module is used for tracking food material targets in the target image sequence based on the food material detection result to obtain a first target track, and tracking hand targets in the target image sequence based on the hand detection result to obtain a second target track; The third processing module is used for determining the interaction state of the food material target and the hand target based on the first target track and the second target track; The fourth processing module is used for obtaining food material taking and placing information of the refrigeration equipment based on the interaction state of the food material target and the hand target, wherein the food material taking and placing information is used for food material management of the refrigeration equipment; the third processing module is configured to determine that the food material target and the hand target are in a contact state when the detection frame intersection ratio of the food material target and the hand target is greater than a first intersection ratio threshold; determining that the interaction state is that the hand target is in contact with the food target under the condition that the similarity of the motion models of the first target track and the second target track is larger than a first similarity threshold; The motion model similarity includes a similarity of motion directions and velocity vectors of the first target track and the second target track.
10. A refrigeration appliance, comprising: An image acquisition device and the food material information identification device as claimed in claim 9, wherein the food material information identification device is connected with the image acquisition device, and the image acquisition device is used for acquiring a target image sequence of the refrigeration device.
11. The refrigeration appliance according to claim 10, wherein the refrigeration appliance includes a drawer, and the food material information identifying means is configured to identify food material pick-and-place information corresponding to the drawer.

Description

Food material information identification method, food material information identification device and refrigeration equipment Technical Field The application belongs to the technical field of refrigeration equipment, and particularly relates to a food material information identification method, a food material information identification device and refrigeration equipment. Background With the development of technology, the food material management function of intelligent home appliances (such as refrigerators and the like) is increasingly popular. The core of food material management relies on the discernment of food material, especially the accurate discernment of food material pick-and-place action. The existing identification scheme has defects, and the food materials are easy to be mistakenly identified when being taken and put for identification, so that the taking and putting result is wrong, and the follow-up quality guarantee period reminding, automatic replenishment and other food material management functions lose reliable data basis. Disclosure of Invention The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the invention provides the food material information identification method, the food material information identification device and the refrigeration equipment, which can improve the identification accuracy of food material taking and placing and provide a reliable data basis for subsequent food material management. In a first aspect, the present application provides a method for identifying food material information, the method being used for a refrigeration device, the method comprising: Acquiring a target image sequence of the refrigeration equipment; Performing target detection on each image frame of the target image sequence to obtain a target detection result of the image frame, wherein the target detection result comprises a food material detection result and a hand detection result; Tracking food material targets in the target image sequence based on the food material detection result to obtain a first target track, and tracking hand targets in the target image sequence based on the hand detection result to obtain a second target track; determining an interaction state of the food material target and the hand target based on the first target track and the second target track; Based on the interaction state of the food material target and the hand target, obtaining food material taking and placing information of the refrigeration equipment, wherein the food material taking and placing information is used for food material management of the refrigeration equipment; Wherein the determining the interaction state of the food material object and the hand object based on the first object track and the second object track includes: Determining that the food material target and the hand target are in a contact state under the condition that the detection frame intersection ratio of the food material target and the hand target is larger than a first intersection ratio threshold; and determining that the interaction state is that the hand target is in a contact state and the food material target is gripped by the hand target under the condition that the similarity of the motion models of the first target track and the second target track is larger than a first similarity threshold. According to the food material information identification method, the hand detection result and the food material detection result are obtained through target detection, the food material targets and the hand targets are tracked through the independent tracking chains, each food material target corresponds to a first target track, each hand target corresponds to a second target track, the influence of shielding phenomena frequently occurring in the food material taking and placing process on tracking stability is reduced, the space-time relationship between the hand targets and the food material targets is dynamically calculated according to the first target track and the second target track, the interaction state of the food material targets and the hand targets is judged, the food material taking and placing information for food material management is obtained through analysis, the identification accuracy of food material taking and placing is improved, and a reliable data basis is provided for subsequent food material management. According to an embodiment of the present application, in a case where the food material object and the hand object are in a contact state, and a similarity of a motion model of the first object track and the second object track is greater than a first similarity threshold, determining that the interaction state is that the hand object grips the food material object includes: Counting continuous frame numbers of which the food material target and the hand target are in a contact state and the similarity of the motion model of the f