US-12626508-B2 - Household appliance video analysis

US12626508B2US 12626508 B2US12626508 B2US 12626508B2US-12626508-B2

Abstract

A video captured by a camera assembly of a household appliance includes a plurality of frames. Methods of analyzing the video may include calculating a plurality of median frames of the video, determining an area of one of the plurality of frames contains an object of interest, and isolating the determined area for analysis of the object of interest. Methods of analyzing the video may include identifying an object of interest in a minimum number of consecutive frames of the video, determining a motion vector of the object of interest based on the consecutive frames of the video, and comparing the motion vector with predetermined in and out vectors to determine whether the object of interest was added to or removed from the household appliance. Such methods may also include adding or removing the object of interest to or from a virtual inventory of the household appliance.

Inventors

Michael Goodman Schroeder

Assignees

HAIER US APPLIANCE SOLUTIONS, INC.

Dates

Publication Date: 20260512
Application Date: 20230705

Claims (10)

1 . A method of analyzing a video captured by a camera assembly of a household appliance, the video comprising a plurality of frames, the method comprising: calculating a plurality of median frames of the video, wherein each median frame is a median frame for a predetermined number of frames of the plurality of frames; calculating frame differences, wherein calculating frame differences comprises subtracting each of the predetermined number of frames from the respective median frame; creating a subtracted image from each of the predetermined number of frames and the respective median frame based on the calculated frame differences; creating a smoothed mask from the subtracted image; determining an area of one of the plurality of frames contains an object of interest; and isolating the determined area for analysis of the object of interest.
2 . The method of claim 1 , further comprising comparing a masked portion of the smoothed mask to a predetermined area threshold, and wherein determining the area of one of the plurality of frames contains the object of interest comprises determining the masked portion is greater than the predetermined area threshold.
3 . The method of claim 1 , wherein isolating the determined area for analysis of the object of interest comprises transmitting the determined area to an artificial intelligence model for object classification and location.
4 . The method of claim 1 , wherein the analysis of the object of interest comprises determining a motion vector of the object of interest.
5 . The method of claim 4 , further comprising comparing the motion vector with predetermined in and out vectors to determine whether the object of interest was added to or removed from the household appliance.
6 . The method of claim 5 , further comprising adding the object of interest to a virtual inventory of the household appliance in response to determining the object of interest was added to the household appliance or removing the object of interest from the virtual inventory in response to determining the object of interest was removed from the household appliance.
7 . The method of claim 4 , wherein the analysis of the object of interest further comprises identifying the object of interest in a minimum number of consecutive frames of the video, wherein the motion vector is determined based on the consecutive frames of the video.
8 . A method of analyzing a video captured by a camera assembly of a household appliance, the video comprising a plurality of frames, the method comprising: identifying an object of interest in a minimum number of consecutive frames of the video; determining a motion vector of the object of interest based on the consecutive frames of the video; comparing the motion vector with predetermined in and out vectors to determine whether the object of interest was added to or removed from the household appliance; calculating a plurality of median frames of the video, wherein each median frame is a median frame for a predetermined number of frames of the plurality of frames, and wherein the object of interest is identified based on the plurality of median frames of the video; calculating frame differences by subtracting each of the predetermined number of frames from the respective median frame; creating a subtracted image from each of the predetermined number of frames and the respective median frame based on the calculated frame differences; creating a smoothed mask from the subtracted image; and updating a virtual inventory of the household appliance in response to determining whether the object of interest was added to or removed from the household appliance.
9 . The method of claim 8 , further comprising comparing a masked portion of the smoothed mask to a predetermined area threshold, and wherein identifying the object of interest comprises determining the masked portion is greater than the predetermined area threshold, wherein the object of interest is depicted within the masked portion.
10 . The method of claim 9 , further comprising classifying and locating the object of interest using an artificial intelligence model.

Description

FIELD OF THE INVENTION The present subject matter relates generally to household appliances, such as refrigerator appliances, and more particularly to systems and methods for analyzing videos captured by such appliances. BACKGROUND OF THE INVENTION Household appliances are utilized generally for a variety of tasks by a variety of users. For example, a household may include such appliances as laundry appliances, e.g., a washer and/or dryer, kitchen appliances, e.g., a refrigerator, an oven, etc., along with other various appliances. Some household appliances may include imaging systems or camera assemblies which capture various images, e.g., still images and/or video in and around the appliance. For example, such systems may be used to identify and track objects added to or removed from the appliance, such as food items in a refrigerator appliance or oven appliance, or laundry articles in a washing machine appliance or dryer appliance. Videos captured by such household appliances generally include background information, such as parts of the appliance itself or static objects, in addition to the object(s) of interest, e.g., food items or laundry articles, that are moved within, into, or out of the household appliance. The background information may not be pertinent, e.g., to inventory identification or tracking analysis, and analyzing such background information may increase the time, processing resources, and complexity of analyzing the video. Accordingly, improved methods of analyzing videos captured by household appliances would be useful. BRIEF DESCRIPTION OF THE INVENTION Aspects and advantages of the invention will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the invention. In an exemplary embodiment, a method of analyzing a video captured by a camera assembly of a household appliance is provided. The video includes a plurality of frames. The method includes calculating a plurality of median frames of the video. Each median frame is a median frame for a predetermined number of frames of the plurality of frames. The method also includes calculating frame differences and determining an area of one of the plurality of frames contains an object of interest. The method further includes isolating the determined area for analysis of the object of interest. In another exemplary embodiment, a method of analyzing a video captured by a camera assembly of a household appliance is provided. The video includes a plurality of frames. The method includes identifying an object of interest in a minimum number of consecutive frames of the video and determining a motion vector of the object of interest based on the consecutive frames of the video. The method further includes comparing the motion vector with predetermined in and out vectors to determine whether the object of interest was added to or removed from the household appliance. The method also includes adding the object of interest to a virtual inventory of the household appliance in response to determining the object of interest was added to the household appliance or removing the object of interest from the virtual inventory in response to determining the object of interest was removed from the household appliance. These and other features, aspects and advantages of the present invention will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS A full and enabling disclosure of the present invention, including the best mode thereof, directed to one of ordinary skill in the art, is set forth in the specification, which makes reference to the appended figures. FIG. 1 provides a front view of a refrigerator appliance according to an exemplary embodiment of the present subject matter. FIG. 2 provides a perspective view of the refrigerator appliance of FIG. 1. FIG. 3 provides a front view of the refrigerator appliance of FIG. 1 with doors thereof in an open position. FIG. 4 provides a front view of another exemplary refrigerator appliance with doors thereof in an open position according to one or more additional exemplary embodiments of the present subject matter. FIG. 5 provides a flow diagram of an exemplary method for operating a refrigerator appliance according to one or more exemplary embodiments of the present subject matter. FIG. 6 provides a flow diagram of another exemplary method for operating a refrigerator appliance according to one or more additional exemplary embodiments of the present subject matter. FIG. 7 illustrates an exemplary median frame according to one or more exemplary embodiments of the present subject matter. FIG. 8 illustrates an exemplary frame according to one or m