US-12620219-B2 - Method and assistance system for checking samples for defects

US12620219B2US 12620219 B2US12620219 B2US 12620219B2US-12620219-B2

Abstract

A method for checking samples for defects is provided, in which image data of the samples are recorded and classified into predeterminable defect categories by a defect detection algorithm, and the samples classified into a defect category are represented in a multi-dimensional confusion matrix as a classification result of the defect detection algorithm, characterized in that miniature images which reproduce the image data are assigned according to the classified defect categories of the image data to segments of the confusion matrix which represent the defect categories, and these miniature images are displayed visually, the miniature image is assigned by an interaction with a user or a software robot to a different segment from the assigned segment of the confusion matrix, and is either provided as training image data for the defect detection algorithm or is output as training image data for the defect detection algorithm.

Inventors

Silvio Becher
Felix Buggenthin
JOHANNES KEHRER
Ingo Thon
Stefan Hagen Weber

Assignees

SIEMENS AKTIENGESELLSCHAFT

Dates

Publication Date: 20260505
Application Date: 20201116
Priority Date: 20191209

Claims (11)

1 . A method for checking samples for defectiveness, the method comprising: recording image data of the samples to be checked; classifying the image data associated with the samples into predefinable defect categories by a defect recognition algorithm; presenting a number of the samples classified in the predefinable defect categories in a multi-dimensional confusion matrix as a classification result of the defect recognition algorithm; assigning miniature images which reproduce the image data associated with the samples according to the predefinable defect categories of the image data, to segments of the multi-dimensional confusion matrix which represent the predefinable defect categories, wherein the miniature images are presented visually within the segments, and, as a function of the assigning the miniature images, a size of the miniature images is adapted to optically fit into the segment, wherein each miniature image is a thumbnail representation of image data of a respective sample that is visually embedded within a matrix segment corresponding to the classified defect category of the respective sample, wherein the multi-dimensional confusion matrix comprises a first axis representing defect categories classified by the defect recognition algorithm and a second axis representing defect categories assigned by a user or software robot, and wherein the miniature images are visually embedded within individual matrix segments defined by intersections of the first axis and the second axis; assigning, by way of an interaction with a user or a software robot, a miniature image into a different segment than the assigned segment of the multi-dimensional confusion matrix, which is either provided as training image data for the defect recognition algorithm or is provided and output as training image data for the defect recognition algorithm, wherein the assigning comprises dragging and dropping the miniature image from one matrix segment to another matrix segment within the multi-dimensional confusion matrix.
2 . The method as claimed in claim 1 , wherein a size of the miniature image is individually adapted after selection of the miniature image.
3 . The method as claimed in claim 1 , wherein a number of miniature images within a segment is optically identified.
4 . The method as claimed in claim 1 , wherein the miniature images are positioned within a segment of the multi-dimensional confusion matrix in a manner sorted according to at least one predefinable criterion, the at least one predefinable criterion being a confidence value, an entropy over all defect categories, a dimension reduction, a similarity, or a distance metric.
5 . The method as claimed in claim 1 , wherein an assignment of one or more miniature images from an assigned segment into a different segment of the multi-dimensional confusion matrix is carried out by way of comparison of selected or selectable miniature image regions on a basis of the at least one criterion or on a basis of at least one further criterion including a poor confidence value, a similar image brightness, a visually similar sample shape and/or a recognizable defect.
6 . An assistance system for checking samples for defectiveness by a defect recognition device, which records image data of the samples to be checked and classifies the image data associated with the samples into predefinable defect categories by a defect recognition algorithm, wherein a number of the samples classified in the defect categories are presentable in a multi-dimensional confusion matrix as a classification result of the defect recognition algorithm, the assistance system comprising: at least one processing unit having at least one storage unit, wherein the processing unit is configured to assign miniature images which reproduce the image data associated with the samples, according to the classified defect categories of the image data, to segments of the multi-dimensional confusion matrix which represent the predefinable defect categories, wherein the multi-dimensional confusion matrix comprises a first axis representing defect categories classified by the defect recognition algorithm and a second axis representing defect categories assigned by a user or software robot, and wherein the miniature images are visually embedded within individual matrix segments defined by intersections of the first axis and the second axis, and to present the miniature images visually within the segments, and, as a function of assigning the miniature images, a size of the miniature images is adapted to optically fit into the segment, wherein each miniature image is a thumbnail representation of image data of a respective sample that is visually embedded within a matrix segment corresponding to the sample's classified defect category, to assign a miniature image, by way of an interaction with a user or a software robot, into a different segment than the assigned segment of the multi-dimensional confusion matrix, wherein the assigning comprises dragging and dropping the miniature image from one matrix segment to another matrix segment within the multi-dimensional confusion matrix, either to provide the miniature image as training image data for the defect recognition algorithm in the at least one storage unit or to provide the miniature image as training image data for the defect recognition algorithm in the at least one storage unit and to output the miniature image at an output unit.
7 . The apparatus as claimed claim 6 , wherein the miniature images are positionable within a segment in a manner sorted according to at least one predefinable criterion.
8 . The apparatus as claimed in claim 6 , wherein the number of miniature images within a segment is optically identifiable.
9 . A computer program product, comprising a non-transitory computer readable hardware storage device having computer readable program code stored therein, said program code executable by a processor of a computer system to implement a method as claimed in claim 1 , comprising program code parts designed to carry out the method.
10 . The method as claimed in claim 1 , wherein the samples are checked for defectiveness while a manufacturing machine is running or after a manufacturing process.
11 . The apparatus as claimed in claim 6 , wherein the samples are checked for defectiveness while a manufacturing machine is running or after a manufacturing process.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to PCT Application No. PCT/EP2020/082279, having a filing date of Nov. 16, 2020, which claims priority to EP Application No. 19214373.3, having a filing date of Dec. 9, 2019, the entire contents both of which are hereby incorporated by reference. FIELD OF TECHNOLOGY The following relates to a method for checking samples for defectiveness, to an assistance system for checking samples for defectiveness, and to a computer program product. BACKGROUND In industrial production and manufacturing, machine learning methods are being used more and more often in order to be able to take automated decisions. One example here is the application of classification methods to recorded image data of samples, which can represent produced parts, in order to assess the quality of the samples and to sort out faulty or defective samples. The recorded image data are assigned to different classes on the basis of common features (e.g. “part OK”, “dispose of part”, “rework necessary”). A further application example is the classification of time series data from sensors. In this case, a time series is interpreted as a sample that ultimately also supplies data. The time series are classified with the aid of an algorithm (e.g. “normal”, “start up machine”, “wear too high”, “abnormal operating state”). On the basis of the result of the classification algorithm, actions can be derived, e.g. stopping the machine, arranging inspection, exchanging tools or adapting machine parameters. The classification is effected with the aid of a classification or defect recognition algorithm, e.g. neural network, support vector machine, etc. In the present case, defectiveness is understood to mean not only the fault or defect on the sample itself, but also a “defective” classification. By way of example, the classification should have been “abnormal operating state” instead of “start up machine”. It is possible to use the following methods: batch labeling: a group of data acquire the same label, without the individual data being viewed. In this case, groups are formed which, with high probability, include exclusively members of one class (e.g. all data from normal operation without incidents).individual labeling: data points are presented successively to an expert for assessment. This is usually done without direct supporting context information (e.g. visualization of all assignments) with respect to the other classes. In the case of classification or defect recognition algorithms, it is customary for the learnt algorithm to be applied to a data set for which the correct classes are known, but which itself would not be used for the training of the algorithm. The result of the algorithm can thus be compared with the so-called ground truth. Metrics, such as e.g. precision, hit rate, accuracy or F-score, give indications about the quality of the algorithm. One conventional presentation here is the confusion matrix, in which the number of correctly or incorrectly classified data can be read. Examples of confusion matrices are known from the patent publications U.S. Pat. No. 9,020,237 B2, U.S. Pat. No. 8,611,675 B2, EP 2277143 A2. In this case, the evaluator obtains a quantitative understanding of the quality of the algorithm. For a domain expert, however, it is essential to consider the associated images/time series in detail in order to acquire deeper insights about why the data were classified incorrectly or correctly. It is possible for the unknown or unclassified image data to be presented clearly to a domain expert in order to classify the image data themselves. An initial training set is obtained with the classified image data. In order to be able to assess the prediction quality of the classification or defect recognition algorithm, an evaluation of the algorithm after the training phase is expedient. However, it is often difficult to decide whether the quality of the classification is sufficient. Moreover, the evaluation is often not carried out by the developer of the classification or defect recognition algorithm, who has the mathematical background, but rather by domain experts. The latter usually have the expertise themselves semantically to understand the images or image data from samples or else time series, to classify them and to compare their knowledge with the results of the defect recognition algorithm. In this case, samples can be various parts that may need to be produced, e.g. housing, screw, gearwheel, etc. Objects such as e.g. blood cells, in medicine are also conceivable, which can be recognized according to their type and correspondingly classified. For this purpose, the image data have to be clearly presented to the evaluator with their respective classification. It is only with the expert's domain knowledge that the defect recognition algorithm can be evaluated and improved. The optimized defect recognition algorithm can then be used in an automated manner i