US-12620200-B2 - Method for operating a technical system and technical system

US12620200B2US 12620200 B2US12620200 B2US 12620200B2US-12620200-B2

Abstract

A method for operating a technical system and a technical system. The method includes providing for at least one class at least one class attribute comprising a description for members of the class, providing features characterizing a digital image, determining a class of the at least one class that classifies the digital image depending on the features, and determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image. The at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image. The method further includes operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute.

Inventors

Jiaojiao Zhao
Sadaf Gulshad
Jan Hendrik Metzen
Smeulders Arnold

Assignees

ROBERT BOSCH GMBH

Dates

Publication Date: 20260505
Application Date: 20230623
Priority Date: 20220630

Claims (10)

1 . A method for operating a technical system, the method comprising the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the determining of the class that classifies the digital image includes: providing at least one attribute query; mapping the features and the at least one attribute query with a decoder to projected features; mapping the projected features with a first layer of at least one neural network to the at least one first attribute; and mapping the features with a second layer of the at least one neural network to the class that classifies the digital image.
2 . The method according to claim 1 , further comprising: training a neural network to determine the class that classifies the digital image and the at least one first attribute, wherein the training includes minimizing a mean square error between the at least one class attribute and the at least one first attribute and/or minimizing a cross-entropy loss that depends on the class that classifies the digital image.
3 . The method according to claim 1 , further comprising: receiving the digital image; and outputting the class that classifies the digital image and the at least one first attribute.
4 . The method according to claim 1 , further comprising: capturing the digital image with a camera and/or a radar sensor and/or a LiDAR sensor and/or an ultrasonic sensor and/or a motion sensor and/or a thermal image sensor; and/or the operating of the technical system includes outputting the class that classifies the digital image and/or the at least one first attribute characterizing a traffic sign and/or a road surface and/or a pedestrian and/or a vehicle.
5 . A method for operating a technical system, the method comprising the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the determining of the class that classifies the digital image includes: providing at least one attribute query; mapping the features and the at least one attribute query with a decoder to projected features; mapping the projected features with a first layer of at least one neural network to the at least one first attribute; and determining the class that classifies the digital image depending on a dot product between the at least one first attribute and the at least one class attribute.
6 . A method for operating a technical system, the method comprising the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the determining of the class that classifies the digital image includes: providing at least one attribute query; mapping the features and the at least one attribute query with a decoder to projected features; mapping the projected features with a first layer of at least one neural network to the at least one first attribute; mapping the projected features with a second layer of the at least one neural network to at least one second attribute; and determining the class that classifies the digital image depending on a dot product between the at least one first attribute and the at least one second attribute.
7 . A method for operating a technical system, the method comprising the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the providing of the features includes: providing the digital image; mapping the digital image with a neural network to at least one token characterizing a local structure in the digital image, or at least one edge or at least one line in the digital image; determining a fixed length output, wherein the determining of the fixed length output includes reshaping the at least one token in a spatial dimension, and/or splitting the at least one token with overlapping and/or padding and a stride, determining the features with an encoder depending on the output.
8 . A technical system for classifying digital images, the technical system comprising: at least one processor; and at least one memory, wherein the at least one memory is configured to store computer-readable instructions that, when executed by the at least one processor, cause the technical system to perform the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the determining of the class that classifies the digital image includes: providing at least one attribute query; mapping the features and the at least one attribute query with a decoder to projected features; mapping the projected features with a first layer of at least one neural network to the at least one first attribute; and mapping the features with a second layer of the at least one neural network to the class that classifies the digital image.
9 . The technical system according to claim 8 , further comprising: a sensor configured to capture the digital image, the sensor including a camera sensor and/or a radar sensor and/or a LiDAR sensor and/or an ultrasonic sensor and/or a motion sensor and/or a thermal image sensor; and/or an output configured to output the class that classifies the digital image and/or the at least one first attribute characterizing a traffic sign and/or a road surface and/or a pedestrian and/or a vehicle.
10 . A non-transitory computer-readable medium on which is stored a computer program including computer-readable instructions for operating a technical system, the instructions, when executed by computer, causing the computer to perform the following steps: providing, for at least one class, at least one class attribute including a description for members of the class; providing features characterizing a digital image; determining a class of the at least one class that classifies the digital image depending on the features; determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute includes an explanation for classifying the digital image with the class that classifies the digital image; and operating the technical system depending on the class that classifies the digital image and/or depending on the at least one first attribute, wherein the determining of the class that classifies the digital image includes: providing at least one attribute query; mapping the features and the at least one attribute query with a decoder to projected features; mapping the projected features with a first layer of at least one neural network to the at least one first attribute; and mapping the features with a second layer of the at least one neural network to the class that classifies the digital image.

Description

CROSS REFERENCE The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application No. DE 10 2022 206 722.3 filed on Jun. 30, 2022, which is expressly incorporated herein by reference in its entirety. FIELD The present invention concerns a method for operating a technical system and technical system. BACKGROUND INFORMATION Neural networks predict classes on images typically without providing an explanation. It has also been shown that predictions of neural networks are not robust, that is: performance deteriorates considerably under domain shift. Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., Xue, H.; “Towards robust vision transformer,” arXiv preprint arXiv:2105.07926 (2021) describes general aspects of neural networks. SUMMARY The present invention provides a method for operating a technical system. According to an example embodiment of the present invention, the method for operating the technical system comprises providing for at least one class at least one class attribute comprising a description for members of the class, providing features characterizing a digital image, determining a class of the at least one class that classifies the digital image depending on the features, and determining at least one first attribute depending on the at least one class attribute provided for the class that classifies the digital image, wherein the at least one first attribute comprises an explanation for classifying the digital image with the class that classifies the digital image, and operating the technical system depending on the class that classifies the digital image and/or depending on the first attribute. A class attribute comprises a description for the members of the class. The at least one first attribute comprises a description of the class that is predicted for the digital image. This description explains the classification result. The at least one first attribute is a localized attribute for this digital image and by this provides an explanation of the prediction. As a by-product, this also improves the robustness of the prediction. According to an example embodiment of the present invention, determining the class that classifies the digital image preferably comprises providing at least one attribute query, mapping the features and the at least one attribute query with a decoder to projected features, mapping the projected features with a first layer of at least one neural network to the at least one first attribute, and mapping the features with a second layer of the at least one neural network to the class that classifies the digital image. This method classifies the digital image with an attribute-guided network comprising the first and second layer. The attribute queries to predict per-image attributes and perform class predictions are learned attribute queries that are provided from within the network. According to an example embodiment of the present invention, determining the class that classifies the digital image preferably comprises providing at least one attribute query, mapping the features and the at least one attribute query with a decoder to projected features, mapping the projected features with a first layer of at least one neural network to the at least one first attribute, and determining the class that classifies the digital image depending on a dot product between the at least one first predicted attribute and the at least one class attribute. This method classifies the digital image with an attribute-embedded network comprising the first layer. The attribute queries to predict per-image attributes and perform class predictions are learned attribute queries that are provided from within the network. According to an example embodiment of the present invention, determining the at least one class that classifies the digital image preferably comprises providing at least one attribute query, mapping the features and the at least one attribute query with a decoder to projected features, mapping the projected features with a first layer of at least one neural network to the at least one first attribute, mapping the projected features with a second layer of the at least one neural network to at least one second attribute, and determining the class that classifies the digital image depending on a dot product between the at least one first attribute and the at least one second attribute. This method classifies the digital image with an auto-attribute network comprising the first and second layer. The attribute queries to predict per-image attributes and perform class predictions are learned attribute queries that are provided from within the network. According to an example embodiment of the present invention, the method may further comprise training a neural network to determine the class that classifies the digital image and the at least one first attribute, wherein training comprises minimizing a mean square error between the at least one class at