EP-4736070-A1 - DEVICE AND METHOD FOR A SIMILARITY EVALUATION

Abstract

The invention relates to a process for determining attention, comprising the following steps: carrying out a first vector-matrix multiplication in such a way that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal; converting each generated electrical signal into a pulse, the occurrence of which depends on the magnitude of the respectively generated signal, wherein the larger the signal, the later a pulse occurs; determining attention from the pulses. A second vector-matrix multiplication can be performed. An apparatus for carrying out a method includes a first electrical circuit configured to carry out a vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals. The apparatus includes a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse. The apparatus includes a third electronic circuit configured to determine attention from the time-dependent electrical pulses. A second vector-matrix multiplication can be carried out with the apparatus.

Inventors

• SIEGEL, SEBASTIAN
• FINKBEINER, Jan Robert

Assignees

• Forschungszentrum Jülich GmbH

Dates

Publication Date

20260506

Application Date

20250520

Claims (1)

1. Forschungszentrum Jülich GmbH G70313WO / PT 1.3110 PCT Claims 1. A method for determining attention comprising the steps of: performing a first vector-matrix multiplication such that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal; converting each generated electrical signal into a pulse whose occurrence depends on the magnitude of the electrical signal, wherein a pulse occurs later the larger the electrical signal; determining attention using the pulses. 2. The method according to claim 1, characterized in that determining attention from the pulses comprises a second vector-matrix multiplication. 3. The method according to any one of the preceding claims, characterized in that determining attention from the pulses comprises a time-dependent reduction of values. 4. The method according to any one of the preceding claims, characterized in that values generated using the pulses are reduced in a time-dependent manner. 5. A method according to the preceding claim, characterized in that the earlier a value was generated, the greater the reduction. 6. A method according to the preceding claim, characterized in that the time dependence of the reduction is exponential. 7. A method according to any one of the preceding claims, characterized in that the first vector-matrix multiplication is performed using an analog circuit. 8. A method according to any one of the preceding claims, characterized in that the second vector-matrix multiplication is performed using an analog circuit. 2 9. A method according to any one of the preceding claims, characterized in that the determination of attention from the pulses is carried out using an analog circuit. 10. A method according to any one of the two preceding claims, characterized in that the analog circuit comprises memristors. 11. A device for carrying out a method according to any one of the preceding claims, comprising a first electrical circuit (1) configured to perform vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals, a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse, and a third electronic circuit (1, 14) configured to determine attention from the time-dependent electrical pulses. 12. A device according to the preceding claim, characterized in that the first electrical circuit and/or the second electrical circuit comprises memristors (r12 to r34). 13. Device according to one of the two preceding claims, characterized in that the second electrical circuit comprises a capacitor (C) for integrating an electrical signal. 14. Device according to the preceding claim, characterized in that the second circuit comprises a discharge device (13) for the capacitor (C). 15. Device according to one of the two preceding claims, characterized in that the second circuit comprises an operational amplifier (op-amp). 16. Device according to one of the two preceding claims, characterized in that the third circuit (14) comprises an electrical resistor (15) and a capacitor (16).

Description

Forschungszentrum Jülich GmbH G70313WO / PT 1.3110 PCT Device and method for a similarity assessment Description The invention relates to a device and a method for determining attention. The following equation can be used to determine attention. where Q = query, K = key, and V = value. QK T is a vector-matrix multiplication. The result is a first vector. The softmax function is an operation on the first vector to transform it into a second vector containing probabilities. The softmax function divides each value of the first vector by the sum of all values in the first vector and stores the result in the new second vector. A concrete example is shown below: Multiplying the operation by V is another vector-matrix multiplication. The equation for Attention(Q,K,V) performs a similarity assessment between queries (Q) and keys (K) to weight the information of the value (V). K can be sequence elements from the future, present, and past, while Q is current input data. The equation for Attention(Q,K,V) outputs a weighted sum of values, allowing you to focus on the important parts of a query. Attention makes it possible to concentrate on specific information and ignore irrelevant parts. In machine learning, attention is used to focus on important elements of input data in order to improve task performance. Instead of the softmax function, other activation functions can also be used to detect attention. Determining attention requires very large matrix operations that consume a lot of power. The object of the invention is to enable the detection of attention with minimal effort. This object can be achieved by a method and a device with the features of the first claim and the dependent claim, respectively. Advantageous embodiments are described in the dependent claims. The object can be achieved as follows: Method for detecting attention comprising the steps of: performing a first vector-matrix multiplication such that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal, such as an electric current; Converting any generated electrical signal into a pulse, the occurrence of which depends on the signal's magnitude, for example, the electrical charge of the signal, with the pulse occurring later the larger the electrical signal (e.g., the greater the charge); determining an action from the pulses. The electrical signal can be a flowing electric current. The electrical signal can be a voltage. The electrical signal can be integrated through a capacitor, thus yielding an action that results from the magnitude of the electrical signal. Determining the action from the pulses can involve a second vector-matrix multiplication. Determining attention from the pulses can involve a time-dependent reduction of values. Values generated using the pulses can be reduced over time. The earlier a value is generated, the greater the reduction can be. The time dependence of the reduction can be exponential. The first vector-matrix multiplication can be performed using an analog circuit. Determining attention from the pulses can be performed using an analog circuit. The analog circuit can include memristors. A device for carrying out the method can include a first electrical circuit configured to perform a vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals. The device can include a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse. The circuit can include a third electronic circuit configured to determine attention from the time-dependent electrical pulses. The first, second, and/or third electrical circuits may include memristors. Circuits with memristors may be configured to perform vector-matrix multiplications. The third circuit may additionally include resistors and capacitors. The components of the third circuit may be configured to derive attention from input values. The resistors and capacitors of the third circuit may be designed to reduce values resulting from vector-matrix multiplications over time. These values may be reduced exponentially and may be values resulting from pulses of the second circuit. The second electrical circuit may include a capacitor for integrating an electrical signal. The second circuit may include a capacitor discharge device. The first circuit can include comparators. The second circuit can include transistors. The components of the second circuit can be configured to generate a time-dependent pulse from the magnitude of a time-dependent signal. The second circuit can include an operational amplifier. The second circuit can reduce a value, for example, linearly. This can be done by a charged capacitor that is discharged through an electrical resistor with a constant current. This can be done digitally, either alternatively or additionally. Depending on the linear reduction, the aforementioned exponential