KR-20260067064-A - MEMORY CONTROLLER FOR PROCESSING IN MEMORY AND MEMORY COMMAND GENERATION METHOD USING THE MEMORY CONTROLLER

Abstract

A PIM command generator for a processing in memory device (PIM device) includes an input buffer, and when the PIM command generator receives a first PIM request from a host instructing to write input data, the PIM command generator stores the input data in the input buffer, receives a second PIM request from the host instructing a processing in memory operation between the input data and the data stored in the PIM device, and in response to the second PIM request, scans the elements of the input data stored in the input buffer to generate a PIM command corresponding to a non-zero element among the input data, skips the generation of a PIM command corresponding to a zero element, and transmits the generated PIM command to the PIM device.

Inventors

• 서승우
• 이선정
• 조연곤

Assignees

• 삼성전자주식회사

Dates

Publication Date

20260512

Application Date

20241105

Claims (20)

1. In a PIM instruction generator for a processing in memory device (PIM device), input buffer Includes, The above PIM command generator is, When a first PIM request (processing in memory request) instructing to write input data is received from the host, the input data is stored in the input buffer, and A second PIM request is received from the host instructing a PIM operation (processing in memory operation) between the input data and the data stored in the PIM device, and In response to the second PIM request, by scanning the elements of the input data stored in the input buffer, a PIM command corresponding to a non-zero element among the input data is generated, and the generation of a PIM command corresponding to a zero element is skipped. Transmitting the above-generated PIM command to the PIM device, PIM command generator.
2. In paragraph 1, The above PIM command generator is, The apparatus further comprises one or more registers storing at least one of tiling information of the PIM device, hardware information of the PIM device, or address mapping information. PIM command generator.
3. In paragraph 1, The above second PIM request is, Indicating a matrix-vector multiplication operation between a matrix stored in the memory of the PIM device and an input vector stored as input data in the input buffer, PIM command generator.
4. In paragraph 3, The above PIM command generator is, In response to the second PIM request, based on the fact that the first element among the elements of the input vector is non-zero, determine the memory address of at least a part of the matrix to be multiplied with the first element, and Based on the fact that the second element among the elements of the above input vector is zero, skipping the determination of at least a part of the memory address of the matrix to be multiplied with the second element in the memory of the PIM device. PIM command generator.
5. In paragraph 4, The above PIM command generator is, Based on the tiling information of the above PIM device, the matrix is divided into one or more memory tiles, and For each of the one or more memory tiles above that has an element to be multiplied with the first element, a PIM command is generated that instructs multiplying the column corresponding to the first element among the memory tiles and the first element, and accumulating each of the multiplication results into an element of the corresponding output vector. PIM command generator.
6. In paragraph 1, The above PIM command generator is, Based on information indicating each non-zero element among the above input data and information related to the PIM device, the address of a memory region corresponding to the element in the memory of the PIM device is determined, and Generating one or more PIM instructions using the above element and the address of the above memory region, PIM command generator.
7. In paragraph 1, The above PIM command generator is, One or more generated PIM instructions passed to the memory instruction queue, PIM command generator.
8. In electronic devices, A PIM command generator that generates one or more PIM commands based on a PIM request received from a host. Includes, The above PIM command generator is, input buffer Includes, The above PIM command generator is, When a first PIM request instructing to write input data is received, the input data is stored in the input buffer, and Receives a second PIM request instructing a PIM operation (processing in memory operation) between the above input data and the data stored in the PIM device, and In response to the second PIM request, by scanning the elements of the input data stored in the input buffer, a PIM command corresponding to a non-zero element among the input data is generated, and the generation of a PIM command corresponding to a zero element is skipped. Transmitting the above-generated PIM command to the PIM device, Electronic device.
9. In paragraph 8, The above electronic device is, An arbiter that classifies a memory request received from the above host into a standard memory request or a PIM request (processing in memory request); and It further includes a standard command generator, and The above arbitrator is, Based on classifying the above memory request as a general memory request, the memory request is passed to the general command generator, and Based on classifying the above memory request as the above PIM request, the above memory request is transmitted to the above PIM command generator, and The above general command generator is, Based on receiving the memory request from the above arbitrator, generate a general memory instruction, and The above PIM command generator is, Based on receiving the memory request from the above arbitrator, generating the one or more PIM commands, Electronic device.
10. In paragraph 8, The above PIM command generator is, The apparatus further comprises one or more registers storing at least one of tiling information of the PIM device, hardware information of the PIM device, or address mapping information. Electronic device.
11. In paragraph 8, The above second PIM request is, Indicating a matrix-vector multiplication operation between a matrix stored in the memory of the PIM device and an input vector stored as input data in the input buffer, Electronic device.
12. In Paragraph 11, The above PIM command generator is, In response to the second PIM request, based on the fact that the first element among the elements of the input vector is non-zero, determine the memory address of at least a part of the matrix to be multiplied with the first element, and Based on the fact that the second element among the elements of the above input vector is zero, skipping the determination of at least a part of the memory address of the matrix to be multiplied with the second element in the memory of the PIM device. Electronic device.
13. In Paragraph 12, The above PIM command generator is, Based on the tiling information of the above PIM device, the matrix is divided into one or more memory tiles, and For each of the one or more memory tiles above that has an element to be multiplied with the first element, a PIM command is generated that instructs multiplying the column corresponding to the first element among the memory tiles and the first element, and accumulating each of the multiplication results into an element of the corresponding output vector. Electronic device.
14. In paragraph 8, The above PIM command generator is, Based on information indicating each non-zero element among the above input data and information related to the PIM device, the address of a memory region corresponding to the element in the memory of the PIM device is determined, and Generating one or more PIM instructions using the above element and the address of the above memory region, Electronic device.
15. In paragraph 8, The above electronic device is, It further includes a memory instruction queue, The above PIM command generator is, One or more PIM commands generated above are transferred to the memory command queue, Electronic device.
16. In paragraph 8, The above electronic device is, General command generator; General memory instruction queue; and Includes more PIM command queues, The above PIM command generator is, One or more of the above-mentioned PIM commands are transmitted to the PIM command queue, and The above general command generator is, After generating a general memory command based on a general memory request received from the above host, the generated general memory command is delivered to the general memory command queue. Electronic device.
17. In Paragraph 16, The above electronic device is, It further includes a scheduler connected to the above general memory command queue and the above PIM command queue, and The above scheduler is, Through scheduling, one of the general memory instruction queue or the PIM instruction queue is determined, and Transmitting at least one of the general memory command or the PIM command received from the above-determined queue to the PIM device, Electronic device.
18. In a memory instruction generation method performed by a PIM instruction generator for a processing in memory device (PIM device), When the PIM command generator receives a first PIM request from a host instructing it to write input data, the step of storing the input data in the input buffer of the PIM command generator; A step of receiving a second PIM request from the host, which instructs a PIM operation (processing in memory operation) between the input data and the data stored in the PIM device by the PIM command generator; A step of generating a PIM command corresponding to a non-zero element among the input data and skipping the generation of a PIM command corresponding to a zero element by scanning the elements of the input data stored in the input buffer in response to the second PIM request by the above PIM command generator; and A step of transmitting the generated PIM command to the PIM device by the above PIM command generator A memory instruction generation method including
19. In Paragraph 11, The above second PIM request is, Indicating a matrix-vector multiplication operation between a matrix stored in the memory of the PIM device and an input vector stored as input data in the input buffer, Method for generating memory instructions.
20. A computer-readable recording medium storing one or more computer programs comprising instructions for performing the method of any one of paragraphs 18 to 19.

Description

Memory Controller for Processing in Memory and Memory Command Generation Method Using the Memory Controller A memory controller for processing in memory is disclosed below. With the emergence of large-scale AI models (e.g., Large Language Models; LLMs), the size of AI models is on the rise. LLMs can be broadly divided into a summarization stage and a generation stage. However, as the scale of AI models increases and the number of tokens generated based on them grows, the processing time of the generation stage can account for the majority of the AI model's total operation time due to insufficient memory bandwidth. Processing in Memory (PIM) is being researched to address this memory bandwidth issue. Processing-in-Memory (Processing-in-Memory) refers to a technology that includes the capability to process data within a memory device, in addition to storing it. Processing-in-Memory aims to improve overall system performance by performing computations near memory to reduce data transfer bottlenecks between memory and the host processor. Memory devices equipped with Processing-in-Memory technology can integrate computational units or processing cores near memory cells to enable processing tasks to be performed within memory without data movement. As a result, the load on the host processor is reduced, power consumption is decreased, and data processing speed can be improved. The aforementioned background technology was possessed or acquired during the process of deriving the present disclosure and cannot be considered as prior art disclosed to the general public prior to the filing of the present disclosure. FIG. 1 is a drawing illustrating examples of memory systems according to various embodiments. FIG. 2 is a flowchart illustrating an example of a method in which zero skip is performed by a PIM generator of a memory controller according to various embodiments. FIG. 3 is a diagram illustrating an example of an operation in which a memory controller according to various embodiments outputs a memory command based on a memory request. FIG. 4a is a diagram illustrating an example of an operation in which a PIM operation generator according to various embodiments generates memory instructions for matrix-vector multiplication operations without performing zero skips. FIG. 4b is a diagram illustrating an example of an operation in which a PIM operation generator according to various embodiments generates memory instructions for matrix-vector multiplication operations while performing zero skips. FIG. 5 is a block diagram illustrating an example configuration of a PIM command generator according to various embodiments. FIG. 6 is a drawing illustrating an example of a memory system including a DMA device according to various embodiments. Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments. Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component. When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or joined to that other component, or that there may be other components in between. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification. Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted. FIG. 1 is a drawing illustrating examples of memory sy