US-12619560-B2 - Computer memory expansion device and method of operation

US12619560B2US 12619560 B2US12619560 B2US 12619560B2US-12619560-B2

Abstract

A memory expansion device operable with a host computer system (host) comprises a non-volatile memory (NVM) subsystem, cache memory, and control logic configurable to receive a submission from the host including a read command and specifying a payload in the NVM subsystem and demand data in the payload. The control logic is configured to request ownership of a set of cache lines corresponding to the payload, to indicate completion of the submission after acquiring ownership of the cache lines, and to load the payload to the cache memory. The set of cache lines correspond to a set of cache lines in a coherent destination memory space accessible by the host. The control logic is further configured to, after indicating completion of the submission and in response to a request from the host to read demand data in the payload, return the demand data after determining that the demand data is in the cache memory.

Inventors

Jordan HORWICH
Jerry ALSTON
Chih-Cheh Chen
Patrick Lee
Scott Milton
Jeekyoung Park

Assignees

NETLIST, INC.

Dates

Publication Date: 20260505
Application Date: 20240813

Claims (20)

1 . A memory expansion device operable in a computer system, the computer system including a host computer (host) and a dedicated bus, the memory expansion device comprising: interface circuitry configured to communicate with the host via the dedicated bus based on a predefined protocol; a non-volatile memory (NVM) subsystem; local memory providing a coherent memory space accessible by the host; cache memory; and control logic coupled to the interface circuitry the cache memory, and the NVM subsystem, wherein the control logic is configurable to: receive a submission from the host, the submission including a read command and a payload specification in the NVM subsystem; determine a priority of the submission based on the payload specification, wherein the submission is of a first priority if the submission is to resolve a page fault at the host, the submission is of a second priority if the submission is to fetch data from storage, and submission is of a third priority if the submission is to prefetch data from storage; and performing a read process of a plurality of read processes in response to determining the priority of the submission, wherein the read process is based on the priority of the submission.
2 . The memory expansion device of claim 1 , further comprising a local memory coupled to the control logic and providing a destination space.
3 . The memory expansion device of claim 2 , further comprising a controller memory buffer (CMB) including submission queues, accessible by the host, the submission queues including at least a first submission queue for queuing submission of the first priority and at least a second submission queue for queuing submissions of the second priority, wherein the CMB corresponds to a CMB memory space in the local memory.
4 . The memory expansion device of claim 3 , further comprising a cNVMe controller to determine the read command, a payload from the payload specification, and one or more hints of the submission, wherein the cNVME controller initiates one or more of the read processes.
5 . The memory expansion device of claim 4 , wherein the read process is a demand read process implemented by cNVMe controller in response to determining the submission is of the first priority, the demand read process comprising requesting ownership of cache lines corresponding to the payload specification in response to reading submission.
6 . The memory expansion device of claim 5 , the demand read process further comprising indicating completion of the submission after acquiring ownership of the cache lines and loading a payload of the payload specification to cache memory, the cache lines corresponding to a coherent destination memory space in local memory accessible by the host.
7 . The memory expansion device of claim 4 , wherein the one or more hints specify how the corresponding cache lines are prepared before the payload is loaded.
8 . The memory expansion device of claim 7 , wherein the read process is a predictive read process implemented by cNVMe controller in response to determining the submission is of the second priority, the predictive read process comprising: requesting ownership of cache lines corresponding to the payload specification in response to reading the submission; preparing the cache lines based on the one or more hints; and determining whether the payload has been prefetched and stored in a private memory space of the local memory.
9 . The memory expansion device of claim 8 , the predictive read process comprising loading a portion of the payload from the private memory into the cache lines corresponding to a coherent destination memory space.
10 . The memory expansion device of claim 9 , wherein the portion of the payload is a first portion, the predictive read process comprising: reading a second portion of the payload from the NVM subsystem; and writing the second portion of the payload to the corresponding cache lines.
11 . The memory expansion device of claim 10 , where in at least a first hint of the one or more hints is used to load the first portion and at least a second hint of the one or more hints is used to load the second portion.
12 . The memory expansion device of claim 3 , wherein the read process is a speculative read process in response to determining the submission is of the third priority, the speculative read process comprising determining whether to fetch a payload of the payload specification based on predefined criteria.
13 . The memory expansion device of claim 12 , wherein the speculative read process comprises: determining that the payload can be fetched from the NVM subsystem and loaded into a private memory of the local memory within a predetermined time period defined by the predefined criteria; and loading the payload into the private memory by fetching the payload from the NVM subsystem using NVM read commands.
14 . A memory expansion device operable in a computer system, the computer system including a host computer (host) and a dedicated bus, the memory expansion device comprising: interface circuitry configured to communicate with the host via the dedicated bus based on a predefined protocol; a non-volatile memory (NVM) subsystem; local memory providing a coherent memory space accessible by the host; cache memory; and control logic coupled to the interface circuitry the cache memory, and the NVM subsystem, wherein the control logic is configurable to: receive a submission from the host, the submission including a read command and a payload defined by a payload specification in the NVM subsystem; determine a priority of the submission based on the payload specification, wherein the submission is of a first priority if the submission is to resolve a page fault at the host, the submission is of a second priority if the submission is to fetch data from storage, and submission is of a third priority if the submission is to prefetch data from storage; and performing a read process of a plurality of read processes in response to determining the priority of the submission, wherein the read process is based on the priority of the submission and is one or more of a demand read process, a predictive read process, and a speculative read process, and each of the read processes includes indicating completion of the submission.
15 . The memory expansion device of claim 14 , wherein the read process is the demand read process in response to determining the submission is of the first priority, the demand read process comprising: requesting ownership of cache lines corresponding to the payload specification in response to reading submission; and indicating completion of the submission after acquiring ownership of the cache lines and loading a payload of the payload specification to cache memory, the cache lines corresponding to a coherent destination memory space in local memory accessible by the host.
16 . The memory expansion device of claim 14 , wherein the read process is the predictive read process in response to determining the submission is of the second priority, the predictive read process comprising: requesting ownership of cache lines corresponding to the payload specification in response to reading the submission; preparing the cache lines based on one or more hints; and determining a portion of the payload has been prefetched and stored in a private memory space of the local memory; and loading a portion of the payload from the private memory into the cache lines corresponding to a coherent destination memory space.
17 . The memory expansion device of claim 14 , wherein the read process is the speculative read process in response to determining the submission is of the third priority, the speculative read process comprising: determining that the payload can be fetched from the NVM subsystem and loaded into a private memory of the local memory within a predetermined time period; and loading the payload into the private memory by fetching the payload from the NVM subsystem using NVM read commands.
18 . A method, comprising: at a memory expansion device coupled to a host computer (host) via a dedicated bus, the memory expansion device comprising interface circuitry configured to communicate with the host via the dedicated bus based on a predefined protocol, a non-volatile memory (NVM) subsystem, local memory providing a coherent memory space accessible by the host, and cache memory, receiving a submission from the host, the submission including a read command and a payload defined by a payload specification in the NVM subsystem; determining a priority of the submission based on the payload specification, wherein the submission is of a first priority if the submission is to resolve a page fault at the host, the submission is of a second priority if the submission is to fetch data from storage, and submission is of a third priority if the submission is to prefetch data from storage; and performing a read process of a plurality of read processes in response to determining the priority of the submission, wherein the read process is based on the priority of the submission and is one or more of a demand read process, a predictive read process, and a speculative read process, and each of the read processes includes indicating completion of the submission in response to acquiring ownership of cache lines in a coherent memory space.
19 . The method of claim 18 , wherein the memory expansion device includes a controller memory buffer (CMB) including submission queues, accessible by the host, the submission queues including at least a first submission queue for queuing submission of the first priority and at least a second submission queue for queuing submissions of the second priority, wherein the CMB corresponds to a CMB memory space in the local memory.
20 . The method of claim 19 , further comprising maintaining selected portions of the CMB memory space in a shared state, wherein, in response to a given cache line of the cache lines being modified by the host causing the shared state corresponding to the given cache line being invalidated, re-acquiring the given cache line to reinstate its shared state.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. patent application Ser. No. 18/000,125, filed Nov. 28, 2022, which is a US national phase application of PCT Application No. PCT/US2021/035317, filed Jun. 1, 2021, which claims priority to U.S. Provisional Application Ser. No. 63/032,484, filed May 29, 2020, entitled “Software-Defined CXL Memory Controllers and Methods of Operation,” each of which is hereby incorporated by reference in its entirety. FIELD OF THE INVENTION The various embodiments described in this document relate in general to computer memory, and more specifically to a computer memory expansion device and method of operation. BACKGROUND Emerging applications, such as cloud computing, artificial intelligence, and machine learning, are driving demand for faster and faster data processing. With the increasing number of cores per socket running at higher clock frequencies, and the aid of accelerators, such as graphic processing units (GPU's), field-programmable gate arrays (FPGA's), data processing units (DPU's), etc., processor speed, and/or the number of active threads per socket, has been doubling every two years. The increasing processor power places increasing demand on memory capacity and memory speed or bandwidth, which unfortunately do not increase at the same rate. Often, higher memory speed means lower memory capacity, and, as memory capacity increases to keep up with the increase in processor speed, memory latency, which is a measure of how long it takes to complete a memory operation, is also increasing at a rate of about 1.1 times every two years. Thus, solving the problem of memory capacity and bandwidth gaps is critical in the performance of data processing systems. Software-defined memory (SDM) expansion using Non-Volatile Memory Express Solid-State Drives (NVMe SSD) provides better economics but has various performance issues, such as lack of efficiency across different workloads, poor quality of predictive prefetching due to high latency, large latency penalty for page faults, and lack of efficiency in moving data into coherent host memory. SUMMARY In some embodiments, a high density, high bandwidth, and low cost memory expansion device includes non-volatile memory (NVM, e.g., NAND Flash) as tier 1 memory for low-cost virtual memory capacity expansion, optional device DRAM as tier 2 coherent memory for physical memory capacity and bandwidth expansion, and device cache as tier 3 coherent memory for low latency. In some embodiments, a memory expansion device is operable in a computer system, the computer system including a host computer (host) and a dedicated bus. The memory expansion device comprises interface circuitry configured to communicate with the host via the dedicated bus based on a predefined protocol, a non-volatile memory (NVM) subsystem, cache memory, and control logic coupled to the interface circuitry the cache memory, and the NVM subsystem. The control logic is configurable to receive a first submission from the host, the first submission including a first read command and specifying a first payload in the NVM subsystem. In response to the first submission being of first priority, the control logic is further configured to request ownership of first cache lines corresponding to the first payload, indicate completion of the first submission after acquiring ownership of the first cache lines, and load the first payload to the cache memory, the first cache lines corresponding to cache lines in a first coherent destination memory space accessible by the host. In some embodiments, the memory expansion device is coupled to the host via a Computer Express Link (CXL) bus, wherein the interface circuitry provides a CXL interface between the control logic and the CXL bus, and wherein the first coherent destination memory space is accessible by the host using a CXL protocol. In some embodiments, the control logic is further configured to request ownership of the first cache lines from a home agent at the host computer. In some embodiments, the first submission further specifies demand data in the first payload. The control logic is configured to, before loading the first payload into the cache memory issue first NVM read commands to read the first payload from the NVM subsystem, the first NVM read commands being written into a command queue associated with the NVM subsystem. The control logic is further configured to prioritize reading the demand data from the NVM subsystem when issuing the first NVM read commands such that a logic block address in the NVM subsystem corresponding to a logic block including the demand data is read before logic block addresses corresponding to other logic blocks in the payload. In some embodiments, the control logic is configured to indicate completion of the first submission before determining that the first payload has been loaded in the cache memory. In some embodiments, the control logic is further configu