Search

EP-4315083-B1 - SYSTEM AND METHOD FOR PROVIDING PAGE MIGRATION

EP4315083B1EP 4315083 B1EP4315083 B1EP 4315083B1EP-4315083-B1

Inventors

  • WHITE, SEAN T.
  • NG, PHILIP

Dates

Publication Date
20260513
Application Date
20220324

Claims (11)

  1. A method for providing page migration of pages among tiered memories comprising: identifying, by one or more processors (112), frequently accessed memory pages in each tiered memory region and generate page hotness ranking information indicating how frequently memory pages are being accessed; providing the page hotness ranking information to at least one of: an operating system or hypervisor executing on the one or more processors (112); issuing a page move command (168), by the operating system or hypervisor executing on the one or more processors, to a hardware data mover, HDM, (126) based on the page hotness ranking information; and in response to processing the page move command (168) from the one or more processors (112), delaying input output memory management unit address mappings while servicing the page move command and moving, by the hardware data mover, a memory page from one memory tier to another memory tier.
  2. The method of claim 1 wherein issuing the page move command (168) comprises issuing command data that causes the hardware data mover (126) to provide a secure nested paging batch move of multiple memory pages to a different memory tier, or provide a secure nested paging move of one or more memory pages securely owned by a guest operating system.
  3. The method of claim 1 wherein issuing the page move command (168) comprises issuing command data that causes the hardware data mover (126) to move contents of one or more pages that may be actively in use by a direct memory access device (135) in a non-secure nested paging system to memory that is either unencrypted or encrypted using an operating system or hypervisor owned key.
  4. The method of claim 1 wherein issuing the page move command (168) comprises issuing command data that causes the hardware data mover (126) to provide a secure nested paging move of input/output device pages that moves pages that may be actively in use by a direct memory access device (135).
  5. The method of claim 1 comprising providing an application programming interface, API, between the operating system and the hardware data mover (126) that provides the move commands to the hardware data mover and provides returned page migration data from the HDM to the operating system.
  6. An apparatus for providing page migration of pages among tiered memories comprising: one or more processors (112) configured to execute stored code that when executed cause the one or more processors to: identify frequently accessed memory pages in each memory tier region and generate page hotness ranking information indicating how frequently memory pages are being accessed; an operating system or hypervisor configured to issue a page move command (168) to a hardware data mover, HDM, (126) based on the page hotness ranking information; and a hardware data mover (126), operatively coupled to the one or more processors, and operative to delay input output memory management unit address mappings while servicing the page move command and move a memory page to a different memory tier in response to processing the page move command received from the one or more processors (112) executing the operating system or hypervisor.
  7. The apparatus of claim 6 wherein the page move command (168) comprises command data that causes the hardware data mover (126) to provide a secure nested paging batch move of multiple memory pages to a different memory tier, or provide a secure nested paging move of one or more memory pages securely owned by a guest operating system.
  8. The apparatus of claim 6 wherein issuing the page move command (168) comprises issuing command data that causes the hardware data mover (126) to move contents of one or more pages that may be actively in use by a direct memory access device (135) in a non-secure nested paging system to memory that is either unencrypted or encrypted using an operating system or hypervisor owned key.
  9. The apparatus of claim 6 wherein the page move command (168) comprises command data that causes the hardware data mover (126) to provide a secure nested paging move of input/output device pages that moves pages that may be actively in use by a direct memory access device (135).
  10. The apparatus of claim 6 wherein the one or more processors (112) execute stored code that when executed cause the one or more processors to provide an application programming interface, API, (160) between the operating system and the hardware data mover (126) that provides the move commands to the hardware data mover and provides returned page migration data from the HDM to the operating system.
  11. The apparatus of claim 6 comprising an input/output memory management unit (124) operatively coupled to the hardware data mover (126) and operative to interface with input/output units that access the tiered memory and wherein the hardware data mover comprises: a security processor, SP, (130); and a direct memory access engine, DMA, (135) in communication with the security processor, and wherein the security processor (130) is operative to process the page move command and wherein the DMA engine is operative to move one or more pages between different tiers of memory.

Description

BACKGROUND ART Memory controllers, including direct memory access (DMA) controllers, are circuits that translate accesses generated by a memory accessing agent such as a data processor into one or more commands that are understood by computer memory. A memory controller can be implemented as a separate chip or integrated with other components such as data processors on a single integrated circuit chip. In the latter case, the memory controller is usually called an integrated memory controller. Integrated memory controllers support various bus protocols, such as the dynamic random-access memory (DRAM) bus protocol and the double data rate dynamic random-access memory (DDR DRAM) bus protocol. The need for tight coupling of memory with computing resources like processors such as central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), and the like pose challenges to the system designer related to memory capacity requirements, memory controller availability, memory lifecycle limitations, and memory bandwidth availability to CPUs. Capabilities such as in-memory workloads and server virtualization drive the need for increasing memory capacity. Moreover, the increasing performance of CPUs (e.g., one or more cores) creates a need for more memory channels per socket. For example, memory capacity requirements are driven by the number of CPUs in order to maintain balanced computational resources for many workloads. Tiered memory systems are composed of two or more regions of memory with different latency and/or bandwidth characteristics. Examples: 2-socket systems have "local" and "remote" tiers of memory, even if all the memory is traditional DRAM. The industry is developing new combinations of tiered memory such as HBM and DRAM, DRAM and CXL Type 3 memory expansion devices, DRAM and NVDIMM-P. Some examples of latency and bandwidth differences may include for example, CXL-attached memory that has a higher latency and different bandwidth characteristics versus DRAM. HBM may have similar latency characteristics to DRAM but much higher bandwidth. Due to performance differences between memory tiers, it is desirable to move frequently used pages from slower memory tiers into faster tiers, and move infrequently used pages from faster memory tiers into slower tiers. For ease of management, such as assigning memory to a process, memory may be divided into pages of various sizes such as 4KiB, 2MiB, 1GiB Using page migration improves performance by making best use of the faster tiers of memory. With the rapid development of multiple competing memory technologies ongoing, there is no clear best technology or solution or interface currently. Hardware-only solutions have been proposed but would be proprietary and unlikely to pick the best memory solution out of current and yet to emerge options. Also, hardware-only solutions for page migration may perform well on some workloads but badly on others and can be difficult to adjust the operation based on behavior. Also, with virtualization and the need for enhanced security between hypervisor and virtual machines, hardware only solutions may not be robust enough to adapt to security needs. Other solutions that perform page migration use software-only solutions that may be more flexible than hardware-only solutions, but software implemented data movement will be lower performance than hardware-implemented data movement. Also known systems may not provide suitable security since, for example, a hypervisor may not be able to migrate pages in use by guest virtual machines that are protected by advanced security features. US 2019/042145 A1 (Pham et al.) discloses a memory controller to couple to a multilevel memory characterized by a faster higher level and a slower lower level. The memory controller has early demotion logic circuitry to demote a page from the higher level to the lower level without system software having to instruct the memory controller to demote the page and before the system software promotes another page from the lower level to the higher level. US 2020/310993 A1 (Kumar et al.) discloses systems and methods sharing memory circuitry between processor memory circuitry and accelerator memory circuitry in each of a plurality of peer-to-peer connected accelerator units. Each of the accelerator units includes physical-to-virtual address translation circuitry and migration circuitry. The physical-to-virtual address translation circuitry in each accelerator unit includes pages for each of at least some of the plurality of accelerator units. The migration circuitry causes the transfer of data between the processor memory circuitry and the accelerator memory circuitry in each of the plurality of accelerator circuits. The migration circuitry migrates and evicts data to/from accelerator memory circuitry based on statistical information associated with accesses to at least one of: processor memory circuitry or accelerator memory circuit