US-12625781-B1 - Fault isolation in an I/O fabric using downstream port containment

Abstract

A technique of non-disruptive fault isolation for an input/output (I/O) subsystem includes processing circuitry at a root complex interposed between a plurality of server platforms and a plurality of I/O switches receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled. In response to receipt of the notification, the processing circuitry notifies the plurality of attached server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter. The processing circuitry waits a containment interval, and thereafter, releases the containment to permit resetting of the I/O adapter.

Inventors

• Seamus J. BURKE
• Louis A. Rasor
• Todd C. Sorenson

Assignees

• INTERNATIONAL BUSINESS MACHINES CORPORATION

Dates

Publication Date

20260512

Application Date

20241114

Claims (19)

1. 1 . A computer-implemented method of non-disruptive fault isolation for an input/output (I/O) subsystem, the method comprising: at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled; in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter; during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
2. 2 . The method of claim 1 , further comprising: the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
3. 3 . The method of claim 1 , further comprising: a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
4. 4 . The method of claim 3 , wherein: receiving notification of downstream port containment includes receiving a message signaled interrupt.
5. 5 . The method of claim 3 , wherein: releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.
6. 6 . The method of claim 3 , wherein providing the notification to the plurality of server platforms includes broadcasting a message signaled interrupt to all of the plurality of server platforms.
7. 7 . The method of claim 3 , wherein the containment interval terminates in response to a message from a controlling server platform among the plurality of server platforms.
8. 8 . The method of claim 3 , wherein, in response to the notification, each of the plurality of server platforms marks the I/O adapter as subject to downstream port containment to refrain from attempting communication with the I/O adapter.
9. 9 . The method of claim 3 , wherein enforcing the downstream port containment comprises: a digital signal processor of a downstream port enforcing port containment on an I/O link responsive to an uncorrectable error signaled as ERR_FATAL by the I/O adapter and reporting the port containment to the root complex in a message signaled interrupt.
10. 10 . A computer program product, comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform computer operations, the computer operations including: at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled; in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter; during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment; and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
11. 11 . The computer program product of claim 10 , wherein the computer operations further comprise: the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
12. 12 . The computer program product of claim 10 , wherein the computer operations further comprise: a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
13. 13 . The computer program product of claim 10 , wherein: receiving notification of downstream port containment includes receiving a message signaled interrupt.
14. 14 . The computer program product of claim 10 , wherein: releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.
15. 15 . A data processing system, comprising: processing circuitry; and one or more computer-readable storage media communicatively coupled to the processing circuitry, wherein the one or more computer-readable storage media includes program instructions to perform operations including: at a root complex interposed between a plurality of server platforms and a plurality of I/O switches, processing circuitry receiving, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled; in response to receipt of the notification, the processing circuitry notifying the plurality of server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter; during an containment interval, a controlling server platform among the plurality of server platforms checking an I/O communication link coupling the I/O adapter and the I/O switch and collecting metadata regarding adapter operation from the I/O adapter for root-cause analysis while the I/O adapter is being subject to downstream port containment; and the processing circuitry waiting a containment interval, and thereafter, releasing the containment to permit resetting of the I/O adapter, wherein the I/O adapter is a storage adapter; providing the notification to the plurality of server platforms comprises an MSI broadcast to all of the plurality of server platforms; and the containment interval terminates in response to a message from the controlling server platform.
16. 16 . The data processing system of claim 15 , wherein the operations further comprise: the controlling server platform resetting the I/O adapter after elapsing of the containment interval.
17. 17 . The data processing system of claim 15 , wherein the operations further comprise: a controlling server platform among the plurality of server platforms determining whether a port containment count for the I/O adapter satisfies a containment threshold; and based on determining that the port containment count satisfies the containment threshold, the controlling server platform fencing the I/O adapter from receiving further I/O traffic and marking for physical replacement a field-replaceable unit including the I/O adapter.
18. 18 . The data processing system of claim 15 , wherein: receiving notification of downstream port containment includes receiving a message signaled interrupt.
19. 19 . The data processing system of claim 15 , wherein: releasing containment includes the processing circuitry signaling release of containment to both the I/O adapter and the plurality of server platforms.

Description

STATEMENT REGARDING PRIOR DISCLOSURES BY THE INVENTOR OR A JOINT INVENTOR The inventors hereof are aware of the following prior public disclosure: “Accelerating Innovation in Mainframe Storage for Next-Generation Workloads with the new IBM DS8000,” Sep. 10, 2024, published online at https://newsroom.ibm.com/blog-accelerating-innovation-in-mainframe-storage-for-next-generation-workloads-with-the-new-ibm-ds8000 BACKGROUND OF THE INVENTION The present invention relates in general to data processing, and more specifically, to fault isolation in an input/output (I/O) fabric using downstream port containment. Large scale cloud-based data centers typically include significant input/output resources, often implemented utilizing hot-pluggable field replaceable units (FRUs) that support efficient maintenance and replacement of I/O devices, such as network adapters and storage controllers. By employing hot-pluggable FRUs, maintenance and replacement of failing FRUs can be accomplished without server downtime. Recently, I/O protocols have been enhanced to provide additional error isolation capabilities. For example, some I/O protocols, such as Peripheral Component Interconnect Express (PCIe) version 4, now support Downstream Port Containment (DPC), which enables I/O traffic at or below a downstream port to be halted in response to detection of an unmasked uncorrectable error. As discussed further herein, the capability of DPC can be leveraged to provide enhanced fault isolation and to prevent spread of I/O errors and data corruption. SUMMARY OF THE INVENTION In accordance with one or more embodiments, a method, system, and/or computer program product can be utilized to implement a technique of non-disruptive fault isolation for an input/output (I/O) subsystem. According to the technique, processing circuitry at a root complex interposed between a plurality of server platforms and a plurality of I/O switches receives, via an I/O switch among the plurality of I/O switches, notification of downstream port containment of an I/O port to which an I/O adapter is coupled. In response to receipt of the notification, the processing circuitry notifies the plurality of attached server platforms of the downstream port containment to cause the plurality of server platforms to refrain from attempting communication of I/O traffic via the I/O adapter. The processing circuitry waits a containment interval, and thereafter, releases the containment to permit resetting of the I/O adapter. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a high-level block diagram of an exemplary data processing environment in accordance with one or more embodiments; FIG. 2 is a high-level block diagram of an exemplary data center embodiment of the computing environment of FIG. 1; and FIG. 3 is a high-level logical flowchart of an exemplary process by which a root complex of an I/O subsystem implements fault containment using downstream port containment in accordance with one or more embodiments; and FIG. 4 is a high-level logical flowchart of an exemplary process by which a controller server implements fault containment using downstream port containment in accordance with one or more embodiments. In accordance with common practice, various features illustrated in the drawings may not be drawn to scale. Accordingly, dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, like reference numerals may be used to denote like or corresponding features in the specification and figures. DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENT Various aspects of the present disclosure are described by narrative text, flowcharts, block diagrams of computer systems and/or block diagrams of the machine logic included in computer program product (CPP) embodiments. With respect to any flowcharts, depending upon the technology involved, the operations can be performed in a different order than what is shown in a given flowchart. For example, again depending upon the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, concurrently, or in a manner at least partially overlapping in time. A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media (also called “mediums”) collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device that can retain and store instructions for use by a computer processor. Without limitation, the computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic