US-12620242-B2 - Systems and methods for printed circuit board netlist extraction from multimodal imagery

Abstract

A system and method for generating a netlist for a printed circuit board. In some embodiments, the method includes capturing image data of a circuit board from a plurality of sensors to generate a set of captured data for each of the plurality of sensors; extracting a plurality of features from the image data using machine learning; and generating a design associated with the circuit board from the plurality of features.

Inventors

• Anthony F. George
• John Timothy BALINT
• Thomas KENT

Assignees

• BATTELLE MEMORIAL INSTITUTE

Dates

Publication Date

20260505

Application Date

20231025

Claims (18)

1. 1 . A method comprising: capturing image data of a circuit board from a plurality of sensors to generate a set of captured data for each of the plurality of sensors; extracting a plurality of features from the image data using machine learning; wherein: the plurality of features include at least one of traces, pins, components, and/or text; the text is extracted using optical character recognition; the pins are extracted using a pin recognition neural network; the components are extracted using a component recognition neural network; and the traces are extracted using a Canny edge detector and a loss function, the loss function including a linear combination between an edge-based loss and a cross entropy loss, wherein the cross entropy loss operates as a pixel-based loss function; and generating a design associated with the circuit board from the plurality of features.
2. 2 . The method of claim 1 , wherein capturing the image data of the circuit board from the plurality of sensors to generate the set of captured data for each of the plurality of sensors further comprises: capturing a multimodal image data of the circuit board, wherein the plurality of sensors configured to acquire a plurality of electromagnetic wavelengths.
3. 3 . The method of claim 1 , wherein extracting the plurality of features from the image data using the machine learning further comprises: fusing the plurality of features from each of the plurality of sensors to create a plurality of segmentation masks, wherein the plurality of features are combined using a feature pyramid network.
4. 4 . The method of claim 3 , wherein the feature pyramid network is an encoder-decoder neural network.
5. 5 . The method of claim 4 , wherein: every group of deconvolutional layers in the encoder-decoder neural network are used in a prediction; and segmentation maps are computed at multiple scales that are concatenated together.
6. 6 . The method of claim 3 , wherein extracting the plurality of features from the image data using the machine learning further comprises: capturing a plurality of polygons; and computing an Intersection over Union between a segmentation mask for each pair of the plurality of polygons.
7. 7 . The method of claim 1 , wherein generating the design associated with the circuit board from the plurality of features further comprises: generating an adjacency matrix from the plurality of features; generating text regions from the plurality of features using a character recognition neural network; and combining the adjacency matrix with the plurality of features to generate the design.
8. 8 . A system comprising: a plurality of sensors configured to capture image data of a circuit board to generate a set of captured data for each of the plurality of sensors; and a processor configured to: extract a plurality of features from the image data of the circuit board using machine learning; wherein: the plurality of features include at least one of traces, pins, components, and/or text; the text is extracted using optical character recognition; the pins are extracted using a pin recognition neural network; the components are extracted using a component recognition neural network; and the traces are extracted using a Canny edge detector and a loss function, the loss function including a linear combination between an edge-based loss and a cross entropy loss, wherein the cross entropy loss operates as a pixel-based loss function; and generate a design associated with the circuit board from the plurality of features.
9. 9 . The system of claim 8 , wherein capture the image data of the circuit board from the plurality of sensors to generate the set of captured data for each of the plurality of sensors further comprises: capture a multimodal image data of the circuit board, wherein the plurality of sensors configured to acquire a plurality of electromagnetic wavelengths.
10. 10 . The system of claim 8 , wherein extract the plurality of features from the image data using the machine learning further comprises: fuse the plurality of features from each of the plurality of sensors to create a plurality of segmentation masks, wherein the plurality of features are combined using a feature pyramid network.
11. 11 . The system of claim 10 , wherein the feature pyramid network is an encoder-decoder neural network.
12. 12 . The system of claim 11 , wherein: every group of deconvolutional layers in the encoder-decoder neural network are used in a prediction; and segmentation maps are computed at multiple scales that are concatenated together.
13. 13 . The system of claim 10 , wherein extract the plurality of features from the image data using the machine learning further comprises: capture a plurality of polygons; and compute an Intersection over Union between a segmentation mask for each pair of the plurality of polygons.
14. 14 . The system of claim 10 , wherein generate the design associated with the circuit board from the plurality of features further comprises: generate an adjacency matrix from the plurality of features; generate text regions from the plurality of features using a character recognition neural network; and combine the adjacency matrix with the plurality of features to generate the design.
15. 15 . A system comprising: a processor; a non-transitory computer-readable storage media; and program instructions stored on the non-transitory computer-readable storage media for execution by the processor, the stored program instructions including instructions to: capture image data of a circuit board from a plurality of sensors to generate a set of captured data for each of the plurality of sensors; extract a plurality of features from the image data using machine learning; wherein: the plurality of features include at least one of traces, pins, components, and/or text; the text is extracted using optical character recognition; the pins are extracted using a pin recognition neural network; the components are extracted using a component recognition neural network; and the traces are extracted using a Canny edge detector and a loss function, the loss function including a linear combination between an edge-based loss and a cross entropy loss, wherein the cross entropy loss operates as a pixel-based loss function; and generate a design associated with the circuit board from the plurality of features.
16. 16 . The system of claim 15 , wherein capture the image data of the circuit board from the plurality of sensors to generate the set of captured data for each of the plurality of sensors further comprises one or more of the following program instructions, stored on the non-transitory computer-readable storage media, to: capture a multimodal image data of the circuit board, wherein the plurality of sensors configured to acquire a plurality of electromagnetic wavelengths.
17. 17 . The system of claim 15 , wherein extracting the plurality of features from the image data using the machine learning further comprises one or more of the following program instructions, stored on the non-transitory computer-readable storage media, to: fuse the plurality of features from each of the plurality of sensors to create a plurality of segmentation masks, wherein the plurality of features are combined using a feature pyramid network.
18. 18 . The system of claim 17 , wherein extracting the plurality of features from the image data using the machine learning further comprises one or more of the following program instructions, stored on the non-transitory computer-readable storage media, to: capture a plurality of polygons; and compute an Intersection over Union between a segmentation mask for each pair of the plurality of polygons.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS The present application claims the benefit of the filing date of U.S. Provisional Application Ser. No. 63/380,804, filed Oct. 25, 2022, the entire teachings of which application is hereby incorporated herein by reference. TECHNICAL FIELD The present application relates generally to systems and methods for printed circuit board netlist extraction from multimodal imagery. BACKGROUND Electronic Commercial Off-The-Shelf (COTS) items are packaged electronic components that are used in custom systems. Usually, an electronic COTS item comprises a printed circuit board (PCB) with various integrated circuits (ICs) and discrete components mounted on the PCB and electrically interconnected by printed circuitry of the PCB. For example, a military air force or civilian airline may use a COTS item as an aircraft component. In many cases, the customer (e.g., air force or airline) will qualify a COTS item for use in a particular system (e.g., a particular model/make aircraft) based on component specifications and/or extensive testing to ensure the COTS item meets stringent performance and reliability criteria. For example, an air force may require all ICs in a COTS item to be “military grade”, for example having an operational temperature range meeting some required standard. It is often critical to have a high level of confidence that COTS items meet design and performance specifications prior to their deployment. The outsourcing implicit in using a COTS item means that the customer does not control the manufacturing/supply chain producing the COTS item, making it extremely difficult to track component and assembly provenance. For example, the manufacturer may make an undocumented revision to the COTS item that may potentially affect the customer. Another concern with employing COTS items in a system is that there is the potential to receive a counterfeit COTS item, which may have substitute (possibly lower quality) parts or may introduce privacy-compromising capabilities or other issues. Another concern with employing COTS items is that the COTS item may not meet performance specifications provided by the supplier. This can occur due to use of lower performing counterfeit parts, use of manufacturing parameters that do not meet performance specifications, failure to provide proper connectivity between components, and so forth. These types of concerns can be alleviated by testing of COTS items. In destructive analysis approaches, the COTS item is disassembled or otherwise destructively analyzed to assure it meets the requisite specifications. Naturally, destructive analysis can only be performed on “extra” COTS items delivered to the customer, which increases cost at the customer end. Furthermore, destructive analysis cannot provide assurance that a particular COTS item that is actually installed in a customer system meets specifications. Nondestructive analysis (NDA) overcomes these deficiencies. However, existing NDA techniques do not accurately and efficiently capture the connectivity of the PCB components to one another. This makes it difficult to confirm a COTS item meets design specifications or to determine the design of a PCB when a COTS manufacturer does not provide design information to the end user. BRIEF DESCRIPTION OF THE DRAWINGS The above-mentioned and other features of this disclosure, and the manner of attaining them, will become more apparent and better understood by reference to the following description of embodiments described herein taken in conjunction with the accompanying drawings, wherein: FIG. 1 diagrammatically illustrates one example of a multimodal inspection system (MIS) consistent with the present disclosure. FIG. 2 illustrates an example MIS consistent with the present disclosure from a front-side-view perspective. FIG. 3 illustrates an example computing device associated with an MIS consistent with the present disclosure. FIG. 4 illustrates a database that may be within an MIS consistent with the present disclosure. FIGS. 5A-5C illustrate detection of a trace on a printed circuit board that is occluded by a logo on the circuit board. FIG. 6 illustrates operation of a feature pyramid network consistent with the present disclosure. FIG. 7 diagrammatically illustrates operation of a loss function consistent with the present disclosure. FIG. 8 illustrates encapsulation of features of a PCB in a system consistent with the present disclosure. FIG. 9 illustrate test results showing performance of a system and method consistent with the present disclosure. FIGS. 10-12 illustrate test results showing performance of a system and method consistent with the present disclosure; and FIG. 13 is a flow diagram illustrating the process flow for one example of a system and method consistent with the present disclosure. DETAILED DESCRIPTION While a number of NDA solutions exist to locate the components on a PCB, there is still a need for solutions to accurately and eff