US-12619862-B2 - Deconvolution by convolutions

Abstract

Disclosed herein are apparatus, method, and computer-readable storage device embodiments for implementing deconvolution via a set of convolutions. For example, an apparatus includes a memory and at least one processor, communicatively coupled with the memory, to perform operations including obtaining a tensor representing image data, generating a deconvolution segment of a plurality of deconvolution segments by performing a convolution to transform the tensor based at least in part on a convolution filter and a stride value, and generating a complete deconvolution for image processing by assembling each deconvolution segment of the plurality of deconvolution segments.

Inventors

• Tom Michiels
• Thomas Julian PENNELLO

Assignees

• SYNOPSYS, INC.

Dates

Publication Date

20260505

Application Date

20240314

Priority Date

20181003

Claims (20)

1. 1 . An apparatus comprising: a memory; and at least one processor, communicatively coupled with the memory, to perform operations comprising: obtaining a representation of image data; identifying a set of deconvolution parameters of a complete deconvolution having a set of deconvolution parameters comprising a deconvolution stride value and a deconvolution kernel size of a deconvolution kernel; dividing the complete deconvolution into a plurality of convolutions, each convolution of the plurality of convolutions having a set of convolution parameters comprising: a respective convolution kernel size determined based on the deconvolution kernel size, the deconvolution stride value and a respective convolution index value of the convolution; and a respective convolution kernel coefficient determined based on the deconvolution kernel; for each convolution of the plurality of convolutions, performing the convolution to transform the representation of image data and generate a convolution output of a plurality of convolution outputs, wherein each convolution output of the plurality of convolution outputs corresponds to a respective position determined based on the deconvolution stride value and the respective convolution index value; generating the complete deconvolution by interleaving each convolution output of the plurality of convolution outputs; and causing at least one of an image processing operation or a computer vision operation to be performed using the complete deconvolution.
2. 2 . The apparatus of claim 1 , wherein causing at least one of the image processing operation or a computer vision operation to be performed using the complete deconvolution comprises outputting the complete deconvolution via at least one of: an output file or an output device.
3. 3 . The apparatus of claim 1 , wherein the representation of image data comprises a tensor, and wherein performing the convolution further comprises expanding the tensor by at least one layer of padding elements.
4. 4 . The apparatus of claim 3 , wherein the operations further comprise truncating each convolution of the plurality of convolutions by a truncation value corresponding to the at least one layer of padding elements.
5. 5 . The apparatus of claim 1 , wherein the representation of image data is an output of a separate convolution.
6. 6 . The apparatus of claim 1 , wherein each convolution is performed using a respective convolution filter that comprises a respective subset of a deconvolution filter.
7. 7 . A method comprising: obtaining, by at least one processing device, a representation of image data; identifying, by at least one processing device, a set of deconvolution parameters of a complete deconvolution having a set of deconvolution parameters comprising a deconvolution stride value and a deconvolution kernel size; dividing, by the at least one processing device, the complete deconvolution into a plurality of convolutions, each convolution of the plurality of convolutions having a set of convolution parameters comprising: a respective convolution kernel size determined based on the deconvolution kernel size, the deconvolution stride value and a respective convolution index value of the convolution; and a respective convolution kernel coefficient determined from a deconvolution kernel; for each convolution of the plurality of convolutions, performing the convolution of a plurality of convolutions to transform the representation of image data based at least in part on a kernel size of the complete deconvolution and a stride value of the complete deconvolution, wherein each convolution of the plurality of convolutions corresponds to a respective deconvolution segment of the plurality of deconvolution segments, and wherein the stride value of the complete deconvolution is equal to a number of convolutions of the plurality of convolutions and generate a convolution output of a plurality of convolution outputs, wherein each convolution output of the plurality of convolution outputs corresponds to a respective position determined based on the deconvolution stride value and the respective convolution index value; generating, by the at least one processing device, the complete deconvolution by interleaving each convolution output of the plurality of convolution outputs; and causing, by the at least one processing device, at least one of an image processing operation or a computer vision operation to be performed using the complete deconvolution.
8. 8 . The method of claim 7 , wherein causing at least one of the image processing operation or a computer vision operation to be performed using the complete deconvolution comprises outputting, by the at least one processing device, the complete deconvolution via at least one of: an output file or an output device.
9. 9 . The method of claim 7 , wherein the representation of image data comprises a tensor, and wherein performing the convolution further comprises expanding the tensor by at least one layer of padding elements.
10. 10 . The method of claim 9 , further comprising truncating, by the at least one processing device, each convolution of the plurality of convolutions by a truncation value corresponding to the at least one layer of padding elements.
11. 11 . The method of claim 7 , wherein the representation of image data is an output of a separate convolution.
12. 12 . The method of claim 7 , wherein each convolution is performed using a respective convolution filter that comprises a respective subset of a deconvolution filter.
13. 13 . A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by at least one computer processor, cause the at least one computer processor to perform operations comprising: obtaining a representation of image data; identifying a set of deconvolution parameters of a complete deconvolution having a set of deconvolution parameters comprising a deconvolution stride value and a deconvolution kernel size; dividing the complete deconvolution into a plurality of convolutions, each convolution of the plurality of convolutions having a set of convolution parameters comprising: a respective convolution kernel size determined based on the deconvolution kernel size, the deconvolution stride value and a respective convolution index value of the convolution; and a respective convolution kernel coefficient determined from a deconvolution kernel; for each convolution of the plurality of convolutions, performing the convolution to transform the representation of image data and generate a convolution output of a plurality of convolution outputs, wherein each convolution output of the plurality of convolution outputs corresponds to a respective position determined based on the deconvolution stride value and the respective convolution index value; generating the complete deconvolution by interleaving each convolution output of the plurality of convolution outputs; and causing at least one of an image processing operation or a computer vision operation to be performed using the complete deconvolution.
14. 14 . The non-transitory computer-readable medium of claim 13 , wherein causing at least one of image processing operation or a computer vision operation to be performed using the complete deconvolution comprises outputting the complete deconvolution via at least one of: an output file or an output device.
15. 15 . The non-transitory computer-readable medium of claim 13 , wherein the representation of image data comprises a tensor, wherein performing the convolution further comprises expanding the tensor by at least one layer of padding elements, and wherein the operations further comprise truncating each convolution of the plurality of convolutions by a truncation value corresponding to the at least one layer of padding elements.
16. 16 . The non-transitory computer-readable medium of claim 13 , wherein the representation of image data is an output of a separate convolution.
17. 17 . The non-transitory computer-readable medium of claim 13 , wherein each convolution is performed using a respective convolution filter that comprises a respective subset of a deconvolution filter, and wherein the stride value further defines a number of convolution results of a set of convolution results generated by performing the convolution.
18. 18 . The apparatus of claim 1 , wherein the set of convolution parameters further comprises a convolution stride value of 1.
19. 19 . The method of claim 7 , wherein the set of convolution parameters further comprises a convolution stride value of 1.
20. 20 . The non-transitory computer-readable medium of claim 13 , wherein the set of convolution parameters further comprises a convolution stride value of 1.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS The present application is a continuation of U.S. patent application Ser. No. 16/592,241 titled “DECONVOLUTION BY CONVOLUTIONS,” filed Oct. 3, 2019, which claims priority to European Patent Application No. EP18198452 titled “Neural Network Deconvolution Implemented by a Set of Convolutions,” filed Oct. 3, 2018, which are hereby incorporated by reference in their entirety. COPYRIGHT NOTICE The assignee of this patent document has no objection to the facsimile reproduction by anyone of the patent document itself, or of the patent application, as it appears in the files of the United States Patent and Trademark Office, but otherwise reserves all rights whatsoever in any included works of authorship protected by copyright. DISCLAIMERS In the following Background, Summary, and Detailed Description, paragraph headings do not signify limitations. In these writings, the citation or identification of any publication signifies neither relevance nor status as prior art. Many paragraphs in this disclosure may be common to multiple Synopsys patent specifications. FIELD(S) OF TECHNOLOGY The following information is solely for use in searching the prior art. This disclosure has significance in the field of electronics in general, including the following topics: electronic design automation, neural networks, convolution and dedicated circuitry therefor, computer algorithms, software programs, and any combination thereof, non-exclusively. BACKGROUND Convolution and deconvolution are mathematical operations that are valuable in a variety of applications that may depend on neural networks, for example. Conventional techniques have given rise to dedicated hardware intended to accelerate convolutional neural network (CNN) operations, such as vector processing or convolution calculation, more efficiently than they could be performed by general-purpose computing hardware. However, such hardware, generic or specialized, may have no native support for deconvolution per se. Referring to image processing using convolution and deconvolution, a convolution may operate on multiple input pixels to result in a single output pixel, for example. By contrast, in deconvolution, multiple output pixels may result from a single input pixel, in some cases. More specifically, a deconvolution operation is not merely a reversal of steps performed in a convolution operation, even if an output of a deconvolution may resemble an input of a convolution. Thus, performing deconvolution is a non-trivial action. Due to at least these aspects of deconvolution, specialized hardware for deconvolution processing is not presently available. As such, performance of deconvolution operations, whether on generic computer hardware or specialized hardware better suited for a different purpose, is inefficient by various measures, including computation speed and resource usage, for example. This inefficiency causes sub-optimal performance of more complex applications that may depend on deconvolutions, such as machine learning, computer vision, etc. In other words, deconvolutions may result in a performance bottleneck for such applications. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are incorporated herein and form a part of the specification. FIG. 1 shows an example 3s2 convolution, according to some embodiments. FIG. 2 shows an example 3s2 deconvolution, according to some embodiments. FIG. 3 shows a generic 1-D convolution, according to some embodiments. FIG. 4 shows part of an example 3s2 convolution, according to some embodiments. FIG. 5 shows an algebraic manipulation of generic deconvolution, according to some embodiments. FIG. 6 shows an example of a transformed 1-D deconvolution, according to some embodiments. FIG. 7 shows example 3s2 deconvolution inputs, according to some embodiments. FIG. 8 shows a further example 3s2 deconvolution, according to some embodiments. FIG. 9 shows an example 3s1p2 deconvolution, according to some embodiments. FIG. 10 shows two related convolution examples, according to some embodiments. FIG. 11 shows an example 5s3 deconvolution, according to some embodiments. FIG. 12 shows some convolution size calculations, according to some embodiments. FIG. 13 shows an example of input and output of testing deconvolution by convolutions, according to some embodiments. FIG. 14 shows an example of Python code loop to assemble convolution outputs into a deconvolution output, according to some embodiments. FIG. 15 shows an example 3s2p1 convolution, according to some embodiments. FIG. 16 shows an algebraic manipulation of generic deconvolution with padding, according to some embodiments. FIG. 17 shows an example of an example of a transformed 1-D deconvolution with padding, according to some embodiments. FIG. 18 shows an example 3s2p1 convolution, according to some embodiments. FIG. 19 shows an example of Python code loop to assemble convolution outputs into a deconvolution output, accounting