US-12621396-B2 - Photo-based workflow initiation

US12621396B2US 12621396 B2US12621396 B2US 12621396B2US-12621396-B2

Abstract

Systems and methods are provided for generating a resource transmission request to initiate a workflow associated with resource transmission. In particular, the disclosed technology is directed to processing image data corresponding to a physical notice of a request for resource transmission to generate an electronic resource transmission request. The system captures image data of the notice and extracts data from the image data. As an example, the system matches the extracted data against predetermined forms and determines whether the notice is in a known format. In instances where there is no match (such that the notice has an unknown format), the system uses one or more of heatmaps, rules of locating field data, and/or a field data extraction model to assign respective field names with data values in the extracted data. The heatmap includes regions in the image data with a likelihood of data values corresponding to particular field names.

Inventors

Alex Kharbush
Bradley Irvin
Sean Tan

Assignees

BillGO, Inc.

Dates

Publication Date: 20260505
Application Date: 20231109

Claims (20)

1 . A computer implemented method, comprising: receiving image data, wherein the image data is representative of a notice of a request for resource transmission in a physical form; processing the image data to determine a set of regions, wherein each region of the set of regions corresponds to a field name and comprises a subset of pixels representative of field data for the corresponding field name; processing each determined region of the image data to extract, from the subset of pixels for the region, the field data for the field name associated with the request for resource transmission, wherein the set of regions includes a region for a recipient of the resource transmission, the processing comprising: matching textual data extracted from image data for a region with a predetermined form in a set of predetermined forms, the matched predetermined form defining a layout for the physical form corresponding to the determined set of regions; assigning a data value in the textual data to a field name according to the matched predetermined form, wherein the matched predetermined form associates the field name with a region in the image data where the data value is extracted from; and generating, based at least in part on the assigned data value, the field data; generating, based on the extracted field data for each of the determined regions, a request for resource transmission to the recipient; and transmitting the generated resource transmission request, thereby causing transmission of a resource to the recipient based on the extracted field data.
2 . The computer implemented method of claim 1 , wherein the field data includes the field name and the data value assigned to the field name.
3 . The computer implemented method of claim 1 , further comprising: obtaining a heatmap associated with the field name in the notice of a request for resource transmission; and assigning, based on the heatmap, a data value in the extracted textual data with the field name, wherein the heatmap indicates a likelihood of the data value extracted from at a region of interest corresponding to the field name.
4 . The computer implemented method of claim 3 , wherein the heatmap includes statistical data based on an aggregate of a set of the textual data corresponding to the field name and a location information of the textual data appearing in the image data.
5 . The computer implemented method of claim 1 , further comprising: assigning, using a field data extraction model, a data value from a region of the set of regions with a corresponding field name, wherein the field data extraction model, when trained, predicts a likelihood of the data value extracted from the region corresponding to the corresponding field name.
6 . The computer implemented method of claim 5 , wherein the field data extraction model includes a neural network with multiple layers of processing with co-efficient values that have been updated based on training using training data.
7 . The computer implemented method of claim 1 , further comprising: assigning, using a predefined rule for identifying a data value associated with a field name, a data value from a region of the set of regions with a corresponding field name, wherein the predefined rule indicates a region of the image data where the data value associated with the field name likely appears.
8 . The computer implemented method of claim 1 , wherein the set of regions comprises one or more of: a region for a sender name, a region for a sender address, a region for a receiver name, a region for a receiver address, a region for an account number, a region for a payer name, a region for a payer address, a region for a payee name, a region for a payee address, a region indicating an amount of resource for transmission, and a region indicating an amount of financial resources for remittance.
9 . The computer implemented method of claim 1 , wherein the set of regions define a layout of a plurality of different layouts for physical forms.
10 . A system, the system comprising: a processor; and a memory storing computer executable instructions that when executed by the processor cause the system to perform a set of operations, the set of operations comprising: obtaining image data, wherein the image data corresponds to a notice of a request for resource transmission in a physical form; processing the image data to determine a format from a set of formats, each format defining a different layout for a physical form, the determined format indicating a set of regions for the image data that each correspond to a field name and comprise a subset of pixels representative of field data for the corresponding field name; processing each region of the image data to extract, from the subset of pixels for the region, the field data for the field name associated with the request for resource transmission, wherein the set of regions includes a region for a recipient of the resource transmission, the processing comprising: matching textual data extracted from image data for a region with a predetermined form in a set of predetermined forms that each correspond to a format of the set of formats; assigning a data value in the textual data to a field name according to the matched predetermined form, wherein the matched predetermined form associates the field name with a region in the image data where the data value is extracted from; and generating, based at least in part on the assigned data value, the field data; generating, based on the extracted field data for the set of regions, a request for resource transmission to the recipient; and transmitting the generated resource transmission request, thereby causing transmission of a resource to the recipient based on the extracted field data.
11 . The system of claim 10 , wherein the field data includes a pair of the field name and the data value assigned to the field name.
12 . The system of claim 10 , the set of operations further comprising: retrieving a heatmap associated with the field name in the notice of a request for resource transmission; assigning, based on the heatmap, a data value in the extracted textual data with the field name, wherein the heatmap indicates a likelihood of the data value extracted from at a region of interest corresponding to the field name.
13 . The system of claim 10 , the set of operations further comprising: assigning, using a field data extraction model, a data value from a region of the set of regions with a corresponding field name, wherein the field data extraction model, when trained, predicts a likelihood of the data value extracted from the region corresponding to the corresponding field name.
14 . The system of claim 10 , the set of operation further comprising: matching, based on fuzzy matching, textual data in a region with a predetermined form in the set of predetermined forms.
15 . The system of claim 10 , wherein the set of regions define a layout of a plurality of different layouts for physical forms.
16 . A computer-implemented method, comprising: receiving image data, wherein the image data corresponds to a notice of a request for resource transmission in a physical form; processing the image data to determine a set of regions of the image data, wherein each region of the set of regions corresponds to a field name and comprises a subset of pixels representative of field data for the corresponding field name; processing each determined region of the image data to extract, from the subset of pixels for the region, the field data corresponding to the field name of and associated with the request for resource transmission, the processing comprising: matching textual data extracted from image data for a region with a predetermined form in a set of predetermined forms, the matched predetermined form defining a layout for the physical form corresponding to the determined set of regions; assigning a data value in the textual data to a field name according to the matched predetermined form, wherein the matched predetermined form associates the field name with a region in the image data where the data value is extracted from; and generating, based at least in part on the assigned data value, the field data; generating, based on the extracted field data from the received image data, a request instruction to perform the requested resource transmission to the recipient; and transmitting the generated resource transmission request instruction, thereby causing transmission of a resource to the recipient based on the extracted field data, wherein the extracted field data includes a recipient of the resource transmission and the field name is part of a set of field names, the set of field names further comprising at least one of: a sender name, a sender address, a receiver name, a receiver address, or an amount of a resource for transmission.
17 . The computer-implemented method of claim 16 , further comprising: assigning, using a field data extraction model, a data value from a region of the set of regions with a corresponding field name, wherein the field data extraction model, when trained, predicts a likelihood of the data value extracted from the region corresponding to the field name.
18 . The computer-implemented method of claim 17 , wherein the field data extraction model includes a neural network with multiple layers of processing with co-efficient values that have been updated based on training using training data.
19 . The computer-implemented method of claim 16 , wherein the set of field names further comprises: an account number, a payer name, a payer address, a payee name, and a payee address.
20 . The method of claim 16 , wherein the set of regions define a layout of a plurality of different layouts for physical forms.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to U.S. Provisional Application No. 63/383,009, titled “Photo-Based Workflow Initiation,” filed on Nov. 9, 2022, the entire disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND Using online systems for processing resource transmission has become a common place in our daily lives. However, such online systems typically have a high degree of variability, which may make it difficult for a user to manually enter information associated with a resource transmission request. These and other difficulties may result in a poor user experience, introduce the potential for human error, and may even causes resources to be transmitted incorrectly. It is with respect to these and other general considerations that the aspects disclosed herein have been made. Although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure. SUMMARY Aspects of the present disclosure relate to receiving image data as input to initiate a workflow. In particular, the workflow relates to tasks including transmission of resources based on a request for resource transmission. The image data may be in a form that is predetermined by a system (e.g., a predefined format) or unknown by the system (e.g., an unknown format). An embodiment captures a notice of a request for resource transmission in a physical form (e.g., a paper form) and generates image data (e.g., photo data). The disclosed technology transforms the image data into a set of fields and data values and generates a request for resource transmission for initiating the workflow. The system identifies whether the extracted data corresponds to a determined form of a notice. The process of identifying includes matching the extracted data and location information of the respective extracted data in the image data against the predetermined forms. A predetermined form includes a field name and location information of a region of interest where a data value associated with the field name appears in the form. Given the identified form, the system assigns data values to respective field names. When the extracted data does not correspond to a predetermined form, the system uses a set of heatmaps associated with the notice to determine a set of field names and associated data values from the extracted data. The set of heatmaps includes one or more heatmaps corresponding to respective field names. For example, a heatmap indicates one or more regions of interest in the image data with probability values. The probability values may indicate a likelihood that a data value associated with a field name appears (e.g., is printed or is otherwise stored) in the respective one or more regions of interest. The set of heatmaps is generated based on statistic data associated with locations of various fields at which data is placed in sample notices (e.g., of a request for resource transmission) having varying formats. The system assigns data values to respective field names using data values in a region that matches with a region of interest with the highest probability value in the respective heatmaps. The process of identifying further includes a use of one or more rules associated with placement of data associated with respective field names. Additionally, or alternatively, the system uses a field data extraction model to determine data values for respective field names to generate a request for resource transmission. The field data extraction model includes a neural network and is trained according to a representative set of training data. The model is used to process image data comprising a notice of a request for resource transmission as an input. Given the input, the model predicts data value assignments to respective field names at various locations within the image data. Training data used for training the model may include sample image data of notices of a request for resource transmission in various formats. The training data may further include a set of pairs of a field name and a data value as ground truth data, which matches with content of the image data. Accordingly, the system generates a request for resource transmission in an electronic form based on information that is extracted from image data according to aspects described herein, thereby reducing the potential for human error, expediting resource transmission processing, and improving an associated user experience, among other benefits. The generated request for resource transmission may thus initiate a workflow for processing the request for resource transmission accordingly. This Summary introduces a selection of concepts in a simplified form, which is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claime