CN-117115836-B - Handwritten character generation method and system for single sample class and electronic equipment

CN117115836BCN 117115836 BCN117115836 BCN 117115836BCN-117115836-B

Abstract

The invention discloses a handwritten character generation method, a system and electronic equipment for single sample class, and relates to the technical field of handwritten character digitalization, the method comprises the steps of training a variation self-encoder by using a sample real handwritten character image to obtain probability distribution of the variation self-encoder in a hidden space of the variation self-encoder, initializing a generator by using a decoder of the trained variation self-encoder, and initializing a discriminator by using the encoder; the method comprises the steps of inputting a target real handwritten character image and a target computer printed character image into the pre-trained encoder to obtain corresponding hidden variables, aligning the mean value and covariance of all hidden variables in a hidden space by utilizing a CORAL transformation technology to obtain an initialized hidden variable set, inputting the initialized hidden variables into a generator in a generating countermeasure network, and generating a new character image. The invention can amplify the data of the hand-written character data set with extremely unbalanced samples, so that the generated small sample samples have correctness and diversity.

Inventors

WANG WEILAN
MAO LEER
LI QIAOQIAO
HU PENGFEI

Assignees

西北民族大学

Dates

Publication Date: 20260508
Application Date: 20230915

Claims (6)

1. A method of generating handwritten characters for a single sample class, comprising: Inputting a sample real handwriting character image into a variation self-encoder to pretrain the variation self-encoder, obtaining probability distribution of the sample real handwriting character image in a hidden space of the variation self-encoder, initializing a generator in an countermeasure network by using a decoder of the trained variation self-encoder, and initializing a discriminator in the countermeasure network by using the encoder of the trained variation self-encoder; Respectively inputting a target real handwritten character image in a target real data set and a corresponding target computer printed character image into an encoder in a trained variable self-encoder to obtain hidden variables of each target real handwritten character image and hidden variables of the corresponding target computer printed character image, wherein the types of each target real handwritten character image in the target real data set are different; The method comprises the steps of obtaining an initialization hidden variable set by aligning hidden variables of all target real handwritten character images and hidden variables of corresponding target computer printed character images in hidden space by means of CORAL transformation technology, and obtaining the initialization hidden variable set by aligning the hidden variables of all target real handwritten character images and hidden variables of corresponding target computer printed character images in hidden space by means of CORAL transformation technology, wherein the method specifically comprises the following steps: Sum formula Obtaining an initialization hidden variable set, wherein, The mean value of the kth category of the target real handwriting character in the hidden space, The mean value of the kth category of the character image in hidden space is printed for the target computer, For the position of the ith target computer print character image belonging to the kth category on the hidden space, Values in hidden space after conversion for the ith target computer print character image, wherein hidden variables of the computer print character image are The hidden variable of the true handwritten character image is , And Respectively represent And Is used for the co-variance matrix of (a), And Respectively are And Is a decomposition of the singular values of (a), , And Is that The first r largest singular values and the corresponding singular vectors; In the generation of the reactive network training phase, training the training of the discriminator by adopting the sampling frequency of the new category so that the loss expectation determined as false by the discriminator for each category does not exceed the loss expectation determined as true; Inputting the initialization hidden variables in the initialization hidden variable set into a generator in a generation countermeasure network to generate a new character image.
2. The method of claim 1, wherein the encoder of the variational self-encoder is composed of a plurality of downsampling layers, the decoder of the variational self-encoder is composed of a plurality of upsampling layers, and the variational self-encoder is used for mapping the character image from the high-dimensional image space to the low-dimensional hidden space.
3. The method for generating handwritten characters for single sample classes according to claim 2, wherein the training process of the variational self-encoder is as follows: training the variation self-encoder by adopting a sample real hand-written character image in a sample real data set, and stopping training when the comprehensive loss value of the variation self-encoder is smaller than a set threshold value to obtain the trained variation self-encoder.
4. A method of generating handwritten characters for a single sample class according to claim 3, wherein the integrated loss value of the variant self-encoder is the sum of the loss value of the variant self-encoder and the cross-entropy loss value.
5. A handwritten character generation system for a single sample class, comprising: The training module is used for inputting the sample real handwritten character image into the variation self-encoder to pretrain the variation self-encoder, obtaining probability distribution of the sample real handwritten character image in a hidden space of the variation self-encoder, initializing a generator in a countermeasure network by using a decoder of the trained variation self-encoder, initializing a discriminator in the countermeasure network by using the encoder of the trained variation self-encoder, and training the discriminator by adopting a sampling frequency of a new category in a training stage of the generated countermeasure network, so that the loss expectation of each category judged as false does not exceed the loss expectation judged as true; The hidden variable determining module is used for respectively inputting target real handwritten character images in target real data sets and corresponding target computer printed character images into the trained encoders of the variable self-encoders to obtain hidden variables of each target real handwritten character image and hidden variables of the corresponding target computer printed character images, wherein the types of each target real handwritten character image in the target real data sets are different; The hidden variable initialization module is used for aligning the hidden variables of all the target real hand-written character images and the hidden variables of the corresponding target computer print character images in the hidden space by utilizing the CORAL transformation technology to obtain an initialized hidden variable set, and aligning the hidden variables of all the target real hand-written character images and the hidden variables of the corresponding target computer print character images in the hidden space by utilizing the CORAL transformation technology to obtain the initialized hidden variable set, and specifically comprises the following steps: Sum formula Obtaining an initialization hidden variable set, wherein, The mean value of the kth category of the target real handwriting character in the hidden space, The mean value of the kth category of the character image in hidden space is printed for the target computer, For the position of the ith target computer print character image belonging to the kth category on the hidden space, Values in hidden space after conversion for the ith target computer print character image, wherein hidden variables of the computer print character image are The hidden variable of the true handwritten character image is , And Respectively represent And Is used for the co-variance matrix of (a), And Respectively are And Is a decomposition of the singular values of (a), , And Is that The first r largest singular values and the corresponding singular vectors; And the character image generation module is used for inputting the initialization hidden variables in the initialization hidden variable set into a generator in the generation countermeasure network to generate a new character image.
6. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the handwritten character generation method for a single sample class according to any one of claims 1 to 4.

Description

Handwritten character generation method and system for single sample class and electronic equipment Technical Field The present invention relates to the field of handwriting character digitizing technology, and in particular, to a method, a system, and an electronic device for generating handwriting characters for single sample class. Background Optical Character Recognition (OCR) can digitize printed or handwritten text from an image. Despite significant advances in recognition accuracy over the years, challenges remain in recognizing text in historical handwritten files. One of the main challenges is the imbalance in the number of samples between character classes. Such class imbalance is very common in characters such as chinese and Tibetan, and seriously hinders further improvement of recognition accuracy of OCR systems based on deep learning. One method of intuitively improving the recognition accuracy of a character classifier is to add more synthetic data to a small sample class. Depth generation models, such as generation of a countermeasure network (GENERATIVE ADVERSARIAL Networks, GAN) and variational self-encoders (Variational Autoencoder, VAE), can automatically generate synthetic samples given a set of training data, either marked or unmarked. In the last decade, GAN has achieved excellent results in generating realistic samples. However, there is a well-known "fidelity-diversity" tradeoff when dealing with data sets that are class-unbalanced, i.e., when certain classes of samples are very scarce. That is, to optimize the loss function, the generator tends to generate exactly the same samples, because the arbiter will consider any generated samples that are different from the true samples as "false", which does not meet the data augmentation requirements. Some research has been directed to solving the category imbalance learning problem by improving the loss function of GAN. For example, MFC-GAN introduces multiple "false" tags on an AC-GAN basis, ensuring fine-grained generation of small sample classes. Ren et al propose an entropy-based weighting strategy to characterize the importance of different classes, namely assigning a higher importance to small sample classes and a lower importance to large sample classes. The importance weighted tag vector and the random noise vector are connected as inputs to WASSERSTEIN GAN. In order to bias the generator towards small sample classes, lin et al propose Rare-GAN that introduce a re-weighting technique in the loss function of AC-GAN, i.e. to assign two different weights to the samples depending on the class to which the samples belong (large sample class or small sample class). In addition, to improve the classification performance of AC-GAN on unbalanced data, they also employ an active learning strategy based on confidence. However, this approach by improving the loss function has limited data augmentation on small sample classes because artificial neural networks follow some "lazy principle" in optimizing the loss function. There are also some studies combining GAN and other depth generation models. BAGAN is proposed, for example, mariani et al. They solve the class imbalance problem by fitting the probability distribution of each class in the hidden space of the pre-trained self-encoder (Autoencoder, AE) and using it as input to the generator. The generator and arbiter are initialized by the decoder and encoder of the pre-trained AE, respectively, prior to the countermeasure training. The goal of the pre-training AE is to learn the low-dimensional manifold for each class in hidden space, enabling it to capture the conditional probability distribution for each class in the event of sample starvation. However, the hidden space of AE training often lacks regularity, i.e., the boundaries of the class cannot be captured correctly. To alleviate this problem, yang et al proposed IDA-GAN, with VAE pre-training instead of AE pre-training in BAGAN. In addition, it blends category labels into the input of the generator to guide the generation process. In BAGAN and IDA-GAN, class conditional distributions are obtained by fitting a gaussian distribution over the hidden variables, however the assumption that the class follows a gaussian distribution in hidden space is somewhat controversial. The above-described study has achieved a certain achievement in unbalanced learning. However, when speaking "small sample classes", they generally refer to classes having at least tens of samples, there is still no effective solution to the problem of generating a single sample class for the extreme case of small sample classes. In the characters such as Chinese and Tibetan, the situation of single sample class is very frequent due to the existence of a large number of rare words. Disclosure of Invention The invention aims to provide a handwritten character generation method, a system and electronic equipment for a single sample class, which are used for carrying out data