Search

CN-122023195-A - Wall painting digital restoration method based on structure guidance and state space model

CN122023195ACN 122023195 ACN122023195 ACN 122023195ACN-122023195-A

Abstract

The invention discloses a wall painting digital restoration method based on a structure guiding and state space model, which comprises the steps of constructing an edge restoration model which is input into a mask image, a damaged edge image and a damaged gray image and is output into an edge restoration image, wherein the edge restoration model is composed of 1 input convolution block, 3 down-sampling convolution blocks, 8 double-channel Mamba modules, 3 up-sampling convolution blocks, 1 output convolution block and 1 discriminator in sequence, constructing a main body restoration model which is input into the damaged wall painting, the mask image and the edge restoration image and is output into a main body restoration image, constructing a detail restoration model which is input into the mask image, the main body restoration image and the edge restoration image and is output into a final restoration image, and applying the trained model to an actual wall painting to be restored.

Inventors

  • CHEN JIANJUN
  • Qiu Yiyin
  • SONG JINGJING
  • WANG FEI
  • YANG XIBEI

Assignees

  • 江苏科技大学

Dates

Publication Date
20260512
Application Date
20260116

Claims (10)

  1. 1. A wall painting digital restoration method based on a structure guiding and state space model is characterized by comprising the following steps: shooting a wall painting to obtain a wall painting picture, processing the wall painting picture to generate a wall painting edge picture, generating a wall painting gray-scale picture based on the wall painting picture, and randomly generating a mask picture on one wall painting picture; obtaining a broken edge image and a broken gray image by multiplying the wall painting edge image and the wall painting gray image with the mask image element by element respectively; obtaining a broken wall painting by multiplying the wall painting by the mask drawing element by element: step 2), constructing an edge restoration model which is input into a mask image, a damaged edge image and a damaged gray image and output into an edge restoration image, wherein the edge restoration model is sequentially composed of 1 input convolution block, 3 downsampling convolution blocks, 8 double-way Mamba modules, 3 upsampling convolution blocks, 1 output convolution block and 1 discriminator; Constructing a main body repair model with the input of a broken wall drawing, a mask drawing and an edge repair drawing and the output of the main body repair drawing, wherein the main body repair model sequentially comprises 1 input convolution block, 3 edge feature fusion modules, 8 double-circuit Mamba modules, 3 convolution up-sampling blocks, 1 output convolution block and 1 discriminator; Step 4) constructing and inputting into a mask chart The main body repair graph and the edge repair graph are output as a detail repair model of the final repair graph, the detail repair model comprises 3 downsampling convolution blocks and 3 upsampling convolution blocks, the 2 nd downsampling convolution block is connected with the 2 nd upsampling convolution block through 1 double-circuit Mamba module, and the 3 rd downsampling convolution block is connected with the 1 st upsampling convolution block through 1 double-circuit Mamba module; And 5) training an edge repair model, a main body repair model and a detail repair model respectively, and applying the trained edge repair model, main body repair model and detail repair model to an actual wall painting needing to be repaired to obtain a final repair picture of the actual wall painting.
  2. 2. The method for digitally repairing wall painting based on structure guide and state space model according to claim 1, wherein in step 1), the method for generating the wall painting edge map by processing the wall painting image is that firstly a Canny algorithm is used for processing the wall painting image to obtain a Canny fine edge map, then a MuGE edge deep learning detection algorithm is used for processing the wall painting image to obtain a contour map, and finally the contour map is obtained Adding the wall painting edge map with the outline map and trimming the upper limit of the pixel value of the image to be not more than 255.
  3. 3. The method for digitally repairing wall painting based on structure guide and state space model according to claim 1, wherein in step 1), the wall painting gray-scale image is generated based on wall painting by using red, green and blue channel pixel values of any wall painting By calculation formula And obtaining the pixel value of any point on the corresponding wall painting gray level diagram.
  4. 4. The method for digital wall painting restoration based on structure guidance and state space model according to claim 1, wherein each convolution block is connected in 1 normalization layer, 1 ReLU activation function and 13×3 convolution sequence, each two-way Mamba module comprises two-way Mamba scanning submodule and feedforward network submodule, each discriminator is composed of 5 downsampled convolution blocks, and each convolution block is connected through 1 LeakyReLU activation function.
  5. 5. The method for digital repairing wall painting based on structure guidance and state space model according to claim 4, wherein said two-way Mamba scan submodule comprises a vertical S-shaped round-trip scan and a horizontal S-shaped round-trip scan, and is used for inputting channel number Height and width of Is flattened into a length after vertical S-shaped round-trip scanning Is a one-dimensional image sequence of (2) Flattened to length after horizontal S-shaped round-trip scanning Is a one-dimensional image sequence of (2) , Generating a length of a triangle function position coding function Position-coding sequences of (c) : Sequence of horizontal and vertical one-dimensional images Respectively corresponding to the position coding sequences Adding and normalizing to obtain the coded characteristic sequence , Is a normalization layer; Feature sequence Sequentially through linear layers And activating function SiLU to obtain a first sub-term ; Feature sequence Sequentially through linear layers One-dimensional convolution SiLU after activating the function and state space model SSM, a second term is obtained , , Dividing the first sub-item into And a second sub-term Multiplying by element and passing through a linear layer Obtaining a feature sequence with enhanced global features , Representing element-wise multiplication and then enhancing the global feature by using the feature sequence And Enhanced length by global features Is a horizontal and vertical one-dimensional feature sequence of (2) And (3) with Reshape into Two-dimensional feature map in form Will be Two-dimensional feature map in form The two are added to obtain a two-dimensional characteristic diagram output by the two-way Mamba scanning sub-module 。
  6. 6. The method for digital repairing of wall painting based on structure guidance and state space model according to claim 5, wherein the two-dimensional feature map is output by a two-way Mamba scanning sub-module Input into the feedforward network sub-module, the feedforward network sub-module first convolves with 1 x 1 convolution 3 X 3 depth separable convolution The GELU activation functions form a gate And then two-dimensional characteristic diagram By 1X 1 convolution Mapped two-dimensional feature map and gating Multiplying by elements, screening redundant dimension information, and obtaining enhanced two-dimensional feature map The output of the feedforward network submodule is a two-dimensional characteristic diagram 。
  7. 7. The method for digital wall painting restoration based on structure guidance and state space model according to claim 1, wherein said edge feature fusion module in step 3) samples an edge restoration map to and inputs a two-dimensional feature map therein The same size, obtain , Through 3 x 3 convolution Processed and inputted with the two-dimensional characteristic diagram Splicing in the channel dimension to obtain a feature map , Splice for channel dimension, then match the feature map Normalized to obtain a feature map Normalized feature map By 1X 1 convolution Mapping to obtain a first sub-term Normalized feature map Sequentially through 1×1 convolution 3 X 3 depth separable convolution Obtaining the second sub-term after GELU activating the function Dividing two items And (3) with Element-by-element multiplication to obtain feature map , Then through a 1X 1 convolution Processing the obtained feature map and two-dimensional feature map New two-dimensional feature map obtained by addition 。
  8. 8. The method for digital wall painting restoration based on structure guidance and state space model according to claim 1, wherein in step 5), a back propagation algorithm is used to optimize a loss function during the training of the edge restoration model , wherein, Is a defect area The loss of the material is controlled by the temperature, In order to combat the loss of this, In order to perceive the loss of the light, In order to achieve a loss of style, For total variation loss, loss function coefficients Defective area Loss of , Representing the multiplication by element, Representing the sum function, The L1 norm is represented by the expression, Representing the edge map of the wall painting, Representing an edge restoration map; Countering losses And (3) with Representing the output of the arbiter of the edge restoration model, Indicating a lot desire to fetch a corresponding data distribution; Challenge training requires training the arbiter to minimize the loss: , Loss of perception , Features are extracted for layer i of the VGG19 network, Representing an accumulation function; Style loss , Representing feature extraction for layer i of VGG19 network Calculating a Gram matrix; Total variation loss , For the pixel coordinates of the picture, Is the total number of picture pixels.
  9. 9. The method for digital wall painting restoration based on structure guidance and state space model according to claim 8, wherein the loss function is optimized by using a back propagation algorithm in the training process of the main body restoration model and the detail restoration model Loss function subentry of body repair model Loss function subentry of detail repair model , wherein, , , , , , , , , , , Representing the output of the discriminant of the subject repair model, Representing wall painting A subject repair map is shown in which, Representing the final repair map.
  10. 10. The method for digital wall painting restoration based on the structure-guided and state-space model according to claim 1, wherein all trainable parameters of the edge restoration model, the body restoration model and the detail restoration model are all initialized to be random values subject to normal distribution with a mean value of 0 and a standard deviation of 0.01, and all trainable parameters are automatically updated and adjusted correspondingly according to a back propagation algorithm.

Description

Wall painting digital restoration method based on structure guidance and state space model Technical Field The invention belongs to the technical field of computer image processing and digital restoration, and particularly relates to a mural digital restoration method based on structural guidance and a state space model. Background The wall painting is taken as a valuable cultural heritage of human beings and carries rich historical, artistic and scientific information. However, a large number of precious wall paintings have suffered from varying degrees of damage, such as cracking, area loss and blurring of content, due to changes in the storage environment, natural weathering attacks (such as cracking, fading, mildew) and artificial damage. How to scientifically and accurately repair the wall paintings to enable the wall paintings to reproduce original appearances is a problem to be solved urgently in the field of cultural relic protection. Traditional wall painting repair mainly relies on manual physical filling and expert copying redrawing, and the methods are long in time consumption, high in cost, extremely dependent on personal experience and art achievement of a repairman and high in subjectivity. In addition, physical repair is often irreversible, and once an error occurs, secondary damage is caused to the mural body. With the development of computer vision and digital technology, digital image restoration technology was introduced into virtual restoration of wall paintings. Early methods, such as patch-based techniques, were primarily used to fill very small areas of missing, and when encountering somewhat larger areas or areas of complex structure, had a tendency to produce blurred, distorted texture replication. In recent years, deep learning techniques typified by convolutional neural networks (Convolutional Neural Networks, CNN) have made remarkable progress in the field of image restoration. The CNN-based model can automatically extract features and generate visually reasonable textures and contents by learning a large amount of data. However, when the standard CNN model is directly applied to wall painting restoration, firstly, the perceived capability of structural information is insufficient, and the CNN model is easy to extract smooth texture features, so that the image edge structural features are difficult to capture. Wall paintings typically contain a large number of well-defined character outlines, building lines, and decorative patterns, which are the core semantic information of the wall painting. Many existing repair methods lack direct utilization of such structures, resulting in lines that tend to be broken, misplaced or blurred when damaged areas span critical structures (e.g., clothing line patterns or architectural contours), severely deviating from the original artistic style of the wall painting. The modeling of global consistency is limited, namely the CNN model relies on stacked convolution kernels to extract features, and the inherent characteristics of the convolution kernels lead to relatively limited feelings, so that the global dependence and long-range structure of an image are difficult to capture. In addition, although the transducer-based model can model the global situation excellently, the Self-Attention (Self-Attention) mechanism of the core has square complexityWhereinIs the length of the input sequence, i.e. the total number of pixels in the image. When the fresco image is processed, a large number of pixels make the calculation and memory expenditure very high, so that the model training and reasoning cost is extremely high, and the application of the fresco image in the repairing task is greatly limited. In contrast, state space Models (STATE SPACE Models, SSM), particularly novel structured SSM represented by Mamba, can reduce computational complexity to nearly linear while modeling very long sequence dependencies by introducing selective mechanisms and efficient hardware-aware design。 Disclosure of Invention The invention aims to provide a mural digital restoration method based on a structure guide and a state space model, which can not only effectively utilize priori structural information of the mural to assist in the mural restoration, but also can efficiently model and realize global context dependence and high-fidelity restoration aiming at the problems of poor mural line restoration condition, unrealistic mural detail restoration and the like in the current mural digital restoration field. The technical scheme adopted by the invention is that the mural digital restoration method based on the structure guiding and state space model comprises the following steps: shooting a wall painting to obtain a wall painting picture, processing the wall painting picture to generate a wall painting edge picture, generating a wall painting gray-scale picture based on the wall painting picture, and randomly generating a mask picture on one wall painting picture; obt