CN-121998871-A - Raw domain mole pattern removing method based on Mamba

CN121998871ACN 121998871 ACN121998871 ACN 121998871ACN-121998871-A

Abstract

The invention discloses a Raw domain moire removing method based on Mamba, and relates to the technical field of image signal processing. A Raw domain mole pattern removing method based on Mamba comprises the following steps of S1, constructing a Raw domain mole pattern removing model based on Mamba, adopting a U-Net type multi-scale encoder-decoder framework, S2, designing a time-space-based selective scanning module, adaptively focusing on a space-time area with remarkable mole patterns, S3, training the model by using a deep learning Pytorch framework, repeatedly traversing VDRawmoire data sets until the model converges, S4, inputting multi-frame Raw domain mole pattern images into the model, and outputting a single-frame RGB image reconstruction result after mole pattern removing, and improving the Raw domain mole pattern removing performance to a new height by using the time-space-based selective scanning.

Inventors

YUE HUANJING
LIU JICHENG
YANG JINGYU

Assignees

天津大学

Dates

Publication Date: 20260508
Application Date: 20260210

Claims (6)

1. The Raw domain mole pattern removing method based on Mamba is characterized by comprising the following steps of: S1, constructing a Mamba-based mole pattern removing model, namely adopting a U-Net-type multi-scale encoder-decoder architecture, extracting multi-scale Raw domain mole pattern characteristics by an encoder, and finally outputting a high-quality mole pattern removing image by fusing multi-level context information through jump connection by a decoder, wherein the encoder and the decoder are constructed based on Mamba modules; S2, designing a space-time scanning module, namely under Mamba framework, introducing a selective scanning mechanism combining a time dimension and a space dimension to enable the constructed mole pattern removing model to be adaptively focused on a space-time region with obvious mole patterns so as to improve modeling and inhibiting capacity of the mole patterns; S3, training a model, namely training the constructed mole pattern removing model by using a deep learning Pytorch framework, and repeatedly traversing VDRawmoire data sets until the model converges; S4, outputting a result, namely inputting the Raw domain moire image in the VDRawmoire dataset into a trained model to obtain an RGB image with moire removed.
2. The Mamba-based Raw domain mole striae removal method as set forth in claim 1, wherein the S1 specifically includes the following: S1.1, given an input moire image I M , converting the moire image I M into embedded dimension features through a shallow feature extraction convolution layer, and simultaneously recording space size information of the input image for subsequent recovery, wherein the function is expressed as follows: ; S1.2, sending the obtained features into an image block embedding module, flattening the two-dimensional feature map into a one-dimensional sequence, and then applying random discarding operation to the sequence so as to enhance the generalization capability of the model; S1.3, sequentially passing the characteristics through a plurality of residual state space groups, wherein each residual state space group comprises a space selective scanning module and a time selective scanning module with specified depths, and the functions are expressed as follows: ; S1.4, after state space processing, fusing original input by residual connection of features, carrying out inter-channel information interaction and importance recalibration by channel attention, and finally converting a 1D sequence back to a 2D feature map by a normalization and patch anti-embedding layer, and obtaining a recovery result by residual connection, wherein the function is expressed as follows: Wherein, the The method comprises the steps of obtaining residual weights, wherein DP represents a random discarding module, SS2D_T represents a time selective scanning module, SS2D_S represents a space selective scanning module; S1.5, inputting the characteristics processed by the space-time scanning module into a channel attention module to enhance the response of the important channel, and adding the response with the original input through residual connection to obtain the final recovery result of the current level.
3. The Mamba-based Raw domain mole-pattern removal method according to claim 1, wherein the S2 specifically includes the following: S2.1, designing a scanning strategy of a space dimension, and carrying out serialization scanning on the feature map by using different serialization strategies to enable a network to learn the texture and structure dependency relationship of the feature map in space; s2.2, designing a scanning strategy of a time dimension, scanning a channel time sequence of each pixel of the feature map, and capturing correlation features between the channel and the time sequence in a sequence modeling mode.
4. A Mamba-based Raw domain mole-removal method as defined in claim 3 wherein the spatial dimension scanning strategy comprises the following: s2.1.1, the input features are expanded to the internal dimension through the linear projection layer and are divided into input features X and gating signals Z, and the functions are expressed as: firstly expanding the dimension through a linear layer Linder e , and then dividing a gating signal Z and an input characteristic X by using Chunk; S2.1.2 input feature X is locally feature enhanced by depth separable convolution and SiLU activation function is applied, the specific function being expressed as: ; s2.1.3 four-way permutation expansion of features using specific expansion method, including original sequence x, transpose x T , flip sequence And transposed post-flip sequences ; S2.1.4, using four linear projection layers to correspond to four arrangement directions, respectively generating parameters of continuous state equation, namely time step The system comprises a state transition matrix A, an input projection matrix B, an output projection matrix C and a jump connection weight D; S2.1.5, state transition matrix a is initialized by generating a base sequence of 1 to d state , broadcasting to each channel d inner times, taking natural logarithms and converting the natural logarithms into learnable parameters, and expressing functions as follows: Wherein, the Index for scanning direction; Index for feature dimension; a log is a trainable parameter of a logarithmic space, which is optimized autonomously in the training process; the input projection matrix B and the output projection matrix C map the input X L to the input X L through linear projection Dimension splitting is carried out through torch. Split, and split is divided into time steps The input projection matrix B, the output projection matrix C, the function is expressed as: Wherein, the Representation of First, the Position of each direction Is characterized by (2); Is the corresponding sub-matrix of the linear projection layer; Is the corresponding sub-matrix of the linear projection layer; Is the corresponding sub-matrix of the linear projection layer; Is a bias vector; Is a linear projection layer matrix after the dimension is compressed; The activation guarantee time step is positive; s2.1.6 applying a selective scanning operation in combination with a time step The discrete state space transformation is completed by the input projection matrix B, the output projection matrix C and the jump connection weight D, and the specific function is expressed as follows: ; s2.1.7 to combine the output results in four directions After layer normalization, the characteristic selection is realized by multiplying the gating signal Z element by element, and finally, the original dimension is mapped back through the output projection layer, and the specific function is expressed as follows: 。
5. the Mamba-based Raw domain mole-pattern removal method as set forth in claim 4, wherein the scanning strategy of the time dimension specifically includes the following: the input feature X is flattened along the time dimension and the channel dimension, and is reversed in the flattened dimension to form two scanning directions, and the rest processing steps are the same as the scanning strategy of the space dimension.
6. The Mamba-based Raw domain mole-pattern removal method according to claim 1, wherein the S3 specifically includes the following: s3.1, reconstructing loss adopts L1 loss, and the function is expressed as: Wherein I G represents Ground Truth, and I DM represents the Moire removal result; S3.2, further introducing a perception loss And color loss To improve visual quality and color fidelity, mixing loss Expressed as: Wherein, the And Representing the weight parameters.

Description

Raw domain mole pattern removing method based on Mamba Technical Field The invention relates to the technical field of image signal processing, in particular to a Raw domain mole pattern removing method based on Mamba. Background Moire is a common artifact in digital imaging systems, particularly when capturing objects with fine textures or periodic patterns. This phenomenon is typically caused by the interaction of the sensor pixel array with the detailed structure of the subject, resulting in irregular waviness or streaking of the image, severely affecting visual quality and subsequent processing. Early de-moire networks focused mainly on the sRGB domain, and researchers proposed various de-moire methods based on the multi-scale nature and frequency domain recognizability of sRGB domain moire. Moire is generated by frequency aliasing between the camera Color Filter Array (CFA) and the display screen grid, while sRGB domain moire is additionally affected by Image Signal Processing (ISP), where demosaicing interpolates pixels, resulting in more complex moire characteristics. Unlike the sRGB domain, the Raw domain moire is only generated by frequency aliasing between the camera photosensitive grid array and the display diode grid array, without ISP nonlinear transformation, there is no moire cross contamination of other channels, which makes the Raw domain moire characteristics simpler and purer. VDRawmoire proposes to remove moire using the difference in the degree of moire contamination of the Raw domain color channel. The potential of Mamba in low-level visual tasks has not been fully explored, but its architectural features naturally meet the core needs of such tasks. Mamba can effectively utilize the consistency and the difference of original photosensitive information among channels, show stronger long-range modeling capability than CNN, and avoid high computational overhead of a transducer, so that the method has great advantages and wide application prospects in the low-level vision field. In order to solve the problems, the method provided by the invention provides a Raw domain mole pattern removing method based on Mamba, so that an optimal mole pattern removing effect is realized. Disclosure of Invention The invention aims to provide a Raw domain mole pattern removing method based on Mamba to solve the problems in the background technology and realize a mole pattern removing effect with higher quality. In order to achieve the above purpose, the present invention adopts the following technical scheme: The Raw domain mole pattern removing method based on Mamba specifically comprises the following steps: S1, constructing a Mamba-based mole pattern removing model, namely adopting a U-Net-type multi-scale encoder-decoder architecture, extracting multi-scale Raw domain mole pattern characteristics by an encoder, and finally outputting a high-quality mole pattern removing image by fusing multi-level context information through jump connection by a decoder, wherein the encoder and the decoder are constructed based on Mamba modules; S2, designing a space-time scanning module, namely under Mamba framework, introducing a selective scanning mechanism combining a time dimension and a space dimension to enable the constructed mole pattern removing model to be adaptively focused on a space-time region with obvious mole patterns so as to improve modeling and inhibiting capacity of the mole patterns; S3, training a model, namely training the constructed mole pattern removing model by using a deep learning Pytorch framework, and repeatedly traversing VDRawmoire data sets until the model converges; S4, outputting a result, namely inputting the Raw domain moire image in the VDRawmoire dataset into a trained model to obtain an RGB image with moire removed. Preferably, the S1 specifically includes the following: S1.1 Moire image I given input MThe convolution layer is extracted through the shallow layer features and is converted into embedded dimension features, meanwhile, the space size information of the input image is recorded for subsequent recovery, and the function is expressed as follows: ; S1.2, sending the obtained features into an image block embedding module, flattening the two-dimensional feature map into a one-dimensional sequence, and then applying random discarding operation to the sequence so as to enhance the generalization capability of the model; S1.3, sequentially passing the characteristics through a plurality of residual state space groups, wherein each residual state space group comprises a space selective scanning module and a time selective scanning module with specified depths, and the functions are expressed as follows: ; S1.4, after state space processing, fusing original input by residual connection of features, carrying out inter-channel information interaction and importance recalibration by channel attention, and finally converting a 1D sequence back to a 2D feature map by a normalization