CN-122023126-A - Image processing method and device and electronic equipment

Abstract

The application discloses an image processing method, an image processing device and electronic equipment, wherein the method comprises the steps of acquiring a first RGB image and a first depth image; the first RGB image and the first depth image correspond to the same target picture, auxiliary data are generated based on the first depth image, the auxiliary data are used for marking target areas and non-target areas ‌ in the first RGB image, fusion data are determined based on the auxiliary data, the first RGB image and the first depth image, the fusion data are processed based on a target model to obtain a second RGB image, and the second resolution of the second RGB image is higher than the first resolution of the first RGB image.

Inventors

• WU GE

Assignees

• 鼎道智芯(上海)半导体有限公司

Dates

Publication Date

20260512

Application Date

20260202

Claims (10)

1. 1. An image processing method, comprising: Acquiring a first RGB image and a first depth image, wherein the first RGB image and the first depth image correspond to the same target picture; Generating auxiliary data based on the first depth image, wherein the auxiliary data is used for marking a target area and a non-target area ‌ in the first RGB image; Determining fusion data based on the auxiliary data, the first RGB image and the first depth image; And processing the fusion data based on the target model to obtain a second RGB image, wherein the second resolution of the second RGB image is higher than the first resolution of the first RGB image.
2. 2. The image processing method of claim 1, the generating auxiliary data based on the first depth image, comprising: Generating first auxiliary data based on the first depth image, wherein the first auxiliary data is used for marking a near view area and a far view area in the first RGB image; the determining fusion data based on the auxiliary data, the first RGB image, and the first depth image includes: Determining a first RGB intermediate image based on the first auxiliary data and the first RGB image, wherein the first RGB intermediate image only contains a close-range region in the first RGB image; And fusing the first RGB intermediate image, the first RGB image and the first depth image to obtain fusion data.
3. 3. The image processing method according to claim 1 or 2, the generating auxiliary data based on the first depth image, comprising: generating second auxiliary data based on the first depth image, wherein the second auxiliary data is used for marking the outline boundary and the non-outline boundary of the target object in the first RGB image; the determining fusion data based on the auxiliary data, the first RGB image, and the first depth image includes: determining a second RGB intermediate image based on the second auxiliary data and the first RGB image, wherein pixels of the contour boundary in the first RGB image correspond to first RGB values, pixels of the contour boundary in the second RGB intermediate image correspond to second RGB values, and the second RGB values are higher than the first RGB values; and fusing at least the second RGB intermediate image and the first depth image to obtain fusion data.
4. 4. The image processing method of claim 2, the generating first auxiliary data based on the first depth image, comprising: Determining a depth global average value of the first depth image based on depth estimation values of all pixels in the first depth image; And generating a binary matrix consistent with the first depth image in size based on the depth global average value, and determining the binary matrix as first auxiliary data, wherein the binary matrix is used for marking a first pixel with a depth estimated value larger than the depth global average value and a second pixel with a depth estimated value not larger than the depth global average value in the first depth image.
5. 5. The image processing method according to claim 3, the generating second auxiliary data based on the first depth image, comprising: And carrying out nonlinear transformation on the depth estimation value of each pixel in the first depth image to obtain a weight matrix consistent with the size of the first depth image, determining the weight matrix as second auxiliary data, and enhancing the RGB value of each pixel in the first RGB image by the weight matrix.
6. 6. The image processing method according to claim 1, wherein the processing the fusion data based on the object model to obtain a second RGB image includes: extracting features of the fusion data to obtain a first feature map, wherein the size of the first feature map is matched with the first resolution; up-sampling the first feature map to obtain a second feature map, wherein the size of the second feature map is matched with the second resolution; and processing the second feature map to obtain a second RGB image.
7. 7. The image processing method according to claim 6, wherein the upsampling the first feature map to obtain a second feature map includes: constructing a feature map to be filled based on the second resolution, wherein features in the feature map to be filled are zero values, and the size of the feature map to be filled is matched with the second resolution; And filling each feature in the first feature map to a corresponding position in the feature map to be filled based on the size relation between the first resolution and the second resolution, and keeping the unfilled position in the feature map to be filled as a zero value to obtain a second feature map.
8. 8. The image processing method according to claim 1, the object model being trained by: Acquiring a sample RGB image pair and a sample depth image pair, wherein the sample RGB image pair comprises a first sample RGB image and a second sample RGB image, the first sample RGB image has the first resolution, the second sample RGB image has the second resolution, the sample depth image pair comprises a first sample depth image and a second sample depth image, the first sample depth image corresponds to the first sample RGB image, and the second sample depth image corresponds to the second sample RGB image; Generating sample auxiliary data based on the first sample depth image, wherein the sample auxiliary data is used for marking a target area and a non-target area in the first sample RGB image; Determining sample fusion data based on the sample assistance data, the first sample RGB image, and the first sample depth image; Processing the sample fusion data based on a target model to obtain an RGB predicted image and a depth predicted image, wherein the RGB predicted image has the second resolution; And adjusting parameters of the target model based on a loss value, wherein the loss value represents the difference between the RGB predicted image and a second sample RGB image corresponding to the first sample RGB image and the difference between the depth predicted image and a second sample depth image corresponding to the first sample depth image.
9. 9. An image processing apparatus comprising: The device comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a first RGB image and a first depth image, and the first RGB image and the first depth image correspond to the same target picture; The generation module is used for generating auxiliary data based on the first depth image, wherein the auxiliary data is used for marking a target area and a non-target area in the first RGB image; A determining module, configured to determine fusion data based on the auxiliary data, the first RGB image, and the first depth image; and the processing module is used for processing the fusion data based on the target model to obtain a second RGB image, and the second resolution of the second RGB image is higher than the first resolution of the first RGB image.
10. 10. An electronic device, comprising: The memory is used for storing a computer program; the processor is configured to execute the computer program to implement the steps of: Acquiring a first RGB image and a first depth image, wherein the first RGB image and the first depth image correspond to the same target picture; generating auxiliary data based on the first depth image, wherein the auxiliary data is used for marking a target area and a non-target area in the first RGB image; Determining fusion data based on the auxiliary data, the first RGB image and the first depth image; Processing the fusion data based on a target model to obtain a second RGB image, wherein the second resolution of the second RGB image is higher than the first resolution of the first RGB image; and the display device is used for receiving and displaying the second RGB image output by the processor.

Description

Image processing method and device and electronic equipment Technical Field The present application relates to the field of computer technologies, and in particular, to an image processing method, an image processing device, and an electronic device. Background In the field of digital image application, under the constraint of limited computational power and power consumption of electronic equipment, the frame rate is often ensured by reducing the resolution and hardware load, and finally, the image definition is insufficient. Disclosure of Invention The technical scheme provided by the application is as follows: the first aspect of the present application provides an image processing method, including: Acquiring a first RGB image and a first depth image, wherein the first RGB image and the first depth image correspond to the same target picture; Generating auxiliary data based on the first depth image, wherein the auxiliary data is used for marking a target area and a non-target area ‌ in the first RGB image; Determining fusion data based on the auxiliary data, the first RGB image and the first depth image; And processing the fusion data based on the target model to obtain a second RGB image, wherein the second resolution of the second RGB image is higher than the first resolution of the first RGB image. In one possible implementation, the generating auxiliary data based on the first depth image includes: Generating first auxiliary data based on the first depth image, wherein the first auxiliary data is used for marking a near view area and a far view area in the first RGB image; the determining fusion data based on the auxiliary data, the first RGB image, and the first depth image includes: Determining a first RGB intermediate image based on the first auxiliary data and the first RGB image, wherein the first RGB intermediate image only contains a close-range region in the first RGB image; And fusing the first RGB intermediate image, the first RGB image and the first depth image to obtain fusion data. In one possible implementation, the generating auxiliary data based on the first depth image includes: generating second auxiliary data based on the first depth image, wherein the second auxiliary data is used for marking the outline boundary and the non-outline boundary of the target object in the first RGB image; the determining fusion data based on the auxiliary data, the first RGB image, and the first depth image includes: determining a second RGB intermediate image based on the second auxiliary data and the first RGB image, wherein pixels of the contour boundary in the first RGB image correspond to first RGB values, pixels of the contour boundary in the second RGB intermediate image correspond to second RGB values, and the second RGB values are higher than the first RGB values; and fusing at least the second RGB intermediate image and the first depth image to obtain fusion data. In one possible implementation, the generating first auxiliary data based on the first depth image includes: Determining a depth global average value of the first depth image based on depth estimation values of all pixels in the first depth image; And generating a binary matrix consistent with the first depth image in size based on the depth global average value, and determining the binary matrix as first auxiliary data, wherein the binary matrix is used for marking a first pixel with a depth estimated value larger than the depth global average value and a second pixel with a depth estimated value not larger than the depth global average value in the first depth image. In one possible implementation, the generating second auxiliary data based on the first depth image includes: And carrying out nonlinear transformation on the depth estimation value of each pixel in the first depth image to obtain a weight matrix consistent with the size of the first depth image, determining the weight matrix as second auxiliary data, and enhancing the RGB value of each pixel in the first RGB image by the weight matrix. In one possible implementation, the processing the fusion data based on the object model to obtain a second RGB image includes: extracting features of the fusion data to obtain a first feature map, wherein the size of the first feature map is matched with the first resolution; up-sampling the first feature map to obtain a second feature map, wherein the size of the second feature map is matched with the second resolution; and processing the second feature map to obtain a second RGB image. In one possible implementation, the upsampling the first feature map to obtain a second feature map includes: constructing a feature map to be filled based on the second resolution, wherein features in the feature map to be filled are zero values, and the size of the feature map to be filled is matched with the second resolution; And filling each feature in the first feature map to a corresponding position in the feature map to be filled based on t