CN-115861430-B - Detection method, device, equipment and system of free view angle system

Abstract

The application discloses a detection method, a device, equipment and a system of a free view angle system, relates to the field of free view angles, and solves the problem of how to accurately detect the offset of shooting equipment in the free view angle system in real time. The free view angle system comprises N shooting devices, and the N shooting devices synchronously shoot the first scene. After the first shooting equipment is shifted, the first shooting equipment can preliminarily determine the shift according to the IMU information, determine the shift amount according to the acquired image after the first shooting equipment is shifted and the images of other shooting equipment which is not shifted in the system, and determine the degree of shifting of the first shooting equipment according to the shift amount. Therefore, in the process of synchronously shooting the first scene by the N shooting devices in the free view angle system, the shooting devices with offset can be accurately detected in real time.

Inventors

• CAO SHIMING
• LI MING

Assignees

• 华为技术有限公司

Dates

Publication Date

20260508

Application Date

20210923

Claims (20)

1. 1. A method of detecting a freeview system, the freeview system including N photographing devices that synchronously photograph a first scene, the N photographing devices including a first photographing device including a camera and an inertial measurement unit IMU, the method being performed by the first photographing device, the method comprising: When IMU information of the first shooting equipment changes, acquiring a first image at a first moment through the camera; Acquiring M images acquired by M shooting devices at the first moment, wherein the M shooting devices are devices except the first shooting device in the free view angle system, IMU information of the M shooting devices is unchanged, N is an integer greater than or equal to 3, M is an integer greater than or equal to 1, and M < N; And determining an offset according to the first image and the M images, wherein the offset is used for indicating the degree of offset of the changing external parameters relative to the initial camera external parameters when the first shooting equipment acquires the first image at the first moment.
2. 2. The method of claim 1, wherein the M photographing devices comprise devices adjacent to the first photographing device.
3. 3. The method of claim 1, wherein the M photographing devices comprise photographing devices within a preset range of positions within the freeview system.
4. 4. A method according to any of claims 1-3, wherein said determining an offset from said first image and said M images comprises: determining a re-projection error according to the first image and the M images, wherein the re-projection error is used for representing coordinate errors of feature points in detection areas in the first image and the M images; and determining the offset according to the reprojection error.
5. 5. A method according to any one of claims 1-3, wherein after said determining an offset from said first image and said M images, the method further comprises: and sending prompt information to the computing device, wherein the prompt information is used for indicating that the gesture of the first shooting device is deviated.
6. 6. The method of claim 1, wherein after determining an offset from the first image and the M images, the method further comprises: And if the offset is smaller than a preset offset, determining a re-standard external parameter of the first shooting device according to the offset and the initial camera external parameter, wherein the re-standard external parameter is used for carrying out post-processing on an image acquired by the first shooting device for generating the free view video of the first scene.
7. 7. The method of claim 1, wherein after determining an offset from the first image and the M images, the method further comprises: If the offset is smaller than a preset offset, determining projection points in M images acquired by the M shooting devices at a second moment according to the background point cloud of the first scene; and determining a re-standard external parameter of the first shooting device according to the characteristic point of the second image at the second moment acquired by the first shooting device and the projection point, wherein the re-standard external parameter is used for carrying out post-processing on the image acquired by the first shooting device for generating the free view video of the first scene.
8. 8. A method according to any one of claims 1-3, wherein after said determining an offset from said first image and said M images, the method further comprises: And if the offset is greater than or equal to a preset offset, adjusting the gesture of the first shooting equipment by a holder where the first shooting equipment is positioned.
9. 9. The method of claim 8, wherein after the adjusting the pose of the first photographing device by the pan-tilt at which the first photographing device is located, the method further comprises: Determining projection points in M images acquired by the M shooting devices at a second moment according to the background point cloud of the first scene; and determining a re-standard external parameter of the first shooting device according to the characteristic point of the second image at the second moment acquired by the first shooting device and the projection point, wherein the re-standard external parameter is used for carrying out post-processing on the image acquired by the first shooting device for generating the free view video of the first scene.
10. 10. The method according to any one of claims 6, 7 or 9, further comprising: And sending the re-standard external parameters to the computing equipment.
11. 11. A method of generating a freeview video, wherein the freeview system includes N capture devices that synchronously capture a first scene, the N capture devices including the first capture device, the method performed by a computing device, the method comprising: Post-processing the images acquired by the N shooting devices according to the camera parameters of the N shooting devices to obtain a first free view video, wherein the camera parameters comprise initial camera internal parameters and initial camera external parameters; If the computing equipment acquires the re-standard external parameters of the first shooting equipment, performing post-processing on the image acquired by the first shooting equipment according to the re-standard external parameters of the first shooting equipment and the initial camera internal parameters to acquire a second free view video, wherein the re-standard external parameters are updated external parameters of the initial camera external parameters; and merging the first free view video and the second free view video to obtain the free view video of the first scene.
12. 12. The method of claim 11, wherein prior to post-processing the image acquired by the first capture device based on the re-standard external parameters and the initial camera internal parameters of the first capture device, the method further comprises: Receiving prompt information sent by the first shooting equipment, wherein the prompt information is used for indicating that the gesture of the first shooting equipment is deviated; And acquiring a re-standard external parameter of the first shooting equipment, wherein the re-standard external parameter is used for carrying out post-processing on an image acquired by the first shooting equipment for generating the free view video of the first scene.
13. 13. The method of claim 12, wherein obtaining the re-standard external parameters of the first photographing device comprises: receiving the re-standard external parameters sent by the first shooting equipment; Or determining projection points in M images acquired by M shooting devices according to the background point cloud of the first scene, wherein the M shooting devices are devices except the first shooting device in the free view angle system, IMU information of the M shooting devices is unchanged, N is an integer greater than or equal to 3, M is an integer greater than or equal to 1, and M < N; And determining the re-standard external parameters of the first shooting equipment according to the characteristic points of the second image acquired by the first shooting equipment and the projection points.
14. 14. The method according to claim 12 or 13, wherein prior to said obtaining the re-labeled external parameters of the first photographing device, the method further comprises: and suspending post-processing of the image sent by the first shooting equipment.
15. 15. A detection apparatus, characterized in that the detection apparatus is applied to shooting equipment in a freeview system, the freeview system includes N shooting equipment, N shooting equipment carries out synchronous shooting to a first scene, N shooting equipment contains first shooting equipment, first shooting equipment contains camera and inertial measurement unit IMU, the detection apparatus includes: when the IMU information of the first shooting equipment is changed, the camera is used for acquiring a first image at a first moment; The image acquisition module is used for acquiring M images acquired by M shooting devices at the first moment, wherein the M shooting devices are devices except the first shooting device in the free view angle system, IMU information of the M shooting devices is unchanged, N is an integer greater than or equal to 3, M is an integer greater than or equal to 1, and M < N; The detection module is used for determining offset according to the first image and the M images, and the offset is used for indicating the degree of offset of the changing external parameters relative to the initial camera external parameters when the first shooting equipment acquires the first image at the first moment.
16. 16. The apparatus of claim 15, wherein the M photographing devices comprise devices adjacent to the first photographing device.
17. 17. The apparatus of claim 15, wherein the M photographing devices comprise photographing devices within a preset range of positions within the freeview system.
18. 18. The apparatus according to any one of claims 15-17, wherein the detection module is configured to, when determining the offset from the first image and the M images: determining a re-projection error according to the first image and the M images, wherein the re-projection error is used for representing coordinate errors of feature points in detection areas in the first image and the M images; and determining the offset according to the reprojection error.
19. 19. The apparatus according to any one of claims 15-17, further comprising a communication module; the communication module is used for sending prompt information to the computing equipment, and the prompt information is used for indicating that the gesture of the first shooting equipment is deviated.
20. 20. The apparatus of claim 15, further comprising a recalibration module; And if the offset is smaller than a preset offset, the recalibration module is used for determining a recalibration external parameter of the first shooting device according to the offset and the initial camera external parameter, wherein the recalibration external parameter is used for carrying out post-processing on an image acquired by the first shooting device for generating the free view video of the first scene.

Description

Detection method, device, equipment and system of free view angle system Technical Field The present application relates to the field of image processing, and in particular, to a method, an apparatus, a device, and a system for detecting a free view angle system. Background The free view angle technology realizes a real-time interactive video mode, so that a user can independently select a 360-degree viewing angle of the free view angle video on the terminal. The free view angle technology is mainly applied to scenes such as program special effect shooting and live broadcasting. The freeview technique may also be referred to as a freeview technique. And triggering a shutter by a plurality of shooting devices in the free view angle system according to the control signal, and synchronously shooting the same scene to obtain multi-frame images with a plurality of angles. The computing device processes multi-frame images of multiple angles according to parameters of the multiple shooting devices, and free view angle videos are generated. In the shooting process, if shooting equipment in the free view angle system collides or shakes, the position, the angle and the like of the shooting equipment are caused to deviate, parameters of the shooting equipment after deviation are utilized to process free view angle video generated by an image, and shake also occurs. In general, the offset detection of a photographing apparatus in a freeview system is determined by a worker from the subjective viewing experience of a finally generated freeview video. The time span in which the detection and photographing apparatus offset occurs is large, and subjective judgment is easily ignored for a minute offset. Therefore, how to accurately detect the offset of the photographing apparatus in the free view angle system in real time is a problem to be solved. Disclosure of Invention The detection method, the device, the equipment and the system for the free view angle system solve the problem of how to accurately detect the offset of shooting equipment in the free view angle system in real time. In a first aspect, a method for detecting a freeview system is provided, where the freeview system includes N photographing devices, where the N photographing devices synchronously photograph a first scene, the N photographing devices include a first photographing device, and the first photographing device includes a camera and an inertial measurement unit (Inertial Measurement Unit, IMU). The method is performed by a first photographing apparatus, which is any one photographing apparatus in the freeview system. When IMU information of a first shooting device changes, acquiring a first image at a first moment through a camera, acquiring M images acquired by M shooting devices at the first moment from M shooting devices, and determining an offset according to the first image and the M images, wherein the offset is used for indicating the degree of offset of a changing external parameter of the first shooting device when the first shooting device acquires the first image at the first moment relative to an initial camera external parameter. The image capturing device comprises M image capturing devices, wherein the M image capturing devices are devices except for a first image capturing device in a free view angle system, IMU information of the M image capturing devices is unchanged, N is an integer greater than or equal to 3, M is an integer greater than or equal to 1, and M < N. Therefore, after the first shooting equipment is shifted, the first shooting equipment can preliminarily determine the shift according to the IMU information, determine the shift amount according to the acquired image after the first shooting equipment is shifted and the images of other shooting equipment which is not shifted in the system, and determine the degree of the shift of the first shooting equipment according to the shift amount. Therefore, in the process of synchronously shooting the first scene by the N shooting devices in the free view angle system, the shooting devices with offset can be accurately detected in real time. If the difference between the M images and the first image is large, the accuracy of the offset determined from the first image and the M images is low. In order to improve the accuracy of the first photographing apparatus to determine the offset amount, the first photographing apparatus may acquire an image from a photographing apparatus which is not offset among the designated photographing apparatuses. For example, the M photographing apparatuses include apparatuses adjacent to the first photographing apparatus. For another example, the M photographing apparatuses include photographing apparatuses within a preset range of positions within the freeview system. In one possible implementation, determining the offset from the first image and the M images includes the first photographing device determining a re-projection error from the firs