CN-122018164-A - Augmented reality near-to-eye display device and method

CN122018164ACN 122018164 ACN122018164 ACN 122018164ACN-122018164-A

Abstract

The invention provides an augmented reality near-to-eye display device and method, wherein the device comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a target scene image; the system comprises a processing module, a fusion module and a display module, wherein the processing module is used for determining target scene information according to a target scene image and generating an auxiliary information hologram corresponding to the target scene according to the target scene information, and the fusion module is used for fusing the auxiliary information hologram to the target scene to perform augmented reality near-eye display. The invention can realize the accurate virtual-real fusion of the auxiliary information and the real scenes with different depths, reduce the volume and weight of the device, improve the wearing comfort, and is suitable for industrial applications such as intelligent manufacturing, warehouse management and the like.

Inventors

CAO LIANGCAI
HE ZEHAO
GAO YUNHUI
LUO YINGJUN

Assignees

清华大学
首都师范大学

Dates

Publication Date: 20260512
Application Date: 20260205

Claims (10)

1. An augmented reality near-eye display device, comprising: The acquisition module is used for acquiring the target scene image; The processing module is used for determining target scene information according to the target scene image and generating an auxiliary information hologram corresponding to the target scene according to the target scene information; and the fusion module is used for fusing the auxiliary information hologram to a target scene and carrying out augmented reality near-eye display.
2. The augmented reality near-eye display device of claim 1, wherein the processing module comprises: the device comprises a scene information determining module, a target scene information processing module and a scene information processing module, wherein the scene information determining module is used for dividing a target scene from the target scene image based on a monocular depth estimation algorithm and determining target scene information, and the target scene information comprises the intensity distribution of the target scene and the three-dimensional coordinates of the target scene relative to eyes of a user; and the hologram generation module is used for generating auxiliary information describing the target scenery according to the intensity distribution of the target scenery and the three-dimensional coordinates of the target scenery relative to eyes of the user based on a hologram generation algorithm, and generating an auxiliary information hologram according to the auxiliary information and the three-dimensional coordinates.
3. The augmented reality near-eye display device of claim 2, wherein the scene information determination module is configured to classify the target scene image according to a blended feature, the blended feature comprising a color, an edge, a contour, a vanishing line, and a vanishing point; and carrying out depth estimation on the distant view image by adopting a horizontal edge gradient method, carrying out depth estimation on the intermediate view image by adopting a vanishing line gradient method, and carrying out depth estimation on the near view image by adopting a weighted depth superposition method to obtain the target scene information.
4. The augmented reality near-eye display device of claim 2, wherein the scene information determination module is configured to input the target scene image into a pre-trained depth gradient-assisted monocular depth estimation network model to obtain the target scene information output by the monocular depth estimation network model; The monocular depth estimation network model is obtained through training of depth estimation training samples and depth gradient auxiliary samples.
5. The augmented reality near-eye display device of any one of claims 2 to 4, wherein the hologram generation module is to: image feature recognition is carried out based on the intensity distribution, and the type features of the target scenery are determined; generating text or graphic auxiliary information containing a type identifier according to the type feature, and generating numerical auxiliary information containing a distance parameter according to a depth component in the three-dimensional coordinate; Inputting the auxiliary information and the three-dimensional coordinates into a pre-trained model without convolution errors to drive a self-coding deep learning network, and obtaining the auxiliary information hologram output by the self-coding deep learning network; The self-coding deep learning network compensates coding phase convolution errors in a decoding stage in a phase expansion mode.
6. The augmented reality near-eye display device of claim 5, wherein the fusion module comprises: A light source module for generating collimated polarized illumination light; The optical modulation module is arranged at the output end of the light source module, the input end of the optical modulation module is electrically connected with the output end of the hologram generating module and is used for loading the auxiliary information hologram, diffracting the auxiliary information hologram after the polarized illumination light irradiates the auxiliary information hologram, and projecting the auxiliary information hologram to a preset position adjacent to a target scene; the waveguide display module is arranged at the diffraction light wave output end of the optical modulation module and used for conducting the diffraction light waves to the eye area of the user so as to perform virtual-real fusion on the auxiliary information hologram and the target scene.
7. The augmented reality near-eye display device of claim 6, wherein the light source module comprises: The laser light source is used for emitting divergent spherical waves; The collimating lens is arranged at the output end of the laser light source and is used for converging the divergent spherical waves and outputting collimated plane waves; the polarizing plate is arranged at the output end of the collimation plane wave of the collimation lens, and the output end of the polarizing plate is used as the output end of the light source module and used for changing the polarization state of the collimation plane wave and outputting the collimated polarized illumination light.
8. The augmented reality near-eye display device of claim 6, wherein the optical modulation module is a spatial light modulator for loading a phase hologram corresponding to the auxiliary information, the phase hologram for changing a surface phase distribution of the spatial light modulator.
9. An augmented reality near-to-eye display method, comprising: Acquiring a target scene image; Determining target scene information according to the target scene image, and generating an auxiliary information hologram corresponding to the target scene according to the target scene information; and fusing the auxiliary information hologram to a target scene to perform augmented reality near-eye display.
10. The augmented reality near-to-eye display method of claim 9, wherein determining target scene information from the target scene image and generating an auxiliary information hologram corresponding to a target scene from the target scene information comprises: dividing a target scene from the target scene image based on a monocular depth estimation algorithm, and determining target scene information, wherein the target scene information comprises the intensity distribution of the target scene and the three-dimensional coordinates of the target scene relative to eyes of a user; Based on a hologram generation algorithm, generating auxiliary information describing the target scenery according to the intensity distribution of the target scenery and the three-dimensional coordinates of the target scenery relative to eyes of a user, and generating the auxiliary information hologram according to the auxiliary information and the three-dimensional coordinates.

Description

Augmented reality near-to-eye display device and method Technical Field The invention relates to the technical field of display, in particular to an augmented reality near-to-eye display device and method. Background The augmented reality (Augmented Reality, AR) near-to-eye display technology presents great potential in industrial applications such as intelligent manufacturing, warehouse management, etc., by superimposing virtual information into a real scene. In the prior art, a micro display chip is used for loading a two-dimensional digital image, and a picture is projected to human eyes through an optical relay system. However, such schemes can only project information to a fixed depth, lack of spatial layering, and cannot realize virtual-real fusion of auxiliary information and scenes with different depths in a real working scene, and the optical calibration unit remarkably increases the volume and weight of the device, so that wearing comfort of a user is reduced. Disclosure of Invention The invention provides an augmented reality near-eye display device and method, which are used for solving the defects that the existing near-eye display device can only project information to a fixed depth, virtual-real fusion with scenes with different depths in a real scene can not be realized, the system is large in size and the wearing comfort is poor. The invention can realize the accurate virtual-real fusion of the auxiliary information and the real scenes with different depths, reduce the volume and weight of the device, improve the wearing comfort, and is suitable for industrial applications such as intelligent manufacturing, warehouse management and the like. The invention provides an augmented reality near-to-eye display device which comprises an acquisition module, a processing module and a fusion module, wherein the acquisition module is used for acquiring a target scene image, the processing module is used for determining target scene information according to the target scene image and generating an auxiliary information hologram corresponding to a target scene according to the target scene information, and the fusion module is used for fusing the auxiliary information hologram into the target scene to perform augmented reality near-to-eye display. The augmented reality near-to-eye display device comprises a scene information determining module, a hologram generating module and a processing module, wherein the scene information determining module is used for dividing a target scene from a target scene image based on a monocular depth estimation algorithm to determine the target scene information, the target scene information comprises the intensity distribution of the target scene and the three-dimensional coordinates of the target scene relative to eyes of a user, the hologram generating module is used for generating auxiliary information describing the target scene according to the intensity distribution of the target scene and the three-dimensional coordinates of the target scene relative to eyes of the user based on a hologram generating algorithm, and generating an auxiliary information hologram according to the auxiliary information and the three-dimensional coordinates. The scene information determining module is used for classifying the target scene image according to mixed characteristics, wherein the mixed characteristics comprise colors, edges, outlines, vanishing lines and vanishing points, the image categories comprise distant view images, middle view images and near view images, the depth of the distant view images is estimated by adopting a horizontal edge gradient method, the depth of the middle view images is estimated by adopting a vanishing line gradient method, and the depth of the near view images is estimated by adopting a weighted depth superposition method, so that the target scene information is obtained. According to the augmented reality near-to-eye display device, the scene information determining module is used for inputting the target scene image into a pre-trained depth gradient assisted monocular depth estimation network model to obtain the target scene information output by the monocular depth estimation network model, wherein the monocular depth estimation network model is obtained through training of a depth estimation training sample and a depth gradient assisted sample. The hologram generation module is used for carrying out image feature recognition based on the intensity distribution to determine type features of the target scenery, generating text or graphic auxiliary information containing type identifiers according to the type features and generating numerical auxiliary information containing distance parameters according to depth components in the three-dimensional coordinates, inputting the auxiliary information and the three-dimensional coordinates into a pre-trained model without convolution errors to drive a self-coding depth learning network to obtain the a