CN-122021678-A - Display method, display device, electronic device, medium and program product applied to mixed reality device

CN122021678ACN 122021678 ACN122021678 ACN 122021678ACN-122021678-A

Abstract

The disclosure provides a display method, a display device, an electronic device, a medium and a program product applied to mixed reality equipment, which can be used in the technical field of computers. The method comprises the steps of obtaining a real-time screen streaming picture of a computing device, wherein the real-time screen streaming picture comprises a first graphic code, the first graphic code has an association relation with the computing device transmitting the real-time screen streaming picture, responding to the fact that a second graphic code exists in a real-time shooting picture, the second graphic code is consistent with the first graphic code, identifying a target screen of the computing device, the real-time shooting picture is obtained by shooting a real physical space where the computing device is located based on a mixed reality device, determining three-dimensional space coordinates of the target screen of the computing device with the current position of the mixed reality device as an origin, and replacing the real-time shooting picture in an area where the three-dimensional space coordinates are located with the real-time screen streaming picture based on the three-dimensional space coordinates so as to be displayed in the mixed reality device.

Inventors

Fu Huanmin
LIN MING
XIONG WEI

Assignees

中国建设银行股份有限公司
建信金融科技有限责任公司

Dates

Publication Date: 20260512
Application Date: 20251230

Claims (12)

1. A display method applied to a mixed reality device, comprising: Acquiring a real-time screen streaming picture of a computing device, wherein the real-time screen streaming picture comprises a first graphic code, and the first graphic code has an association relationship with the computing device transmitting the real-time screen streaming picture; In response to the existence of a second graphic code in a real-time shooting picture, wherein the second graphic code is consistent with the first graphic code, a target screen of the computing equipment is identified, and the real-time shooting picture is obtained by shooting a real physical space where the computing equipment is located based on the mixed reality equipment; determining three-dimensional space coordinates of a target screen of the computing device with the current position of the mixed reality device as an origin; And replacing a real-time shooting picture in the area where the three-dimensional space coordinates are located with the real-time screen streaming picture based on the three-dimensional space coordinates so as to be displayed in the mixed reality equipment.
2. The method of claim 1, wherein the obtaining real-time screen streaming pictures of a computing device comprises: in response to successful connection of the picture series flow between the mixed reality equipment and the computing equipment, triggering the computing equipment to adjust to a target state of continuously sending the real-time screen series flow picture to the mixed reality equipment; In the target state, the real-time screen streaming picture sent by the computing device is received, wherein the first graphic code in the real-time screen streaming picture is embedded in the form of a video stream overlay.
3. The method according to claim 1 or 2, further comprising: shooting a real physical space where the computing equipment can cover by the mixed reality equipment to obtain the real-time shooting picture; And carrying out pattern recognition on the real-time shooting picture to determine whether the second pattern code exists in the real-time shooting picture, wherein the second pattern code is embedded in the form of a video stream covering layer.
4. The method of claim 3, wherein the first graphic code comprises a first logo graphic and a second logo graphic, the second graphic code comprises a third logo graphic and a fourth logo graphic, the method further comprising: Extracting a single video frame from the real-time shooting picture, and positioning a target area containing the second graphic code from the video frame; Matching the second graphic code in the target area with the first mark graphic in the first graphic code; In response to the presence of the third logo graphic in the second logo code matching the first logo graphic, matching the second logo graphic in the target area with the second logo graphic in the first logo code; And determining that the second graphic code is consistent with the first graphic code in response to the fourth graphic code being present in the second graphic code in matching correspondence with the second graphic code.
5. The method of claim 4, wherein said matching the second graphic code in the target area with the first logo graphic in the first graphic code comprises: Extracting key points of the first mark patterns in the first pattern codes to obtain key point characteristics of the first mark patterns; extracting key points of the second graphic code in the target area to obtain key point characteristics of the second graphic code; And calculating the similarity of the key point features of the first marked graph and the key point features of the second graph code, and matching the second graph code in the target area with the first mark graph in the first graph code based on the similarity calculation result.
6. The method of claim 5, wherein the determining three-dimensional spatial coordinates of the target screen of the computing device with the current location of the mixed reality device as an origin comprises: acquiring a physical size of the first logo graphic in the first graphic code and an observed size of the third logo graphic in the second graphic code, wherein the observed size and the physical size are determined based on the same dimension size; determining a distance of a target screen of the computing device relative to the mixed reality device based on camera internal parameters of a camera device in the mixed reality device, the physical size and the observed size; determining angle information of a target screen of the computing device relative to the mixed reality device based on the acquired perspective distortion information of the third logo graphic in the second graphic code in the real-time shooting picture; And determining three-dimensional space coordinates of the target screen of the computing device by taking the current position of the mixed reality device as an origin according to the distance and angle information of the target screen of the computing device relative to the mixed reality device.
7. The method of claim 1, wherein the replacing the real-time shot in the area where the three-dimensional spatial coordinates are located with the real-time screen streaming picture for display in the mixed reality device based on the three-dimensional spatial coordinates comprises: And in response to the geometric vertex coordinates of the real-time screen series flow pictures and the three-dimensional space coordinates meet a threshold error condition, continuously projecting the real-time screen series flow pictures in the area where the three-dimensional space coordinates are located, so that the real-time shooting pictures in the area where the three-dimensional space coordinates are located are replaced by the real-time screen series flow pictures.
8. The method as recited in claim 7, further comprising: And responding to that the geometric vertex coordinates of the real-time screen series flow pictures and the three-dimensional space coordinates do not meet a threshold error condition, performing geometric transformation processing on the real-time screen series flow pictures so as to replace the real-time shooting pictures in the area where the three-dimensional space coordinates are located with the real-time screen series flow pictures.
9. A display device applied to a mixed reality apparatus, comprising: The system comprises a streaming picture acquisition module, a streaming picture generation module and a processing module, wherein the streaming picture acquisition module is used for acquiring a real-time screen streaming picture of the computing device, the real-time screen streaming picture comprises a first graphic code, and the first graphic code has an association relationship with the computing device for transmitting the real-time screen streaming picture; The shooting picture identification module is used for responding to the existence of a second graphic code in a real-time shooting picture, the second graphic code is consistent with the first graphic code, a target screen of the computing equipment is identified, and the real-time shooting picture is obtained by shooting a real physical space where the computing equipment is located based on the mixed reality equipment; the space coordinate determining module is used for determining three-dimensional space coordinates of a target screen of the computing device by taking the current position of the mixed reality device as an origin; And the picture replacing module is used for replacing the real-time shooting picture in the area where the three-dimensional space coordinates are positioned with the real-time screen streaming picture based on the three-dimensional space coordinates so as to be displayed in the mixed reality equipment.
10. An electronic device, comprising: one or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8.
11. A computer-readable storage medium, on which a computer program or instructions is stored, which, when executed by a processor, carries out the steps of the method according to any one of claims 1-8.
12. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Description

Display method, display device, electronic device, medium and program product applied to mixed reality device Technical Field The present disclosure relates to the field of computer technology, and more particularly, to a display method, apparatus, electronic device, medium, and program product applied to a mixed reality device. Background With the advent of Mixed Reality (MR) device technology, various Mixed Reality devices have grown. It can be used as a supplemental device to mix and link with the display screen of a conventional computing device. In the process of realizing the inventive concept of the present disclosure, the inventor finds that at least the following technical problems exist that when a user writes a document, looks at a video, and the like on a traditional computing device, the user interrupts the operations after wearing the mixed reality device, the traditional computing device cannot be used continuously, the user needs to restart on the mixed reality device, and the screen content is particularly blurred to be unavailable through the mixed reality device, so that the linkage experience of multiple devices of the user is reduced. Disclosure of Invention In view of the foregoing, the present disclosure provides a display method, apparatus, electronic device, medium, and program product applied to a mixed reality device. According to a first aspect of the disclosure, a display method applied to a mixed reality device is provided, and the method comprises the steps of obtaining a real-time screen series flow picture of the computing device, wherein the real-time screen series flow picture comprises a first graphic code, the first graphic code has an association relation with the computing device transmitting the real-time screen series flow picture, responding to the fact that a second graphic code exists in the real-time shooting picture, the second graphic code is consistent with the first graphic code, identifying a target screen of the computing device, the real-time shooting picture is obtained by shooting a real physical space where the computing device is located based on the mixed reality device, determining three-dimensional space coordinates of the target screen of the computing device with the current position of the mixed reality device as an origin, and replacing the real-time shooting picture in an area where the three-dimensional space coordinates are located with the real-time screen series flow picture based on the three-dimensional space coordinates so as to be displayed in the mixed reality device. According to the embodiment of the disclosure, the method for acquiring the real-time screen streaming picture of the computing device comprises the steps of triggering the computing device to adjust to a target state of continuously transmitting the real-time screen streaming picture to the mixed reality device in response to successful picture streaming connection between the mixed reality device and the computing device, and receiving the real-time screen streaming picture transmitted by the computing device in the target state, wherein a first graphic code in the real-time screen streaming picture is embedded in a video streaming cover layer mode. According to the embodiment of the disclosure, the method comprises the steps of shooting a real physical space where the computing device which can be covered by the mixed reality device is located to obtain a real-time shooting picture, and carrying out pattern recognition on the real-time shooting picture to determine whether a second pattern code exists in the real-time shooting picture, wherein the second pattern code is embedded in a video stream covering layer mode. According to the embodiment of the disclosure, the first graphic code comprises a first mark graphic and a second mark graphic, the second graphic code comprises a third mark graphic and a fourth mark graphic, the method further comprises the steps of extracting a single video frame from a real-time shooting picture, locating a target area containing the second graphic code from the video frame, matching the second graphic code in the target area with the first mark graphic in the first graphic code, matching the second graphic code in the target area with the second mark graphic in the first graphic code in response to the existence of the third mark graphic matched with the first mark graphic in the second graphic code, and determining that the second graphic code is matched with the first graphic code in response to the existence of the fourth mark graphic matched with the second mark graphic in the second graphic code. According to the embodiment of the disclosure, the matching of the second graphic code in the target area and the first mark graph in the first graphic code comprises the steps of extracting key points of the first mark graph in the first graphic code to obtain key point characteristics of the first mark graph, extracting key points of t