KR-102962377-B1 - DEVICE FOR PROVIDING AUGMENTED REALITY AND METHOD FOR AUGMENTED PROVIDING AUGMENTED REALITY USING THE SAME

KR102962377B1KR 102962377 B1KR102962377 B1KR 102962377B1KR-102962377-B1

Abstract

The present invention relates to an augmented reality providing device capable of accurately recognizing a writing tool, such as an electronic pen, and implementing movement information of the writing tool in augmented reality content, and an augmented reality providing method using the same. An augmented reality providing device according to one embodiment of the present invention includes a support frame supporting at least one transparent lens, at least one display module displaying augmented reality content through at least one transparent lens, a sensing module generating front-facing image data of the support frame, and a control module receiving movement detection signals of an electronic pen to detect movement information of the electronic pen and modulating the augmented reality content so that the movement information of the electronic pen is written in the augmented reality content.

Inventors

이태희
양병춘
조주완
최병화
최해윤

Assignees

삼성디스플레이 주식회사

Dates

Publication Date: 20260508
Application Date: 20210902

Claims (20)

A support frame that supports at least one transparent lens; At least one display module that displays augmented reality content through the above at least one transparency lens; A sensing module that generates front-direction image data of the support frame; and It includes a control module that receives motion detection signals of an electronic pen, detects motion information of the electronic pen, and modulates the augmented reality content so that the motion information of the electronic pen is written into the augmented reality content. The above control module is An augmented reality providing device that continuously calculates position coordinate information of the electronic pen using the motion detection signal and generates movement path data of the electronic pen connected to the position coordinate information of the electronic pen.
In Article 1, The above at least one display module is An augmented reality providing device assembled on one or both sides of the support frame or formed integrally with the support frame, which displays an image of the augmented reality content through an image transmission member and reflective members of the transparency lens.
In Article 1, The above sensing module is It is assembled to the above support frame or formed integrally with the above support frame, An augmented reality providing device that detects the distance to an object in the front direction of the support frame and front direction image data using a depth sensor and an image sensor, and detects the movement of the user's eyeball or pupil using first and second biosensors.
In Paragraph 3, The above control module is The front-facing image data of the above-mentioned sensing module is divided into a plurality of preset block regions, and the coordinates of the user's gaze direction are detected according to the analysis results of pupil detection signals through the first and second biosensors. An augmented reality providing device that transmits a drawing start signal to the electronic pen by detecting the electronic pen in an image of a segmented block area corresponding to the user's gaze direction coordinates.
In Article 4, The above control module is Augmented reality content data is supplied to the at least one display module so that the at least one display module displays an image of the augmented reality content, and An augmented reality providing device that, when motion detection signals are received from the electronic pen in response to the drawing start signal, modulates the augmented reality content data so that the movement information of the electronic pen is written on the image of the augmented reality content and supplies it to the at least one display module.
In Article 5, The above control module is The motion detection signals of the electronic pen are analyzed in real time to continuously calculate position coordinate information of the electronic pen, and the movement path data of the electronic pen connected to the position coordinate information of the electronic pen is generated. An augmented reality providing device that inputs movement path data of the electronic pen into the augmented reality content data to modulate the augmented reality content data, and transmits the modulated augmented reality content data to at least one display module and at least one content display device located externally.
In Article 6, The above at least one content display device is An augmented reality providing device that displays the augmented reality content data shared from the control module on a screen, and displays the modulated augmented reality content data at the same timing as the at least one display module when the modulated augmented reality content data is received.
In Article 1, The above electronic pen is A motion detection module provided within a housing shaped like a writing instrument and generating the motion detection signals; At least one interface module that generates a drawing start signal or a drawing stop signal according to user input information; and An augmented reality providing device that transmits the motion detection signals above via wireless communication or receives the drawing start signal or the drawing stop signal through the control module.
In Article 1, The above control module is A data correction unit that modulates the augmented reality content data so that the movement path data of the electronic pen is entered into the augmented reality content data; An image display control unit that controls the at least one display module so that the augmented reality content data or the augmented reality content data modulated by the data correction unit is displayed through the at least one display module and the transparency lens; and An augmented reality providing device comprising a second wireless communication unit that transmits the above-mentioned modulated augmented reality content data to at least one external content display device.
In Article 9, The above control module is Receiving the motion detection signal including an acceleration signal, an angular velocity signal, and a tilt signal that vary as the electronic pen moves in the X-axis, Y-axis, and Z-axis directions, and An augmented reality providing device that continuously calculates the X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the electronic pen according to changes in distance, speed, and inclination of the electronic pen moving in the X-axis, Y-axis, and Z-axis directions.
In Article 9, The above control module is The front direction image data of the above-mentioned sensing module is divided into a plurality of preset block regions, and the coordinates of the user's gaze direction are detected according to the matrix arrangement information of the infrared sensors included in the above-mentioned sensing module and the movement analysis result of the pupil detection signals according to the matrix arrangement information. An augmented reality providing device that detects the electronic pen in an image of a segmented block area corresponding to the user's gaze direction coordinates, and transmits a drawing start signal to the electronic pen through a first wireless communication unit.
In Article 11, The above control module is An augmented reality providing device that detects an image of an electronic pen shape according to at least one of the analysis results of the difference in grayscale or luminance between pixels of a segmented block area corresponding to the user's viewing direction coordinates, or the analysis results of the contrast between pixel data and a mask of the electronic pen shape.
In Article 11, The above control module is When the electronic pen is detected in the image of the divided block area, distance information between the sensing module and the electronic pen is calculated according to the distance detection signal from the sensing module, and An augmented reality providing device that transmits a drawing start signal to the electronic pen via the first wireless communication unit when the distance information between the sensing module and the electronic pen is less than or equal to preset reference distance information.
In Article 9, The above control module is An augmented reality providing device that generates movement path data of the electronic pen by continuously storing the X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the electronic pen, which are continuously calculated from a location information calculation unit, into pre-set coordinate input space data or block data of the size of augmented reality content.
In Article 10, The above control module is An augmented reality providing device that corrects the size and resolution of the movement path data of the electronic pen to correspond to the frame-by-frame size and resolution characteristics of the augmented reality content data and transmits it to the data correction unit.
In Article 15, The above data correction unit The above augmented reality content data is stored by frame, and An augmented reality providing device that modulates augmented reality content data by sequentially writing movement path data of the electronic pen, in which the size and resolution characteristics are modulated, into augmented reality content data separated by frame.
A step of displaying augmented reality content through at least one display module and a transparency lens; A step of transmitting a drawing start signal generated in an image processing unit to an electronic pen via a first wireless communication unit; A step of continuously calculating position coordinate information of the electronic pen in a position information calculation unit using motion detection signals received from the electronic pen; A step of generating movement path data of the electronic pen using the image processing unit connected to the position coordinate information of the electronic pen; A step of modulating the augmented reality content data in a data correction unit so that the movement path data of the electronic pen is entered into the augmented reality content data; A step of displaying the augmented reality content data, on which the movement path data of the electronic pen is written, using at least one display module; and A method for providing augmented reality, comprising the step of transmitting the augmented reality content data, on which the movement path data of the electronic pen is entered, to at least one external content display device via a second wireless communication unit.
In Article 17, The step of transmitting the above drawing start signal to the electronic pen A step of detecting front-facing image data of an augmented reality providing device and dividing it into a plurality of pre-set block regions; A step of detecting the coordinates of the user's gaze direction according to matrix arrangement information of infrared sensors included in first and second biosensors that detect pupils, and the result of movement analysis of pupil detection signals according to the matrix arrangement information; A step of detecting the electronic pen in an image of a segmented block area corresponding to the user's gaze direction coordinates; and A method for providing augmented reality comprising the step of generating a drawing start signal and transmitting it to the electronic pen when the electronic pen is detected in an image of the divided block area.
In Article 17, The step of generating the movement path data of the above electronic pen is A step of continuously storing the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate of the electronic pen in preset coordinate entry space data or in block data of the size of augmented reality content; and A method for providing augmented reality, comprising the step of correcting the size and resolution of electronic pen movement path data, in which the X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the electronic pen are arranged continuously, to correspond to the frame-by-frame size and resolution characteristics of the augmented reality content data.
In Article 19, The step of modifying the augmented reality content data above A step of storing the above augmented reality content data by frame; A step of storing movement path data of the electronic pen modulated with the size and resolution characteristics above; and A method for providing augmented reality comprising the step of sequentially writing, modifying, and storing the movement path data of the electronic pen, in which the size and resolution characteristics are modulated, into the augmented reality content data separated by frame.

Description

Device for providing augmented reality and method for providing augmented reality using the same The present invention relates to an augmented reality providing device and a method for providing augmented reality using the same. Recently, interest in virtual reality (VR) has been growing as electronic devices and display devices capable of implementing VR have been developed. Research is also being conducted on technologies that can realize augmented reality (AR) and mixed reality (MR) as the next steps after virtual reality. Unlike virtual reality, which is based on a completely virtual world, augmented reality is a display technology that enhances the effect of reality by overlaying virtual objects or visual information onto a real-world environment. While virtual reality has been applied restrictively to fields such as games and virtual experiences, augmented reality has the advantage of being applicable to various real-world environments. In particular, augmented reality is attracting attention as a next-generation display technology suitable for ubiquitous environments and the Internet of Things (IoT). This type of augmented reality can be considered an example of mixed reality in that it combines and displays additional information from the real and virtual worlds. FIG. 1 is a drawing showing an augmented reality providing device according to one embodiment of the present invention. FIG. 2 is a perspective view specifically showing the augmented reality providing device illustrated in FIG. 1. Figure 3 is an exploded perspective view of the augmented reality providing device shown in Figure 2. FIG. 4 is a frontal exploded perspective view of the augmented reality providing device shown in FIG. 2 and FIG. 3. FIG. 5 is a layout diagram specifically showing the display module of the augmented reality providing device illustrated in FIG. 2. Figure 6 is a layout diagram showing area A of Figure 5 in detail. Figure 7 is a layout diagram showing in detail the pixels illustrated in area B of Figure 6. FIG. 8 is a cross-sectional view showing an example of a display panel cut along line II' of FIG. 7. Figure 9 is an enlarged cross-sectional view showing in detail an example of the light-emitting element of Figure 8. FIG. 10 is a block diagram schematically showing the configuration of an electronic pen according to one embodiment. FIG. 11 is a block diagram schematically showing a control module equipped in the augmented reality providing device of FIG. 2 to FIG. 4. FIG. 12 is a flowchart illustrating a method for providing augmented reality content of an augmented reality providing device according to one embodiment. FIG. 13 is a diagram showing a method of using an electronic pen and an augmented reality providing device according to one embodiment. FIG. 14 is a flowchart illustrating a method for generating a drawing start signal through electronic pen recognition according to one embodiment. FIG. 15 is a diagram showing an electronic pen recognition method using an augmented reality providing device in one embodiment. FIG. 16 is a diagram showing an electronic pen recognition method through pupil tracking and image analysis according to one embodiment. FIG. 17 is a drawing showing a display screen of augmented reality content and electronic pen recognition status of an augmented reality providing device according to one embodiment. FIG. 18 is a flowchart illustrating a method for inserting electronic pen movement information and displaying augmented reality content through electronic pen movement recognition according to one embodiment. FIG. 19 is a diagram showing a method for detecting and transmitting a sensing signal of an electronic pen according to one embodiment. FIG. 20 is a diagram illustrating a method for generating coordinate information of an electronic pen according to one embodiment. FIG. 21 is a diagram illustrating a method for generating movement path data of an electronic pen and a method for modulating data size according to one embodiment. FIG. 22 is a diagram illustrating a method for matching movement path data of an electronic pen and augmented reality content according to one embodiment. FIG. 23 is an augmented reality content display screen implemented by matching movement path data of an electronic pen and augmented reality content according to one embodiment. The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. When elements or layers are referred to as b