CN-115981492-B - Three-dimensional handwriting generation method, device and system

CN115981492BCN 115981492 BCN115981492 BCN 115981492BCN-115981492-B

Abstract

The invention relates to a three-dimensional handwriting generating method, equipment and a system, which belong to the technical field of man-machine interaction, and can draw and write in real time without any specific hardware equipment, the cost is low, a pen point detection model is constructed in advance through a deep learning model so as to detect a pen point, the position of the pen point is mapped into a three-dimensional space to obtain three-dimensional space coordinate information, the coordinate information of the three-dimensional space is tracked in real time to obtain a predicted three-dimensional coordinate, and a three-dimensional coordinate graph of handwriting movement is obtained by drawing, so that the error is small, the effect is good, and the universality is realized.

Inventors

WANG WENTONG
ZHAO JIE
WANG MENGKUI
XU FENG
TENG DA
LIU QIANG

Assignees

北京石油化工学院

Dates

Publication Date: 20260512
Application Date: 20230103

Claims (10)

1. A method of generating three-dimensional handwriting, comprising: Acquiring an operation image of pen gesture interaction operation performed by a user holding an interaction pen based on a depth camera, wherein the operation image comprises a hand and the interaction pen; determining a hand detection frame in the operation image based on a hand expansion rule, wherein the hand detection frame comprises a complete interactive pen; Determining the position of a pen point in the hand detection frame according to a preset pen point detection model; Positioning the position of the pen point to a three-dimensional space to obtain coordinate information of the pen point in the three-dimensional space; Tracking coordinate information of the pen point in a three-dimensional space in real time by a tracking algorithm based on a 3D Kalman filter to obtain a predicted three-dimensional coordinate; and drawing a three-dimensional coordinate graph of handwriting movement by using 3D software according to the predicted three-dimensional coordinates.
2. The method of claim 1, wherein determining the position of the pen tip within the hand detection frame according to a preset pen tip detection model comprises: Extracting image features in the hand detection frame according to the preset nib detection model, calculating boundary frame information of a nib in the hand detection frame based on the image features to obtain a boundary frame, and taking the boundary frame and the boundary frame as nib positions, wherein the boundary frame information comprises U coordinates, V coordinates and width and height of the nib boundary frame.
3. The method according to claim 2, wherein positioning the position of the pen tip in a three-dimensional space to obtain coordinate information of the pen tip in the three-dimensional space includes: determining the two-dimensional coordinates of the centroid of the pen point according to the position of the pen point; acquiring depth information of a pixel point corresponding to the two-dimensional coordinate of the centroid of the pen point in a depth map; And obtaining coordinate information of the three-dimensional space of the pen point according to the depth information.
4. A method according to claim 3, further comprising: And aligning the depth map data acquired in the depth camera with the color image data.
5. The method of claim 4, wherein the aligning the depth map data acquired in the depth camera with color image data comprises: Converting the 2D points of the depth map into 3D points of a world coordinate system; A 3D point of the world coordinate system is projected onto the color image.
6. The method according to claim 1, wherein the tracking algorithm based on the 3D kalman filter tracks coordinate information of the pen tip in a three-dimensional space in real time to obtain predicted three-dimensional coordinates, and the method comprises: Determining coordinate information of a three-dimensional space of a pen point of a current operation image and coordinate information of a three-dimensional space of a pen point of a previous operation image; And predicting the track state of the nib in the three-dimensional space from the last operation image to the current operation image based on the tracking algorithm of the 3D Kalman filter, the coordinate information of the three-dimensional space of the nib of the current operation image and the coordinate information of the three-dimensional space of the nib of the last operation image, so as to obtain predicted three-dimensional coordinates.
7. The method of claim 1, wherein the drawing a three-dimensional graph of handwriting movement using 3D software from the predicted coordinates comprises: And drawing a three-dimensional coordinate graph of handwriting movement by using 3D software according to the coordinate information of the pen point in the three-dimensional space and the predicted coordinates.
8. The method of claim 1, wherein the predetermined nib detection model comprises a YOLOv convolutional neural network model, wherein the YOLOv convolutional neural network model is trained based on the interactive pen sample image and a predetermined interactive pen nib label.
9. A three-dimensional handwriting generating apparatus, comprising a processor and a memory, the processor being coupled to the memory: The processor is used for calling and executing the program stored in the memory; the memory for storing the program at least for executing the three-dimensional handwriting generating method according to any one of claims 1 to 8.
10. A three-dimensional handwriting generating system, comprising a depth camera and the three-dimensional handwriting generating apparatus of claim 9, which are connected to each other; the depth camera is used for shooting an operation image of the interactive pen with a preset distance.

Description

Three-dimensional handwriting generation method, device and system Technical Field The invention relates to the technical field of man-machine interaction, in particular to a three-dimensional handwriting generation method, equipment and a system. Background Human-computer interaction technology (Human-Computer Interaction Techniques, HCI) establishes "communication" between a person and a machine on the basis of computer technology. The man-machine interaction technology sends instruction information to the machine through certain man-machine interaction actions and programs, so that the machine feeds back a result according to the received instruction information, and the requirements of people are met. In various man-machine interaction devices, man-machine interaction pens are important carriers for realizing mutual communication and feedback of man-machine interaction. In the construction of a traditional man-machine interaction pen, a large number of inertial sensing devices are generally adopted for information acquisition. However, a large number of sensing devices are often high in cost, various noise, interference from the outside and the like exist in the working process, and errors occur in the interaction process due to the messy signals, so that the handwriting generation accuracy of the interaction pen is reduced. Therefore, the technical problems of high manufacturing cost and low generation accuracy of the human-computer interaction pen handwriting acquisition exist in the prior art. Disclosure of Invention In view of the above, the invention aims to provide a three-dimensional handwriting generating method, device and system, so as to solve the technical problems of high cost and low generating accuracy of the traditional human-computer interaction pen handwriting acquisition. In order to achieve the above purpose, the invention adopts the following technical scheme: In one aspect, a method for generating three-dimensional handwriting includes: Acquiring an operation image of pen gesture interaction operation performed by a user holding an interaction pen based on a depth camera, wherein the operation image comprises a hand and the interaction pen; determining a hand detection frame in the operation image based on a hand expansion rule, wherein the hand detection frame comprises a complete interactive pen; Determining the position of a pen point in the hand detection frame according to a preset pen point detection model; Positioning the position of the pen point to a three-dimensional space to obtain coordinate information of the pen point in the three-dimensional space; Tracking coordinate information of the pen point in a three-dimensional space in real time by a tracking algorithm based on a 3D Kalman filter to obtain a predicted three-dimensional coordinate; And drawing a three-dimensional coordinate graph of handwriting movement by using 3D software according to the predicted coordinates. Optionally, the determining the position of the pen tip in the hand detection frame according to the preset pen tip detection model includes: Extracting image features in the hand detection frame according to the preset nib detection model, calculating boundary frame information of a nib in the hand detection frame based on the image features to obtain a boundary frame, and taking the boundary frame and the boundary frame as a nib position, wherein the boundary frame information comprises U coordinates U i, V coordinates V i, width w i and height h i of the nib boundary frame. Optionally, the positioning the position of the pen tip to a three-dimensional space to obtain coordinate information of the pen tip in the three-dimensional space includes: determining the two-dimensional coordinates of the centroid of the pen point according to the position of the pen point; acquiring depth information of a pixel point corresponding to the two-dimensional coordinate of the centroid of the pen point in a depth map; And obtaining coordinate information of the three-dimensional space of the pen point according to the depth information. Optionally, the method further comprises: And aligning the depth map data acquired in the depth camera with the color image data. Optionally, the aligning the depth map data acquired in the depth camera with color image data includes: Converting the 2D points of the depth map into 3D points of a world coordinate system; A 3D point of the world coordinate system is projected onto the color image. Optionally, the tracking algorithm based on the 3D kalman filter tracks coordinate information of the pen tip in a three-dimensional space in real time to obtain a predicted three-dimensional coordinate, which includes: Determining coordinate information of a three-dimensional space of a pen point of a current operation image and coordinate information of a three-dimensional space of a pen point of a previous operation image; And predicting the track state of the nib in the three-dimension