Search

CN-116724341-B - System for recognizing online handwriting

CN116724341BCN 116724341 BCN116724341 BCN 116724341BCN-116724341-B

Abstract

The invention relates to a system (1) for recognition of online handwriting, comprising-a handwriting instrument (2) comprising a body (3) extending longitudinally between a first end (4) and a second end (5), the first end (4) having a writing tip (6) capable of writing on a support, the handwriting instrument (2) further comprising a module (17) comprising at least one motion sensor (7) configured to obtain motion data about the handwriting of a user while the user is writing a sequence of characters using the handwriting instrument (2), -a computing unit (8) in communication with the at least one motion sensor (7) and configured to analyze the motion data by means of a machine learning model, the machine learning model being trained in a multitasking manner such that it is capable of simultaneously performing at least two tasks, the machine learning model being configured to deliver as output the sequence of characters written by the user using the handwriting instrument.

Inventors

  • Eric hubert
  • Arthur Belhomme
  • Emily Calderon

Assignees

  • 毕克有限公司

Dates

Publication Date
20260508
Application Date
20211022
Priority Date
20201026

Claims (16)

  1. 1. A system (1) for recognizing online handwriting, comprising: a handwriting instrument (2) comprising a body (3) extending longitudinally between a first end (4) and a second end (5), the first end (4) having a writing tip (6) capable of writing on a support, and a module (17) comprising at least one motion sensor (7) being a tri-axial accelerometer and/or tri-axial gyroscope configured to obtain motion data concerning handwriting of a user while the user is writing a sequence of characters using the handwriting instrument (2), A computing unit (8) in communication with the motion sensor (7) and configured to analyze the motion data by means of a machine learning model trained in a multitasking manner such that it is capable of simultaneously performing a stroke segmentation task and a character classification task, The machine learning model is configured to deliver the sequence of characters written by the user using the handwriting instrument as output, wherein: -the machine learning model comprises a convolutional neural network performing hidden feature extraction to obtain intermediate features, the convolutional neural network being a multi-spatial context full convolutional recursive network MC-FCRN or a full convolutional network FCN; the stroke segmentation task is performed according to intermediate features using an additional convolutional neural network with upsampling, which is a multi-spatial context full convolutional recursive network MC-FCRN or full convolutional network FCN with upsampling, to segment strokes on paper and air movements; The character classification task is performed using intermediate features with a recurrent neural network, which is a BLSTM or converter trained using a join-sense time-classified CTC loss function.
  2. 2. The system (1) according to claim 1, characterized in that said module (17) is embedded in said second end (5) of said handwriting instrument.
  3. 3. The system according to claim 1, characterized in that the module (17) is placed on an external surface of the second end (5) of the handwriting instrument (2).
  4. 4. The system according to claim 1, characterized in that the module (17) further comprises the computing unit (8).
  5. 5. The system (1) according to claim 1, wherein the module (17) further comprises a short-range radio communication interface (9) configured to communicate raw motion data acquired by a motion sensor (7) to a mobile device (11) comprising the computing unit (8) via a communication interface (12) of the mobile device (11).
  6. 6. The system (1) according to claim 1, characterized in that the characters in the character sequence comprise numbers and/or letters.
  7. 7. The system (1) according to claim 1, wherein the computing unit (8) comprises a volatile memory to store data acquired by the motion sensor.
  8. 8. The system (1) according to claim 1, wherein the computing unit (8) comprises a non-volatile memory to store the machine learning model implementing handwriting recognition.
  9. 9. The system (1) according to claim 1, the trained machine learning model being stored in the computing unit (8).
  10. 10. The system (1) of claim 9, wherein the stroke segmentation task includes marking the acquired motion data in at least one of a stroke on paper or an over-the-air movement.
  11. 11. The system (1) of claim 9, wherein the trained machine learning model is further configured such that the acquired motion data is pre-processed prior to use in the stroke segmentation task, wherein pre-processing includes windowing raw motion data in a time frame of N samples, wherein the samples are marked in one of a stroke or an over-the-air movement on the paper.
  12. 12. The system (1) according to claim 10, wherein the air movement marker comprises at least two sub-markers, forward air movement and backward air movement.
  13. 13. The system (1) according to claim 9, wherein the trained machine learning model is further trained to enable the two tasks to be performed simultaneously.
  14. 14. The system (1) according to claim 1, wherein the module (17) further comprises a battery (10) configured to provide power to at least the motion sensor (7) when the user is using the handwriting instrument.
  15. 15. The system (1) according to claim 14, wherein the battery (10) is further configured to provide power to other components included in the module (17).
  16. 16. The system (1) according to claim 15, wherein the battery (10) is further configured to provide power to the computing unit (8) or to a short range radio communication interface (9).

Description

System for recognizing online handwriting The present application claims partial priority from the european patent application EP20306281.5, filed on 10/26/2020, claims 1-8, which is incorporated herein by reference. Technical Field The present disclosure relates to the field of systems for recognizing online handwriting. More specifically, the present disclosure relates to the field of systems for online and continuous recognition of handwriting. Background Currently, there are several systems or methods for recognizing handwriting of a user. One type of handwriting recognition, known as online handwriting recognition, consists of performing recognition of a character or sequence of characters while the user is writing the character or sequence of characters. Another type of handwriting recognition, known as offline handwriting recognition, is based on analysis of pictures of the presented text. For example, it is known from patent document US10126825 to perform online handwriting recognition by using a device such as a mobile terminal and three sensors as an accelerometer, a gyroscope and a magnetometer. The method described in this document uses a neural network of the two-way long short-term memory (BLSTM) type. The data acquired by the three sensors is submitted to several pre-processes for analysis by the neural network. In addition, the neural network is trained using a dictionary containing predetermined words. The neural network is trained such that it can identify the beginning and end of the word being written, and on this basis, the neural network can determine which word in the dictionary has been written. One disadvantage of this approach is the use of three sensors, which results in a costly device construction. Another disadvantage of this document is that the neural network is trained to recognize predetermined words. The identification is not continuous. A method for detecting online handwriting is also known from scientific publications published in IEEE model analysis and machine intelligence journal (IEEE Transactions on PATTERN ANALYSIS AND MACHINE INTELLIGENCE) by Xie (Xie) et al, 8, through full convolution recursive network learning spatial semantic context （Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition） for online handwriting chinese text recognition. The method uses four neural networks, a multi-space context full convolution recursive network (MC-FCRN), a Full Convolution Network (FCN), long Short Term Memory (LSTM), and join-sense time classification (CTC). The neural network is trained jointly. The method uses a path signature method applied over a reduced data window. This publication does not describe how to obtain data effectively, but using a path signature method means using a position sensor such as a camera or touch screen. Another disadvantage is the need to use four neural networks, each with entries of the previous neural network for chain training. An Offline handwriting Chinese text recognition (Offline HANDWRITTEN CHINESE Text Recognition with Convolutional Neural Networks) using a convolutional neural network is also known from the publication published in ArXiv by Liu (Liu) et al in 2020, for Offline handwriting recognition, which cuts a text line image into small time steps, which are then fed into a feature extractor neural network. However, the present publication performs offline handwriting recognition, which is not suitable for online handwriting recognition. Disclosure of Invention It is an object of the present disclosure to improve this situation. A system for recognizing online handwriting is presented, comprising: A handwriting instrument comprising a body extending longitudinally between a first end and a second end, the first end having a writing tip capable of writing on a support, the handwriting instrument further comprising a module comprising at least one motion sensor configured to acquire motion data regarding handwriting of a user while the user is writing a sequence of characters using the handwriting instrument, -A computing unit in communication with the at least one motion sensor and configured to analyze the acquired motion data by a machine learning model trained in a multitasking manner, the machine learning model configured to deliver as output a sequence of characters written by a user using a handwriting instrument. In an embodiment, the module is embedded in the second end of the handwriting instrument. In an embodiment, the module is different from the handwriting instrument, i.e. the module is a separate element. The module may be placed on an external surface of the second end of the handwriting instrument. In an embodiment, the module further comprises a computing unit. In an embodiment, the module further includes a short-range radio communication interface configured to communicate raw motion data acquired by the motion sensor to a mobile device co