US-12626525-B2 - System for recognizing online handwriting

Abstract

This invention concerns a system for recognizing online handwriting comprising: a handwriting instrument including a body extending longitudinally between a first end and a second end, the first end having a writing tip which is able to write on a support, the handwriting instrument further including a module comprising at least one motion sensor configured to acquire motion data on the handwriting of a user when the user is writing a characters sequence with the handwriting instrument, a calculating unit communicating with the at least one motion sensor and configured to analyze the motion data by a machine learning model trained in a multitask way such that it is capable of performing at least two tasks at the same time, the machine learning model being configured to deliver as an output the characters sequence which was written by the user with the handwriting instrument.

Inventors

• Eric Humbert
• Arthur Belhomme
• Amélie Caudron

Assignees

• SOCIéTé BIC

Dates

Publication Date

20260512

Application Date

20211022

Priority Date

20201026

Claims (17)

1. 1 . A system for recognizing an online handwriting of a user comprising: a handwriting instrument including a body extending longitudinally between a first end and a second end, the first end having a writing tip which is able to write on a support, a module comprising at least one motion sensor configured to acquire motion data on the online handwriting of the user when the user is writing a characters sequence with the handwriting instrument, a calculating unit comprising a memory and a processor, the calculating unit communicating with the at least one motion sensor and configured to analyze the motion data by a machine learning model trained in a multitask way such that it is capable of performing a stroke segmentation task and a character classification task at a same time, wherein: the machine learning model comprises a shared backbone neural network configured to perform hidden features extraction of the motion data to obtain intermediate features, the stroke segmentation task is performed on the intermediate features, and the character classification task is performed using a recursive neural network using the intermediate features, and the machine learning model being configured to deliver as an output the characters sequence which was written by the user with the handwriting instrument.
2. 2 . The system according to claim 1 , wherein the module is embedded in the second end of the handwriting instrument.
3. 3 . The system according to claim 1 , wherein the module is placed on an outside surface of the second end of the handwriting instrument.
4. 4 . The system according to claim 1 , wherein the module further comprises the calculating unit.
5. 5 . The system according to claim 1 , wherein the module further includes a short-range radio communication interface configured to communicate raw motion data acquired by the at least one motion sensor to a mobile device comprising the calculating unit via a communication interface of the mobile device.
6. 6 . The system according to claim 1 , wherein the at least one motion sensor is a three-axis accelerometer.
7. 7 . The system according to claim 6 , wherein the module further comprises a second motion sensor being a three-axis gyroscope.
8. 8 . The system according to claim 1 , wherein characters of the characters sequence comprise numbers.
9. 9 . The system according to claim 1 , wherein the calculating unit comprises a volatile memory to store the motion data acquired by the at least one motion sensor.
10. 10 . The system according to claim 1 , wherein the calculating unit comprises a non-volatile memory to store the machine learning model enabling the recognizing of the online handwriting of the user.
11. 11 . The system according to claim 1 , wherein the trained machine learning model is stored in the calculating unit.
12. 12 . The system according to claim 1 , wherein the stroke segmentation task comprises labeling the acquired motion data in at least one of following classes: on-paper stroke or in-air movement.
13. 13 . The system according to claim 12 , wherein the trained machine learning model is further configured such that the acquired motion data is pre-processed before being used in the stroke segmentation task, wherein pre-processing comprises windowing raw motion data in time frames of N samples, wherein the samples are labeled in one of the on-paper stroke or the in-air movement.
14. 14 . The system according to claim 12 , wherein an in-air movement label comprises at least two sub-labels, forward in-air movement and backward in-air movement.
15. 15 . The system according to claim 1 , wherein the module further includes a battery configured to provide power to the at least one motion sensor when the user is using the handwriting instrument.
16. 16 . The system according to claim 15 , wherein the battery is further configured to provide power to other components comprised in the module.
17. 17 . The system according to claim 1 , wherein characters of the characters sequence comprise letters.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a National Stage Application under 35 U.S.C. § 371 of International Application No. PCT/EP2021/079431, filed Oct. 22, 2021, now published as WO 2022/090096 A1, which claims partial priority from European patent application No. 20306281.5, filed on Oct. 26, 2020, their content being incorporated herein by reference. TECHNICAL FIELD This disclosure pertains to the field of systems for recognizing online handwriting. More precisely, the disclosure pertains to the field of systems for online and continuous recognition of handwriting. BACKGROUND ART Currently, several systems or methods exist for recognizing the handwriting of a user. One type of handwriting recognition, called online handwriting recognition, consists of performing the recognition of a character, or a sequence of characters, while the user is writing it. Another type of handwriting recognition, called offline handwriting recognition, is based on the analysis of a picture showing text. For example, it is known from patent document U.S. Ser. No. 10/126,825 to perform online handwriting recognition by using a device such as a mobile terminal and three sensors being an accelerometer, a gyroscope and a magnetometer. The method described in this document uses a neural network of BLSTM type (Bidirectional Long-short term memory). The data acquired by the three sensors are submitted to several pre-processes in order to be analyzed by the neural network. Moreover, the neural network is trained with a dictionary containing predetermined words. The neural network is trained such that it is able to recognize the beginning and the end of a word being written and, on this basis, the neural network is able to determine which word of the dictionary has been written. One drawback of this method is then the use of three sensors which leads to a device expensive to build. Another drawback of this document is that the neural network is trained to recognize predetermined words. The recognition is not made continuously. It is also known from the scientific publication Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition, Xie et al. published in August 2018 in IEEE Transactions on Pattern Analysis and Machine Intelligence, a method for detecting online handwriting. The method uses four neural networks: MC-FCRN (multi-spatial-context fully convolutional recurrent network), FCN (fully convolutional network), LSTM (long-short term memory) and CTC (connectionist temporal classification). The neural networks are jointly trained. The method uses the path signature method, applied on reduced windows of data. This publication does not describe how the data are effectively acquired, however using path signature method implies to use positional sensors like a camera or a touchscreen. Another drawback is the need of using four neural networks, each being trained in chain with the entry of the preceding neural network. It is also known from the publication Offline Handwritten Chinese Text Recognition with Convolutional Neural Networks, Liu et al., published in 2020 in ArXiv, a method for offline handwriting recognition which cut line-of-text images into small time-steps before feeding them in a feature extractor neural network. However, this publication performs offline handwriting recognition, which is not adapted for the online handwriting recognition. SUMMARY One purpose of this disclosure is to improve the situation. It is proposed a system for recognizing online handwriting comprising: a handwriting instrument including a body extending longitudinally between a first end and a second end, the first end having a writing tip which is able to write on a support, the handwriting instrument further including a module comprising at least one motion sensor configured to acquire motion data on the handwriting of a user when the user is writing a characters sequence with the handwriting instrument,a calculating unit communicating with the at least one motion sensor and configured to analyze the acquired motion data by a machine learning model trained in a multitask way, the machine learning model being configured to deliver as an output the characters sequence which was written by the user with the handwriting instrument. In embodiments, the module is embedded in the second end of the handwriting instrument. In embodiments, the module is distinct from the handwriting instrument, i.e. it is a separate element. The module may be placed on an outside surface of the second end of the handwriting instrument. In embodiments, the module further comprises the calculating unit. In embodiments, the module further includes a short-range radio communication interface configured to communicate raw motion data acquired by the motion sensor to a mobile device comprising the calculating unit via a communication interface of the mobile device. In embodiments, the motion