US-12625584-B2 - Multimodal trajectory information processing and analysis system based on man-machine interaction terminal

US12625584B2US 12625584 B2US12625584 B2US 12625584B2US-12625584-B2

Abstract

Provided is a multimodal trajectory information processing and analysis system based on a man-machine interaction terminal. The system includes a collection device and an information processing device. The collection device is configured to collect, analyze and process finger operation trajectory information and send it to the information processing device, and the information processing device is configured to receive a display interface of a device under test in real time, and is configured to fuse and display the collected finger operation trajectory information corresponding to the display interface of the device under test, and obtain an evaluation system of man-machine interaction experience by analyzing a finger operation trajectory. The system achieves the simultaneous collection of the movement trajectory data and interaction data of the target object, which is convenient for accurately analyzing the rules and habits of the interactions between the users and the information systems.

Inventors

Qichao Zhao
Ran YANG

Assignees

KINGFAR INTERNATIONAL INC.

Dates

Publication Date: 20260512
Application Date: 20241210
Priority Date: 20231229

Claims (11)

1 . A multimodal trajectory information processing and analysis system based on man-machine interaction terminal, comprising: a collection sensor, an information processing device, and data transmission interface; wherein the collection sensor is configured to collect, finger operation trajectory information comprising real-time operation interaction information, and send the collected finger operation trajectory information to the information processing device through the data transmission interface; the information processing device is configured to receive a display interface of a device under test in real time, and is configured to fuse and display the collected finger operation trajectory information corresponding to the display interface of the device under test, and obtain an evaluation system of man-machine interaction experience by analyzing finger operation trajectories; and a bracket and a lifting device; wherein the bracket is configured to place the device under test and support an upper limb of a tester; the lifting device is connected to the bracket and is configured to adjust a location of the bracket; and the collection sensor is fixed arranged on the lifting device at an end away from the bracket.
2 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 1 , further comprising: a distance measuring device and a controller; wherein the distance measuring device is configured to detect a distance between the collection sensor and the device under test, and output a distance detection signal; and the lifting device is connected to the controller and is configured to adjust the distance between the collection sensor and the device under test in response to receiving a control signal.
3 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 1 , wherein the collection sensor is configured to collect and output the real-time operation interaction information, and the operation interaction information at least comprises interaction locations, durations of the interaction locations and the number of the interaction locations; and the information processing device is connected to the collection sensor and is configured to receive the operation interaction information in real time, and generate the finger operation trajectory information based on the received operation interaction information.
4 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 3 , wherein the collection sensor comprises a touch sensor unit, and the touch sensor unit is located above the display interface of the device under test.
5 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 4 , wherein the touch sensor unit is made of transparent plate material, and the touch sensor unit covers the display interface of the device under test.
6 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 4 , wherein the touch sensor unit is a transparent film, and is closely attached to the display interface of the device under test.
7 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 3 , wherein the optical sensor unit comprises an infrared transceiver sensor.
8 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 7 , wherein for collection of the interaction locations by the collection sensor, the collection sensor is configured to: obtain a plurality of light intensity signals, wherein the light intensity signals are configured to reflect light intensity of infrared light received by any infrared receiver in a group of first receivers or any infrared receiver in a group of second receivers, and the first receivers and second receivers are infrared receivers arranged along two adjacent sides of the device under test; determine a finger area based on a location of an infrared receiver with a weaker output light intensity signal and a location of an infrared transmitter that emits the infrared light; and determine an effective finger area of the finger area to obtain interaction coordinate information.
9 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 8 , wherein for the determination of the finger area based on the location of the infrared receiver with the weaker output light intensity signal and the location of the infrared transmitter that emits the infrared light, the collection sensor is configured to: determine a plurality of blocking paths based on the location of the infrared transmitter that emits the infrared light at each moment and the location of the infrared receiver with the weaker output light intensity signal; and determine an overlapping area of the plurality of blocking paths and recording the overlapping area as the finger area.
10 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 9 , wherein for the determination of the effective finger area in the finger area, the collection sensor is configured to: determine whether an area of each finger area does not exceed 0.5 cm 2 and is not less than 0.01 cm 2 ; and determine whether each finger area is quasi-circular; wherein in response to the finger area simultaneously satisfying the above two conditions, the finger area is regarded as the interaction coordinate information.
11 . The multimodal trajectory information processing and analysis system based on man-machine interaction terminal according to claim 10 , wherein for the determination of whether each finger area is quasi-circular, the collection sensor is configured to: obtain coordinate information of each point in the finger area; determine a centroid coordinate of the finger area based on the coordinate information; determine distances from the centroid coordinate to edges of the finger area based on the centroid coordinate and the coordinate information of each point in the finger area; and calculate a variance value based on the distances from the centroid coordinate to the edges of the finger area; wherein in response to the variance value being less than or equal to a preset threshold, the finger area is quasi-circular; and in response to the variance value being greater than the preset threshold, the finger area is non-quasi-circular.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS The present application claims priority to Chinese Patent Application No. 202311865562.8, filed on Dec. 29, 2023, and to Chinese Patent Application No. 202311865497.9, filed on Dec. 29, 2023, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD The present disclosure relates to the field of man-machine interaction technologies, and in particular to a multimodal trajectory information processing and analysis system based on man-machine interaction terminal. BACKGROUND With the continuous technological development, the interaction methods between users and information systems have expanded from traditional means such as buttons, handles, mice, and keyboards to multi-channel man-machine interaction methods such as touch screens, voice, and gestures. In order to understand the rules and habits of the interactions between users and information systems, it is necessary to collect and analyze the finger interaction trajectories of users during the touch screen interaction process. In the related technologies, for the finger trajectory analysis method, the movement trajectory data of fingers is collected and analyzed, rather than directly collecting the data of the interaction between fingers and information. For the interaction analysis method, the data of the interaction between fingers and information products is first collected, and then the finger interaction trajectory is analyzed, but the trajectory data of fingers is not collected. Therefore, the collected data is not comprehensive enough, which is not conducive to accurately analyzing the rules and habits of the interactions between users and information systems. SUMMARY In order to simultaneously collect the movement trajectory data and interaction data of the target object, the present disclosure provides a multimodal trajectory information processing and analysis system based on man-machine interaction terminal. According to a first aspect, the present disclosure provides a multimodal trajectory information processing and analysis system based on a man-machine interaction terminal, including a bracket, a lifting device, a collection device and an information processing device. The collection device is configured for collecting, and analyzing and processing finger operation trajectory information including real-time operation interaction information, and sending the collected finger operation trajectory information to the information processing device. The information processing device is configured for receiving a display interface of a device under test in real time, and is configured for fusing and displaying the collected finger operation trajectory information corresponding to the display interface of the device under test, and obtaining an evaluation system of man-machine interaction experience by analyzing finger operation trajectories. By adopting the above technical solutions, the collection device can collect and analyze the finger operation trajectory information and transmit it to the information processing device, and the information processing device can receive the display interface of the device under test in real time, thereby achieving the simultaneous collection of the movement trajectory data and interaction data of the target object, which is convenient for accurately analyzing the rules and habits of the interactions between the users and the information systems. Optionally, the multimodal trajectory information processing and analysis system based on a man-machine interaction terminal further includes a bracket and a lifting device. The bracket is configured for placing the device under test and for supporting an upper limb of a tester. The lifting device is connected to the bracket and is configured for adjusting a location of the bracket. The collection device is fixedly arranged on the lifting device at an end away from the bracket. Optionally, the collection device is configured for collecting and outputting the real-time operation interaction information, and the operation interaction information at least includes interaction locations, a duration of the interaction locations and the number of the interaction locations. The information processing device is connected to the collection device and is configured for receiving the operation interaction information in real time, and generating finger operation trajectory information based on the received operation interaction information. Optionally, the collection device includes a touch sensor unit, and the touch sensor unit is located above the display interface of the device under test. Optionally, the touch sensor unit is made of transparent plate material, and the touch sensor unit covers the display interface of the device under test. Optionally, the touch sensor unit is a transparent film, and is closely attached to the display interface of the device under test. Optionally, the collection device includes