Search

CN-122018727-A - Non-visual non-authority interface sensing method based on radio frequency/millimeter wave

CN122018727ACN 122018727 ACN122018727 ACN 122018727ACN-122018727-A

Abstract

The invention discloses a radio frequency/millimeter wave-based non-visual non-authority interface sensing method, and belongs to the technical field of non-authority AI interaction and non-visual sensing. The method comprises the steps of disposing a radio frequency reflection sensor or a millimeter wave scanning sensor as a non-visual perception module, wherein the non-visual perception module is not electrically connected with a touch screen equipment system, does not depend on system authority and an API interface, the perception module transmits an adaptive radio frequency/millimeter wave signal, extracts characteristics such as amplitude, phase and frequency offset after being reflected by a screen interface to form interface characteristic data, analyzes the screen interface information through an interface characteristic recognition model formed by a convolutional neural network and an attention mechanism after noise reduction and interference resistance treatment, generates operation guidance, and achieves non-authority AI interaction through passive optical guidance or physical triggering. The invention adapts the full-product touch screen equipment and the multi-operation system, can stably work under the visual perception failure scene such as strong light, shielding, strong electromagnetic and the like, ensures the recognition accuracy to be more than or equal to 95 percent and the control position recognition error to be less than or equal to 1.5mm through the designs such as dynamic threshold adaptation, distance-precision association, interference self-adaptive adjustment and the like, builds a non-visual perception defense system without dead angles of the full scene, has extremely strong legal stability and technical feasibility, and replaces avoidance risk by an effective blocking perception mode.

Inventors

  • CHANG LE

Assignees

  • 常乐

Dates

Publication Date
20260512
Application Date
20260322

Claims (10)

  1. 1. The non-visual non-authority interface sensing method based on the radio frequency/millimeter wave is characterized by comprising the following steps of: The method comprises the steps that a non-visual perception module is deployed outside a touch screen device or in a device shell, wherein the non-visual perception module comprises a radio frequency reflection sensor or a millimeter wave scanning sensor, is not electrically connected with a touch screen device system, and does not depend on any system authority, API (application program interface) or hardware drive of the touch screen device; the non-visual perception module transmits radio frequency signals or millimeter wave signals, and the signals cover a screen display area of the touch screen device; After the signals are reflected by a screen interface of the touch screen device, a non-visual perception module receives the reflected signals, and the characteristics of the reflected signals are extracted to form interface characteristic data; Carrying out noise reduction and anti-interference treatment on the interface characteristic data, and reserving effective characteristics related to display contents of a screen interface; transmitting the processed effective characteristic data to an AI unit, analyzing screen interface information through an interface characteristic identification model, and generating interface information containing operation coordinate references; The AI unit generates operation guide information based on the interface information, and realizes non-authority AI interaction in a passive optical guiding or physical triggering mode, wherein the touch screen equipment comprises various touch screen equipment such as a smart phone, a tablet computer, an industrial touch screen, a vehicle-mounted touch screen, a smart watch and the like, and is compatible with operating systems such as a hong, an android, an iOS, a Windows and the like.
  2. 2. The method of claim 1, wherein the step 1 of "no electrical connection" includes establishing data communication with the touch screen device system without any wired or wireless means such as USB, serial port, UART, OTG interface, bluetooth, wi-Fi, etc., and the non-visual perception module operates only by external power supply or self-contained power supply, and no data transmission channel exists between the non-visual perception module and the touch screen device system.
  3. 3. The method of claim 1, wherein the radio frequency reflection sensor has an operating frequency range of 1-10GHz, a transmitting signal power of-10 dBm-0 dBm and a scanning period of 10-50ms, the millimeter wave scanning sensor has an operating frequency range of 24-77GHz, a transmitting signal power of 0 dBm-5 dBm and a scanning period of 5-20ms, the overlap ratio of a signal coverage area and a screen display area of a touch screen device is more than or equal to 95%, and the transmitting power and the operating frequency of the radio frequency reflection sensor and the millimeter wave scanning sensor are configured to ensure that the amplitude difference of the reflected signals meets the judgment condition of claim 4.
  4. 4. The method of claim 1, wherein the interface characteristic data in the step 3 comprises the difference of reflected signal amplitude values of different display areas of a screen interface, the difference of phase shift between a control and a background area and the change of frequency shift of a dynamic interface element, wherein the difference of amplitude values is more than or equal to 3dB, the difference of amplitude values is judged to be different display areas, the threshold value can be dynamically adjusted to 1-2dB under special scenes such as an OLED dark mode, and the like, and the difference of phase shift is more than or equal to 10 degrees, and the boundary between the control and the background is judged.
  5. 5. The method of claim 1, wherein the noise reduction and anti-interference processing in step 4 includes eliminating the environmental radio frequency interference by adopting an adaptive filtering algorithm, eliminating the equipment shell reflection interference by signal power threshold screening, eliminating the screen refreshing interference by using a time synchronization technology, and improving the signal-to-noise ratio of the processed signal by more than or equal to 15dB compared with the original signal.
  6. 6. The method of claim 1, wherein the interface feature recognition model comprises a Convolutional Neural Network (CNN) feature extraction network and an attention mechanism interface analysis network, wherein when the distance between the non-visual perception module and a screen is less than or equal to 20mm, the control position recognition error is less than or equal to 0.5mm, when the distance is 20-50mm, the error is less than or equal to 1.5mm, and the interface content recognition accuracy is more than or equal to 95%.
  7. 7. The method of claim 1, further comprising a sensing parameter adaptive adjustment mechanism, wherein the sensing parameter adaptive adjustment mechanism measures interference intensity within + -10 MHz bandwidth of a sensor operating frequency band through an environmental interference detection unit, and when the environmental interference is more than or equal to-20 dBm, the transmitting power is increased by 1-3dBm, the scanning frequency is increased to 150% of the original frequency, and the signal bandwidth is reduced to 50% of the original bandwidth.
  8. 8. The method of claim 1, wherein the non-visual perception module is deployed in a mode of independent external arrangement, integrated in a device shell and integrated in an external accessory, the deployment position and the screen distance are 5-50mm, the standby power consumption of the module is less than or equal to 10 mu W, the working power consumption is less than or equal to 1mW, and the size is less than or equal to 20mm multiplied by 5mm.
  9. 9. The method of claim 1, wherein the multi-sensor cooperative work is supported, wherein the radio frequency reflection sensor and the millimeter wave scanning sensor can be used alternatively or work simultaneously, the identification accuracy is improved by more than or equal to 5% compared with a single sensor after the cooperative output characteristic data are fused, and the implementation mode of the unauthorized AI interaction comprises any one of optical signal physical triggering, ultrasonic auxiliary triggering and passive optical guiding triggering.
  10. 10. The method of claim 1, wherein the adaptation for different scenarios: The vehicle-mounted scene comprises the steps of adopting a millimeter wave scanning sensor, wherein the working frequency is 60-77GHz, the anti-electromagnetic interference capability is more than or equal to 20 dB@60-77 GHz frequency band, the scanning period is 5-10ms, and the control identification error is less than or equal to 1.0mm; The industrial control scene is that a radio frequency reflection sensor is adopted, the working frequency is 5-10GHz, the transmitting power is 0 dBm-3 dBm, the capability of penetrating oil dirt and dust is more than or equal to 80%, and the sensing error is less than or equal to 1mm; The small-size equipment scene adopts a miniature millimeter wave sensor, the working frequency is 24-30GHz, the module size is less than or equal to 10mm multiplied by 3mm, and the control identification error is less than or equal to 0.3mm.

Description

Non-visual non-authority interface sensing method based on radio frequency/millimeter wave Technical Field The invention relates to the technical field of non-authority interaction and non-visual perception of intelligent terminals, in particular to a radio frequency/millimeter wave-based non-visual non-authority interface perception method, which belongs to the technical field of non-authority AI interaction and non-visual perception, is suitable for various touch screen devices such as intelligent mobile phones, tablet computers, touch screen notebook computers, integrated computers, industrial control touch screens, vehicle-mounted touch screens, intelligent watches, intelligent household appliance touch panels and the like, is compatible with mobile operating systems such as hong, android, iOS and the like and Windows, mac OS, linux desktop/industrial control/vehicle-mounted operating systems, can stably work under visual perception failure scenes such as strong light, weak light, greasy dirt shielding, severe environment and the like, and provides a reliable interface information acquisition scheme for non-authority AI interaction. Background Along with the popularization of intelligent terminals, the non-authority AI interaction technology is widely applied to scenes such as cross-platform interaction, industrial control, vehicle-mounted intelligence and the like due to the advantages of no dependence on equipment system authority, no access to an API interface, strong compatibility and the like. The current interface perception mode without permission AI interaction mainly depends on visual recognition, such as shooting a screen interface through a camera, screen projection and acquisition of display content, but visual perception has obvious scene limitation that image acquisition definition is greatly reduced under strong light backlight and weak light dark environments, so that interface recognition is invalid, greasy dirt and dust on the surface of equipment can shield a visual acquisition path to influence perception accuracy in industrial scenes, and vibration and electromagnetic interference can cause blurring and distortion of images acquired by the vision in vehicle-mounted and industrial electromagnetic environments, so that perception stability is further reduced. Meanwhile, a defending gap exists in a non-authority interaction technology system which only depends on visual perception, and an adversary can avoid related patent protection in an alternative perception mode, so that the applicability and the safety of an interaction scheme are limited. In addition, in the existing non-visual perception scheme, part of parameter expression logic is ambiguous, for example, the expression of 'scan frequency is increased by 50%' can generate double interpretation, and is inconsistent with other explicit parameter adjustment expressions, although the technical feasibility is not affected, legal disputes can be caused, and the stability of the patent is reduced. In order to solve the above problems, a non-visual non-authority interface sensing method which is independent, stable and strict in logic is needed, the gap of visual sensing scenes is filled, the blocking sensing mode replaces avoidance risks, and reliability and legal stability of non-authority AI interaction in a complex environment are ensured. Disclosure of Invention First, the technical problem to be solved The existing interface perception mode of the non-authority AI interaction mainly depends on visual recognition (such as camera shooting and screen throwing acquisition), but visual perception is easy to fail in complex scenes such as strong light backlight, weak light darkness, greasy dirt dust shielding, industrial electromagnetic interference, vehicle-mounted vibration and the like, and a technical short plate with insufficient perception stability exists. After multiple rounds of optimization, the proposal is found to have the tiny problem of logic ambiguity of parameter expression, namely the expression of 'scan frequency is improved by 50 percent' in the original design can generate double interpretation, which is inconsistent with the explicit expression of 'signal bandwidth is reduced to 50 percent of the original bandwidth', and the technical feasibility is not affected, but the pursuit is extremely strict, the expression mode is required to be unified, and the potential dispute is eliminated. The invention aims to solve all the problems, and provides a non-visual non-authority interface perception method based on radio frequency/millimeter waves, which realizes stable interface information acquisition in a complex environment by definitely defining absolute boundaries without electrical connection, dynamically adapting perception threshold values, associating deployment distance and precision, defining interference measurement references, eliminating parameter contradictions and uniformly expressing logic, fills up the b