CN-121979433-A - IOS device intelligent rolling screenshot and evidence fixing system based on Wi-Fi and Bluetooth cooperation

CN121979433ACN 121979433 ACN121979433 ACN 121979433ACN-121979433-A

Abstract

The invention discloses an iOS device intelligent scrolling screenshot and fixing system based on Wi-Fi and Bluetooth cooperation. The system comprises an iOS device, a PC running a certification program, and a wireless communication module. The fixed evidence program comprises a protocol analysis module, an audio and video processing module, an AI image processing module and a Bluetooth control module, wherein the protocol analysis module is used for establishing screen projection connection through a reverse AirPlay protocol and receiving a screen coding stream, the audio and video processing module is used for decoding the coding stream and generating screenshot and video, the AI image processing module is used for analyzing screen image frames and generating a rolling control instruction, and the Bluetooth control module is used for simulating man-machine interface equipment through Bluetooth to drive an iOS equipment screen to automatically roll. The system circularly executes the processes of image analysis, decision scrolling, control execution and screenshot saving through cooperation of the modules until the complete screenshot of the long content is completed. The invention realizes the automatic, high-precision and whole-process rolling screenshot and evidence solidification of the screen content of the iOS equipment.

Inventors

CHEN MINGXIAO
CAO RONGXIN
HUANG JIAHONG
CHEN MINGHUI

Assignees

厦门市美亚柏科信息安全研究所有限公司

Dates

Publication Date: 20260505
Application Date: 20251208

Claims (10)

1. The intelligent rolling screenshot and evidence fixing system of the iOS equipment based on the cooperation of Wi-Fi and Bluetooth is characterized by comprising the iOS equipment, a PC running an evidence fixing program and a wireless communication module connected with the iOS equipment and the PC, wherein the wireless communication module comprises a Wi-Fi module and a Bluetooth module, the Wi-Fi module is used for establishing a high-bandwidth data channel, and the Bluetooth module is used for establishing a reverse control channel of the PC to the iOS equipment; The fixing and verifying program running in the PC comprises the following steps: the protocol analysis module is used for establishing encryption screen-throwing connection with the iOS equipment through a reverse AirPlay protocol, and receiving and decrypting a real-time screen coding stream from the iOS equipment; The audio and video processing module is used for decoding and extracting frames from the real-time screen coding stream, outputting screen image frames and generating single-frame screenshot or screen recording files based on the screen image frames; The AI image processing module is used for analyzing the image frame of the screen, identifying the target content area, the interface element and the rolling state, judging the starting point and the ending point of the rolling screenshot and the accurate distance of each rolling based on an image similarity algorithm, and generating a control instruction of screen rolling; The Bluetooth control module is used for simulating man-machine interface equipment through a Bluetooth low-power-consumption protocol, converting a control instruction into a mouse control event which can be identified by the iOS equipment, and driving the screen of the iOS equipment to automatically scroll; The intelligent scrolling screenshot is realized by cooperation of the modules according to the following procedures that a protocol analysis module acquires a screen coding stream, an audio and video processing module decodes the screen coding stream into screen image frames, an AI image processing module analyzes the image frames and generates an instruction to trigger Bluetooth control module iOS equipment to drive a screen to scroll when the screen is required to scroll, after the scrolling is stopped and the picture is stable, the audio and video processing module executes screenshot storage, and the AI image processing module continuously analyzes new image frames and circularly executes the scrolling and screenshot process until the screenshot is judged to be completed.
2. The method according to claim 1, wherein the protocol parsing module is specifically configured to: Simulating the PC end as an AirPlay receiving end, and broadcasting simulated AirPlay receiving end equipment information in a local area network through Bonjour service; Performing TLS handshake with the iOS equipment initiating the screen image, and establishing an encrypted communication channel; receiving a real-time screen coding stream which is sent by the iOS equipment through a UDP protocol and is packaged into an RTP data packet; and decapsulating the RTP data packet to separate a video coding stream and an audio coding stream, wherein the video coding format is H.264 or H.265.
3. The system of claim 2, wherein the audio-video processing module comprises: A decoding unit, configured to decode the separated video encoding stream, and convert video data in h.264 or h.265 format into an original image frame in YUV or RGB format; The screenshot unit is used for encoding the original image frame into a picture file in a JPG or PNG format for storage; The encoding and packaging unit is used for encoding the decoded video frames, converting the synchronously received audio encoding stream into an AAC format, and finally packaging the encoded video data and the audio data into an MP4 container to form an MP4 video file.
4. The method of claim 1, wherein the AI image processing module comprises: An element detection unit for identifying specific interface elements and position coordinates thereof in the screen image frame by adopting a target detection model or an image segmentation model based on deep learning, wherein the specific interface elements comprise, but are not limited to, chat bubbles, list item boundaries, application navigation bars or page scroll bars; The rolling decision unit is connected with the element detection unit and is used for calculating the accurate displacement required by the interface area where the specific interface element is positioned to roll to the next area to be captured according to the identified element position and in combination with a preset capture strategy; The completion judging unit is used for adopting an image similarity algorithm to compare screen images before and after the Bluetooth control module triggers the scrolling operation, judging that no new effective content is generated by the current scrolling when the similarity between the images after the scrolling and the images before the scrolling is higher than a preset threshold value, thereby ending the screenshot process, or ending the screenshot process when a preset ending mark is identified.
5. The system of claim 4, wherein the AI image processing module is configured to execute a targeted screenshot strategy comprising: A context continuous screenshot strategy for controlling the screen to scroll to ensure continuous chat bubbles are completely intercepted in sequence when the chat session interface is identified; a list complete screenshot strategy for controlling the screen to scroll to intercept the complete list content from the head to the tail when the list or the form interface is identified; And the key region priority strategy is used for preferentially ensuring that the region containing the preset key element type is completely captured based on the identification result of the element detection unit.
6. The system of claim 4, wherein the predetermined termination mark comprises at least one of: a specific application interface bottom mark or end mark identified by the element detection unit; after continuous scrolling for a plurality of times, the number of times that the completion judging unit judges that no new valid content is added reaches a preset threshold value; and detecting that the screen content is restored to the initial picture or homepage of the screenshot task.
7. The system according to claim 1, wherein the bluetooth control module is specifically configured to: Configuring a Bluetooth protocol stack provided by an operating system at a PC end, simulating a man-machine interface device conforming to a Bluetooth low-power consumption specification, and configuring a report descriptor of the man-machine interface device as a mouse type; broadcasting a simulated man-machine interface device for the iOS device to find and complete pairing; After the pairing is successful, the control data conforming to the mouse report format of the human-computer interface device is sent to the iOS device through the notification channel of the universal attribute configuration file so as to simulate and execute the mouse rolling, clicking or dragging event.
8. The system of claim 4, wherein the bluetooth control module is further configured to: After executing the scrolling operation according to the instruction of the AI image processing module, triggering a delay waiting for the interface rendering to be stable, and informing the audio/video processing module to execute screenshot after the interface is stable; The AI image processing module is further configured to control the Bluetooth control module to perform a reverse small-amplitude scroll check to confirm whether to truly scroll to the bottom when the completion determination unit determines that a single scroll does not generate valid newly added content.
9. The system of claim 1, wherein the certification program further comprises an evidence chain management module for: automatically generating a metadata file containing a time stamp, a device identifier and an operation log for a complete scrolling screenshot task; Performing associated storage on all screenshot files, screen recording files and metadata files generated by the task, and calculating hash values of the screenshot files, the screen recording files and the metadata files; And storing the hash value into a blockchain or carrying out digital signature to form a tamper-proof evidence packet.
10. An iOS device intelligent scrolling screenshot and evidence fixing method based on Wi-Fi and Bluetooth cooperation is characterized by comprising the following steps: a Wi-Fi-based high-bandwidth data channel and a Bluetooth-based reverse control channel which is controlled by the PC to the iOS device are established between the PC running the certification program and the iOS device; establishing encryption screen-throwing connection with the iOS equipment through a reverse AirPlay protocol, and receiving and decrypting a real-time screen coding stream from the iOS equipment; decoding and frame extraction processing is carried out on the real-time screen coding stream, and continuous screen image frames are output; analyzing the image frames of the screen in real time, identifying a target content area, interface elements and a rolling state, judging a starting point and an ending point of a rolling screenshot and the accurate distance of each rolling based on an image similarity algorithm, and generating a control instruction containing rolling parameters when judging that the rolling is needed; The control instruction is converted into a simulated mouse control event which can be identified by the iOS equipment through a Bluetooth low-power-consumption protocol, and the simulated mouse control event is sent to the iOS equipment so as to drive a screen of the iOS equipment to automatically scroll; After one-time scrolling operation is finished and the content of the screen image frame tends to be stable, capturing a picture from the current image frame and storing the picture as a screenshot file; and circularly executing the processes of analyzing, deciding, controlling scrolling and capturing based on the new screen image frame until the completion of capturing the target content is judged.

Description

IOS device intelligent rolling screenshot and evidence fixing system based on Wi-Fi and Bluetooth cooperation Technical Field The invention relates to the technical field of cooperation of computer software and mobile equipment, in particular to an iOS (integrated operation system) intelligent rolling screenshot and evidence fixing system based on Wi-Fi and Bluetooth cooperation. Background With the wide popularization of mobile devices and the continuous evolution of mobile application ecology, iOS devices have become a core evidence source bearing key clues in digital evidence collection investigation by virtue of their huge user base. In various investigation scenarios such as case investigation and electronic data fixation, the investigator often needs to completely capture application content beyond the visual range of the screen, such as long bills, chat records, emails or social media dynamics, and these content often directly relate to case core facts. However, the iOS system does not provide a native rolling screenshot function for a long time, the technical blank brings a significant obstacle to digital evidence collection work, and the traditional computer adopts the implementation scheme that a single screen screenshot is added with a manual sliding screen to continuously repeat the operation until the screenshot is finished, so that the evidence collection efficiency is low due to complicated operation, the integrity of evidence of missing and destruction of manual operation can also occur, and the legal availability of the evidence in a judicial program is influenced. Disclosure of Invention The invention aims to realize the automation, high-precision rolling screenshot and solid evidence of contents such as long pages, chat records and the like on the premise of not carrying out any software installation or system setting (such as starting a developer mode) on the iOS equipment. The embodiment of the invention provides an iOS device intelligent scrolling screenshot and security system based on Wi-Fi and Bluetooth cooperation, which comprises iOS devices, a PC running a security program and a wireless communication module connecting the iOS devices and the PC, wherein the wireless communication module comprises a Wi-Fi module and a Bluetooth module, the Wi-Fi module is used for establishing a high-bandwidth data channel, and the Bluetooth module is used for establishing a reverse control channel of the PC to the iOS devices; The fixing and verifying program running in the PC comprises the following steps: the protocol analysis module is used for establishing encryption screen-throwing connection with the iOS equipment through a reverse AirPlay protocol, and receiving and decrypting a real-time screen coding stream from the iOS equipment; The audio and video processing module is used for decoding and extracting frames from the real-time screen coding stream, outputting screen image frames and generating single-frame screenshot or screen recording files based on the screen image frames; The AI image processing module is used for analyzing the image frame of the screen, identifying the target content area, the interface element and the rolling state, judging the starting point and the ending point of the rolling screenshot and the accurate distance of each rolling based on an image similarity algorithm, and generating a control instruction of screen rolling; The Bluetooth control module is used for simulating man-machine interface equipment through a Bluetooth low-power-consumption protocol, converting a control instruction into a mouse control event which can be identified by the iOS equipment, and driving the screen of the iOS equipment to automatically scroll; The intelligent scrolling screenshot is realized by cooperation of the modules according to the following procedures that a protocol analysis module acquires a screen coding stream, an audio and video processing module decodes the screen coding stream into screen image frames, an AI image processing module analyzes the image frames and generates an instruction to trigger Bluetooth control module iOS equipment to drive a screen to scroll when the screen is required to scroll, after the scrolling is stopped and the picture is stable, the audio and video processing module executes screenshot storage, and the AI image processing module continuously analyzes new image frames and circularly executes the scrolling and screenshot process until the screenshot is judged to be completed. Optionally, the protocol parsing module is specifically configured to: Simulating the PC end as an AirPlay receiving end, and broadcasting simulated AirPlay receiving end equipment information in a local area network through Bonjour service; Performing TLS handshake with the iOS equipment initiating the screen image, and establishing an encrypted communication channel; receiving a real-time screen coding stream which is sent by the iOS equipment through a UDP proto