CN-122002007-A - Synchronous acquisition and fusion communication method and system for dual video streams of intelligent glasses and mobile phone

CN122002007ACN 122002007 ACN122002007 ACN 122002007ACN-122002007-A

Abstract

The invention discloses a method and a system for synchronously acquiring, fusing and communicating double video streams of intelligent glasses and mobile phones, which relate to the technical field of video communication and live broadcasting, and comprise the following steps that S1, the intelligent glasses and a mobile terminal establish a special communication link through Wi-Fi to finish equipment pairing and communication parameter adaptation; S2, the mobile terminal sends a synchronous acquisition instruction through the link, and triggers the second image acquisition module to start simultaneously with the first image acquisition module of the intelligent glasses, so as to acquire the third person and the first person visual angle videos respectively. According to the intelligent glasses and mobile phone dual video stream synchronous acquisition fusion communication method and system, synchronous capturing and seamless fusion of dual-view video are achieved, immersion and interactivity of video communication and live broadcast are remarkably enhanced, visual angle limitation of a traditional single camera device is broken through wireless cooperative work of the intelligent glasses and mobile phone, and richer and comprehensive visual experience is provided for audiences.

Inventors

WANG YONGWEN
WANG HAO
XU YUNJIA

Assignees

瑟尔新数字科技(江苏)有限公司

Dates

Publication Date: 20260508
Application Date: 20260122

Claims (10)

1. The intelligent glasses and mobile phone double video stream synchronous acquisition fusion communication method is characterized by comprising the following steps of: s1, establishing a special communication link between the intelligent glasses and the mobile terminal through Wi-Fi to finish equipment pairing and communication parameter adaptation; s2, the mobile terminal sends a synchronous acquisition instruction through the link, and triggers a second image acquisition module of the mobile terminal and a first image acquisition module of the intelligent glasses to be started simultaneously so as to acquire visual angle videos of a third person and a first person respectively; s3, the intelligent glasses encode the collected first video stream in real time and transmit the first video stream to the mobile terminal through Wi-Fi; S4, the mobile terminal receives the first video stream and the second video stream acquired by the mobile terminal, and synchronous calibration is carried out on the two paths of video streams by utilizing a local time stamp and frame level alignment technology; s5, the mobile terminal provides a plurality of layout modes and supports a user to switch the display priority of the main attached picture; S6, selecting a vector marking tool by touch operation on a preview or live broadcast interface of the mobile terminal by a user, and overlapping the marking on a video picture in real time; s7, the mobile terminal recodes the fusion video stream after the synchronization, layout and marking processes and sends the recoding video stream to the receiving end through a network.
2. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phones according to claim 1, wherein S1 comprises: the mobile terminal starts Wi-Fi function, searches the special communication equipment signal around, screens out the broadcasting signal of the target intelligent glasses and initiates a connection request; after receiving a connection request of the mobile terminal, the intelligent glasses send pairing verification information to the mobile terminal, wherein the verification mode is that the unique identification of the equipment is compared or a preset password is input by a user, and the pairing and binding of the equipment are completed after verification is passed; After successful pairing, the two parties automatically negotiate a transmission protocol; synchronizing system clock references of the two devices; Negotiating a video coding format, detecting the current network environment, setting a data transmission bandwidth threshold and delay compensation parameters, generating a communication parameter configuration file and synchronizing the communication parameter configuration file to the two devices.
3. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phones according to claim 2, wherein S2 comprises: The mobile terminal generates a synchronous acquisition instruction containing acquisition resolution, frame rate and exposure parameters based on the established special communication link, and embeds a local reference time stamp; The mobile terminal transmits the synchronous acquisition instruction to the intelligent glasses through a special link, and simultaneously starts the instruction sending timing; After the intelligent glasses receive the instruction, decoding and extracting configuration information and a reference time stamp, triggering a first image acquisition module to start by a main processing chip, and initializing an acquisition state according to configuration parameters; after the intelligent glasses start to collect, a ready signal is fed back to the mobile terminal, and the mobile terminal immediately activates a second image collecting module after receiving the feedback signal; The mobile terminal judges the starting time difference of the two parties through the timing difference value, and if the difference value exceeds a set threshold value, the mobile terminal resends the synchronous instruction; After the two-party acquisition modules work stably, the intelligent glasses acquire the first-person visual angle video, and the mobile terminal acquires the third-person visual angle video.
4. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phone according to claim 3, wherein S3 comprises: the main processing chip of the intelligent glasses receives the original video data transmitted by the first image acquisition module and starts a hardware coding engine; The coding engine performs frame-level compression on the original video data according to the negotiated format, and simultaneously distributes a unique frame number and an acquisition time stamp for each frame of video, and associates the unique frame number and the acquisition time stamp with the corresponding video frame; Starting a transmission buffer queue, and storing the coded video frames into the queue according to the sequence of frame numbers; Based on the real-time transmission state of the Wi-Fi link, video frames in the queue are processed in a slicing mode, and each piece of data is transmitted after a check code is added; and setting a data retransmission mechanism, and if the mobile terminal feeds back that a certain fragment is lost, the intelligent glasses call the corresponding fragment to retransmit from the buffer queue.
5. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phones according to claim 4, wherein S4 comprises: the mobile terminal receives a first video stream transmitted by the intelligent glasses through a special link, simultaneously collects a second video stream of the mobile terminal, and respectively establishes frame data storage buffer areas of two paths of video streams; extracting acquisition time stamps from each frame of data of two paths of video streams, and generating a first time stamp sequence and a second time stamp sequence; Calculating the time difference value of corresponding frames in the two time stamp sequences, setting a synchronous threshold value, and screening out frame data of which the time difference value exceeds the threshold value; The super-threshold frames are adjusted in a frame delay buffering or redundant frame discarding mode, and frame-level time stamps of the first video stream and the second video stream are matched and aligned frame by frame; and performing picture dislocation calibration on the two paths of aligned video streams.
6. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phone according to claim 5, wherein S5 comprises: The control APP of the mobile terminal presets a plurality of layout modes, and corresponding layout selection controls are set on the user interaction interface; The user selects a target layout mode through a touch control, and after receiving a selection instruction, the APP reads screen resolution parameters of the mobile terminal; distributing display areas of two paths of video streams according to the selected layout mode and screen resolution; Performing self-adaptive scaling on the picture sizes of the two paths of video streams, and keeping the original picture proportion; and rendering and displaying the two paths of adjusted video stream pictures according to the selected layout mode.
7. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phone according to claim 6, wherein S5 further comprises: setting a main picture switching control on the interactive interface of the mobile terminal, and associating display priority identifiers of the first video stream and the second video stream; Triggering a switching control by a user, and reading the display state of the current main attached picture after the APP receives the switching instruction; Exchanging the display priority of two paths of video streams, switching the original auxiliary picture into a main picture and expanding the main picture to a corresponding display area, switching the original main picture into the auxiliary picture and adjusting the size; Updating the interface display state.
8. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phone according to claim 7, wherein S6 comprises: A marking toolbar is arranged on a preview or live broadcast interface of the mobile terminal, and comprises a plurality of vector marking tools, and a user selects a target tool through touching; the user touches and slides on the interface to draw a mark track, and the APP captures touch coordinate data in real time to generate a continuous track point sequence; converting the track point sequence into Bezier curve vector graphic data, and removing redundant coordinate points; providing a marking style custom option; The processed vector marking data are overlapped to the corresponding coordinate position of the video picture in real time; And providing a mark revocation and clearing function.
9. The method for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phone according to claim 8, wherein S7 comprises: The mobile terminal recodes the fusion video stream with synchronous calibration, layout adjustment and mark superposition completed; Detecting the current network bandwidth state in real time, and dynamically adjusting the coding rate according to the bandwidth data; Selecting a corresponding transmission protocol; Encapsulating the coded fusion video stream according to a protocol format, and transmitting the encapsulated fusion video stream to a receiving end in fragments; monitoring the network transmission state in real time, and adjusting the video resolution according to the bandwidth condition; if the feedback data of the receiving end is lost, triggering a local retransmission mechanism and supplementing the lost fragments.
10. Intelligent glasses and mobile phone double video stream synchronous acquisition and fusion communication system, which is characterized by comprising: the system comprises a communication connection module, a video acquisition synchronization module, a video fusion processing module, a user interaction module and a code transmission module; The communication connection module is used for setting up a special link between the intelligent glasses and the mobile terminal through Wi-Fi to finish equipment pairing and communication parameter adaptation, and provides a basis for double video streaming; The video acquisition synchronization module is electrically connected with the communication connection module and is used for receiving a mobile terminal synchronization instruction, triggering the two image acquisition modules to be started simultaneously, respectively acquiring the double-view video and realizing transmission synchronization; the video fusion processing module is electrically connected with the video acquisition synchronization module and is used for receiving two paths of video streams, calibrating the video streams through a time stamp and frame level alignment technology and combining the video streams with a selected layout mode to finish video fusion; The user interaction module is electrically connected with the video fusion processing module and is used for providing layout selection and main attached picture switching functions, and supporting a user to add vector marks through touch operation and superimpose in real time; the coding transmission module is electrically connected with the user interaction module and is used for recoding the fused video stream and sending the recoded video stream to the opposite terminal of the call or the receiving terminal of the live broadcast platform through a network.

Description

Synchronous acquisition and fusion communication method and system for dual video streams of intelligent glasses and mobile phone Technical Field The invention relates to the technical field of video communication and live broadcast, in particular to a method and a system for synchronously acquiring and integrating double video streams of intelligent glasses and mobile phones. Background With the continuous development of video communication and live broadcasting technology, the application scene of the video communication and live broadcasting technology is widely used in a plurality of fields such as teaching demonstration, product evaluation, outdoor activities and the like. In the actual use process, users gradually generate the core requirement of double-view synchronous display, so that the users not only hope to present the first-person real-time picture seen by themselves and intuitively transfer scene and details, but also need to display the third-person state of themselves in the environment, so that the audience can more comprehensively know the whole situation, and the display mode with the complementary double views can remarkably improve the content expressive force and interactive experience. Currently, the mainstream video acquisition scheme mostly depends on a single camera device, the visual angle is relatively fixed, and the requirement of synchronous presentation of double visual angles is difficult to meet. Even if partial schemes support multi-camera switching, the multi-camera switching method is also mostly limited between different cameras of the same equipment, and obvious limitation exists in view angle expansion. Meanwhile, a scheme for conveniently realizing the collaborative work of the intelligent glasses and the mobile terminal and other double independent devices is lacking in the prior art, a mature solution is not formed in the aspects of synchronous acquisition, low-delay fusion, stable transmission and the like of double video streams, and the use requirements of users in diversified scenes cannot be fully adapted. In this regard, we propose a method and a system for synchronous acquisition and fusion communication of dual video streams of smart glasses and mobile phones. Disclosure of Invention In order to solve the technical problems, the technical scheme solves the problems that the prior art relies on single camera equipment or multiple cameras of the same equipment, the visual angle is limited, double visual angle synchronous presentation is difficult to realize, the cooperative scheme of double independent equipment such as intelligent glasses and mobile terminals is lacking, the mature solution is not available in the aspects of double video stream synchronous acquisition, low-delay fusion and stable transmission, and the requirement of diversified scenes of users cannot be adapted. In order to achieve the above purpose, the invention adopts the following technical scheme: the intelligent glasses and mobile phone dual video stream synchronous acquisition fusion communication method comprises the following steps: s1, establishing a special communication link between the intelligent glasses and the mobile terminal through Wi-Fi to finish equipment pairing and communication parameter adaptation; s2, the mobile terminal sends a synchronous acquisition instruction through the link, and triggers a second image acquisition module of the mobile terminal and a first image acquisition module of the intelligent glasses to be started simultaneously so as to acquire visual angle videos of a third person and a first person respectively; s3, the intelligent glasses encode the collected first video stream in real time and transmit the first video stream to the mobile terminal through Wi-Fi; S4, the mobile terminal receives the first video stream and the second video stream acquired by the mobile terminal, and synchronous calibration is carried out on the two paths of video streams by utilizing a local time stamp and frame level alignment technology; s5, the mobile terminal provides a plurality of layout modes and supports a user to switch the display priority of the main attached picture; S6, selecting a vector marking tool by touch operation on a preview or live broadcast interface of the mobile terminal by a user, and overlapping the marking on a video picture in real time; s7, the mobile terminal recodes the fusion video stream after the synchronization, layout and marking processes and sends the recoding video stream to the receiving end through a network. Preferably, the S1 includes: the mobile terminal starts Wi-Fi function, searches the special communication equipment signal around, screens out the broadcasting signal of the target intelligent glasses and initiates a connection request; after receiving a connection request of the mobile terminal, the intelligent glasses send pairing verification information to the mobile terminal, wherein the verification mode is that the