US-20260127397-A1 - BIOLOGICALLY INSPIRED ACTIVE VISUAL COMMUNICATION SYSTEM AND METHOD

US20260127397A1US 20260127397 A1US20260127397 A1US 20260127397A1US-20260127397-A1

Abstract

An exemplary system and method are disclosed for providing visual communication between agents in a multi-agent system. In some implementations, the exemplary system and method are configured to (i) receive and decode a data packet or message visually received from other communication targets and (ii) visually transmit a reply data packet or message, in response to the communication targets, using a digital or electromechanical display.

Inventors

Bert BRAS
Marc Weissburg
Bryan Cochran
Hanlong LI
Bryan Starbuck

Assignees

GEORGIA TECH RESEARCH CORPORATION

Dates

Publication Date: 20260507
Application Date: 20251103

Claims (20)

1 . A receiver comprising: a camera configured to acquire an image of a scene; and a receiver controller comprising: a receiver processor; and a receiver memory having receiver instructions stored thereon, wherein execution of the receiver instructions causes the receiver processor to: receive an image of the scene; determine presence of a first fiducial marker in the received image, wherein the first fiducial marker has an encoded data packet, transmitted optically via a transmitter, having a message; and determine, via a trained artificial intelligence (AI) model, one or more observable marker elements of the first fiducial marker, wherein the one or more observable marker elements are used to determine a packet structure for a reply message or a decoding of the first fiducial marker.
2 . The receiver of claim 1 , wherein the trained AI model was trained to detect and classify observable marker elements using a set of fiducial markers acquired from a set of images or a training dataset, wherein the AI model was trained using fiducial markers and an associated number of observable marker elements as the training data.
3 . The receiver of claim 1 , wherein the packet structure is configured to be used in a packet transmitted to a target, wherein the packet structure includes an indicator that the target does not need to respond, wherein the packet structure is mapped to an arrangement of the one or more observable marker elements.
4 . The receiver of claim 1 , wherein the packet structure is configured to be used in a packet transmitted to a target, wherein the packet structure includes an indicator that the target generates a one-time response, wherein the packet structure is mapped to an arrangement of the one or more observable marker elements.
5 . The receiver of claim 1 , wherein the packet structure is configured to be used in a packet transmitted to a target, wherein the packet structure includes an indicator that the target initiates a subsequent serial exchange of messages with the receiver, wherein the packet structure is mapped to an arrangement of the one or more observable marker elements.
6 . The receiver of claim 1 , wherein the first fiducial marker is generated or decoded according to a protocol defined by an adjustable AprilTag.
7 . The receiver of claim 1 , wherein the first fiducial marker is generated or decoded according to an adjustable protocol selected from or based on the group consisting of a QR code, a Ju marker, a Chroma tag, a Vu mark, a Topo tag, an S tag, and an ArUco tag.
8 . The receiver of claim 1 , wherein the one or more observable marker elements are subsequently used by a transmitter, the transmitter comprising: a transmitter controller comprising: a transmitter processor; and a transmitter memory having transmitter instructions stored thereon, wherein execution of the transmitter instructions causes the transmitter processor to: receive the one or more observable marker elements of the first fiducial marker; determine a packet structure of the encoded data packet using the one or more observable marker elements of the first fiducial marker, wherein the determined packet structure is mapped to the one or more observable marker elements of the first fiducial marker; determine a second fiducial marker having a second encoded data packet, wherein the second encoded data packet has same packet structure as the determined packet structure and includes the reply message; and demonstrate the second fiducial marker on a display, wherein the second fiducial marker is subsequently imaged for extracting the second encoded data packet.
9 . The receiver of claim 8 , wherein the display is an electromechanical display comprising: a plurality of tiles, each being configured to show an observable marker element of the second fiducial marker; and one or more actuators operatively coupled to the plurality of tiles, the one or more actuators being configured to flip the plurality of tiles in accordance with an arrangement of the one or more observable marker elements of the second fiducial marker.
10 . The receiver of claim 9 , wherein each tiles is a flapper or a printed placard.
11 . The receiver of claim 8 , wherein the display is a digital display having a plurality of pixels, each pixel being configured to show an observable marker element of the second fiducial marker.
12 . A transmitter comprising: a transmitter controller comprising: a transmitter processor; and a transmitter memory having transmitter instructions stored thereon, wherein execution of the transmitter instructions causes the transmitter processor to: receive one or more observable marker elements of a first fiducial marker from an external device or from a camera, wherein the first fiducial marker has an encoded data packet having a message; determine a packet structure of the encoded data packet using the one or more observable marker elements of the first fiducial marker, wherein the determined packet structure is mapped to the one or more observable marker elements of the first fiducial marker; determine a second fiducial marker having a second encoded data packet, wherein the second encoded data packet has same packet structure as the determined packet structure and includes a reply message; and demonstrate the second fiducial marker on a display, wherein the second fiducial marker is subsequently imaged for extracting the second encoded data packet.
13 . The transmitter of claim 10 , wherein the display is an electromechanical display comprising: a plurality of tiles, each being configured to show an observable marker element of the second fiducial marker; and one or more actuators operatively coupled to the plurality of tiles, the one or more actuators being configured to flip the plurality of tiles in accordance with an arrangement of the one or more observable marker elements of the second fiducial marker.
14 . The transmitter of claim 13 , wherein each tile is a flapper or a printed placard.
15 . The receiver of claim 12 , wherein the display is a digital display having a plurality of pixels, each pixel being configured to show an observable marker element of the second fiducial marker.
16 . A non-transitory computer-readable medium having instructions stored thereon, wherein execution of the instructions causes a receiver processor to: receive an image of a scene acquired by a camera; determine presence of a first fiducial marker in the received image, wherein the first fiducial marker has an encoded data packet, transmitted optically via a transmitter, having a message; and determine, via a trained artificial intelligence (AI) model, one or more observable marker elements of the first fiducial marker, wherein the one or more observable marker elements are used to determine a packet structure for a reply message or a decoding of the first fiducial marker.
17 . The non-transitory computer-readable medium of claim 16 , wherein the trained AI model was trained to detect and classify observable marker elements using a set of fiducial markers acquired from a set of images or a training dataset, wherein the AI model was trained using fiducial markers and an associated number of observable marker elements as the training data.
18 . The non-transitory computer-readable medium of claim 16 , wherein the first fiducial marker is generated or decoded according to a protocol defined by an adjustable AprilTag.
19 . The non-transitory computer-readable medium of claim 16 , wherein the execution of the instructions further causes a transmitter processor to: receive the one or more observable marker elements of the first fiducial marker; determine a packet structure of the encoded data packet using the one or more observable marker elements of the first fiducial marker, wherein the determined packet structure is mapped to the one or more observable marker elements of the first fiducial marker; determine a second fiducial marker having a second encoded data packet, wherein the second encoded data packet has same packet structure as the determined packet structure and includes the reply message; and demonstrate the second fiducial marker on a display, wherein the second fiducial marker is subsequently imaged for extracting the second encoded data packet
20 . The non-transitory computer-readable medium of claim 19 , wherein the display is an electromechanical display comprising: a plurality of tiles, each being configured to show an observable marker element of the second fiducial marker; and one or more actuators operatively coupled to the plurality of tiles, the one or more actuators being configured to flip the plurality of tiles in accordance with an arrangement of the one or more observable marker elements of the second fiducial marker.

Description

RELATED APPLICATION This application claims priority to, and the benefit of, U.S. Provisional Patent Application No. 63/715,082, filed Nov. 1, 2024, entitled “BIOLOGICALLY INSPIRED ACTIVE VISUAL COMMUNICATION SYSTEM,” which is incorporated by reference herein in its entirety. BACKGROUND Communication systems rely on analog or digital modalities to transmit information between entities. Analog communication conveys signals in a continuous form, while digital communication encodes information into discrete packets transmitted across structured networks. In digital communication, a data packet is a predefined unit of data that includes both payload and control information, facilitating reliable and efficient transmission. Packet-based communication is foundational to modem networking, facilitating distributed systems to coordinate actions, share data, and respond to dynamic environments. Multi-agent systems (e.g., fleets of autonomous robots or distributed platforms) operate in environments where current communication channels (e.g., radio frequency, audio) may be unreliable, bandwidth-limited, or subject to interference, hindering exchanges of data packets or messages between the agents. Therefore, an alternative modality for inter-agent communication is desirable. There is a benefit to developing a communication system and method that can function in constrained or degraded environments to provide coordination and interaction among autonomous agents. SUMMARY An exemplary system and method are disclosed for providing visual communication between agents in a multi-agent system. In some implementations, the exemplary system and method are configured to (i) receive and decode a data packet or message visually received from other communication targets and (ii) visually transmit a reply data packet or message, in response to the communication targets, using a digital or electromechanical display. Inspired by biological examples of non-verbal communication (e.g., signaling behaviors of honeybees, primates, and other animals), the exemplary system and method encode information into visually perceptible formats (e.g., a fiducial marker on a display) that can be interpreted among agents in a multi-agent system. The exemplary system and method facilitate agents to (i) visually receive and decode a data packet or message from one another, and (ii) visually transmit, to one another, a reply data packet or message using a digital or electromechanical display. By adopting visual communication, the exemplary system and method improve the robustness and adaptability of inter-agent communication, particularly in environments where current communication channels (e.g., radio frequency, audio) are unreliable or unavailable. This represents an improvement in computer and communication technology by introducing a visual communication that enhances coordination and interaction in distributed autonomous systems. The exemplary system and method can be employed for communication between agents in a multi-agent system, including but not limited to (i) unmanned aerial vehicles (UAVs) (e.g., drones, quadcopters, etc.), (ii) autonomous ground vehicles (e.g., autonomous tractors, autonomous haulage vehicles, etc.), (iii) unmanned sea vehicles (USVs) (e.g., saildrones, autonomous submarines, etc.), (iv) autonomous or teleoperated robots, and (v) manned communication operation (e.g., personnel equipped with transceivers or user interfaces). The exemplary system is operable in environments where current communication methods (e.g., radio, audio) are not feasible, and can interface with either software-based agents or human-operated devices to encode and decode visual messages for bidirectional communication. In an aspect, a receiver is disclosed comprising: a camera configured to acquire an image of a scene; and a receiver controller comprising: a receiver processor; and a receiver memory having receiver instructions stored thereon, wherein execution of the receiver instructions causes the receiver processor to: receive an image of the scene; determine presence of a first fiducial marker in the received image, wherein the first fiducial marker has an encoded data packet, transmitted optically via a transmitter, having a message; and determine, via a trained artificial intelligence (AI) model (e.g., CNN), one or more observable marker elements of the first fiducial marker, wherein the one or more observable marker elements are used to determine a packet structure for a reply message or a decoding of the first fiducial marker. In some embodiments, the trained AI model was trained to detect and classify observable marker elements using a set of fiducial markers (e.g., QR code, etc.) acquired from a set of images or a training dataset, wherein the AI model was trained using fiducial markers and an associated number of observable marker elements as the training data. In some embodiments, the packet structure is configured to be used in a packet t