US-12617093-B2 - Real-time robotic control using digital twin models

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing custom real-time control using a digital twin model. One of the methods include receiving a definition of a custom real-time action to be performed in real time by the real-time robotic control system to control a robot; and executing the custom real-time action according to the definition at each tick of a real-time control cycle to effectuate real-time operation of the robot, including: obtaining, by a real-time control process, values of one or more digital twin variables injected from the digital twin model into memory accessible by the real-time control process, computing updated real-time command information according to the definition of the custom real-time action and the values of the one or more digital twin variables injected from the digital twin model, and controlling the robot using the computed real-time command information.

Inventors

• Michael Beardsworth
• Stoyan Gaydarov

Assignees

• INTRINSIC INNOVATION LLC

Dates

Publication Date

20260505

Application Date

20231219

Claims (20)

1. 1 . A computer-implemented method performed by a real-time robotic control system, the method comprising: receiving a definition of a custom real-time action to be performed in real time by the real-time robotic control system to control a robot, wherein the definition of the custom real-time action references one or more variables of a digital twin model maintained in a non-real-time process; and executing the custom real-time action according to the definition at each tick of a real-time control cycle to effectuate real-time operation of the robot, including: obtaining, by a real-time control process, values of one or more digital twin variables injected from the digital twin model into memory accessible by the real-time control process, computing real-time command information according to the definition of the custom real-time action and the values of the one or more digital twin variables injected from the digital twin model, and controlling the robot using the computed real-time command information.
2. 2 . The method of claim 1 , wherein the one or more digital twin variables represent a non-robot state of an object in an operating environment.
3. 3 . The method of claim 2 , wherein the one or more digital twin variables represent a position of the object in the environment.
4. 4 . The method of claim 1 , further comprising: repeatedly injecting, by the non-real-time process, values of the one or more digital twin variables into the memory accessible by the real-time control process.
5. 5 . The method of claim 4 , wherein repeatedly injecting the values occurs at a rate that is slower than a rate of the real-time control cycle.
6. 6 . The method of claim 1 , wherein obtaining, by the real-time control process, the values of the one or more digital twin variables comprises obtaining the values during idle time of the real-time control cycle.
7. 7 . The method of claim 1 , wherein the definition of the custom real-time action comprises source code having variable names that reference objects maintained by the digital twin model in the non-real-time process.
8. 8 . The method of claim 7 , wherein the definition of the custom real-time action also references objects belonging to the real-time control process.
9. 9 . A system comprising: one or more processors and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a definition of a custom real-time action to be performed in real time by the real-time robotic control system to control a robot, wherein the definition of the custom real-time action references one or more variables of a digital twin model maintained in a non-real-time process; and executing the custom real-time action according to the definition at each tick of a real-time control cycle to effectuate real-time operation of the robot, including: obtaining, by a real-time control process, values of one or more digital twin variables injected from the digital twin model into memory accessible by the real-time control process, computing real-time command information according to the definition of the custom real-time action and the values of the one or more digital twin variables injected from the digital twin model, and controlling the robot using the computed real-time command information.
10. 10 . The system of claim 9 , wherein the one or more digital twin variables represent a non-robot state of an object in an operating environment.
11. 11 . The system of claim 10 , wherein the one or more digital twin variables represent a position of the object in the environment.
12. 12 . The system of claim 9 , wherein the operations further comprise: repeatedly injecting, by the non-real-time process, values of the one or more digital twin variables into the memory accessible by the real-time control process.
13. 13 . The system of claim 12 , wherein repeatedly injecting the values occurs at a rate that is slower than a rate of the real-time control cycle.
14. 14 . The system of claim 9 , wherein obtaining, by the real-time control process, the values of the one or more digital twin variables comprises obtaining the values during idle time of the real-time control cycle.
15. 15 . The system of claim 9 , wherein the definition of the custom real-time action comprises source code having variable names that reference objects maintained by the digital twin model in the non-real-time process.
16. 16 . The system of claim 15 , wherein the definition of the custom real-time action also references objects belonging to the real-time control process.
17. 17 . A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a definition of a custom real-time action to be performed in real time by the real-time robotic control system to control a robot, wherein the definition of the custom real-time action references one or more variables of a digital twin model maintained in a non-real-time process; and executing the custom real-time action according to the definition at each tick of a real-time control cycle to effectuate real-time operation of the robot, including: obtaining, by a real-time control process, values of one or more digital twin variables injected from the digital twin model into memory accessible by the real-time control process, computing real-time command information according to the definition of the custom real-time action and the values of the one or more digital twin variables injected from the digital twin model, and controlling the robot using the computed real-time command information.
18. 18 . The non-transitory computer storage medium of claim 17 , wherein the one or more digital twin variables represent a non-robot state of an object in an operating environment.
19. 19 . The non-transitory computer storage medium of claim 18 , wherein the one or more digital twin variables represent a position of the object in the environment.
20. 20 . The non-transitory computer storage medium of claim 18 , wherein the operations further comprise: repeatedly injecting, by the non-real-time process, values of the one or more digital twin variables into the memory accessible by the real-time control process.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63/435,677, filed on Dec. 28, 2022. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND This specification relates to frameworks for software control systems. Real-time software control systems are software systems that must execute within strict timing requirements to achieve normal operation. The timing requirements often specify that certain actions must be executed or outputs must be generated within a particular time window in order for the system to avoid entering a fault state. In the fault state, the system can halt execution or take some other action that interrupts normal operation. Such real-time software control systems are often used to control physical machines that have high precision and timing requirements. As one example, a workcell of industrial robots can be controlled by a real-time software control system that requires each robot to repeatedly receive commands at a certain frequency, e.g., 1, 10, or 100 kHz. If one of the robots does not receive a command during one of the periodic time windows, the robot can enter a fault state by halting its operation or by automatically executing a recovery procedure to return to a maintenance position. In this specification, a workcell is the physical environment in which a robot will operate. Workcells have particular physical properties, e.g., physical dimensions that impose constraints on how robots can move within the workcell. Due to such timing requirements, software control systems for physical machines are often implemented by closed software modules that are configured specifically for highly-specialized tasks. For example, a robot that picks components for placement on a printed circuit board can be controlled by a closed software system that controls each of the low-level picking and placing actions. SUMMARY This specification describes a real-time robotic control framework that implements a digital twin model that corresponds to an operating environment. The digital twin model stores a set of digital twin variables each representing a state of an aspect of the operating environment. In this specification, a framework is a software system that allows a user to provide higher level program definitions while implementing the lower level control functionality of a real-time robotic system. In this specification, the operating environment includes multiple subsystems, each of which can include one or more real-time robots, one or more computing devices having software or hardware modules that support the operation of the robots, or both. The framework provides mechanisms for bridging, communication, or coordination between the multiple systems, including forwarding control parameters from a robot application system, providing sensor measurements to a real-time robotic control system for use in computing the custom action, and receiving hardware control inputs computed for the custom action from the real-time robotic control system, all while maintaining the tight timing constraints of the real-time robot control system, e.g., at the order of one millisecond. Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The disclosed real-time robotic control framework provides capabilities for a robot to react in a timely and comprehensive manner. By internally maintaining not only the kinematic variables of the robots in an operating environment but also pose variables of objects in the operating environment, the control framework stores a more accurate digital twin model that comes closer to a one-to-one correspondence with the actual operating environment. With this digital twin model, the control framework can provide robot commands by reasoning about the relationship between objects in the operating environment and the state of the robot itself even within tight timing constraints. The disclosed control framework thus facilitates easier and faster development of custom actions and reactions to be executed by the robot that fit the needs of particular robotic tasks. These actions and reactions can be defined in customized terms to different processes being implemented, rather than in abstract and/or obscure sensor signals. Compared to existing solutions, some implementations of the framework allow a user to define such custom actions and reactions with less amount of code, as well as with reusable variable names that improve code maintainability. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF