CN-115775364-B - Artificial intelligence method, system and storage medium for predicting from image sequence

CN115775364BCN 115775364 BCN115775364 BCN 115775364BCN-115775364-B

Abstract

Embodiments of the present disclosure provide artificial intelligence methods and systems for prediction from a sequence of images. The method may include receiving a sequence of images acquired at different points in time. The method may further include applying a stabilization model to process the image sequence for prediction. The working model is trained with the stabilization model and the plastic model. The training enhances the consistency between the working model, the stabilizing model and the plastic model. The working model is trained using a loss function that includes a cross entropy loss of the combination of training batches and memory samples and a consistency loss across memory samples.

Inventors

Elie Alani
Fahd safraz
Bahram zonuz

Assignees

北京四维图新科技股份有限公司

Dates

Publication Date: 20260508
Application Date: 20220613
Priority Date: 20210908

Claims (9)

1. An artificial intelligence method of predicting from a sequence of images, comprising: Receiving said sequence of images acquired at different points in time, and Applying a stabilization model to process the image sequence for the prediction, wherein a working model is trained with the stabilization model and a plastic model, wherein the training enhances consistency between the working model, the stabilization model, and the plastic model, wherein the working model is trained using a loss function comprising a combined cross entropy loss of training batches and memory samples and a consistency loss across the memory samples; wherein a training batch is received from a data stream and a memory sample is received from a context memory; Training the working model based on a loss function that emphasizes consistency of the working model, the stabilization model, and the plastic model over the memory samples, wherein the loss function comprises a cross entropy loss of a combination of the training batch and the memory samples and a consistency loss over the memory samples; updating the stability model and the plasticity model based on the working model; determining that the stability model satisfies training conditions, and Providing the stable model as the artificial intelligence inference model; The system comprises a working model, a stability model, a plastic model and a working model, wherein the working model is used for memorizing events similar to a plot and learning a statistical structure of a perception event, the stability model is used for slowly learning structural knowledge, keeping long-term semantic memory of experience events and providing an effective representation of cross tasks, the plastic model is used for quickly learning the latest experience, keeping short-term semantic memory of experience events and providing an adaptive and effective representation of the latest tasks, the stability model and the plastic model are complementary working models, and the working model receives feedback from the stability model and the plastic model.
2. The artificial intelligence method of claim 1, wherein the training updates the stable model based on the working model at a first training rate and updates the plastic model at a second training rate, the first training rate being slower than the second training rate.
3. The artificial intelligence method of claim 1 or 2, wherein the training batch is from a data stream and the memory samples are from a context memory bank.
4. The artificial intelligence method of claim 1 or 2, wherein the consistency penalty is based on logits generated by the working model on the memory sample and a replay logits selected for the memory sample from the plastic model or the stable model.
5. The artificial intelligence method of claim 4, wherein the loss function is a weighted combination of the cross entropy loss and the consistency loss, wherein the consistency loss is a mean square error between the logits generated by the working model on the memory sample and the replay logits from the plastic model or the stable model.
6. The artificial intelligence method of claim 2, wherein the parameters of the stability model are updated using an exponentially weighted average of the parameters of the working model, the working model having a first decay parameter at the first training rate; The parameters of the plastic model are updated using an exponentially weighted average of the parameters of the working model, the working model having a second decay parameter at the second training rate.
7. An artificial intelligence system for predicting a sequence of images acquired from an image acquisition device at different points in time, comprising: A storage device for storing a stability model, wherein a working model is trained with the stability model and a plasticity model, wherein the training enhances consistency between the working model, the stability model, and the plasticity model, wherein the working model is trained using a loss function comprising a cross entropy loss of a combination of training batches and memory samples and a consistency loss across the memory samples, and A processor for: applying the stabilization model to process the sequence of images to make the prediction; wherein a training batch is received from a data stream and a memory sample is received from a context memory; Training the working model based on a loss function that emphasizes consistency of the working model, the stabilization model, and the plastic model over the memory samples, wherein the loss function comprises a cross entropy loss of a combination of the training batch and the memory samples and a consistency loss over the memory samples; updating the stability model and the plasticity model based on the working model; determining that the stability model satisfies training conditions, and Providing the stable model as the artificial intelligence inference model; The system comprises a working model, a stability model, a plastic model and a working model, wherein the working model is used for memorizing events similar to a plot and learning a statistical structure of a perception event, the stability model is used for slowly learning structural knowledge, keeping long-term semantic memory of experience events and providing an effective representation of cross tasks, the plastic model is used for quickly learning the latest experience, keeping short-term semantic memory of experience events and providing an adaptive and effective representation of the latest tasks, the stability model and the plastic model are complementary working models, and the working model receives feedback from the stability model and the plastic model.
8. The artificial intelligence system of claim 7, wherein the stability model is updated at a first training rate and the plasticity model is updated at a second training rate, the first training rate being slower than the second training rate.
9. A non-transitory computer readable storage medium storing instructions which, when executed by a processor, are configured to implement the method of any one of claims 1-6.

Description

Artificial intelligence method, system and storage medium for predicting from image sequence Technical Field The present disclosure relates to methods and systems for predicting from a sequence of images using a training model. The present disclosure also relates to methods and systems for training such predictive models. More particularly, the present disclosure relates to artificial intelligence inference models trained using a stability model and a plasticity model for making predictions. The model is applicable to computer vision applications. The present disclosure also relates to continuously acquiring and consolidating knowledge from non-stationary data streams. Background Dynamic image processing techniques are widely used in applications such as autopilot, surveillance, medical imaging, and the like. The dynamic image data essentially comprises a sequence of images acquired at different points in time of capturing the dynamic environment. Machine learning methods such as deep neural networks (deep neural network, DNN) have been developed in the field of computer vision to process images, such as still images, and make intelligent predictions based thereon. However, most DNN methods do not fully utilize the knowledge obtained by processing the previous image frames, resulting in the use of a continuous learning method. Humans are adept at continuously learning and accumulating and consolidating knowledge from ever changing environments, which remains a challenge for DNN. Continuous learning (continual learning, CL) refers to the ability of a learning agent to continually interact with a dynamic environment and process information streams to obtain new knowledge while consolidating and retaining previously acquired knowledge. One of the major challenges in implementing continuous learning in DNN is that continuous acquisition of incremental availability information from a non-stationary data distribution often results in catastrophic forgetting or interference, whereby the performance of the model on previously learned tasks drops dramatically as new tasks are learned. The continuous learning method aims to solve the problem of catastrophic forgetting in DNN and realize effective continuous learning. Several approaches have been proposed to address the catastrophic forgetfulness problem in CL. These methods can be broadly classified into regularization-based methods that penalize network weight changes (regularization-based methods), network expansion-based methods that dedicate a different set of network parameters to different tasks (network expansion-based methods), and exercise-based methods that maintain memory buffers and replay samples in previous tasks (rehearsal-based methods). Among other things, exercise-based methods have proven to be more effective in challenging CL tasks. In particular, current experience replay methods, dark experience replay (Dark Experience Replay, DER) preserve the network response during the entire optimization trajectory and increase the consistency penalty on the basis of experience replay (Experience Replay, ER). However, the best way to replay memory samples and constraint model updates to effectively accumulate knowledge remains a problem that remains open. Disclosure of Invention To address these and other problems with existing DNN approaches, the present disclosure provides improved methods and systems that train and use artificial intelligence inference models that can leverage interactions between fast instance-based learning and slow structured learning. Novel methods and systems for a complementary learning system based experience replay (CLS-ER) method are disclosed. In one aspect, embodiments of the present disclosure provide an artificial intelligence method of predicting from a sequence of images. The method may include receiving a sequence of images acquired at different points in time. The method may further include applying a stabilization model to process the image sequence for prediction. The working model is trained with the stabilization model and the plastic model. The training enhances the consistency between the working model, the stabilizing model and the plastic model. The working model is trained using a loss function that includes a cross entropy loss of the combination of training batches and memory samples and a consistency loss across memory samples. In another aspect, embodiments of the present disclosure provide an artificial intelligence system for predicting a sequence of images acquired from an image acquisition device at different points in time. The system may include a storage device for storing a stability model with which the working model is trained. The training enhances the consistency between the working model, the stabilizing model and the plastic model. The working model is trained using a loss function that includes a cross entropy loss of the combination of training batches and memory samples and a consistency loss