US-12620224-B2 - Bird detection and species determination

Abstract

Methods of determining the species of birds in flight are provided along with corresponding systems. A method may include capturing a video stream of a bird in flight using at least one camera, generating a first species probability estimate by delivering images from the video stream to a neural network that has been trained to recognize species of birds from images, obtaining additional parameters from the video stream or from additional data, generating a second species probability estimate by delivering the additional parameters as input to a domain knowledge module with a domain knowledge statistical model, and generating a final species probability estimate by combining the first species probability estimate and the second species probability estimate. The additional parameters may include geometry features related to movement of the bird in flight, or parameters relating to the environment.

Inventors

• Ask HELSETH
• Helge REIKERÅS

Assignees

• SPOOR AS

Dates

Publication Date

20260505

Application Date

20220412

Priority Date

20210415

Claims (20)

1. 1 . A method of determining the species of birds in flight, comprising: capturing at least one video stream of a bird in flight using at least one camera; generating a first species probability estimate by delivering images from the at least one video stream as input to an artificial neural network that has been trained to recognize species of birds from images; delivering images from the at least one video stream as input to a computer-executed geometry feature extraction software module that outputs extracted geometric features related to the bird in flight in the at least one video stream; delivering the extracted geometric features outputted from the computer-executed geometry extraction software module as input to a computer-executed domain knowledge software module utilizing a domain knowledge statistical model to output a second species probability estimate based on the extracted geometric features; and generating a final species probability estimate by combining the first species probability estimate and the second species probability estimate, wherein the extracted geometric features outputted from the computer-executed geometry extraction software module are selected from the group consisting of: size, positions, acceleration, vertical motion, flight trajectory, and wingbeat frequency.
2. 2 . The method according to claim 1 , further comprising: generating the first species probability estimate by delivering extracted features from the artificial neural network and the extracted geometric features outputted from the computer-executed geometry feature extraction software module as input to a shallow neural network that has been trained to generate bird species probabilities based on features extracted by an artificial neural network combined with observed geometric features.
3. 3 . The method according to claim 2 , wherein the extracted geometric features are obtained based on identification of the same bird in a sequence of images from the at least one video stream, and estimating motion based on the change of the identified bird's position between images in the sequence of images.
4. 4 . The method according to claim 2 , wherein the at least one camera is two or more cameras and the at least one video stream is two or more video streams; wherein the extracted geometric features are obtained based on a known position of each camera, identification of the same bird in two or more sequences of images from two or more concurrent video streams, determination of the position of the identified bird in the respective images of the respective video streams, and using multi-view geometry analysis to determine 3D coordinates representative of positions of the identified bird relative to the positions of the cameras from the determined positions in the respective images of the respective video streams.
5. 5 . The method according to claim 2 , wherein one extracted geometric feature is a wingbeat frequency determined by performing Fourier analysis on a sequence of images from the at least one video stream, and identifying a dominant frequency component that is inside a frequency interval consistent with wingbeat frequencies for birds.
6. 6 . The method according to claim 1 , further comprising: training the artificial neural network by delivering a dataset including labeled images of relevant bird species as input to the artificial neural network.
7. 7 . The method according to claim 1 , further comprising: performing object detection on images from the at least one video stream and annotating the images with bounding boxes drawn around each object that is identified as a bird.
8. 8 . The method according to claim 7 , wherein object detection is performed using a second artificial neural network.
9. 9 . The method according to claim 1 , further comprising: providing the species with the highest determined final species probability as output.
10. 10 . The method according to claim 9 , further comprising using the output to control a means of deterrent or curtailment in order to reduce a risk that the bird of the determined species is injured by a wind farm installation.
11. 11 . The method according to claim 1 , wherein the domain knowledge statistical model is a Bayesian belief network and/or one or more artificial neural networks are convolutional neural networks.
12. 12 . A system for determining the species of birds in flight, comprising: at least one video camera; one or more computer processors programmed to execute an artificial neural network configured to receive video images from the at least one video camera and trained to recognize species of birds from images; a geometry feature extraction software module configured to receive at least one video stream from the at least one video camera and to output extracted geometric features related to birds captured in flight in the at least one video stream; a domain knowledge software module with a domain knowledge statistical model, configured to receive the extracted geometric features outputted by the geometry feature extraction software module and to generate a probability of observing respective species of birds given the extracted geometric features; and a species determination software module configured to receive a first species probability estimate based on output from the artificial neural network and a second species probability estimate based on output from the domain knowledge software module and to generate a final species probability estimate, wherein the extracted geometric features outputted from the computer-executed geometry extraction software module are selected from the group consisting of: size, positions, acceleration, vertical motion, flight trajectory, and wingbeat frequency.
13. 13 . The system according to claim 12 , further comprising: a shallow neural network configured to receive extracted features from the artificial neural network and extracted geometry features outputted from the geometry feature extraction software module, and to generate the first species probability estimate.
14. 14 . The system according to claim 13 , wherein the geometry feature extraction software module is configured to received data related to at least one video stream, extract geometric features based on identification of the same bird in a sequence of images from the at least one video stream, and to estimate motion based on the change of the identified bird's position between images in the sequence of images.
15. 15 . The system according to claim 13 , wherein the at least one camera is two or more cameras and the at least one video stream is two or more video streams; the one or more computer programs being programmed to further execute a multi-view geometry analysis software module configured to receive a known position of each camera, receive data related to at least two concurrent video streams, determine a position of a bird identified in the respective images of the respective video streams, determine the position of the identified bird in the respective images of the respective video streams, and use multi-view geometry analysis to determine 3D coordinates representative of positions of the identified bird relative to the positions of the cameras from the determined positions in the respective images of the respective video streams.
16. 16 . The system according to claim 13 , wherein the geometry features extraction software module is further configured to determine a wingbeat frequency by performing Fourier analysis on a sequence of images from the at least one video stream and identifying a dominant frequency component that is inside a frequency interval consistent with wingbeat frequencies for birds.
17. 17 . The system according to claim 12 , wherein the one or more computer processors is programmed to further execute a bird detection and tracking software module configured to receive input from at least one video camera and perform object detection and to annotate images by drawing bounding boxes around each object that is identified as a bird.
18. 18 . The system according to claim 17 , wherein the bird detection and tracking software module includes a second artificial neural network.
19. 19 . The system according to claim 12 , wherein the species determination software module is further configured to deliver the final species probability estimate as output to be stored, displayed, or used to control a process of deterrence or curtailment in order to reduce a risk that the bird of the determined species is injured by a wind farm installation.
20. 20 . The system according to claim 12 , wherein the domain knowledge statistical model is a Bayesian belief network and/or one or more artificial neural networks are convolutional neural networks.

Description

TECHNICAL FIELD The present invention relates to automatic detection and species determination of birds, and in particular to detection and species identification using image recognition and machine learning. BACKGROUND As the world progresses towards less dependence on fossil energy sources, wind power is among the technologies that are becoming important alternatives. There are, however, certain disadvantages associated with wind power. Among these are the impact on biodiversity, and particularly the danger wind farms represent to bird populations. The wind power industry may have to conduct surveys of bird populations prior to establishing wind farms in order to estimate how the wind farms will impact those populations, and it may also become necessary to monitor bird population patterns and their developments around established wind farms. In most cases, bird observations have to be made manually, since current systems for automatic monitoring does not provide sufficiently high-quality data and is expensive in terms of necessary equipment and processing power. For obvious reasons, manual observations are impractical and cannot be part of a continuous monitoring of bird populations over time. Consequently, there is a need for better systems based on computer vision, image processing and statistic processing and modelling. Among the more specific needs of the industry are systems for detection, tracking and classification of species of birds in the vicinity of wind farms. Availability of such systems will greatly improve the effectiveness and efficiency of performing environmental impact assessment (EIAs) pre- and post-construction of wind farms. Additionally, such a system can assist operators of wind farms to implement mitigating measures to prevent birds from colliding with wind turbines during operation or pre-construction through adjusting the farm layout, controlling operation based on currently or recently observed bird species and behavior, and other mitigating measures. In order to develop such systems, a number of technical challenges have to be overcome. Such challenges may relate to the requirement for high quality data input including determining which data to obtain as well as how accurate or detailed the data needs to be. Other challenges relate to the methods required for processing the input data in order to extract features indicative of the presence and behavior of birds, statistical models for interpreting the extracted features correctly, initiating appropriate mitigating measures, and more. SUMMARY OF THE DISCLOSURE This specification discloses methods, devices, and systems that address a number of the requirements discussed above in order to facilitate better mitigation of the risk wind turbines represent to birds in general, and vulnerable or endangered species of birds in particular. In particular, the invention addresses the problem of observing birds, detecting their presence, determining the species of individual birds, and developing statistics. Such results can be stored, displayed or delivered as input to controlling processes in order to better plan construction of wind farms, control their operation, and initiate deterrence and curtailment. According to a first aspect of the invention a method is provided for determining the species of birds in flight. The method comprises capturing at least one video stream of a bird in flight using at least one camera, generating a first species probability estimate by delivering images from the at least one video stream as input to an artificial neural network that has been trained to recognize species of birds from images, obtaining additional parameters from the at least one video stream or from at least one additional data source, generating a second species probability estimate by delivering the obtained additional parameters as input to a domain knowledge module with a domain knowledge statistical model, and generating a final species probability estimate by combining the first species probability estimate and the second species probability estimate. The domain knowledge statistical model may be an influence diagram, for example a Bayesian belief network. The additional parameters may be derived from the video stream, or they may be obtained from other data sources, such as additional sensors, or services accessible over a network such as the Internet. Embodiments of the invention may further comprise extracting geometric features related to the bird in flight by delivering images from the at least one video stream as input to a geometry feature extraction module. The output from this process may be used in one, or both, of the following ways. The extracted geometric features, which may be related to how the bird moves in flight, may be used to contribute to the generation of the first species probability estimate by being delivered, together with extracted features from the artificial neural network, as input to a shallow neural n