US-12620254-B2 - Systems and methods for classifying mosquito larvae based on extracted masks of anatomical components from images

Abstract

Images of an insect larva are subjected to at least a first convolutional neural network to develop feature maps based on anatomical pixels at corresponding image locations in the respective feature maps. The anatomical pixels correspond to a body part of the insect larva. A computer uses a plurality of feature maps form an integrated feature map and bounding boxes on anatomical pixels corresponding to a head, thorax, abdomen, or terminal abdomen area of the insect larva. A classification network identifies the genus, the species, the sex, or the life stage of the insect larva.

Inventors

• Ryan M. CARNEY
• Sriram CHELLAPPAN

Assignees

• UNIVERSITY OF SOUTH FLORIDA

Dates

Publication Date

20260505

Application Date

20220218

Claims (19)

1. 1 . A system for identifying a genus and species of a mosquito larva, the system comprising: an imaging device configured to generate images of the mosquito larva; a computer processor connected to memory storing computer implemented commands in software, the memory receiving the images, wherein the software implements the following computerized method with respective images: applying a first convolutional neural network to the respective images to develop at least a first feature map and a second feature map based on anatomical pixels at corresponding image locations in the respective feature maps, said anatomical pixels corresponding to a body part of the mosquito larva; calculating an outer product of the first feature map and the second feature map; forming an integrated feature map from the first feature map and the second feature map; extracting fully connected layers from respective sets of integrated feature maps that have had the first convolutional neural network applied thereto; applying the fully connected layers to a classification network for identifying the genus and the species of the mosquito larva; and stopping the application of the first convolutional neural network upon a validation accuracy converging to a threshold value.
2. 2 . The system of claim 1 , further comprising calculating respective first feature maps and respective second feature maps for a plurality of corresponding image locations, calculating respective outer products from the respective first feature maps and respective second feature maps, and developing respective integrated feature maps from the outer products.
3. 3 . The system of claim 2 , wherein the plurality of corresponding image locations collectively comprise pixels corresponding to at least one of a head of the mosquito larva, the thorax of the mosquito larva, the abdomen of the mosquito larva, or a terminal abdominal segment of the mosquito larva.
4. 4 . The system of claim 1 , wherein the first feature map and the second feature map comprise pixels from the digital images that encompass terminal abdominal segments of the mosquito larva.
5. 5 . A system for identifying a genus and species of an insect larva, the system comprising: an imaging device configured to generate images of the insect larva; a computer processor connected to memory storing computer implemented commands in software, the memory receiving the images, wherein the software implements the following computerized method with respective images: applying a first convolutional neural network to the respective images to develop at least a first feature map directed to anatomical pixels at corresponding image locations in the respective images, said anatomical pixels corresponding to a body part of the insect larva; applying a second convolutional neural network to the respective images to develop at least a second feature map directed to anatomical pixels at corresponding image locations in the respective images, said anatomical pixels corresponding to a body part of the insect larva; forming an integrated feature map from the first feature map and the second feature map; applying bounding boxes to portions of the images; identifying a region of interest from the bounding boxes; extracting fully connected layers from respective sets of integrated feature maps that have had the first convolutional neural network and the second convolutional neural network applied thereto; applying the fully connected layers to a classification network for identifying the genus and the species of the insect larva; and stopping the application of the first or second convolutional neural network upon a validation accuracy converging to a threshold value.
6. 6 . The system of claim 5 , further comprising calculating respective first feature maps and respective second feature maps for a plurality of instances of the region of interest from a plurality of the images.
7. 7 . The system of claim 6 , wherein the plurality of instances of the region of interest from a plurality of the images comprise respective sets of pixels corresponding to at least one of a head of the insect larva, the thorax of the insect larva, the abdomen of the insect larva, or a terminal abdominal segment of the insect larva.
8. 8 . The system of claim 7 , wherein the insect larva is a mosquito larva.
9. 9 . The system of claim 5 , wherein the integrated feature maps are each one dimensional feature maps applicable to a respective region of interest on the insect larva.
10. 10 . The system of claim 9 , wherein the respective region of interest is a terminal abdominal area of the insect larva.
11. 11 . The system of claim 10 , wherein the one dimensional feature maps include hyperparameters for use in the first convolutional neural network and the second convolutional neural network, and wherein the classification network comprises computer implemented software to identify landmarks and semi-landmarks on the respective images.
12. 12 . The system of claim 11 , further comprising applying respective weights to the first feature map and the second feature map.
13. 13 . The system of claim 11 , further comprising identifying the landmarks and semi-landmarks on portions of the respective images corresponding to at least one of an eye of the insect larva, an antenna of the insect larva, an anal brush of the insect larva, anal gills of the insect larva, comb scales of an insect larva, a ventral brush of the insect larva, anal papillae of the insect larva, pectin of the insect larva, a saddle of the insect larva, and a siphon of the insect larva.
14. 14 . The system of claim 5 , further comprising, for each of the first convolutional neural network and the second convolutional neural network, reducing a respective learning rate by a common factor that depends upon an error rate threshold.
15. 15 . A computerized method of identifying at least one of a genus of an insect larva, a species of an insect larva, a sex of an insect larva, or a life stage of an insect larva, the method comprising: acquiring images of an insect larva and storing pixel data from the images in a computer memory in data communication with a computer processor; using the computer processor to perform the following steps of the method: applying at least one convolutional neural network to the respective images to develop a plurality of feature maps of hyperparameters based on anatomical pixels at corresponding image locations in respective feature maps, said anatomical pixels corresponding to at least one landmark on the insect larva that is identifiable from the images; forming an integrated feature map from the plurality of feature maps; extracting fully connected layers from respective sets of integrated feature maps that have had at least a first convolutional neural network applied thereto; applying the fully connected layers to a classification network for classifying the insect larva from anatomical pixels that correspond to at least one of a head of the insect larva, a thorax of the insect larva, an eye of the insect larva, an antenna of the insect larva, an anal brush of the insect larva, anal gills of the insect larva, comb scales of an insect larva, a ventral brush of the insect larva, anal papillae of the insect larva, pectin of the insect larva, a saddle of the insect larva, and a siphon of the insect larva; and stopping the application of the at least one convolutional neural network upon a validation accuracy converging to a threshold value.
16. 16 . The computerized method of claim 15 , wherein the computer processor further performs additional steps comprising: computing an importance factor for each feature map in an output image that has been a subject of the classification network; using a selected output image with the highest importance factor, compare the selected output image with respective images to evaluate an accuracy level of the classification network.
17. 17 . The computerized method of claim 15 , wherein the step of applying at least one convolutional neural network comprises applying at least one Bi-Linear Convolutional Neural Network, and wherein the at least one landmark on the insect larva further comprises semi-landmarks on the insect larva that are identifiable from the images.
18. 18 . The computerized method of claim 17 , wherein applying the Bi-Linear Convolutional Neural Network comprises applying the first feature map and the second feature map to identify respective features from the images and applying a pooling function to an output of the Bi-Linear Convolutional Neural Network before applying the fully connected layers to the classification network.
19. 19 . The computerized method of claim 15 , further comprising: applying the at least one convolutional neural network to respective sets of anatomical pixels with each of the respective sets corresponding to a respective body part of the insect larva, wherein the respective body part is one of the head of the insect larva, the thorax of the insect larva, the eye of the insect larva, the antenna of the insect larva, the anal brush of the insect larva, the anal gills of the insect larva, the comb scales of the insect larva, the ventral brush of the insect larva, the anal papillae of the insect larva, the pectin of the insect larva, the saddle of the insect larva, and the siphon of the insect larva; and using a plurality of outputs and making an aggregated decision on at least one of the genus of the insect larva, the species of the insect larva, the sex of the insect larva, or the life stage of the insect larva.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application Ser. No. 63/200,883 filed on Apr. 1, 2021 and titled Systems and Methods for Classifying Mosquito Larvae From Images Based on Localized Anatomical Components and Their Extracted Masks and Landmarks, which is incorporated by reference herein. STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH This application is a United States National Phase Patent Application of International Patent Application Number PCT/US2022/017089, filed on Feb. 18, 2022, which claims priority to U.S. Provisional Patent Application Ser. No. 63/200,883 filed on Apr. 1, 2021 and titled Systems and Methods for Classifying Mosquito Larvae From Images Based on Localized Anatomical Components and Their Extracted Masks and Landmarks, which are incorporated by reference herein in their entireties. BACKGROUND Taxonomy is the process of classifying organisms in nature. Entomology is the study of insect organisms. Taxonomy in the context of entomology is a relatively obscure discipline in the era of modern sciences. Very few people want their professional careers spent with hours peering through a microscope trying to identify what genus and species an insect is. In the context of mosquitoes, there are close to 4500 different species of mosquitoes, and training to identify all of these mosquitoes is hard if not impossible. In countries like India, Bangladesh and even the US, it is simply not possible to train professionals to identify all mosquitoes that are endemic in these countries (e.g., there are 400 species of mosquitoes endemic to India; and about 150 species in the US). With increasing travel and global connectivity among nations, mosquitoes can invade to newer places, and identifying the “new” mosquitoes becomes impossible by local professionals. Mosquitos and other insects are considered “vectors” because they can carry viruses, bacteria, and strains of diseases and transmit them to humans. The term “vector” is therefore given its broadest meaning in the art of infectious diseases. Modern entomology updates have focused on eliminating or minimizing human involvement in classifying genus and species of mosquitoes during disease outbreak. There are close to 4500 different species of mosquitoes in the world spread across 45 or so genera. Out of these, only handfuls of species across three genus types spread the deadliest diseases. These mosquitoes belong to Aedes (Zika, Dengue, Chikungunya, Yellow Fever), Culex (West Nile Virus, and EEE), and Anopheles (Malaria). Within these three genera, the deadliest species are Aedes aegypti, Aedes albopictus, Culex nigripalpus, Anopheles gambiae and Anopheles stephensi. When a mosquito-borne disease, say Dengue affects a region, then identifying the presence of the particular vectors for Dengue (i.e., Aedes aegyptiand Aedes albopictus) becomes important. This is hard and expensive. For instance in India, there are close to 450 types of mosquitoes spread all over. Accordingly, public health experts lay traps in disease prone areas, and sometimes hundreds of mosquitoes get trapped. Now, however, they can identify which of those is the genus and species they are looking for. Because, once they identify the right mosquitoes, they can then take those mosquitoes to the lab for DNA testing etc. to see if the pathogen (i.e., virus) is there within the trapped mosquito. Naturally, if they find a reasonable large number of those mosquitoes with the virus in them, there is a public health crisis, and corrective action needs to be taken. Other efforts have focused on detecting foreign mosquitoes at borders. This is a problem that is attracting a lot of global attention—the need to identify if a mosquito in borders of a nation (land or sea or air or road) is a foreign mosquito. For instance, consider a scenario in which mosquitos, e.g., both a domestic vector and one non-native to the US, are on a vehicle entering the US borders. Assuming that borders do have mosquito traps, it is likely that this “new” breed of mosquito could get trapped along with other local mosquitoes. The question here is how public health authorities identify that a “foreign” mosquito is in one such trap. Current entomology classification systems would require going periodically to these traps, collecting and studying subjects through a microscope, and identifying specimens one by one. This is impossibly cumbersome if the goal is to only detect a particular type of “foreign” mosquito. Current disease models rely upon proper classification of infection vectors. The entomology classification systems need to be improved for use in specialized and detail intensive instances, such as the hypothetical above. A need exists in the art of entomological classification to include algorithms that are adaptable for use in resolving important, yet hard to pinpoint issues, such as identifying the hypothetical “foreign” mosquito that did indeed get