Search

CN-121981987-A - Plant diagnosis method and maintenance system

CN121981987ACN 121981987 ACN121981987 ACN 121981987ACN-121981987-A

Abstract

The present disclosure relates to plant diagnostic methods and maintenance systems. A plant diagnostic method includes obtaining information to be processed, extracting a target feature vector from the information via a feature vector extraction model, determining a species, a physiological state, and a scene of a plant based on the target feature vector via a multi-modal large model, determining an initial intent based on the species and the scene of the plant and a ternary knowledge graph in response to determining the scene as a first type of scene, determining a user intent based on the physiological state and the initial intent of the plant, invoking a respective first diagnostic model of a plurality of first diagnostic models based on the scene and the user intent via a rules engine, and determining a diagnostic result based on the species, the physiological state, and the scene of the plant via the respective first diagnostic model, determining a diagnostic result based on the target feature vector via a second diagnostic model in response to determining the scene as a second type of scene, and presenting a diagnostic report to a user based on the diagnostic result.

Inventors

  • XU QINGSONG
  • HE TAO

Assignees

  • 杭州睿胜软件有限公司

Dates

Publication Date
20260505
Application Date
20260121

Claims (10)

  1. 1. A plant diagnostic method comprising: Acquiring information to be processed, wherein the information comprises an image, and the image comprises plants and surrounding environments where the plants are located; extracting a target feature vector from the information via a feature vector extraction model; determining species, physiological states, and scenes of the plant based on the target feature vector via a multi-modal large model; In response to determining that the scene is a first type of scene, Determining an initial intent based on the species and scene of the plant, a ternary knowledge-graph indicating a relationship between the species, scene and intent of the plant, Determining a user intent based on the physiological state of the plant and the initial intent, Invoking, via a rules engine, respective ones of a plurality of first diagnostic models based on the scene and the user intent, and Determining a diagnostic result based on the species, physiological status and scene of the plant via the respective first diagnostic model; responsive to determining that the scene is a second type of scene, determining a diagnostic result based on the target feature vector via a second diagnostic model, and A diagnostic report is presented to a user based on the diagnostic result.
  2. 2. The method of claim 1, wherein, Optionally, the scenes include indoor scenes including home balconies and offices, outdoor scenes including gardens, lawns and orchards, non-target plant scenes including weeds and wild plants, and plant-related environmental scenes including soil, flowerpots and large-area scenes; Optionally, the first type of scene includes other scenes than the outdoor scene and the large area scene among the scenes, The second class of scenes includes the outdoor scenes and the large area scenes.
  3. 3. The method of claim 1, wherein, Optionally, the feature vector extraction model includes a visual encoder configured to perform feature extraction for an object in the image resulting in an image feature vector, wherein the object includes the plant and at least one of soil, a flower pot, and a large area scene indicated by the surrounding environment; Optionally, the visual encoder comprises a lightweight multitasking computer vision CV model configured to perform at least one of: Feature extraction is performed on the ecological features of the plants in the image based on an instance segmentation algorithm, Feature extraction is performed for humidity features and texture features of soil in the image based on a color histogram and a texture analysis algorithm, Extracting features according to the size features, the number features and the position features of the drainage holes of the flowerpot in the image, or Extracting features of plant distribution features in a large-area scene in the image based on a semantic segmentation algorithm; optionally, the information further comprises text information and environmental information, wherein the text information comprises an additional description about the plant, the environmental information comprises at least one of location information, time information, and weather information; optionally, the feature vector extraction model further comprises a text encoder, an embedded layer, a projector, and a multimodal fusion model, wherein, The text encoder is configured to extract text feature vectors from the text information, The embedding layer is configured to extract an ambient feature vector from the ambient information, The projector is configured to map the image feature vector, the text feature vector, and the environmental feature vector to a unified potential space, and The multimodal fusion model is configured to fuse the mapped image feature vector, text feature vector, and environmental feature vector to determine a target feature vector; optionally, the method satisfies at least one of: The text encoder is constructed based on a BERT model and is obtained by fine tuning of corpus in the plant diagnosis field, or The multi-mode fusion model is constructed based on FLAVA architecture and is obtained by fine tuning of corpus in the plant diagnosis field.
  4. 4. The method of claim 1, wherein, Optionally, the multi-modal large model is constructed based on a pre-trained multi-modal large model and is obtained by fine adjustment through a corpus in the plant diagnosis field and a scene feature library, wherein the scene feature library comprises a plurality of sample feature vectors marked with scene labels; Optionally, the pre-trained multimodal big model comprises a CLIP-VIT-L/14 model; Optionally, determining a scene based on the target feature vector via a multi-modal large model comprises: Matching the target feature vector with each sample feature vector in the scene feature library to output at least one scene tag and its confidence, and In response to determining that the at least one scene tag includes a scene tag whose confidence meets a scene confidence requirement, determining that the scene is the scene indicated by the scene tag; Optionally, determining a scene based on the target feature vector via a multi-modal large model further comprises: In response to determining that the at least one scene tag does not include a scene tag whose confidence meets the scene confidence requirement, interacting with a user to obtain supplemental information and further determining a scene based on the supplemental information; Optionally, determining the user intent includes: Determining an initial intent and its probability based on the species of the plant, the scene and the ternary knowledge-graph via an intent prediction model, Correcting the initial intent and its probability based on the physiological state of the plant via the multimodal big model to obtain a corrected intent and its probability, and Determining that the user intent is the modified intent in response to the probability of the modified intent meeting a probability requirement; optionally, the intent prediction model is constructed based on a bayesian probability model; optionally, in the case that the correction intention includes a plurality of correction intents: In response to determining that a first corrective intent of the plurality of corrective intentions, whose probability is highest, meets the probability requirement, determining that the user intent is the first corrective intent, or In response to determining that a difference in probability between a first corrective intent having a highest probability and a second corrective intent having a second highest probability of the plurality of corrective intentions does not exceed a threshold, interacting with a user to obtain supplemental information and determining the user intent from the first corrective intent and the second corrective intent further based on the supplemental information.
  5. 5. The method of claim 1, wherein, Optionally, each first diagnostic model of the plurality of first diagnostic models is fine-tuned with a first training data set comprising a quaternary knowledge-graph, wherein the quaternary knowledge-graph indicates a relationship between species of plant, pest, environmental pathogenic agent, and control scheme; optionally, the second diagnosis model is constructed based on a pre-trained visual language model and is obtained by fine tuning through a second training data set, wherein the second training data set comprises a ternary knowledge graph, a quaternary knowledge graph and a corpus in the plant diagnosis field, and the quaternary knowledge graph indicates the relationship among species, diseases and insect pests, environmental pathogenic factors and prevention and treatment schemes of plants; optionally, the pre-trained visual language model comprises a BLIP-2 model; optionally, the diagnostic result comprises at least one of a species, physiological status, pest, etiology, and control regimen of the plant; optionally, presenting the diagnostic report to the user based on the diagnostic result includes: Converting the diagnostic result into a diagnostic report via a natural language generation model, the diagnostic report comprising a core conclusion module, a cause analysis module, an operation guideline module, and an attention module, wherein, The core conclusion module is obtained based on the species, physiological states and diseases and insect pests of the plants in the diagnosis results, the cause analysis module is obtained based on the diseases and insect pests and causes in the diagnosis results, the operation guidance module is obtained based on the control scheme in the diagnosis results, and the notice module is obtained based on the control scheme in the diagnosis results; optionally, presenting the diagnostic report to the user based on the diagnostic result further comprises at least one of: Under the condition that the scene is determined to be a large-area scene, the areas with diseases and insect pests in the image are marked with colors and priorities based on a semantic segmentation algorithm, wherein the areas with different diseases and insect pests are marked with different colors, In the case where the scenario is determined to be an orchard, the diagnostic report further includes an annual pest management calendar determined based on the control scheme and month in the diagnostic result, or In the case that the scene is determined to be soil, the diagnostic report further includes a soil property radar map indicating current and recommended values of humidity, ph, air permeability of the soil; Optionally, the method further comprises: After presenting the diagnostic report to the user, interacting with the user to obtain a result indicating whether the user is satisfied with the diagnostic report and to take the result as a label of the diagnostic report, and At least one of the feature vector extraction model, the multi-modal large model, the ternary knowledge-graph, the quaternary knowledge-graph, the first diagnostic model, and the second diagnostic model is adjusted based on a labeled diagnostic report.
  6. 6. The method according to claim 1, Optionally, the method further comprises: Generating a maintenance schedule based on the diagnosis result, the maintenance schedule including a maintenance task and an identification of a maintenance device for performing the maintenance task, and Outputting the maintenance scheme; Optionally, the method further comprises: And controlling the corresponding maintenance device according to the identification of the maintenance device in the maintenance scheme to complete the maintenance task.
  7. 7. An electronic device, comprising: processor, and A memory storing computer-executable instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 6.
  8. 8. A computer readable storage medium having stored thereon computer executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 6.
  9. 9. A computer program product comprising instructions which, when executed by a processor, implement the method according to any one of claims 1 to 6.
  10. 10. A maintenance system, comprising: the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is configured to acquire information to be processed, the information comprises an image, and the image comprises a plant and a surrounding environment where the plant is located; A diagnostic module configured to: extracting a target feature vector from the information via a feature vector extraction model; determining species, physiological states, and scenes of the plant based on the target feature vector via a multi-modal large model; In response to determining that the scene is a first type of scene, Determining an initial intent based on the species and scene of the plant, a ternary knowledge-graph indicating a relationship between the species, scene and intent of the plant, Determining a user intent based on the physiological state of the plant and the initial intent, Invoking, via a rules engine, respective ones of a plurality of first diagnostic models based on the scene and the user intent, and Determining a diagnostic result based on the species, physiological status and scene of the plant via the respective first diagnostic model; Responsive to determining that the scene is a second class of scenes, determining, via a second diagnostic model, a diagnostic result based on the target feature vector; A presentation module configured to present a diagnostic report to a user based on the diagnostic result; A maintenance scheme generation module configured to generate a maintenance scheme based on the diagnosis result, the maintenance scheme including a maintenance task and an identification of a maintenance device for performing the maintenance task; A control module configured to control the corresponding curing device to complete the curing task according to the identification of the curing device in the curing scheme, and An execution module including a maintenance device communicatively coupled with the control module, the maintenance device configured to execute maintenance tasks in response to receiving commands from the control module.

Description

Plant diagnosis method and maintenance system Technical Field The present disclosure relates to the field of information processing technology, and more particularly, to a plant diagnostic method and associated electronic device, computer readable storage medium and computer program product, and also to a maintenance system. Background With the improvement of living standard, more and more users start to maintain plants. Since plants may undergo various growth events during growth, it is necessary to pay attention to these growth events at all times in order to better maintain the plants. Disclosure of Invention The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. It should be understood that this summary is not an exhaustive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. Its purpose is to present some concepts related to the disclosure in a simplified form as a prelude to the more detailed description that is presented later. According to a first aspect of the present disclosure, there is provided a plant diagnosis method comprising obtaining information to be processed, the information comprising an image comprising a plant and an ambient environment in which the plant is located, extracting a target feature vector from the information via a feature vector extraction model, determining a species, a physiological state and a scene of the plant based on the target feature vector via a multi-modal large model, determining an initial intent based on the species and the scene of the plant and a ternary knowledge graph indicating a relationship between the species, the scene and the intent of the plant in response to determining the scene as a first type of scene, determining a user intent based on the physiological state and the initial intent of the plant, invoking respective first diagnostic models of a plurality of first diagnostic models based on the scene and the user intent via a rules engine, and determining a diagnostic result based on the species, the physiological state and the scene of the plant via the respective first diagnostic models, determining a diagnostic result based on the target feature vector via the second diagnostic model in response to determining the scene as a second type of scene, and presenting a diagnostic report to a user based on the diagnostic result. In some embodiments, the scenes include indoor scenes including home balconies and offices, outdoor scenes including gardens, lawns, and orchards, outdoor scenes including weeds and wild plants, non-target plant scenes including soil, flowerpots, and large area scenes, and plant-related environmental scenes. In some embodiments, the first type of scene includes other ones of the scenes except for the outdoor scene and the large area scene, and the second type of scene includes the outdoor scene and the large area scene. In some embodiments, the feature vector extraction model includes a visual encoder configured to perform feature extraction for an object in the image resulting in the image feature vector, wherein the object includes at least one of a plant and soil, a flower pot, and a large area scene indicated by the surrounding environment. In some embodiments, the visual encoder includes a lightweight multitasking Computer Vision (CV) model configured to perform at least one of feature extraction for ecological features of plants in the image based on an instance segmentation algorithm, feature extraction for humidity features and texture features of soil in the image based on a color histogram and texture analysis algorithm, feature extraction for size features, number of drain holes features, and drain hole position features of flowerpots in the image, or feature extraction for plant distribution features in a large area of scene in the image based on a semantic segmentation algorithm. In some embodiments, the information further comprises text information and environmental information, wherein the text information comprises additional descriptions about the plant, and the environmental information comprises at least one of location information, time information, and weather information. In some embodiments, the feature vector extraction model further comprises a text encoder, an embedding layer, a projector, and a multimodal fusion model, wherein the text encoder is configured to extract text feature vectors from the text information, the embedding layer is configured to extract ambient feature vectors from the ambient information, the projector is configured to map the image feature vectors, the text feature vectors, and the ambient feature vectors to a unified potential space, and the multimodal fusion model is configured to fuse the mapped image feature vectors, the text feature vectors, and the ambient feature vectors to determine the