Search

CN-122002130-A - Adaptive fixation-point image sensor for near-eye devices

CN122002130ACN 122002130 ACN122002130 ACN 122002130ACN-122002130-A

Abstract

The application provides an adaptive fixation-point image sensor for near-eye devices. Embodiments for an adaptive gaze-spotting image sensor are provided. An embodiment includes an image sensor system for adaptive gaze-point processing including an image sensor to generate a set of signals by imaging an environment, and processing circuitry configured to receive gaze-point region of interest (ROI) information and ambient light information, determine a processing mode based on the ambient light information, compress the set of signals from the image sensor based on the determined processing mode and the gaze-point ROI information to generate a compressed set of pixels, and output the compressed set of pixels.

Inventors

  • LIU CHENG
  • MENG XIAOZHOU
  • LI YONGJUN

Assignees

  • 脸萌有限公司

Dates

Publication Date
20260508
Application Date
20250902
Priority Date
20241107

Claims (20)

  1. 1. An image sensor system for adaptive gaze dotting processing, the image sensor system comprising: An image sensor for generating a set of signals by imaging an environment, and The processing circuitry is configured to process the data, the processing circuitry is configured to: receiving information of a region of interest (ROI) of fixation localization and ambient light information; determining a processing mode based on the ambient light information; Compressing the set of signals from the image sensor based on the determined processing mode and the gaze-point ROI information to generate a compressed set of pixels, and The compressed set of pixels is output.
  2. 2. The image sensor of claim 1, wherein the compressed set of pixels is output using a mobile industrial processor interface MIPI.
  3. 3. The image sensor of claim 1, wherein the processing mode is further determined based on one or more of a predefined illuminance threshold or a predefined spatial frequency threshold.
  4. 4. The image sensor of claim 3, wherein the processing mode is further determined based on the predefined spatial frequency threshold for gaze-spotting regions of interest determined based on the gaze-spotting ROI information.
  5. 5. The image sensor of claim 1, wherein the set of signals comprises signals of a plurality of frames, wherein the processing mode is determined on a per frame basis, and wherein compressing the set of signals is performed on a per frame basis.
  6. 6. The image sensor of claim 1, wherein upon determining that the processing mode is a low resolution mode, compressing the set of signals comprises: One or more of analog combining, digital combining, analog sub-sampling, or digital sub-sampling is performed to generate the compressed set of pixels.
  7. 7. The image sensor of claim 1, wherein upon determining that the processing mode is a gaze-spotting ROI mode, compressing the set of signals comprises: Applying a gaze point map to the set of signals to generate the compressed set of pixels, wherein the gaze point map comprises: Full resolution region determined using the gaze-spotted ROI information, and A compressed region different from the full resolution region.
  8. 8. The image sensor of claim 7, wherein applying the gaze point map comprises performing analog compression and performing digital compression after performing the analog compression, wherein the analog compression comprises analog combining or analog sub-sampling, and wherein the digital compression comprises digital combining or digital sub-sampling.
  9. 9. The image sensor of claim 1, wherein the gaze-dotted ROI information includes coordinates describing one or more ROIs.
  10. 10. The image sensor of claim 1, wherein the image sensor is implemented in a head mounted display device.
  11. 11. A method implemented on an image sensor system for adaptive gaze pointedness processing, the method comprising: generating a set of signals by imaging an environment; receiving information of a region of interest (ROI) of fixation localization and ambient light information; determining a processing mode based on the ambient light information; Compressing the set of signals based on the determined processing mode and the gaze-pointized ROI information to generate a compressed set of pixels, and The compressed set of pixels is output.
  12. 12. The method of claim 11, wherein the compressed set of pixels is output using a mobile industrial processor interface MIPI.
  13. 13. The method of claim 12, wherein dummy data is added to the compressed set of pixels prior to using the MIPI output.
  14. 14. The method of claim 11, wherein the processing mode is further determined based on one or more of a predefined illuminance threshold or a predefined spatial frequency threshold.
  15. 15. The method of claim 11, wherein the set of signals comprises signals of a plurality of frames, wherein the processing mode is determined on a per frame basis, and wherein compressing the set of signals is performed on a per frame basis.
  16. 16. The method according to claim 11, wherein: Upon determining that the processing mode is a low resolution mode, compressing the set of signals includes: one or more of analog combining, digital combining, analog sub-sampling, or digital sub-sampling is performed, To generate the compressed set of pixels, and Upon determining that the processing mode is a gaze-spotting ROI mode, compressing the signal set comprises: a gaze point map is applied to the set of signals to generate the compressed set of pixels.
  17. 17. A method implemented on an image sensor for adaptive gaze pointedness processing, the method comprising: generating a set of signals by imaging an environment; Receiving information of a region of interest (ROI) of fixation localization; Compressing the set of signals based on the gaze-point ROI information to generate a compressed set of pixels by applying a gaze point map, wherein the gaze point map comprises: Full resolution region determined using the gaze-spotted ROI information, and A compressed region different from the full resolution region, and The compressed set of pixels is output using a mobile industrial processor interface MIPI.
  18. 18. The method of claim 17, wherein the set of signals comprises signals of a plurality of frames, and wherein compressing the set of signals is performed on a per frame basis.
  19. 19. The method of claim 17, wherein applying the gaze point map comprises performing analog compression and performing digital compression after performing the analog compression.
  20. 20. The method of claim 17, wherein the compressed set of pixels is output to a processor capable of rendering the compressed set of pixels for display.

Description

Adaptive fixation-point image sensor for near-eye devices The present application claims priority from U.S. patent application Ser. No. 18/940,740, app. day 2024, 11/7, entitled "adaptive gaze pointedness image sensor for near eye devices (ADAPTIVE FOVEATED IMAGE SENSORS FOR NEAR-EYE DEVICES)", which is incorporated herein by reference in its entirety. Background Near-eye devices are display devices, such as head mounted displays, used in various augmented reality/mixed reality/virtual reality (AR/MR/VR) applications. These wearable devices utilize an image generator and imaging optics to provide image content to the eyes of a user. Different configurations of sensors and instruments can implement various functionalities. For AR/MR applications, computer generated content may be imposed on the user's eyes, combined with a real-world view through a transparent display. In VR applications, the device immerses the user in the virtual environment by projecting image content to the entire field of view of the user or a significant portion thereof. In some applications, VR image content is computer generated and virtual. In video feed-through applications, video feed content surrounding a user is captured by an installed camera and displayed to the user. Computer generated content, whether interactive or non-interactive, may also be overlaid on the video feed content. Disclosure of Invention Embodiments of an adaptive gaze-spotting image sensor are provided. One embodiment includes an image sensor system for adaptive gaze point processing, the image sensor system including an image sensor to generate a set of signals by imaging an environment, and processing circuitry configured to receive gaze point region of interest (ROI) information and ambient light information, determine a processing mode based on the ambient light information, compress the set of signals from the image sensor based on the determined processing mode and gaze point ROI information to generate a compressed set of pixels, and output the compressed set of pixels. This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. Drawings Fig. 1 shows a schematic view of an example near-eye device implementing an example adaptive gaze-spotting image sensor. Fig. 2 shows an example flowchart for implementing different processing modes that may be performed using the example near-eye device of fig. 1. Fig. 3A and 3B illustrate an example sub-sampling process that may be performed using the example near-eye device of fig. 1. Fig. 4 illustrates an example merging process that may be performed using the example near-eye device of fig. 1. Fig. 5 illustrates an example hybrid analog-to-digital sampling process for generating gaze point maps, which may be performed using the example near-eye device of fig. 1. Fig. 6 illustrates an example image in which gaze point mapping is applied, which may be performed using the example near-eye device of fig. 1. Fig. 7 shows how the example gaze point map is divided into three different virtual channels, which may be performed using the example near-eye device of fig. 1. Fig. 8A-8C illustrate an example process for adding dummy data for different virtual channels, which may be performed using the example near-eye device of fig. 1. Fig. 9 shows a flowchart of an example method of adaptive fixation point processing that may be performed using the example near-eye device of fig. 1. Fig. 10 shows a schematic view of an example computing system including the near-eye device of fig. 1. Fig. 11 illustrates an example form factor of the near-eye device of fig. 1 in the form of AR/VR glasses. Detailed Description Near-eye wearable devices implementing AR/MR/VR applications may provide image content to users through different methods of various applications, including consumer and industrial applications. With more advanced designs and increased functionality, the development of near-eye devices has involved significant challenges in providing adequate computing power and power economy in limited form factors. For example, near-eye devices may be expected to provide high resolution image content, such as high resolution computer generated content, high resolution video feed content of a user environment, and the like. In addition to processing and rendering high resolution content, other functionalities such as eye tracking techniques may also place demands on computing power. These demands often result in increased demands on battery capacity, computing power, and thermal management, which can result in a more bulky, h