Search

US-12625550-B2 - Electronic devices with display operation based on eye activity

US12625550B2US 12625550 B2US12625550 B2US 12625550B2US-12625550-B2

Abstract

An electronic device may have a display for displaying image content. Head-mounted support structures in the device may be used to support the display. The electronic device may have an eye monitoring system that detects eye saccades and eye blinks. Control circuitry in the electronic device may coordinate operation of the display with periods of suppressed visual sensitivity that are associated with the saccades and blinks. By making adjustments to display circuitry and image content during periods of suppressed visual sensitivity, potentially visually obtrusive changes to displayed images can be hidden from a user of the electronic device. Adjustments to display operation may help reduce burn-in effects, may help reduce power consumption, and may otherwise improve device performance.

Inventors

  • Mehmet N. Agaoglu
  • Andrew B. Watson

Assignees

  • APPLE INC.

Dates

Publication Date
20260512
Application Date
20241203

Claims (20)

  1. 1 . An electronic device, comprising: a camera configured to capture an image; a display configured to display the image; an eye monitoring system configured to detect an eye activity of a blink or a saccade; and control circuitry configured to perform a transformation of the image in coordination with the eye activity, wherein the transformation of the image accommodates changes in a user's perspective of the image.
  2. 2 . The electronic device of claim 1 , wherein the control circuitry is configured to mitigate parallax mismatch by performing the transformation of the image in coordination with the eye activity.
  3. 3 . The electronic device of claim 1 , wherein the transformation comprises a projective or perspective transform.
  4. 4 . The electronic device of claim 1 , wherein the transformation comprises a homographic transform.
  5. 5 . The electronic device of claim 1 , wherein the transformation comprises a shifting or shearing transform.
  6. 6 . The electronic device of claim 1 , wherein the transformation comprises a geometric transformation.
  7. 7 . The electronic device of claim 1 , wherein the control circuitry is configured to perform the transformation of the image during the eye activity.
  8. 8 . The electronic device of claim 1 , wherein the camera is a forward-facing camera.
  9. 9 . The electronic device of claim 8 , wherein the image is a real-world image.
  10. 10 . The electronic device of claim 9 , wherein the control circuitry is further configured to generate computer-generated content and to merge the computer-generated content with the real-world image to produce mixed reality content.
  11. 11 . The electronic device of claim 10 , wherein the display is further configured to display the mixed reality content and the control circuitry is further configured to perform a geometric transformation of the mixed reality content in coordination with the eye activity.
  12. 12 . The electronic device of claim 1 , wherein changes in the user's perspective of the image comprise changes between first and second viewing orientations.
  13. 13 . The electronic device of claim 1 , further comprising: an accelerometer, a gyroscope, or a compass.
  14. 14 . The electronic device of claim 1 , wherein the eye monitoring system comprises a visible camera or an infrared camera.
  15. 15 . An electronic device, comprising: a display comprising an array of pixels configured to display an image; an eye monitor configured to detect an eye activity of a blink or saccade; and control circuitry configured to adjust the display by shifting pixels of the displayed image in response to the eye activity.
  16. 16 . The electronic device of claim 15 , wherein the control circuitry is configured to adjust the display in response to the blink, wherein the blink comprises a time in which an eye is closed, and wherein the control circuitry is configured to adjust the display entirely within the time.
  17. 17 . The electronic device of claim 15 , wherein the control circuitry is configured to adjust the display in response to the saccade and wherein the control circuitry is configured to adjust the display entirely during the saccade.
  18. 18 . The electronic device of claim 15 , wherein the control circuitry is configured to adjust the display by shifting the pixels of the displayed image by at least 10 pixels.
  19. 19 . The electronic device of claim 15 , wherein the eye monitor comprises an infrared light source and an infrared light detector.
  20. 20 . The electronic device of claim 15 , wherein the control circuitry is configured to adjust the display by shifting the pixels of the displayed image based on a pixel shifting value.

Description

This application is a continuation of U.S. patent application Ser. No. 18/456,307, filed Aug. 25, 2023, which is a continuation of U.S. patent application Ser. No. 17/063,382, filed Oct. 5, 2020, now U.S. Pat. No. 11,782,503, which is a continuation of U.S. patent application Ser. No. 16/443,214, filed Jun. 17, 2019, now U.S. Pat. No. 10,802,585, which claims the benefit of provisional patent application No. 62/697,286, filed Jul. 12, 2018, which are hereby incorporated by reference herein in their entireties. BACKGROUND This relates generally to electronic devices and, more particularly, to electronic devices with displays. Electronic devices such as head-mounted devices and computers have displays. During operation of an electronic device, display adjustments may be made. For example, the brightness of a display may be adjusted. Challenges can arise in making display adjustments during device operation. If care is not taken, display adjustments may be obtrusive to a user of an electronic device. SUMMARY An electronic device may have a display for displaying image content. The electronic device may be a head-mounted device or other electronic device. Head-mounted support structures may be provide in the device. The head-mounted support structures in the device may be used to support the display and other device components. An eye monitoring system may be supported by the head-mounted support structures. The eye monitoring system may be used in detecting eye saccades and eye blinks of a user. The human retina is non-uniform in distribution of photoreceptors, and hence, visual acuity drops with distance from the retinal location with highest photoreceptor density, often referred to as the fovea. Saccades are fast, jerky, and mostly ballistic eye rotations. Humans make several saccadic eye movements per second to utilize this highest-resolution part of the retina to look at the object of interest. Blinks are brief closures of eyelids, fully or partially, to clean, moisten, and oxygenate the cornea. Blinking is a semi-automatic behavior, mostly done reflexively to avoid irritants, or routinely to spread the tear across the surface of the cornea. Humans blink many times per minute depending on various conditions. During saccades and blinks, a user's visual sensitivity is temporarily suppressed. Control circuitry in the electronic device may take advantage of momentary suppression of the user's visual sensitivity to make adjustments to the operation of the display. For example, the control circuitry can coordinate operation of the display with periods of suppressed visual sensitivity that are associated with the saccades and blinks to reduce power consumption and/or to make potentially obtrusive image changes. By making adjustments during periods of suppressed visual sensitivity, these adjustments can be hidden from the user of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram of an illustrative electronic device such as a head-mounted display in accordance with an embodiment. FIG. 2 is a top view of an illustrative eye monitoring system in accordance with an embodiment. FIG. 3 is a diagram of an illustrative display system in accordance with an embodiment. FIG. 4 is a graph showing an illustrative eyelid position as a function of time while a user is viewing a display in accordance with an embodiment. FIG. 5 is a flow chart of illustrative operations involved in using an electronic device with a display and eye monitoring system in accordance with an embodiment. DETAILED DESCRIPTION Electronic devices may be provided with displays. Sensors such as eye monitoring sensors may gather information on eye activity such as information on saccades and blinks. During operation of an electronic device, control circuitry in the electronic device may make display adjustments in coordination with periods of time that a user has decreased visual sensitivity. For example, display adjustments may be made during periods of suppressed visual sensitivity that are associated with eye blinks and saccades. Electronic devices that include displays and eye monitoring systems may include, for example, head-mounted displays (e.g., head-mounted devices such as virtual reality or augmented reality glasses), cellular telephones, tablet computers, head-up displays in vehicles and other environments, laptop computers, desktop computers, televisions, wristwatches, and other suitable electronic equipment. In devices such as head-mounted displays, the general location of a user's eyes will be relatively constant during operation, facilitating accurate eye monitoring. In general, however, any suitable electronic device may be provided with an eye monitoring system (e.g., a stand-alone computer, a head-up display, a portable device, etc.). A schematic diagram of an illustrative electronic device with a display and an eye monitoring system is shown in FIG. 1. Electronic device 10 may be a head-mounted device (h