CA-3179379-C - AUTOMATED TOOLS FOR INCREMENTAL GENERATION OF BUILDING MAPPING INFORMATION

Abstract

Techniques are described for computing devices to perform automated operations related to using images acquired in a building as part of an automated generation process of a floor plan or other mapping information for the building, in some cases without using depth information from depth-sensing equipment about distances from the images' acquisition locations to objects in the surrounding building, and for subsequent use in further automated manners, such as controlling navigation of mobile devices and/or for display to end users in a corresponding graphical user interface. The automated analysis and assessment of the building images may, for example, include incrementally generating partial versions of the mapping information, determining acquisition instructions related to further acquisition of additional images and/or other types of data (e.g., to reduce uncertainty associated with portions of the partial mapping information or otherwise obtain missing data}, and providing corresponding guidance in or more manners.

Inventors

• Zhiqiang Wan
• Will A. Hutchcroft
• Manjunath Narayana
• Ivaylo Boyadzhiev
• Yuguang Li

Assignees

• ZILLOW, INC.

Dates

Publication Date

20260505

Application Date

20221019

Priority Date

20220725

Claims (20)

1. 87 CLAIMS What is claimed is: 1. A system comprising: one or more hardware processors of one or more computing devices; and one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one computing device of the one or more computing devices to perform automated operations including at least: obtaining, for a building with multiple rooms, two or more first images acquired at two or more first acquisition locations in two or more rooms of the multiple rooms, wherein each of the first images has visual coverage of at least some walls of one of the two or more rooms containing the respective first acquisition location for that first image; analyzing visual data of the first images to generate initial room shapes of the two or more rooms, including, for each of the first images, generating a shape of the one room in which the respective first acquisition location for that first image is located and that includes a representation of the at least some walls of that one room; generating, a partial floor plan for the building with defined room shapes of the two or more rooms positioned relative to each other, including: determining relative positioning of the two or more rooms based at least in part on the visual data of the first images, and positioning the generated initial room shapes of the two or more rooms relative to each other based on the determined relative positioning; and generating a segmentation mask of an area of the positioned generated initial room shapes of the two or more rooms; determining a polygon shape for the generated segmentation mask that represents the two or more rooms positioned relative to each other; and 88 separating the determined polygon shape into the defined room shapes of the two or more rooms; determining, based at least in part on information associated with the generated partial floor plan, guidance related to one or more additional acquisition locations for the building at which to acquire one or more additional images; and providing information that includes the generated partial floor plan and the determined guidance related to the one or more additional acquisition locations, to cause additional images to be acquired at the one or more additional acquisition locations for use in generating a complete floor plan for the building.
2. 2. The system of claim 1 wherein the at least one computing device is a mobile computing device with one or more cameras, wherein the obtaining of the first images includes acquiring the first images by the mobile computing device under control of a user, and wherein the providing of the information includes displaying, in a graphical user interface and to the user, a visual representation of the generated partial floor plan and the determined guidance.
3. 3. The system of claim 1 wherein the generating of the segmentation mask of the area of the positioned generated initial room shapes of the two or more rooms incudes determining separate binary segmentation masks for each of the two or more rooms and combining the separate binary segmentation masks to generate an overall binary segmentation mask of the area of the positioned generated initial room shapes of the two or more rooms.
4. 4. The system of claim 1 wherein the analyzing of the visual data of the first images further includes determining visual overlap between the first images, and wherein the determining of the relative positioning of the two or more rooms based at least in part on the visual data of the first images includes using the determined visual overlap to identify the relative positioning. 89
5. 5. The system of claim 1 wherein the analyzing of the visual data of the first images includes, for each of the two or more rooms, determining uncertainty information for positioning of one or more walls of that room, and wherein the determining of the guidance related to one or more additional acquisition locations includes selecting at least one of the additional acquisition locations to reduce the determined uncertainty information for at least one wall of at least a first room of the two or more rooms.
6. 6. The system of claim 5 wherein the generated shapes for each of the first images is at least a partial room shape for the one room in which the respective first acquisition location for that first image is located, wherein the first room contains only one of the two or more first acquisition locations at which only one of the first images is acquired, wherein a second room of the two or more rooms contains multiple of the two or more first acquisition locations at which multiple of the first images are acquired, wherein generating of the initial room shapes of the two or more rooms further includes: combining, to produce the initial room shape of the second room, the at least partial room shapes for the multiple images acquired at the multiple first acquisition locations contained in the second room, including to reduce the determined uncertainty information for at least one wall of the produced initial room shape based on the combining; and selecting, to produce the initial room shape of the first room, the at least partial room shape for the only one first image acquired at the only one first acquisition location contained in the first room to use as the initial room shape of the first room, including retaining, in the produced initial room shape of the first room, the determined uncertainty information for the one or more walls of the first room, and wherein the providing of the information that includes the visual representation of the generated partial floor plan further includes presenting a visual representation of the generated partial floor plan with visual indications of uncertainty information about the positioning of walls for at least the first room, and 90 presenting visual indications overlaid on the generated partial floor plan related to the one or more additional acquisition locations.
7. 7. The system of claim 1 wherein the analyzing of the visual data of the first images further includes, for each of the two or more rooms, determining positions of at least one of a doorway or a non-doorway wall opening in that room, wherein the generating of the partial floor plan for the building involves including information in the defined room shapes of the two or more rooms to identify, for each of the two or more rooms, positions in that room of walls of that room and of the at least one of the doorway or the non-doorway wall opening for that room, and wherein the providing of the information further includes presenting a visual representation of the generated partial floor plan with visual indications of the added information to identify the positions in each of the two or more rooms of the walls of that room and of the at least one of the doorway or the non-doorway wall opening for that room.
8. 8. The system of claim 1 wherein the two or more rooms include first rooms that are a subset of the multiple rooms, and wherein the automated operations further include, after the providing of the information that includes the visual representation of the generated partial floor plan: acquiring a plurality of additional second images at a plurality of second acquisition locations in additional second rooms of the building, wherein the second acquisition locations include the additional acquisition locations; analyzing additional visual data of the additional second images to generate additional room shapes of the additional second rooms; generating a second partial floor plan for the building that includes the additional room shapes of the additional second rooms positioned relative to each other; determining that the visual data of the first images and the additional visual data of the additional second images lack visual overlap information between the first rooms and the additional second rooms; 91 determining additional guidance related to one or more further acquisition locations for the building at which to acquire further images to provide the visual overlap information between the first rooms and the additional second rooms; and providing updated information that includes visual representations of the generated partial floor plan and of the generated second partial floor plan, and that further includes the determined additional guidance related to the one or more further acquisition locations.
9. 9. The system of claim 8 wherein the automated operations further include, after the providing of the updated information: acquiring a plurality of additional third images at a plurality of additional third acquisition locations, wherein the third acquisition locations include the further acquisition locations; analyzing further visual data of the additional third images, including to determine the visual overlap information between the first rooms and the additional second rooms; generating a completed floor plan for the building that includes the generated partial floor plan and the generated second partial floor plan positioned relative to each other; and providing a visual representation of the generated completed floor plan, to enable further navigation of the building.
10. 10. The system of claim 1 wherein the determining of the guidance related to the one or more additional acquisition locations for the building is based on one or more problems identified in the visual data of the first images and includes generating one or more instructions to use in correcting the one or more problems, and wherein identifying of the one or more problems in the visual data of the first images is based on, for at least one of the first acquisition locations, at least one of a distance to at least one wall from the at least one first acquisition location, or at least one door blocking a view of another room separate from the room containing the at least one first acquisition location, or at least one object in the room containing 92 the at least one first acquisition location that blocks at least some of that room, or at least one type of room that is not included in the two or more rooms, or a lack of inter-image line-of-sight between visual data of at least two of the first images.
11. 11. The system of claim 1 wherein the providing of the information includes presenting, to a user participating in acquiring of images for the building, the generated partial floor plan and the determined guidance related to the one or more additional acquisition locations, including to present at least one of a location on the presented generated partial floor plan of one of the additional acquisition locations, or a direction on the presented generated partial floor plan to move toward one of the additional acquisition locations, or a location on the presented generated partial floor plan of a door to open, or a location on the presented generated partial floor plan of an obstacle to move, or a location on the presented generated partial floor plan of an obstacle to move around, or textual directions describing a location on the presented generated partial floor plan of one of the additional acquisition locations, or textual directions describing a direction to move toward one of the additional acquisition locations, or textual directions describing a door to open, or textual directions describing an obstacle to move, or textual directions describing an obstacle to move around, or textual directions describing at least one type of room that is not included in the two or more rooms and in which to acquire at least one additional image.
12. 12. The system of claim 1 wherein the automated operations further include at least one of: determining uncertainty information associated with one or more walls of the two or more rooms, and excluding the one or more walls from the generated initial room shapes based at least in part on the determined uncertainty information; or determining uncertainty information associated with one or more walls of the two or more rooms, and representing the determined uncertainty information for the one or more walls in the generated initial room shapes; or 93 performing the generating of the initial room shapes of the two or more rooms by, for each of the two or more rooms, using one or more trained machine learning models to identify a boundary between at least one of a floor or a ceiling of that room and walls of that room, and creating the initial room shape for that room as a twodimensional representation of the walls of that room; or performing the generating of the initial room shapes of the two or more rooms by, for each of the two or more rooms, generating a first three-dimensional model with planar surfaces that represent walls of that room, and using the generated first three-dimensional model as the initial room shape for that room; or performing the generating of the initial room shapes of the two or more rooms by, for each of the two or more rooms, generating a second three-dimensional model having a plurality of three-dimensional data points that represent points on walls of that room, and using the generated second three-dimensional model as the initial room shape for that room; or fitting the generated partial floor plan to a portion of a first outline of the building visible on an overhead image of the building, and performing the providing of the information by presenting, to a first user participating in acquiring of images for the building and based on the fitting, the generated partial floor plan overlaid on the overhead image of the building; or fitting the generated partial floor plan to a portion of a second outline of the building visible on an overhead image of the building, and performing the providing of the information by presenting, to a second user participating in acquiring of images for the building and based on the fitting, the determined guidance overlaid on the overhead image of the building at one or more positions related to the one or more additional acquisition locations; or fitting the generated partial floor plan to a portion of one of a plurality of defined plan templates, and performing the providing of the information by presenting, to a third user participating in acquiring of images for the building and based on the fitting, the generated partial floor plan overlaid on the one defined plan template; or 94 fitting the generated partial floor plan to a portion of one of a plurality of defined plan templates, and performing the providing of the information by presenting, to a fourth user participating in acquiring of images for the building and based on the fitting, the determined guidance overlaid on the one defined plan template at one or more positions related to the one or more additional acquisition locations.
13. 13. A computer-implemented method comprising: acquiring, by a computing device having one or more cameras and for a building with multiple rooms, two or more first images for two or more rooms of the multiple rooms, wherein the first images include an image at each of two or more first acquisition locations in the two or more rooms, and wherein each of the first images has visual coverage of at least some walls of one of the two or more rooms in which the respective first acquisition location for that first image is located; analyzing, by the computing device, visual data of the first images to generate at least partial room shapes of the two or more rooms and to determine visual overlap between the first images, including, for each of the first images, generating at least a partial room shape of the one room in which the respective first acquisition location for that first image is located and that includes at least a two-dimensional (2D) representation of the at least some walls of that one room; generating, by the computing device, a partial floor plan for the building with room shapes of the two or more rooms positioned relative to each other, including: determining, by the computing device, relative positioning of the two or more rooms based at least in part on the visual overlap between the first images; positioning, by the computing device, and using the determined relative positioning and generated at least partial room shapes from the first images, the at least partial room shapes of the two or more rooms relative to each other; generating, by the computing device, a segmentation mask of an area of the positioned at least partial room shapes of the two or more rooms, including combining separate segmentation masks for each of the two or more rooms; 95 determining, by the computing device, a polygon shape for the generated segmentation mask that represents the two or more rooms positioned relative to each other; and separating, by the computing device, the determined polygon shape into the room shapes of the two or more rooms positioned relative to each other; determining, by the computing device and based at least in part on the generated partial floor plan, guidance related to one or more additional acquisition locations for the building at which to acquire one or more additional images; and presenting, by the computing device, information that includes a visual representation of the generated partial floor plan and the determined guidance related to the one or more additional acquisition locations, to cause additional images to be acquired at the one or more additional acquisition locations for use in generating a complete floor plan for the building.
14. 14. The computer-implemented method of claim 13 wherein the analyzing of the visual data of the first images further comprises, for each of the two or more rooms, determining uncertainty information for positioning of one or more walls of that room, and wherein the determining of the guidance related to one or more additional acquisition locations for the building at which to acquire additional images includes identifying at least one of the additional acquisition locations as being at a position to reduce the determined uncertainty information for at least one wall of at least a first room of the two or more rooms.
15. 15. The computer-implemented method of claim 14 wherein the first room contains only one of the two or more first acquisition locations at which only one of the first images is acquired, wherein a second room of the two or more rooms contains multiple of the two or more first acquisition locations at which multiple of the first images are acquired, wherein generating of the at least partial room shapes of the two or more rooms further comprises: 96 combining, by the computing device and to produce the partial room shape of the second room, the generated at least partial room shapes for the multiple images acquired at the multiple first acquisition locations contained in the second room, including to reduce the determined uncertainty information for at least one wall of the produced partial room shape based on the combining; and selecting, by the computing device and to produce the partial room shape of the first room, the generated at least partial room shape for the only one first image acquired at the only one first acquisition location contained in the first room to use as the partial room shape of the first room, including retaining, in the produced partial room shape of the first room, the determined uncertainty information for the one or more walls of the first room, and wherein the presenting of the visual representation of the generated partial floor plan further includes presenting visual indications of uncertainty information about the positioning of walls for at least the first room, and presenting visual indications overlaid on the generated partial floor plan related to the one or more additional acquisition locations.
16. 16. The computer-implemented method of claim 13 wherein the analyzing of the visual data of the first images further comprises, for each of the two or more rooms, determining positions of at least one of a doorway or a non-doorway wall opening in that room, wherein the generating of the partial floor plan for the building includes adding information to the determined polygon shape to identify, for each of the two or more rooms, positions in that room of walls of that room and of the at least one of the doorway or the non-doorway wall opening for that room, wherein the presenting of the information that includes the visual representation of the generated partial floor plan further includes presenting the added information to identify the positions in each of the two or more rooms of the walls of that room and of the at least one of the doorway or the non-doorway wall opening for that room, 97 wherein the two or more rooms include first rooms that are a subset of the multiple rooms, and wherein the method further comprises, after the presenting of the information that includes the visual representation of the generated partial floor plan: acquiring, by the computing device, a plurality of additional second images at a plurality of second acquisition locations in additional second rooms of the building, wherein the second acquisition locations include the additional acquisition locations; analyzing, by the computing device, additional visual data of the additional second images to generate additional room shapes of the additional second rooms; generating, by the computing device, a second partial floor plan for the building that includes the additional room shapes of the additional second rooms positioned relative to each other; determining, by the computing device, that the visual data of the first images and the additional visual data of the additional second images lack visual overlap information between the first rooms and the additional second rooms, to inhibit positioning the partial floor plan relative to the second partial floor plan; determining, by the computing device, additional guidance related to one or more further acquisition locations for the building at which to acquire further images to provide the visual overlap information between the first rooms and the additional second rooms; presenting, by the computing device, updated information that includes visual representations of the generated partial floor plan and of the generated second partial floor plan, and that further includes the determined additional guidance related to the one or more further acquisition locations; acquiring, by the computing device, a plurality of additional third images at a plurality of additional third acquisition locations, wherein the third acquisition locations include the further acquisition locations; 98 analyzing, by the computing device, further visual data of the additional third images, including to determine the visual overlap information between the first rooms and the additional second rooms; generating, by the computing device, a completed floor plan for the building that includes the generated partial floor plan and the generated second partial floor plan positioned relative to each other; and presenting, by the computing device, a visual representation of the generated completed floor plan, to enable further navigation of the building.
17. 17. A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations including at least: acquiring, by one of the computing devices that has one or more cameras, two or more first images at two or more first acquisition locations in a building having multiple rooms, wherein each of the first images has visual coverage of at least some walls of one of the rooms in which the respective first acquisition location for that first image is located; analyzing, by the one or more computing devices, visual data of the first images to determine visual overlap between the first images and to generate one or more initial room shapes, including: generating, for each of the first images, a shape of the one room in which the respective first acquisition location for that first image is located and that includes a representation of at least some walls of that one room; and combining, for at least one room containing at least two of the first acquisition locations, the generated shapes of that room for the first images acquired at the at least two acquisition locations to produce one of the initial room shapes that is for that room; generating, by the one or more computing devices, partial mapping information for the building using one or more defined room shapes of one or more rooms containing the first acquisition locations, including: 99 generating, by the one or more computing devices, a segmentation mask of an area of the generated one or more initial room shapes at one or more positions that reflect the determined visual overlap; determining, by the one or more computing devices, a polygon shape for the generated segmentation mask that represents the one or more rooms; and producing, by the one or more computing devices, the one or more defined room shapes from the determined polygon shape; determining, by the one or more computing devices and based at least in part on information associated with the generated partial mapping information, guidance related to one or more additional acquisition locations for the building at which to acquire one or more additional images; and providing, by the one or more computing devices, information that includes the generated partial mapping information and the determined guidance related to the one or more additional acquisition locations, to enable additional images to be acquired at the one or more additional acquisition locations for use in generating completed mapping information for the building.
18. 18. The non-transitory computer-readable medium of claim 17 wherein the one computing device with the one or more cameras is a mobile computing device and the one or more computing devices include only that mobile computing device, wherein the acquiring of the first images is performed under control of a user, and wherein the providing of the information includes displaying, in a graphical user interface and to the user, a visual representation of the generated mapping information and the determined guidance.
19. 19. The non-transitory computer-readable medium of claim 17 wherein the two or more first acquisition locations are in two or more rooms of the building, wherein generating of the one or more initial room shapes involves generating initial room shapes for the two or more rooms, including producing the initial room shape for the at least one room using the combining and further including producing an additional initial room shape for another room of the two or more rooms 100 that contains only a single one of the first acquisition locations by selecting the shape generated for the first image acquired at that single one acquisition location as the additional initial room shape for the another room, wherein the generated partial mapping information includes a partial floor plan of the building with defined room shapes of the two or more rooms positioned relative to each other, wherein the generating of the partial mapping information further includes: determining, by the one or more computing devices, relative positioning of the two or more rooms based at least in part on the visual overlap between the first images; and positioning, by the one or more computing devices, and using the determined relative positioning, the generated initial room shapes of the two or more rooms relative to each other, wherein the generating of the segmentation mask is for an area of the positioned generated initial room shapes of the two or more rooms, and wherein the producing of the one or more defined room shapes from the determined polygon shape includes separating the determined polygon shape into the defined room shapes of the two or more rooms.
20. 20. The non-transitory computer-readable medium of claim 17 wherein the automated operations further include acquiring the additional images and using additional visual data of the additional images to complete generation of the mapping information for the building and presenting the mapping information to enable interactions with the presented mapping information by one or more users, and wherein the mapping information for the building includes at least one of a threedimensional model of an interior of the building or a virtual tour having userselectable inter-connections between at least some of the first and additional images that represent at least inter-image directions.

Description

AUTOMATED TOOLS FOR INCREMENTAL GENERATION OF BUILDING MAPPING INFORMATION TECHNICAL FIELD 100011 The following disclosure relates generally to using automated tools and associated techniques to generate mapping information for a defined area based at least in part on images acquired in the defined area, such as to automatically and incrementally generate a floor plan for a building using images that are acquired at acquisition locations for the building in accordance with automatically determined instructions for further image acquisitions, as well as subsequently using the generated mapping information. BACKGROUND 100021 In various fields and circumstances, such as architectural analysis, property inspection, real estate acquisition and development, remodeling and improvement services, general contracting and other circumstances, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to and enter the building, including to determine actual as-built information about the building rather than design information from before the building is constructed. However, it can be difficult to effectively capture, represent and use such building interior information, including to display visual information captured within building interiors to users at remote locations (e.g., to enable a user to fully understand the layout and other details of the interior, including to control the display in a user-selected manner). In addition, while a floor plan of a building may provide some information about layout and other details of a building interior, such use of floor plans has some drawbacks in certain situations, including that floor plans can be difficult to construct and maintain, to accurately scale and populate with information about room interiors, to visualize and otherwise use, etc. BRIEF DESCRIPTION OF THE DRAWINGS 100031 Figure 1 includes diagrams depicting an exemplary building environment and computing system(s) for use in embodiments of the present disclosure, including to generate and present information representing the building. Date Rei;:ue/Date Received 2022-10-19 100041 Figures 2A-2N illustrate examples of automated operations involved in acquisition, analysis, and use of images for a building, such as for resulting generation of a floor plan for the building in accordance with automatically determined acquisition instructions. 1ooos1 Figure 3 is a block diagram illustrating computing systems suitable for executing embodiments of one or more systems that perform at least some of the techniques described in the present disclosure. 1ooos1 Figure 4 illustrates an example embodiment of a flow diagram for an Automated Building Information Acquisition (ABIA) system routine in accordance with an embodiment of the present disclosure. 100011 Figure 5 illustrates an example embodiment of a flow diagram for an Image Capture and Analysis (ICA) system routine in accordance with an embodiment of the present disclosure. 1oooa1 Figures 6A-6B illustrate an example embodiment of a flow diagram for a Mapping Information Generation Manager (MIGM) system routine in accordance with an embodiment of the present disclosure. 100091 Figure 7 illustrates an example flow diagram for a Building Information Access system routine in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION 100101 The present disclosure describes techniques for using one or more computing devices to perform automated operations related to, as part of automatically generating mapping information of a defined area for subsequent use in one or more further automated manners, performing automated analyses of visual data of images acquired in the defined area in order to incrementally generate the mapping information and to determine acquisition instructions related to further acquisition of additional images and/or other types of data for use in further generation of the mapping information. In at least some embodiments, the defined area includes an interior of a multi-room building (e.g., a house, office, etc.}, the images include panorama images acquired at the building (e.g., 360° panorama images acquired at various acquisition locations within rooms of the building}, and the generated information includes a 2 Date Rei;:ue/Date Received 2022-10-19 floor plan of the building, such as a 20 (two-dimensional) overhead view (e.g., an orthographic top view) of a schematic floor map that is generated using information from the images - in at least some such embodiments, the generating of the mapping information is further performed without having or using depth information acquired from depth-sensing equipment about distances from the images' acquisition locations to walls or other objects in the surrounding building interior. The automated analysis of a building image may include determining an initial shape of a surrounding room (e.g., for at least some walls of