US-20260127138-A1 - FINE-GRAINED FILE SYSTEM AND FILE READING AND WRITING METHOD

US20260127138A1US 20260127138 A1US20260127138 A1US 20260127138A1US-20260127138-A1

Abstract

A fine-grained file system and a file reading and writing method are provided. When one or more first files require subfile reading or writing, a corresponding reading or writing operation is performed on subfiles of each first file by utilizing a double-layer index structure of this first file, implementing concurrent reading and writing functionality for large individual files, resulting in faster speeds and improved efficiency for file reading and writing.

Inventors

Shu Yin
Yanjie SONG
Guancheng LI

Assignees

SHANGHAITECH UNIVERSITY

Dates

Publication Date: 20260507
Application Date: 20230331
Priority Date: 20230328

Claims (8)

1 . A fine-grained file system, including: an index structure setting module, configured to set, for each file, a double-layer index structure associated with subfiles in the file, and store the double-layer index structure in both a non-volatile memory device and a dynamic random access memory; and a file reading and writing module, connected to the index structure setting module, wherein when one or more first files require subfile reading or writing, the file reading and writing module is configured to, for each of the first files, locate a corresponding first subfile based on the double-layer index structure of this first file, and load data in a data block storing the first subfile into the dynamic random access memory, so as to perform a reading or writing operation on the first subfile.
2 . The fine-grained file system according to claim 1 , wherein the double-layer index structure of each file is configured by: establishing a first mapping relationship between an inode constructed for this file and sub-inodes respectively constructed for the subfiles in the file, so as to obtain a first layer index structure; establishing a second mapping relationship between the sub-inodes mapped to the inode and data blocks storing the subfiles corresponding to the sub-inodes, so as to obtain a second layer index structure; combining the first layer index structure and the second layer index structure to obtain the double-layer index structure.
3 . The fine-grained file system according to claim 2 , wherein the inode includes a key-value mapping table; wherein the key-value mapping table includes one or more subfile numbers and one or more sub-inodes, with each of the subfile numbers corresponding one-to-one with one of the sub-inodes; the fine-grained file system obtains, based on subfile numbers of the first subfiles, sub-inodes that respectively correspond to the subfile numbers of the first subfiles according to the key-value mapping table.
4 . The fine-grained file system according to claim 3 , wherein for each subfile, its sub-inode includes a first pointer pair; wherein the first pointer pair is configured to determine a location of a file log of the subfile, so as to update information of the file log after a writing operation on the subfile is completed.
5 . The fine-grained file system according to claim 4 , wherein the file log includes a second pointer pair; wherein the second pointer pair is configured to obtain a physical address of the subfile when undergoing a reading or writing operation, and update the physical address; the second pointer pair is further configured to collect a file garbage after the physical address is updated.
6 . The fine-grained file system according to claim 5 , wherein the sub-inode includes a lock; wherein the lock is configured to lock the subfile when a reading operation is performed on the subfile, and unlock the subfile when the reading operation is completed.
7 . The fine-grained file system according to claim 5 , further including a free list associated with a plurality of data blocks for file management; wherein the fine-grained file system creates a new file when the plurality of data blocks managed by the free list are not fully utilized.
8 . A file reading and writing method, including: configuring, for each file, a double-layer index structure associated with subfiles in the file through a fine-grained file system; when one or more first files require subfile reading or writing, for each of the first files, locating a corresponding first subfile based on the double-layer index structure of this first file through the fine-grained file system, and performing a reading or writing operation on the first subfile.

Description

FIELD OF THE INVENTION The present disclosure relates to the field of file systems, and in particular, to a fine-grained file system and a file reading and writing method. BACKGROUND OF THE INVENTION Emerging non-volatile memory technologies come with impressive features such as low latency and high read-write bandwidth, making them well-suited for use as storage devices in existing data centers. These new hardware devices significantly boost the performance of storage subsystems, providing users with an improved experience. However, when considering the use of these novel non-volatile memory devices as storage, proper file system support becomes crucial. Unfortunately, existing file systems have not been specifically optimized for these new non-volatile devices. While current non-volatile memory file systems focus on optimizing overall bandwidth, read-write latency, and multi-core scalability for multi-file operations, there remains an unmet need. Specifically, in scenarios like scientific computing, artificial intelligence (AI) training, and sensor data collection, effective optimization strategies for concurrent read-write operations on large individual files are required. Therefore, while existing non-volatile memory file systems address certain aspects, there is still room for efficient solutions tailored specifically to concurrent read-write scenarios involving large individual files. SUMMARY OF THE INVENTION In view of the above-mentioned shortcomings, the present disclosure aims to provide a fine-grained file system and a file reading and writing method, which solve the problems in the related technology, specifically the inability to support concurrent read-write operations on large individual files and the lack of specialized optimization for non-volatile memory devices within traditional file systems. The present disclosure provides a fine-grained file system, including an index structure setting module, configured to set, for each file, a double-layer index structure associated with subfiles in the file, and store the double-layer index structure in both a non-volatile memory device and a dynamic random access memory; and a file reading and writing module, connected to the index structure setting module, wherein when one or more first files require subfile reading or writing, the file reading and writing module is configured to, for each of the first files, locate a corresponding first subfile based on the double-layer index structure of this first file, and load data in a data block storing the first subfile into the dynamic random access memory, so as to perform a reading or writing operation on the first subfile. In some embodiments of the present disclosure, the double-layer index structure of each file is configured by: establishing a first mapping relationship between an inode constructed for this file and sub-inodes respectively constructed for the subfiles in the file, so as to obtain a first layer index structure; establishing a second mapping relationship between the sub-inodes mapped to the inode and data blocks storing the subfiles corresponding to the sub-inodes, so as to obtain a second layer index structure; combining the first layer index structure and the second layer index structure to obtain the double-layer index structure. In some embodiments of the present disclosure, the inode includes a key-value mapping table. The key-value mapping table includes one or more subfile numbers and one or more sub-inodes, with each of the subfile numbers corresponding one-to-one with one of the sub-inodes. The fine-grained file system obtains, based on subfile numbers of the first subfiles, sub-inodes that respectively correspond to the subfile numbers of the first subfiles. In some embodiments of the present disclosure, for each subfile, its sub-inode includes a first pointer pair. The first pointer pair is configured to determine a location of a file log of the subfile, so as to update information of the file log after a writing operation on the subfile is completed. In some embodiments of the present disclosure, the file log includes a second pointer pair. The second pointer pair is configured to obtain a physical address of the subfile when undergoing a reading or writing operation, and update the physical address. The second pointer pair is further configured to collect a file garbage after the physical address is updated. In some embodiments of the present disclosure, the sub-inode includes a lock. The lock is configured to lock the subfile when a reading operation is performed on the subfile, and unlock the subfile when the reading operation is completed. In some embodiments of the present disclosure, the fine-grained file system further includes a free list associated with a plurality of data blocks for file management. The fine-grained file system creates a new file when the plurality of data blocks managed by the free list are not fully utilized. To achieve the above and other rela