Search

CN-121996655-A - Data processing method, apparatus, device, computer readable storage medium and product

CN121996655ACN 121996655 ACN121996655 ACN 121996655ACN-121996655-A

Abstract

The embodiment of the disclosure provides a data processing method, a device, equipment, a computer readable storage medium and a product, wherein the method comprises the steps of obtaining search information input by a user, determining at least one data fragment matched with the search information in a key value database, storing index data of a plurality of users in the key value database, carrying out fragment storage on the index data by taking a preset time interval as a fragment granularity, and adjusting the fragment granularity of at least one data fragment in response to the at least one data fragment meeting a preset adjustment condition to obtain the adjusted at least one data fragment so as to enable the association parameters of the adjusted data fragments to meet the preset condition. Therefore, the method and the device can automatically combine the data fragments with small capacity, reduce the storage cost, automatically split the data fragments with large capacity, and reduce the problems of large service bandwidth and high delay occupied by the data fragments with large capacity in the data reading and writing process.

Inventors

  • WANG JUNXIONG
  • ZHANG CHUANG

Assignees

  • 北京字跳网络技术有限公司

Dates

Publication Date
20260508
Application Date
20241108

Claims (13)

  1. 1. A method of data processing, comprising: acquiring search information input by a user, wherein the search information comprises identification information of the user and a time range; Determining at least one data fragment matched with the search information in a preset key value database, wherein index data of a plurality of users are stored in the key value database, and the index data are stored in fragments by taking a preset time interval as a fragment granularity; And responding to the at least one data fragment meeting a preset adjustment condition, adjusting the fragment granularity of the at least one data fragment to obtain at least one adjusted data fragment, so that the association parameters of the adjusted data fragments meet the preset condition.
  2. 2. The method of claim 1, wherein adjusting the slice granularity of the at least one data slice in response to the at least one data slice satisfying a preset adjustment condition comprises: Triggering a slicing adjustment task in response to the at least one data slicing meeting a preset adjustment condition, and sending the slicing adjustment task to a preset message queue; and controlling a preset slicing service module to asynchronously acquire a slicing adjustment task from the message queue, and adjusting the slicing granularity of the at least one data slicing.
  3. 3. The method of claim 1, wherein adjusting the slice granularity of the at least one data slice in response to the at least one data slice satisfying a preset adjustment condition comprises: determining the corresponding slicing capacity of each data slice; calculating a capacity mean value corresponding to the at least one data fragment; and adjusting the slicing granularity of the at least one data slicing according to the fact that the capacity average value is not in the preset capacity range.
  4. 4. The method of claim 3, wherein adjusting the fragmentation granularity of the at least one data fragment in response to the capacity mean value not being within a preset capacity range comprises: If the capacity mean value exceeds the capacity range, splitting the at least one data slice to reduce the slice granularity corresponding to the at least one data slice; and if the capacity mean value is lower than the capacity range, merging the at least one data slice to improve the slice granularity corresponding to the at least one data slice.
  5. 5. The method of claim 4, wherein the splitting the at least one data slice comprises: For each data fragment, splitting the data fragment into a preset number of sub-fragments, and obtaining at least one adjusted data fragment so that capacity information associated with the at least one adjusted data fragment is in a preset capacity range; or the splitting operation is performed on the at least one data slice, including: Determining a target fragment with a capacity exceeding the capacity range in the at least one data fragment; and splitting the target fragments according to the capacity range so as to ensure that the capacity of the split target fragments is within the capacity range.
  6. 6. The method of claim 4, wherein the merging the at least one data slice comprises: combining a preset number of data fragments into an adjusted data fragment; or the splitting operation is performed on the at least one data slice, including: determining a target fragment with capacity lower than the capacity range in the at least one data fragment; And carrying out merging operation on the target fragments according to the capacity range so as to enable the capacity of the merged target fragments to be in the capacity range.
  7. 7. The method according to any one of claims 1-6, wherein adjusting the fragmentation granularity of the at least one data fragment, after obtaining the adjusted at least one data fragment, further comprises: And updating metadata in the preset fragments based on the adjustment operation for the at least one data fragment, wherein the metadata comprises the current adjustment state, the fragment granularity and the adjustment time of each data fragment, and the adjustment state comprises a normal state, a split state and a merging state.
  8. 8. The method of claim 7, further comprising, after the updating the metadata in the preset tile: Responding to the content searching operation triggered by the user, and acquiring metadata from the preset fragments; Searching the key value database for a target data fragment matched with the search information based on the fragment granularity in the metadata and the search information associated with the content search operation.
  9. 9. The method of claim 7, wherein the method further comprises: Determining at least one adjustment fragment in the key value database which is currently in a split state or a merging state according to the metadata; determining the corresponding adjustment time length of each adjustment fragment according to the adjustment time and the current time; and if the adjustment time length exceeds a preset time length threshold value, carrying out adjustment operation on the adjustment fragments again.
  10. 10. A data processing apparatus, comprising: The acquisition module is used for acquiring search information input by a user, wherein the search information comprises identification information of the user and a time range; the determining module is used for determining at least one data fragment matched with the search information in a preset key value database, wherein index data of a plurality of users are stored in the key value database, and the index data are stored in fragments by taking a preset time interval as a fragment granularity; The adjusting module is used for adjusting the fragment granularity of the at least one data fragment in response to the at least one data fragment meeting a preset adjusting condition, so as to obtain at least one adjusted data fragment, and the associated parameters of the adjusted data fragments meet the preset condition.
  11. 11. An electronic device is characterized by comprising a processor and a memory; The memory stores computer-executable instructions; the processor executing computer-executable instructions stored in the memory, causing the processor to perform the data processing method of any one of claims 1 to 9.
  12. 12. A computer-readable storage medium, in which computer-executable instructions are stored which, when executed by a processor, implement the data processing method of any one of claims 1 to 9.
  13. 13. A computer program product comprising a computer program which, when executed by a processor, implements the data processing method according to any one of claims 1 to 9.

Description

Data processing method, apparatus, device, computer readable storage medium and product Technical Field The embodiment of the disclosure relates to the technical field of data processing, in particular to a data processing method, a device, equipment, a computer readable storage medium and a product. Background A Key-Value (KV) database is a storage system based on Key-Value pairs, and data is stored in the form of Key-Value pairs. Each key is unique and corresponds to a particular value (data slice). In the related art, data fragments corresponding to different key values in a KV database may be stored in uneven sizes. Therefore, when data is read and written to the data fragments with larger capacity, the service bandwidth is often occupied, the delay is high, and the read and write performance is poor. When data is read and written for the data fragments with smaller capacity, one call flow is generated, and the service cost is increased. Disclosure of Invention The embodiment of the disclosure provides a data processing method, a device, equipment, a computer readable storage medium and a product, which are used for solving the technical problem that the system capacity, the data read-write cost and the performance are negatively influenced under the condition of uneven data slicing capacity and density in a KV database. In a first aspect, an embodiment of the present disclosure provides a data processing method, including: acquiring search information input by a user, wherein the search information comprises identification information of the user and a time range; Determining at least one data fragment matched with the search information in a preset key value database, wherein index data of a plurality of users are stored in the key value database, and the index data are stored in fragments by taking a preset time interval as a fragment granularity; And responding to the at least one data fragment meeting a preset adjustment condition, adjusting the fragment granularity of the at least one data fragment to obtain at least one adjusted data fragment, so that the association parameters of the adjusted data fragments meet the preset condition. In a second aspect, embodiments of the present disclosure provide a data processing apparatus, including: The acquisition module is used for acquiring search information input by a user, wherein the search information comprises identification information of the user and a time range; the determining module is used for determining at least one data fragment matched with the search information in a preset key value database, wherein index data of a plurality of users are stored in the key value database, and the index data are stored in fragments by taking a preset time interval as a fragment granularity; The adjusting module is used for adjusting the fragment granularity of the at least one data fragment in response to the at least one data fragment meeting a preset adjusting condition, so as to obtain at least one adjusted data fragment, and the associated parameters of the adjusted data fragments meet the preset condition. In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory; The memory stores computer-executable instructions; The processor executes computer-executable instructions stored in the memory to cause the at least one processor to perform the data processing method as described above in the first aspect and the various possible designs of the first aspect. In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, implement the data processing method according to the first aspect and the various possible designs of the first aspect. In a fifth aspect, embodiments of the present disclosure provide a computer program product comprising a computer program which, when executed by a processor, implements the data processing method according to the first aspect and the various possible designs of the first aspect. The data processing method, the device, the equipment, the computer readable storage medium and the product provided by the embodiment store index data of a plurality of users in a key value database, and store the index data in a slicing way by taking a preset time interval as a slicing granularity. The data capacity of different data slices is different. In order to avoid negative effects of data fragments with uneven capacity and density on system capacity, cost and performance, if at least one data fragment obtained based on search information query is determined to meet preset adjustment conditions in the process of data query, the fragment granularity of the at least one data fragment is adjusted so that the association parameters of the adjusted data fragments meet preset conditions. Therefore, the granularity of the fragments of the pluralit