CN-121985027-A - Excel file asynchronous importing method, device and system based on EventSource and OSS object storage

CN121985027ACN 121985027 ACN121985027 ACN 121985027ACN-121985027-A

Abstract

The disclosure relates to the technical field of data import, in particular to an Excel file asynchronous import method, device and system based on EventSource and OSS object storage, wherein the disclosure establishes EventSource connection between a client and a server through an EventSource protocol based on HTTP by a user; the method comprises the steps that a client uploads an Excel file to be imported into a file temporary storage medium through calling a preset file uploading interface, the file temporary storage medium is an OSS, an asynchronous import flow is triggered, a server pushes file import progress information to a corresponding client through established EventSource connection, the server downloads, analyzes and checks the Excel file to be imported in the file temporary storage medium in a streaming mode, and when verification fails, automatically generated error report files are pushed to the client, so that the method can achieve light-weight communication, safely and efficiently support import of large-volume files, and can generate visual error reports when the import fails, and therefore the problem of quick positioning and repairing is solved, and user experience is further improved.

Inventors

WU ZEHUA
LIU HUAIJIN
CHEN LINING
ZHANG BINGYU

Assignees

杉数科技(北京)有限公司
上海杉数网络科技有限公司
北京杉数智能科技有限公司
广州杉数科技有限公司

Dates

Publication Date: 20260505
Application Date: 20260126

Claims (10)

1. An Excel file asynchronous importing method based on EventSource and OSS object storage is characterized by comprising the following steps: The user establishes EventSource connection between the client and the server through an HTTP-based EventSource protocol; The client uploads an Excel file to be imported into a temporary file storage medium by calling a preset file uploading interface, wherein the temporary file storage medium is an OSS object storage medium; triggering an asynchronous import flow, and pushing file import progress information to the corresponding client by the server through the established EventSource connection; the server downloads, analyzes and checks the Excel file to be imported in the temporary file storage medium in a streaming manner, When the verification is successful, the Excel file to be imported is stored, and information of successful file import is pushed to the client through the EventSource connection; and when verification fails, pushing an automatically generated error report file to the client through the EventSource connection.
2. The asynchronous importing method of Excel file stored based on EventSource and OSS object according to claim 1, wherein the step of establishing EventSource connection between client and server by the user through EventSource protocol based on HTTP further comprises: the client page initiates a connection request to the designated server by using an EventSource API; and the server receives the connection request, and creates and maintains EventSource connection between the client and the server.
3. The asynchronous importing method of an Excel file stored based on EventSource and OSS objects according to claim 1, wherein the step of uploading the Excel file to be imported to a temporary file storage medium by the client by calling a preset file uploading interface further comprises: After the user selects the Excel file to be imported, the client uploads the Excel file to be imported to a temporary file storage medium by calling a preset file uploading interface, wherein the temporary file storage medium is an OSS object storage medium; After the file temporary storage medium successfully receives the Excel file to be imported, sending temporary storage information of the Excel file to be imported to the client, wherein the temporary storage information comprises at least one of information such as file id and storage address; and the client receives the temporary storage information of the Excel file to be imported.
4. The asynchronous importing method of Excel file stored based on EventSource and OSS object according to claim 1, wherein the step of triggering an asynchronous importing process to push file importing progress information to the corresponding client by the server through the established EventSource connection further comprises: The client uses the obtained temporary storage information of the Excel file to be imported as a parameter, and invokes a file import trigger interface of the server through an HTTP request; The file import triggering interface verifies the validity of the HTTP request; The file import trigger interface pushes file import progress information to the client through the established EventSource connection; And the file import triggering interface carries out asynchronous import processing on an import task submission thread pool, pushes the import task submission information to the client, and releases the HTTP request.
5. The asynchronous importing method of an Excel file stored based on EventSource and OSS objects according to claim 1, wherein the server downloads, parses and verifies the Excel file located in the temporary file storage medium in a streaming manner, and further comprises: after receiving the importing task, the asynchronous task processor of the server downloads the Excel file to be imported from the file temporary storage medium in a streaming mode according to the temporary storage information of the Excel file to be imported; Analyzing the input stream of the Excel file to be imported into a Java object set based on a preset Java entity class with @ ExcelProperty annotation by using an Easy Excel library; And the asynchronous task processor circularly traverses the acquired Java object set, performs piece-by-piece verification and error information collection on each object, and automatically generates a visual error report file.
6. The asynchronous importing method of Excel file based on EventSource and OSS object storage according to claim 5, wherein the asynchronous task processor circularly traverses the obtained Java object set, performs piece-by-piece verification and error information collection on each object, and automatically generates a visual error report file, and further comprises: sequentially performing basic format verification, business logic verification, error information collection and error report generation, wherein, The execution basic format verification comprises calling a validator validate of JSR-380 specification, automatically verifying annotation information such as @ NotNull, @ Size, @ Pattern and the like marked on the object, and capturing corresponding error information if the basic format verification fails; The service logic verification is executed, wherein the service logic verification comprises at least one of database uniqueness check and service rule compliance check, and if the service logic verification fails, corresponding error information is captured; The performing error information collection includes collecting error information related to the failure of the basic format check and/or the failure of the business logic check.
7. The asynchronous importing method of Excel file stored based on EventSource and OSS object according to claim 6, wherein when verification fails, the step of pushing the automatically generated error report file to the client through the EventSource connection further comprises: Traversing the Java object set, and setting a corresponding error information character string into an additional extension field of each Java object instance with failed importing according to a preset error information index table to form an error information item; based on original Excel file stream information, automatically adding the formed error information item on the basis of an original data column to generate a new Excel file input stream, wherein the error information item is verification information of corresponding data; uploading the formed new Excel file input stream to a file temporary storage medium to obtain a temporary downloading address; And pushing an error report file to the client through the EventSource connection, wherein the error report file comprises the temporary downloading address of the abnormal file in the file temporary storage medium.
8. An Excel file asynchronous importing device based on EventSource and OSS object storage, which is characterized by comprising: The EventSource connection establishment module is used for establishing EventSource connection between the client and the server through an HTTP-based EventSource protocol by a user; The uploading file temporary storage module is used for uploading an Excel file to be imported into a file temporary storage medium by calling a preset file uploading interface by the client, wherein the file temporary storage medium is an OSS object storage medium; triggering an asynchronous import flow, and pushing file import progress information to the corresponding client by the server through the established EventSource connection; The server downloads, analyzes and checks the Excel file to be imported in the temporary file storage medium in a streaming manner, When the verification is successful, the Excel file to be imported is stored, and information of successful file import is pushed to the client through the EventSource connection; and when verification fails, pushing an automatically generated error report file to the client through the EventSource connection.
9. An Excel file asynchronous import system based on EventSource and OSS object storage, which is characterized by comprising a memory and a processor: the memory is used for storing program codes and transmitting the program codes to the processor; The processor is configured to execute the Excel file asynchronous import method according to any one of claims 1-7 based on EventSource and OSS object storage according to the instruction in the program code.
10. A computer readable storage medium storing program code for performing the Excel file asynchronous import method based on EventSource and OSS object storage of any one of claims 1-7.

Description

Excel file asynchronous importing method, device and system based on EventSource and OSS object storage Technical Field The disclosure relates to the technical field of data import, in particular to an Excel file asynchronous import method, device and system based on EventSource and OSS object storage. Background In Web applications, bulk data importation (e.g., via Excel, CSV files) is a very common business requirement. Conventional data import schemes typically employ a synchronous request-response mode based on the HTTP protocol. In this mode, the front end sends the entire file directly to the back end server as an HTTP request body (e.g., multi part/form-data). After the back-end server receives the file, the file is analyzed, checked (including format check, service logic check, etc.) and subjected to persistent warehousing operation in a memory or temporary storage. After the whole processing procedure is finished, the back end returns the final result (success or failure) to the front end through one HTTP response. This synchronous processing mode exposes significant drawbacks in the face of large files, complex business checks, or high concurrency scenarios. First, HTTP connections have timeout limitations, long processing can cause connection breaks, and the user experience is very poor. Second, the user is in a "black box" waiting state during the entire processing period, and cannot know the processing progress (e.g., how many pieces have been processed, success rate, etc.). Finally, when verification fails, the traditional scheme only returns a simple error prompt, so that a user is difficult to quickly locate and repair specific errors in hundreds of lines of data in the file, repeated attempts are needed, and the efficiency is low. The related art currently existing is as follows: WebSocket is a protocol that performs full duplex communication over a single TCP connection. The method and the system enable data exchange between the client and the server to be simpler, and allow the server to actively push data to the client. In the WebSocket API, the browser and the server only need to complete one handshake, and a persistent connection can be created between the two and bidirectional data transmission can be performed. The method is suitable for applications with high real-time requirements, such as chat rooms, real-time games and the like. EventSource is an API provided in HTML5 for receiving Server send events (Server-SENT EVENTS, SSE). It allows clients to receive event streams sent in real time by the server over HTTP connections. Unlike WebSocket, eventSource is unidirectional, only supporting servers to send messages to clients. The method is simple to use, automatically reconnects, supports event ID and message type, and is suitable for scenes which need server pushing but the client does not need to send complex messages, such as news pushing, real-time state updating and the like. SSE (Server-SENT EVENTS) SSE is a technique in which a Server pushes events to clients in real time. It is realized by long connections and specific MIME types (text/event-stream) based on the HTTP protocol. The server may continue to send data segments, each beginning with "data:" and ending with two wraps. SSE supports event types and reconnection mechanisms, but allows only one-way communication of the server to the client. The compatibility is good, and the method is suitable for scenes such as stock quotation, log stream and the like. The OSS object storage OSS (Object Storage Service) is a cloud storage service which is massive, safe, low in cost and high in reliability, and is suitable for storing any type of files. It provides RESTful API, supports uploading and downloading data whenever and wherever, and has high availability and persistence. The OSS adopts a distributed architecture, has no limit on capacity and file quantity, and is suitable for storing pictures, audios and videos, backup files and the like. In Web applications, it is common to act as a static resource store or file transfer station to relieve server stress. JSR-380 Specification JSR-380 is Java Bean Validation 2.0.0 Specification defining a set of APIs for validating Java object data. It declares constraints on the entity class field through annotations (e.g. @ NotNull, @ Size, @ Email, etc.), and then checks through the Validator interface. The specification supports grouping verification, custom constraint and internationalization information, and is widely applied to data verification scenes, such as Web request parameter verification, database entity verification and the like, so that the integrity and accuracy of data are ensured. Among them, a document importing method, device, storage medium and processor (patent number CN119967046 a) in the prior art disclose a document importing method, device, storage medium and processor. In the scheme, a connection establishment request sent by a client is received, a WebSocket conn