US-12619941-B2 - Technical candidate certification system

US12619941B2US 12619941 B2US12619941 B2US 12619941B2US-12619941-B2

Abstract

Systems and methods to asynchronously vet, appraise, grade, rank and certify technical candidates for hiring are described. The technical candidate certification system administers and evaluates a set of specialized and customized evaluations of a technical candidate to generate a technical candidate ranking value, the ranking value typically provided relative to other technical candidates. The set of evaluations are customized to the job requirements of the client employer and to the profile of the candidate employee, and may be specialized to the hiring of software developer candidate employees. In one aspect, the technical candidate ranking system administers a software coding test and a technical interview, and provides an assessment and measure of a candidate's language proficiency. The client employer is provided with a ranking value of the recommended employee candidates.

Inventors

Michael Frank Piccolo
David Matthew Jackson

Assignees

FullStack Labs, Inc.

Dates

Publication Date: 20260505
Application Date: 20240515

Claims (20)

1 . A system for certifying a candidate comprising: a system database configured to: i) store a notional coding file comprising endpoints and software bugs; ii) store a candidate updated coding file and a candidate new coding file; iii) store a set of system scoring parameters comprising a candidate code execution time threshold value; iv) store a set of technical interviews comprising a set of technical interview questions; and v) store a candidate interview audio answer data; a user interface configured to: i) provide the notional coding file to the candidate; ii) receive from the candidate the candidate new coding file and the candidate updated coding file, the notional coding file changed by the candidate to create the candidate updated coding file; iii) present to the candidate a particular technical interview; and iv) receive the candidate interview audio answer data; a processor configured to: i) receive a set of candidate requirements; ii) retrieve the candidate new coding file and the candidate updated coding file from the system database; iii) retrieve the set of system scoring parameters from the system database; iv) record the candidate interview audio answer data; and v) record a candidate verbal comment data generated by the candidate while the candidate created the candidate updated coding file; wherein the processor operates to: compile and execute the candidate new coding files and the candidate updated coding file; execute an end to end test on the candidate updated coding file and measure an execution time of the candidate new coding file; transcribe the candidate verbal comment data into digitally transcribed candidate verbal comment data; based on the end to end test, determine if the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; calculate a time difference between the execution time of the candidate new coding file and the candidate code execution time threshold value; calculate a candidate coding score based on at least the time difference and whether the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; execute the technical interview to generate the candidate interview audio answer data; digitally transcribe the candidate interview audio answer data into transcribed candidate interview answer data associated with the set of technical interview questions; calculate, using at least one of an AI model and an LLM model, a candidate technical interview score based on the transcribed candidate interview answer data; and generate a ranking score and a certification decision for the candidate based at least on the candidate coding score and the candidate technical interview score.
2 . The system of claim 1 , wherein the candidate technical interview score is established using at least both an AI model and an LLM model.
3 . The system of claim 2 , wherein the coding score is also at least based on a determination of the candidate updated coding file correcting the software bugs.
4 . The system of claim 1 , wherein the coding score is also at least based on a code assessment of at least one of code smells, code vulnerabilities, code reliability, and code linting.
5 . The system of claim 1 , wherein a code assessment is performed using an AI model.
6 . The system of claim 1 , wherein the transcribed candidate interview answer data is transformed into a set of vector representations.
7 . The system of claim 6 , wherein the set of vector representations are used as input into an AI model to establish the candidate technical interview score.
8 . The system of claim 1 , wherein the candidate coding score is also based at least on an evaluation of at least one of test code execution speed, test code security, and test code linting by one or both of an AI model and an LLM model.
9 . A method for certifying a candidate comprising: providing a system database configured to: i) store a notional coding file comprising endpoints and software bugs; ii) store a candidate updated coding file and a candidate new coding file; iii) store a set of system scoring parameters comprising a candidate code execution time threshold value; iv) store a set of technical interviews comprising a set of technical interview questions; and v) store a candidate interview audio answer data; providing a user interface configured to: i) provide the notional coding file to the candidate; ii) receive from the candidate the candidate new coding file and the candidate updated coding file, the notional coding file changed by the candidate to create the candidate updated coding file; iii) present to the candidate a particular technical interview; and iv) receive the candidate interview audio answer data; providing a processor configured to: i) receive a set of candidate requirements; ii) retrieve the candidate new coding file and the candidate updated coding file from the system database; iii) retrieve the set of system scoring parameters from the system database; iv) record the candidate interview audio answer data; and v) record a candidate verbal comment data generated by the candidate while the candidate created the candidate updated coding file; wherein the processor operates to: compile and execute the candidate new coding files and the candidate updated coding file; execute an end to end test on the candidate updated coding file and measure an execution time of the candidate new coding file; transcribe the candidate verbal comment data into digitally transcribed candidate verbal comment data; based on the end to end test, determine if the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; calculate a time difference between the execution time of the candidate new coding file and the candidate code execution time threshold value; calculate a candidate coding score based on at least the time difference and whether the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; execute the technical interview to generate the candidate interview audio answer data; digitally transcribe the candidate interview audio answer data into transcribed candidate interview answer data associated with the set of technical interview questions; calculate, using at least one of an AI model and an LLM model, a candidate technical interview score based on the transcribed candidate interview answer data; and generate a ranking score and a certification decision for the candidate based at least on the candidate coding score and the candidate technical interview score.
10 . The method of claim 9 , wherein the candidate technical interview score is established using at least both an AI model and an LLM model.
11 . The method of claim 10 , wherein the coding score is also at least based on a determination of the candidate updated coding file correcting the software bugs.
12 . The method of claim 9 , the coding score is also at least based on a code assessment of at least one of code smells, code vulnerabilities, code reliability, and code linting.
13 . The method of claim 9 , wherein a code assessment is performed using an AI model.
14 . The method of claim 9 , wherein the transcribed candidate interview answer data is transformed into a set of vector representations.
15 . The method of claim 14 , wherein the set of vector representations are used as input into an AI model to establish the candidate technical interview score.
16 . The method of claim 9 , wherein the candidate coding score is also based at least on an evaluation of at least one of test code execution speed, test code security, and test code linting by one or both of an AI model and an LLM model.
17 . A system for certifying a candidate comprising: a system database configured to: i) store a notional coding file comprising endpoints; ii) store a candidate updated coding file and a candidate new coding file; iii) store a set of technical interviews comprising a set of technical interview questions; and iv) store a candidate interview audio answer data; a user interface configured to: i) provide the notional coding file to the candidate; ii) receive from the candidate the candidate new coding file and the candidate updated coding file, the notional coding file changed by the candidate to create the candidate updated coding file; iii) administer to the candidate a particular technical interview; and iv) receive the candidate interview audio answer data; a processor configured to: i) receive a set of candidate requirements; ii) retrieve the candidate new coding file and the candidate updated coding file from the system database; iii) record the candidate interview audio answer data; and iv) record a candidate verbal comment data generated by the candidate while the candidate created the candidate updated coding file; wherein the processor operates to: compile and execute the candidate new coding files and the candidate updated coding file; execute an end to end test on the candidate updated coding file and measure an execution time of the candidate new coding file; transcribe the candidate verbal comment data into digitally transcribed candidate verbal comment data; based on the end to end test, determine if the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; calculate a candidate coding score based on whether the candidate updated coding file does or does not integrate with the endpoints of the notional coding file; execute the technical interview to generate the candidate interview audio answer data; digitally transcribe the candidate interview audio answer data into transcribed candidate interview answer data associated with the set of technical interview questions; calculate, using at least one of an AI model and an LLM model, a candidate technical interview score based on the transcribed candidate interview answer data; and generate a ranking score and a certification decision for the candidate based at least on the candidate coding score and the candidate technical interview score.
18 . The system of claim 17 , wherein the candidate technical interview score is established using at least both an AI model and an LLM model.
19 . The system of claim 17 , wherein the coding score is also at least based on a code assessment of at least one of code smells, code vulnerabilities, code reliability, and code linting performed by an AI engine.
20 . The system of claim 17 , wherein the candidate coding score is also based at least on an evaluation of at least one of test code execution speed, test code security, and test code linting by one or both of an AI model and an LLM model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional patent application of and claims the benefit of U.S. Provisional Patent Application No. 63/466,958 titled “Technical Candidate Ranking System” filed May 16, 2023, and U.S. Provisional Patent Application No. 63/634,262 titled “Technical Candidate Interview and Coding Scoring System of a Technical Candidate Ranking System” filed Apr. 15, 2024, the disclosures of both are hereby incorporated herein by reference in entirety. FIELD The disclosure relates generally to systems and methods involving technical candidate certification for hiring, and specifically to systems and methods to asynchronously vet, grade, rank and certify technical candidates for hiring. BACKGROUND Conventional approaches to the hiring of technical candidates are at best slow and inaccurate and at worst frustrate if not anger the employee candidate and the hiring employer. The hiring of technical candidates, such as software developers, is particularly challenging given the specialized skills involved and the limited hiring pool of employee candidates. Many employee candidates are non-native language speakers of the hiring employer, presenting a further challenge to assess language proficiency. Given such challenges, many employers outsource technical candidate hiring recommendations to third party technical candidate outsourcing entities. Employee candidates and hiring parties (referred to as “clients”) generally prefer a streamlined hiring process that saves time and effort compared with conventional interviewing that frequently does not involve skill tests. Some candidates and/or clients prefer automated processes where there is limited to no direct verbal communication. However, if verbal communication is involved, a party may prefer a limited number of interviews and no skill tests. Broadly, junior candidates are more accepting of skill tests than senior candidates (who often have limited time available.) Candidate identification and skill verification (aka vetting) companies may help speed up the hiring process and bring more candidates into the pipeline, but they typically do fewer tests than necessary, which significantly affects the analysis of the candidates. This precision in assessing the candidates' profiles is termed “accuracy.” If the candidate identification and skill verification (aka “outsourcing”) company doesn't provide accuracy and the client understands the importance of it, they will spend more time deciding on a candidate and more money analyzing and filtering out a good one from the others. Also, if an outsourcing company takes too long to make a decision about the candidates, the candidate may take on a new position with another company, and, in the meantime, the client may decide to cancel the deal for various reasons, such as having already found what they need with other outsourcing companies. The clients save time if the candidates' profiles contain all the details they need to know. However, it doesn't matter how much data there is if it is not accurate or searchable, which is why the attribute depends directly on accuracy and platform. Lastly, some companies, such as very large software-intensive companies, may perform excessive testing (i.e., several hours with multiple participating teams) which can be inefficient and limit the candidate pool. Conventional technical outsourcing entities vary widely in approaches to handling technical candidate hiring recommendations. Most approaches require sequential processing of a candidate applicant, meaning one part of the review process may not occur until another is complete, leading to unnecessary delays that frequently cause the applicant to disengage. Many processes produce inaccurate results in that applicants are recommended for hire who are not qualified for the position, creating disappointed employer clients. Furthermore, conventional processes and systems result in (candidate) assets that are not both distributable and visible (in contrast, the technical candidate ranking system of the disclosure provides assets that are both distributable and visible). The technical candidate ranking system of the disclosure solves these challenges with traditional approaches, systems, and methods to identify certified job candidates. (The term “certified” means genuine or authentic). The technical candidate ranking system administers and evaluates a set of specialized and customized evaluations of a technical candidate to generate a technical candidate ranking value, the ranking value typically provided relative to other technical candidates. The set of evaluations are customized to the job requirements of the client employer and to the profile of the candidate employee, and specialized to the hiring of software developer candidate employees. In one aspect, the technical candidate ranking system administers and autonomously grades a software coding test and a technical interview, and provides an a