Andrew Wegner | Ponderings of an Andy - ai-interviews

I Proved AI Could Beat Our Technical Interviews. Then I Had to Fix Them.

2026-03-25T15:00:00-05:00

The interview process I used to run¶

For years, I used timed online assessments as part of my hiring process. At PacketFabric, when we needed to scale the engineering team rapidly during our Series B, I helped evaluate several assessment platforms to find one that balanced candidate experience with the quality of signal we received as hiring managers. We landed on Woven, and it worked. The assessments helped us move quickly through a high volume of candidates and gave us comparable data points across applicants.

I had doubts, though. I've been opposed to LeetCode style interviews forever. Woven wasn't LeetCode as it provided amazing feedback to the candidate on what did and did not work. I liked the company so much that I joined Woven for a while. But even with the best assessment platform I'd found, I saw cracks. The candidates who passed the assessments and joined the team didn't always succeed in the ways I expected. Some produced clean code under time pressure but struggled to collaborate with the team. Others passed technical screens convincingly but couldn't translate that skill into shipping features in a real codebase with real constraints. The assessment told me they could solve a problem in isolation. It told me very little about whether they could do the actual job.

Then ChatGPT arrived, and my doubts became certainties.

In late 2022 and early 2023, I ran ChatGPT through technical assessments on LeetCode, TestGorilla, CodeSignal, Codility, HackerRank, and CoderByte. It passed most of them. Not marginally; it produced solutions that would have advanced through our pipeline. That wasn't a theoretical concern anymore. If a freely available tool could pass our technical assessment, we weren't evaluating engineering skill. We were evaluating test-taking ability, and we now had a machine that was better at test-taking than many humans.

That was 2022 and 2023. We are now several AI model generations beyond that early ChatGPT. Tools like Claude Code, Codex, Cursor, and GitHub Copilot are used by millions of developers daily. These aren't novelty chatbots any longer. They're integrated development environments that write, debug, and refactor production code. The gap between what AI could do when I ran those tests and what it can do today is enormous. Any assessment that was vulnerable to early ChatGPT is trivially solvable now.

I had to rebuild how I hire.

What timed assessments and take-homes actually measure¶

Timed assessments reward speed and pattern recognition under artificial pressure. That correlates poorly with the actual work. Nobody ships production code in 45 minutes with a countdown timer and no access to documentation. The skills that make someone fast on a Codility problem like memorized algorithms, familiarity with specific puzzle patterns, comfort performing under observation, are not the same skills that make someone effective on a team building real products.

Take-home projects have the opposite problem. Without time constraints or observation, there's no way to verify who actually did the work, how long it took, or what tools they used. Even before AI, take-homes had issues: they disproportionately disadvantage candidates with families or other time commitments, and they're difficult to evaluate consistently across candidates.

Both formats share a deeper flaw though. They optimize for a narrow technical signal while telling you almost nothing about how a person thinks, communicates, or works with others. I've seen candidates pass assessments convincingly and then struggle on the job because they couldn't explain their decisions to a colleague, couldn't adapt when requirements changed, or couldn't take ownership of a system beyond the specific code they wrote. The assessment didn't predict any of that. It just told me they could solve a contrived problem under contrived conditions.

I want to be fair to these tools, though. They feel rigorous and objective on the surface. That's why they're popular. The scores are comparable, the process is standardized, and it feels like you're making data-driven decisions. But the data is measuring the wrong thing.

What I look for instead¶

My core philosophy has stayed the same across companies even though the specific steps change based on the team, the roles, and the company's needs. The philosophy is this: I want to understand how someone thinks, how they communicate, and whether they take genuine ownership of their work. The specific technology matters, but it matters less than those three things.

How I evaluate depends on the level of the role.

Mid-level and senior engineers¶

For experienced candidates, I focus on past project deep-dives. I ask the candidate to describe something they built - not a rehearsed elevator pitch, but a real system they worked on. Then I poke at it.

The questions I ask depend on what the candidate brings up, and I rotate through several lines of inquiry based on what I'm hearing. If someone describes a system they architected, I'll ask about the tradeoffs they made and why. If they describe a project that clearly had rough patches, I'll ask what went wrong and dig into how they handled it. This isn't because "tell me about a failure" is a novel question, but because the follow-up conversation reveals depth. The most telling question, though, is usually some form of "what would you do differently if you built this again?" Candidates who were deeply involved in a project have learned something from it. They have opinions about what they'd change. Candidates who are exaggerating their involvement tend to stumble here because they haven't actually wrestled with the consequences of the original decisions.

About half of the candidates I interview can genuinely defend their decisions in this format. The other half can describe what they built but can't articulate why they made the choices they did, or what they learned from the experience. That's the signal. I'm not looking for perfect decisions. I'm looking for evidence that they understand systems deeply enough to reason about them, not just implement them.

For more senior and architecturally-focused roles, I lean harder on design tradeoff discussions. For mid-level engineers, I focus more on the "what went wrong" and "what would you change" angles, which are more accessible and still reveal a lot.

Junior engineers¶

Past project deep-dives work less well for junior candidates. They don't have enough experience to defend decisions they haven't had the opportunity to make yet. Asking a recent graduate to explain the architectural tradeoffs of their senior capstone project isn't a fair evaluation.

Instead, I use a small live coding exercise. Something that can be written in 5-10 lines of code, with multiple valid approaches. For example, a problem that can be solved iteratively and recursively. I ask them to write it one way and we talk through it for a minute or two. Then I ask them to write it the other way. I don't tell them in advance that I'll ask for the alternative approach. I want to see what they do naturally when the problem shifts.

The code itself is almost secondary. What I'm evaluating is the conversation. Can they reason about the difference between the two approaches? Can they articulate why one might be preferable in a given context? Do they get flustered when asked to think differently, or do they engage with the challenge? A junior engineer who writes imperfect code but can reason clearly about alternatives and ask good questions is far more promising than one who produces a clean solution but can't explain their thinking.

Leads and directors¶

For leadership roles, I shift to scenario and system design conversations. Less about code, more about organizational and architectural judgment. How would you structure a team to build a particular kind of system? How would you evaluate and prioritize a backlog of technical debt? How do you handle a situation where product and engineering disagree on timeline or scope? The signal here is similar to the engineering deep-dives. I'm listening for whether a candidate can reason through ambiguity, make a decision with incomplete information, and explain their thinking clearly. A strong candidate will ask clarifying questions, acknowledge tradeoffs, and arrive at a defensible position. A weak candidate will give a textbook answer that doesn't account for the messy reality of the scenario.

Technical product managers¶

I also interview Technical Product Managers (TPM), and the approach shares the same philosophy as the engineering deep-dives. I'll present a feature or problem that has already been solved so that I have detailed knowledge of what has worked. I ask them to walk through how they'd approach it. I'm watching how they think through the problem, what questions they ask, what tradeoffs they identify, and how they communicate their reasoning. Then during the conversation, I poke at their decisions the same way I would if I were working through the problem with a TPM on my team. Can they explain why they prioritized one approach over another? Can they articulate the user impact? Can they adjust their thinking when I introduce a constraint they hadn't considered? This fits directly into the communication signal I look for across every role which is the ability to reason through a problem, defend your position, and adapt when new information arrives.

The soft skills that predict long-term success¶

Technical skill gets someone through the interview. Soft skills determine whether they succeed on the team. I assess these through a combination of specific questions, observing behavior throughout the interview, and paying close attention to how candidates describe the systems and processes they've worked in.

Three signals have been the strongest predictors across my career:

Ownership without ego. I want someone who takes responsibility for the system, not just the lines of code they wrote. But ownership can become toxic when it turns into "this is my baby and it's always right and the user is wrong." The best engineers own the outcome for the user, not just the implementation. When something breaks, they don't deflect. But they also don't get defensive when someone suggests a different approach or when a user reports that the system isn't working the way they intended. You can often hear the difference in how candidates describe past work: do they talk about what "I built" in isolation, or do they talk about how the system served its users and how the team improved it over time?

Communication. Can they explain a technical decision to someone who isn't an engineer? Can they disagree with a colleague constructively? The interview itself is a communication exercise and I'm paying attention to how clearly they explain their past projects, how they handle follow-up questions, and whether they can adjust their level of detail based on the conversation. An engineer who can only communicate with other engineers who share their context will hit a ceiling quickly, regardless of their technical ability.

Curiosity and willingness to learn. The technology stack will change. The industry is already changing and AI has introduced so much disruption in such a short time that it is vital candidates are willing to learn and adapt. I've worked across many industries in my career, and the engineers who thrived through transitions were the ones who were genuinely energized by learning something new rather than threatened by it. In interviews, I listen for whether candidates describe their learning in terms of genuine interest or obligation. "I had to learn Kubernetes for the role" sounds very different from "I got interested in how our deployment pipeline could be more reliable, which led me to Kubernetes."

Designing interviews that AI can't shortcut¶

The ChatGPT experiment wasn't just a blog series - it fundamentally changed how I approach hiring. My current process is designed so that AI assistance is either irrelevant or immediately visible.

You can't have ChatGPT defend your past project decisions in a live conversation. When I ask a candidate what they'd do differently about a system they built two years ago, there's no prompt that generates an authentic answer. The deep-dive format is inherently AI-resistant because it's evaluating lived experience, not producible knowledge.

For the junior coding exercises, the code portion could theoretically be AI-assisted. But the follow-up conversation, "now write it the other way, and explain why you'd choose one approach over the other", reveals immediately whether the candidate understands what they wrote. I've caught candidates who clearly used AI for their solution and then could not explain their own code when I asked follow-up questions. The conversation is the evaluation, not the code.

My position on AI use in interviews is nuanced. I don't ban AI tools, but I do expect candidates to disclose if they're using them. If a candidate uses AI and tells me about it, we can have a productive conversation about how they used it and what their own contribution was. If we've asked, and they haven't disclosed, that's a failure of integrity that outweighs technical competence. When you're hiring someone to join a team, trust matters as much as skill.

This isn't a hypothetical concern. I've encountered candidates who were clearly using AI during interviews and couldn't explain their own answers. I'm not alone. CNBC reported in 2025 that tools like Interview Coder and Cluely now use invisible screen overlays that are undetectable by standard screen sharing, and hiring managers describe the telltale pattern of a pause, an "Hmm," and then a suspiciously perfect answer. An analysis of over 19,000 interviews by Fabric found that cheating adoption more than doubled from 15% to 35% between June and December 2025, with technical roles showing a 48% cheating rate.

I've also encountered the broader trend of candidates working at multiple companies simultaneously. This is a real and growing problem in remote engineering. A ResumeBuilder.com survey found that 37% of remote workers hold two full-time jobs, with AI tools making it increasingly feasible to juggle multiple roles within a standard workweek. Both of these trends make it even more critical that your interview process evaluates the actual human sitting across from you, not just the output they produce.

What I haven't solved yet¶

No hiring process is perfect, and I don't want to pretend mine is. There are areas I'm still actively working to improve.

Evaluating candidates from very different technology stacks remains a challenge. When someone's entire career has been in a language or ecosystem that's different from what we use, the past project deep-dive still works for assessing how they think, but it's harder to gauge how quickly they'll become productive in a new environment. I haven't found a clean solution for this that doesn't fall back on the kind of generalized assessments I've moved away from.

Scaling the process is another challenge. Deep-dive conversations take time and they're significantly more time-intensive per candidate than a timed assessment that can be administered asynchronously. When you're hiring for many roles simultaneously, the time cost is real. I've mitigated this by involving more of the team in interviews, but that creates its own overhead.

And the overall hiring pipeline is still slower than I'd like. The industry-wide process of sourcing, screening, interviewing, and closing candidates has friction at every stage. Improving my interview process doesn't fix the weeks lost to scheduling coordination, offer negotiations, and notice periods.

What actually matters¶

I've been hiring engineers across industries for nearly two decades and for companies ranging from pre-seed startups to Fortune 500 corporations. I've been wrong about candidates in both directions; people I was confident about who didn't work out, and people I had reservations about who became some of the strongest members of the team.

What I've learned is that the interview process should evaluate what actually matters on the job: Can this person reason about complex systems? Can they communicate clearly and work constructively with others? Do they take ownership of outcomes, not just code? Are they honest about what they know and what they don't?

No timed assessment answers those questions. A conversation does.

CoderByte assessments fall to ChatGPT

2023-02-04T23:30:00-06:00

Introduction¶

Today I'll be evaluating CoderByte and their publicly posted challenges against ChatGPT. I selected an easy, a medium and the single hard question from their list of open challenges and ran those through ChatGPT.

In previous articles in this series, I've shown how ChatGPT can easily solve interview assessments from LeetCode, TestGorilla, CodeSignal, Codility and HackerRank[hackerrankg]. I've said it in those articles, but I'll say it again: Companies need to evaluate their engineering candidates - all candidates, actually - appropriately when they are hiring. Your engineers are using many tools to do their jobs and AI Tooling is one of the newest tools on the market. Removing a tool from your engineers is going to make them less efficient, and not testing your candidates on how they use these tools is going to mean you hire someone that had not fully demonstrated their capabilities.

ChatGPT should be treated as a tool, but it's also helpful to think of it as a junior level engineer. You need to check its work. ChatGPT is known to be inaccurate, which is why it remains banned on Stack Overflow. There are a few instances where I had to do that below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. I do not remember CoderByte being mentioned during my tenure and CoderByte does not seem to consider Woven a competitor, as Woven isn't listed as an alternative on their pricing FAQ page.

Easy - React Button Toggle¶

While I've tried to be mostly random in my selection of assessments during these tests, I intentionally selected this react assessment because I don't know react. I've liked these types of assessment questions in the previous articles because it means I can't apply my experience to "nudge" ChatGPT toward a solution.

The challenge here is to toggle a button between "On" and "Off". You're presented with a very small amount of boiler plate code.

I simply copy/pasted the prompt and the template into ChatGPT. It sent back this:

import React, { useState } from 'react';
import ReactDOM from 'react-dom';

function Toggle() {
const [toggle, setToggle] = useState(true);

function handleClick() {
    setToggle(!toggle);
}

return (
    <button onClick={handleClick}>{toggle ? 'ON' : 'OFF'}</button>
);
}

ReactDOM.render(
<Toggle />,
document.getElementById('root')
);

I copied that into the IDE provided by CoderByte and submitted the code.

This test took two minutes to complete. The bulk of that time was spot checking that ChatGPT didn't mess with the boiler plate code too badly.

The interesting this about the results is that even with the 10/10, the system says it only scored higher than 12% of other users. Does this mean that only 12% of the users passing this assessment are getting lower than a 10/10? To me, that's the sign of a bad assessment.

Medium - SQL Member Count¶

The medium challenge I selected was an SQL question. SQL is interesting, in my experience, in that the results returned on an assessment are a lot more uniform compared to a non-SQL coding assessment. This one wasn't a difficult one with joins.

The goal of this assessment was to return a list of names in alphabetical order, a count of the number of people that report to them, and the average age of their team members. I pasted the prompt, the expected table layout for the output and the table layout being selected from.

The first response was

SELECT ReportsTo, COUNT(ID) AS Members, AVG(Age) AS Average_Age
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

This is where experience, and checking the results are important. The expected output showed that the average age should be an integer. It also showed that the average age column should be named Average Age not Average_Age (notice the lack of the underscore). I provided this guidance to ChatGPT

The AVG(Age) should be an integer not a float and The "Average_Age" field should be "Average Age" without an underscore.

It corrected both problems with a new query which I submitted.

SELECT ReportsTo, COUNT(ID) AS Members, ROUND(AVG(Age)) AS "Average Age"
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

ChatGPT scored another 10/10.

Even with that extra round trip with ChatGPT, it took less than a minute to complete this assessment.

This assessment seems to have a slightly better split of user scores, which is progress I suppose.

Hard - Bracket Combinations¶

There is only one hard assessment available publicly. The goal for this assessment is to return the number of combinations you can make with a number of pairs of parenthesis. The example provided is:

The possible combinations of 3 pairs of parenthesis, namely: ()()(), are ()()(), ()(()), (())(), ((())), and (()())

I copied the prompt and boiler plate code into ChatGPT and it returned the following response:

def BracketCombinations(num):

    from math import factorial
    return factorial(2 * num) // (factorial(num) * factorial(num + 1))

# keep this function call here 
print(BracketCombinations(input()))

Experience here says that is odd. Technically speaking, Python will allow you to have that import statement inside a function, but if I saw one of my developers do it, I'd mention it during a code review. I was curious though and submitted it as is.

I also went back and told ChatGPT:

Write the function without importing factorial

It replied with this function:

def BracketCombinations(num):
    def fact(n):
        result = 1
        for i in range(1, n + 1):
            result *= i
        return result

    return fact(2 * num) // (fact(num) * fact(num + 1))

# keep this function call here 
print(BracketCombinations(int(input())))

I submitted this response too.

I only have one set of images for this challenge, because they both output exactly the same responses. Both solutions scored the same. Both solutions were better than 35.1% of other users. Both solutions had an O(n) complexity.

Showing the complexity is a nice touch. I will admit that I was expecting the solution that imported factorial to be rejected because it's using a built in function instead of rolling your own. Good for CoderByte for allowing a developer to use the built in libraries as an engineer would do in a production environment, instead of adding a false constraint.

Copy pasting to CoderByte kept this close to a minute.

Final Thoughts¶

Another code only assessment tool has been shown to be ineffective. The one thing this one does, that others don't show publicly, is the time it took to solve a problem. I'm very surprised that a majority of these were solved in under ten minutes. Even so, solving a problem in less than a minute will probably raise a flag of some kind for a hiring manager giving this type of assessment. I guarantee that time is visible to them.

These three assessments show how well ChatGPT can be in being your virtual junior engineer. With the react question, I have no idea if it's the more efficient way to solve the problem but it works. With the SQL question, experience identified two small problems and allowed me to provide feedback and get a corrected query in seconds. The third question turned in two different ways of solving the problem. Apparently, according to CoderByte, both of which perform exactly the same.

The total time it took me to run these three tests was under five minutes. That's a great tool to have in my pocket.

ChatGPT easily solves HackerRank questions

2023-01-08T01:15:00-06:00

Introduction¶

It's important that companies evaluate their engineering candidates appropriately when they are hiring. Do other areas of your company evaluate their potential hires by restricting access to tools they'll use to perform the job? I suspect not. Why do the same to your software engineers?

AI Tooling - things like GitHub's Copilot and now ChatGPT - are tools that your engineers can use to do their job. During the hiring process, engineers should be evaluated on the skills they have, and that includes the ability to utilize all available tools. I expect AI assistance to become as common place as using Stack Overflow to solve a problem. ChatGPT, Copilot, and any future tools doing similar tasks are going to be used frequently in the near future.

In this article, I'm testing ChatGPT against one of the most popular coding assessment services: HackerRank. I've already covered other assessment services like LeetCode, TestGorilla, CodeSignal and Codility in this series. I recommend taking a look at those too.

To the engineers that have found this post and are looking for solutions to your coding assessments, I want to warn you, ChatGPT is an assistant. Think of it as a junior level engineer. You may need to check its work. ChatGPT is known to be inaccurate, which is why it was banned on Stack Overflow. Don't take its responses as gospel. Evaluate the responses and make improvements as needed. You'll see I had to do that a couple of times with the questions below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. HackerRank is mentioned multiple times in the Woven blog and is a competitor to Woven.

Final note before I begin, HackerRank presented a couple annoyances during this test because they use LaTeX formatting in their prompts. Simply copy and pasting prompts to ChatGPT was not possible, because key parts of the problem would be missing. This meant that I spent several minutes per prompt cleaning it up and reinserting the LaTeX characters in plain text. This accounts for the higher solve times compared to the competing services in other articles.

Triangle Quest¶

I selected a random Medium difficulty Python question for my first prompt: Triangle Quest. This one was unique in this series of tests. The expected solution was to add only a single line of code and I could only use arithmetic operations, a single for loop and print statement.

As previously mentioned, there were a few LaTeX things to clean up, but then ChatGPT returned a single line of code to add to the for loop provided. The whole code is:

for i in range(1,int(input())): #More than 2 lines will result in 0 score. Do not leave a blank line also
    print((i * (10 ** i - 1)) // 9)

I submitted the code and passed all test cases.

I was surprised by this one. I was expecting to encounter some problems with the constraint of only being able to use arithmetic operations. It turns out, ChatGPT caught that constraint and solved the problem without any issue.

Iterables and Iterators¶

I wanted to try another Medium difficulty problem, that added more than a single line of code. The constraints on the previous question put the problem well into "puzzle" territory, and I am not a fan of puzzles in technical interviews. The second medium question selected was Iterables and Iterators.

The first result returned didn't take input. Instead, it hard coded values like so:

# given list of letters
letters = ['a', 'a', 'c', 'd']

# number of indices to select
k = 2

The interesting thing with this solution was that it ignored the first input row - "The first line contains the integer N, denoting the length of the list". I don't disagree with this change. We should be able to determine the length of the list from the list itself.

However, the solution required taking input from STDIN, so I let ChatGPT know this by pasting in the only bit of information provided in the code here:

Enter your code here. Read input from STDIN. Print output to STDOUT

ChatGPT made the necessary change and provided a new code block.

import itertools

# read input from stdin
n = int(input())
letters = input().split()
k = int(input())

# number of combinations of indices
total_combinations = len(list(itertools.combinations(range(1, len(letters)+1), k)))

# number of combinations of indices that do not contain the letter 'a'
combinations_without_a = list(filter(lambda x: 'a' not in [letters[i-1] for i in x], itertools.combinations(range(1, len(letters)+1), k)))

# probability that none of the selected indices contain the letter 'a'
probability_without_a = len(combinations_without_a) / total_combinations

# probability that at least one of the selected indices contain the letter 'a'
probability_with_a = 1 - probability_without_a

# print the probability to three decimal places
print("{:.3f}".format(probability_with_a))

I submitted this code and it also passed each of the test cases. This solution also ignored using the n input, but did accept it as input.

Validating Postal Codes¶

At this point, I've shown that ChatGPT can do medium level problems. So, I selected a Hard problem: Validating Postal Codes. The interesting constraint on this one is that if statements aren't allowed. The goal here is to provide two regular expressions to solve the problem.

I've built some nasty regular expressions over the years. I've talked about SmokeDetector in the past, and it's role in removing spam from the Stack Exchange network. It is built on regular expressions. Lots and lots of regular expressions. So, I felt pretty comfortable with this question and wanted to see how ChatGPT would handle it.

The second regular expression, the alternating repeating digit pair, didn't work the first time.

regex_alternating_repetitive_digit_pair = r'(\d)\d\1'

For those that know regular expressions, that is not going to work. I pointed this out and it corrected itself, and explained the meaning of the new regular expression. The two final regular expressions are

regex_integer_in_range = r"^[1-9]\d{5}$"    # Do not delete 'r'.
regex_alternating_repetitive_digit_pair = r"(\d)(?=\d\1)"   # Do not delete 'r'.

This is using a positive lookahead assertion, which ChatGPT explains as:

This regular expression will match any digit (\d) followed by another digit, with the same digit (\1) appearing immediately after that. The (?=...) syntax is a positive lookahead assertion, which means that it will only match a digit if it is followed by the specified pattern, but the pattern will not be included in the overall match. The parentheses around \d capture the digit so that it can be referred to as \1 later on.

I submitted the two regexes above, and passed the test cases.

Final Thoughts¶

The majority of my time on these was spent cleaning up the LaTeX formatting. I mentioned a few prompt clarifications needed to solve problems in the returned code too - one required accepting user input and the other was an invalid regular expression. Pointing this out to ChatGPT got the problem resolved immediately.

This is exactly the type of thing that I'd expect an engineer to need to do with the AI assistant (or more junior engineer). Hiring processes must adapt to the existence of AI assistants. Evaluate your candidates and their ability to do more than simply write code.

Taking Codility's sample interview exam using ChatGPT

2023-01-05T01:00:00-06:00

Introduction¶

It was announced on January 3, 2023 that New York City has banned students and teachers from using ChatGPT, because of "concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content". This continues the daily news stories about ChatGPT that have come out over the past several weeks. I suspect that 2023 is going to be a massive year for public awareness of "AI". Especially with GPT-4's expected release in a month or so. In the meantime, ChatGPT remains banned over on Stack Overflow. The accuracy of ChatGPT was also mentioned by New York City in their banning of the tool in the school system.

My perspective is that this is a tool that software engineers can utilize, but they have to understand that it is just that - a tool. It is only as good as the engineer that wields it. A good engineer will be able to utilize it very effectively. Hiring managers need to be aware of what this tool is capable of, though. I've already shown that it can easily solve common interview questions that the software industry uses in their hiring process. I showed that questions from LeetCode, TestGorilla, and CodeSignal are easily solved by this tool in previous articles in this series.

Today, I'm going to look at another service, similar to these: Codility. Much like CodeSignal, Codility sits on both sides of the interview process. It has a developer facing set of challenges and learning material. It also sells services to companies seeking to streamline a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of Codility.

The question in this post is taken from Codility's sample test. I will not be sharing the exact prompt from the sample test, as Codility has a disclaimer in the prompt clearly stating that disclosure is prohibited. Fair enough.

It'd be a shame if they hid all of their technical exam questions behind such a disclaimer though. Especially when this sample question sits on Stack Overflow with over 400k views and has over 100 answers. It also generated its own meta question.

This is just one reason that these types of questions aren't great for technical interview questions. Exam questions will always escape the exam environment. A student, a job candidate, or someone interested in the content will be able to find it and the answers after it's out there.

The problem¶

If you want to see the full prompt, take a look at the Stack Overflow question. But, briefly, the problem to solve is to find the smallest positive integer that does not appear in an array of numbers. We are provided three sample arrays with positive, negative and duplicate entries.

I copy and pasted the prompt into ChatGPT and got a short function back.

def solution(A):
    # Create a set of all the positive integers in the list
    positive_int_set = set([x for x in A if x > 0])

    # Iterate through the positive integers, starting at 1
    i = 1
    while i in positive_int_set:
        i += 1

    # Return the first integer that is not in the set
    return i

I pasted this back into the testing environment and ran the code.

Compilation successful.

Example test:   [1, 3, 6, 4, 1, 2]
OK

Example test:   [1, 2, 3]
OK

Example test:   [-1, -3]
OK

Your code is syntactically correct and works properly on the example test.
Note that the example tests are not part of your score. On submission at least 8 test cases not shown here will assess your solution.

Results¶

The built in test cases worked as expected. After I submitted the code, I need to wait about 5 minutes for the hidden test cases to be processed and receive my results.

I passed the 3 example test cases, 5 additional "correctness" test cases and 4 performance test cases. 100%

Hooray! ChatGPT wins again. Total time taken - 4 minutes (plus 5 minutes for Codility to do its evaluation).

Change your interview processes¶

Engineers are smart individuals. They, generally, enjoy new technologies too. ChatGPT is a new technology that is going to be a valuable tool in their future. GitHub has had Copilot for a while now. ChatGPT brings a new tool to the toolbelt. It should be expected by hiring managers that your current engineering team and engineers you hire will use such tools.

Interview processes need to adapt to the existence of these tools. If you are asking a question that a simple tool can solve, are you really determining an engineer's ability? Do your counterparts ask someone in finance to calculate a percentage and not expect the candidate to use, at least, a calculator?

Hiring processes must adapt. Evaluate your candidates, especially the more experienced candidates, with something other than an exam question that their newest tool can solve. Your engineer is going to be doing more than writing code and implementing an algorithm. Evaluate those skills. Those are the ones you really need on your team.

If you are curious about the other common interview systems and how they perform against ChatGPT, read one of the articles in the series below.

Running CodeSignal questions through ChatGPT

2023-01-01T17:00:00-06:00

Introduction¶

ChatGPT has taken over a lot of technology news in the last month. Within days, Stack Overflow banned ChatGPT, due to subtle errors that the chatbot is prone to make. I've also written about how I hope that ChatGPT's existence will change the hiring processes in the technology industry, because of how easily it can solve LeetCode and TestGorilla technical interview problems.

Today, I'm going to turn my attention to another company that does interview tests: CodeSignal. CodeSignal sits on both sides of the interview process, with a LeetCode style developer facing set of challenges. Solve problems, increase your score. It also sells services to companies looking to outsource a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of CodeSignal.

The questions in this post are taken from CodeSignal's blog, showing off four of their software engineering questions. CodeSignal provides an explanation of the question, and a sample response with a quick analysis.

Array Manipulation¶

Surprise! Another array question. I suppose it wouldn't be a generic software engineering interview without a question related to arrays.

Getting started...

You can see that it covers a basic array traversal and manipulation

A small nitpick - depending on the hiring process, this sentence can come across as condescending. Depending on where a technical interview fits in the process, it's possible that a non-engineer is the one that sees these results first. I would not count on my HR coworkers to understand "basic array traversal and manipulation", because it's not part of their skillset.

Let's see how ChatGPT does. I copy and pasted the prompt into ChatGPT and waited a few seconds for a result.

def solution(a):
    b = []
    for i in range(len(a)):
        # If it's the first element, use 0 as the value for a[i - 1]
        if i == 0:
        b.append(0 + a[i] + a[i + 1])
        # If it's the last element, use 0 as the value for a[i + 1]
        elif i == len(a) - 1:
        b.append(a[i - 1] + a[i] + 0)
        # Otherwise, use the values of a[i - 1], a[i], and a[i + 1]
        else:
        b.append(a[i - 1] + a[i] + a[i + 1])
    return b

It doesn't match the exact response that CodeSignal provides, but it functions with their test case. I'd count that as a pass.

String Pattern Matching¶

Another common theme in these recent tests I've run is pattern matching. This one, however, at least is more interesting than others. It still doesn't seem to reflect a real world problem, and is just a puzzle, but at least it's more unique.

It has the advantage of testing several fundamental programming skills at once: traversing multiple arrays with nested loops, working with subarrays, and performing basic collections/string operations.

We're still sitting in the "basic" tier here, so I suspect that ChatGPT will have no problem. Time to check that assumption by pasting the prompt to the bot. One thing I did have to modify here is the exponents. They pasted as non-exponents, which would have messed up those constraints at the end. This will be a similar problem with a later question. I adjusted those lines slightly to make the numbers exponents.

And, the response back was:

def solution(pattern, source):
    # Initialize the count of matching substrings
    count = 0
    # Iterate through the substrings of source with length equal to pattern
    for i in range(len(source) - len(pattern) + 1):
    # Initialize a flag to track whether the current substring matches the pattern
        match = True
    # Iterate through the characters in the pattern and the corresponding characters in the substring
        for j in range(len(pattern)):
    # If the character in the pattern is 0 (vowel), check if the corresponding character in the substring is a vowel
            if pattern[j] == '0':
                if source[i + j] not in 'aeiouy':
                    match = False
                    break
    # If the character in the pattern is 1 (consonant), check if the corresponding character in the substring is a consonant
            else:
                if source[i + j] in 'aeiouy':
                    match = False
                    break
    # If the substring matches the pattern, increment the count
        if match:
            count += 1
    return count

A big difference between the response generated and the response provided, is that CodeSignal's response breaks out a portion of the code into a second function. The ChatGPT response does everything in a single, larger, loop. Personally, I'd take off a point for not breaking this into another function. CodeSignal's response is better, but the ChatGPT one functions and passes the test. While the second function is a good idea, not having it is not a deal breaker for me. This is especially true of a "basic" test, which is probably being used for less senior roles.

Two-Dimensional Array Traversal¶

Now we're onto the harder questions. This is a long prompt, with multiple animated gifs, and is multiple pages of scrolling. I'm not going to paste the prompt here because the animation is important to understanding the algorithm that needs to be implemented. The general goal though is to build a very small scale Tetris board.

This Tetris-inspired question is challenging. Candidates will have to traverse and manipulate two-dimensional arrays in a non-conventional way. As it’s an advanced question, it tests how well an engineer can understand a complex set of constraints and proceed with a non-trivial implementation.

From my reading of the prompt, and looking at the solution logic, I agree. While this is still a puzzle type question, I don't recall seeing anything like this previously. The uniqueness, yet similarity to a game that most engineers will know, makes the question interesting to me.

The animated gifs are useful for a human to see what is expected in the solution, but shouldn't be needed by ChatGPT, provided the prompt is descriptive enough. Copying and pasting the text into ChatGPT, provides the following response:

def solution(field, figure):
    # Get the height and width of the field
    height = len(field)
    width = len(field[0])
    # Iterate through the columns of the field
    for col in range(width - 2):
        # Iterate through the rows of the field from the bottom up
        for row in range(height - 1, -1, -1):
        # Check if the current cell is occupied
        if field[row][col] == 1:
            # If the cell is occupied, stop checking this column
            break
        # If the figure fits in the current position, check if a full row is formed
        elif all(field[row + i][col + j] + figure[i][j] <= 1 for i in range(3) for j in range(3)):
            # If a full row is formed, return the current column
            if any(all(field[row + i][col + j] == 1 for j in range(3)) for i in range(3)):
                return col
    # If no full rows are formed, return -1
    return -1

Unfortunately, test cases aren't provided for this one. However, looking at the two responses, there are a couple obvious differences. ChatGPT doesn't handle the size of the board at all. CodeSignal uses this size in multiple places of their code. ChatGPT also uses all() and any() checks, where CodeSignal uses longer and more complicated if statements. Personally, I'd prefer the simple if statements for readability.

I'm unsure how to score this one without the test cases, but will note that ChatGPT failed CodeSignal's first sentence in the solution explanation:

This solution starts by defining some dimensions that will be important for the problem.

I'm skeptical that without the size of the figure this will work as expected.

Lookup Table¶

Lastly, we get to a question with lookup tables. These types of problems can usually be solved via brute force, at the expense of performance. They make good questions for a more senior developer. Note that this prompt also has exponents that I needed to adjust when pasting to the chat bot.

This question tests whether candidates have the problem-solving and data structures skills to use a lookup table (hash set/dictionary) in their programming language of choice. It also involves a bit of tricky logic to avoid double-counting pairs. Finally, this question asks candidates to pay close attention to constraints, testing a key skill for real-world development.

After copying and pasting the prompt into ChatGPT, it spit out a response that attempted to brute force the problem. Depending on how CodeSignal presents the problem to a candidate, this could easily be worked around by modifying the prompt to indicate you are looking for a problem with a hash table. If that type of response isn't provided by CodeSignal though, ChatGPT would not have presented the expected solution, though it did still return expected results. The time trade off would have been seen with different test cases.

I modified the prompt slightly to indicate that I wanted a response with a hash table. ChatGPT complied.

def solution(numbers):
    # Initialize a hash table to store the numbers that have been seen
    seen = {}
    # Initialize a count of the number of pairs that sum to a power of 2
    count = 0
    # Iterate through the numbers
    for i, x in enumerate(numbers):
        # If x is a power of 2, increment the count
        if x & (x - 1) == 0:
            count += 1
        # Check if x - y is a power of 2 for any y that has been seen
        for y in seen:
        # If x - y is a power of 2, increment the count
            if (x - y) & (x - y - 1) == 0:
                count += 1
        # Add x to the hash table
        seen[x] = True
    return count

This passes the test cases as well.

Results¶

Of the four sample questions provided by CodeSignal, the "basic" two questions are solved without any difficulty. A third takes a little prompt tweaking to indicate that we want a hashtable instead of a simple brute forced solution. The last one is difficult to evaluate without test cases. So, of the 3 that can be evaluated all pass easily. Two with very little modification (converting exponents). The third needs a small nudge toward the type of solution desired.

What's this mean?¶

I've looked at common interview questions from LeetCode, TestGorilla and now CodeSignal. As a hiring manager, I can see that a simple technical assessment of skills isn't enough to evaluate a candidate. Fortunately, I knew that already, but I'm hoping that others in the industry are seeing that tools exist for developers to make their day to day lives easier. Those tools can, literally, do the work for them if you are presenting them with basic puzzles during the interview process.

I can already hear people saying that they do face to face coding, so this would never work in their process. You may be right, but ask yourselves this: do your developers do face to face coding in their day to day work? If not, why are you evaluating your candidates with that type of pressure?

Adjust your interview process to reflect what you actually need your candidates to demonstrate. A puzzle isn't real work. Give them a real problem to talk through with your team. Evaluate their ability to solve a real problem so that you can learn about their thought processes, how they approach a problem, and what resources they utilize.

Breaking TestGorilla interview questions with ChatGPT

2022-12-26T10:45:00-06:00

Introduction¶

I've written quite a bit about ChatGPT in the last month. As of this post, it is still banned on Stack Overflow. I believe I showed why with my previous post. In it, I showed that ChatGPT is prone to adding subtle errors (and in some cases not so subtle), which is bad on a Q&A site.

Today's post follows in the footsteps of my writing about how I hope ChatGPT inspires software engineering hiring managers to change their hiring process. I wrote about how ChatGPT can solve LeetCode problems. Today's post is going to focus on a company called TestGorilla and a couple of their assessments.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of TestGorilla. That said, I do not recall TestGorilla being discussed during my employment at Woven. I only became aware of them during my job search this year. I'd be happy to do this same type of analysis on a Woven assessment, but as far as I am aware, their prompts are not presented publicly. TestGorilla's prompts used in this article are taken directly from their website. I did not speak with either company prior to writing this article.

Assessment One - PHP Arrays¶

The first assessment I want to look at is related to PHP Arrays. The prompt itself is fairly simple.

This is billed as:

PHP (coding): intermediate-level algorithms This PHP test evaluates candidates' coding skills through a short coding assignment. The test requires candidates to write code that meets certain requirements, and candidates can run their code to test if the result matches their expectations.

The estimated time for this test is 30 minutes.

Well, let's see if ChatGPT is an intermediate level PHP programmer. Pasting the prompt into ChatGPT quickly spits out a result, in Python. I'm not surprised since almost all of my testing has been in Python. I provided one additional set of directions for ChatGPT to utilize, based on the boilerplate code provided.

Write this in PHP using the following as a template:

<?php function sum_array( $no1, $no2 ) {

$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
    //Insert your code here 
}
?>

The result provided looks functional, based on very outdated PHP syntax knowledge.

<?php

function sum_array($no1, $no2) {
$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];

if ($no1 < 0 || $no2 < 0) {
    return -1;
} else if ($no1 > $no2) {
    return 0;
} else {
    $sum = 0;
    for ($i = 0; $i < count($array); $i++) {
    if ($array[$i] >= $no1 && $array[$i] <= $no2) {
        $sum += $array[$i];
    }
    }
    return $sum;
}
}

?>

I copy and pasted that into TestGorilla's preview page and ran it to see how it'd do.

Task executed successfully.

---------------------------------
    Input parameters: ((int) 10, (int) 20)
Result: (30)
OK
---------------------------------
    Input parameters: ((int) 10, (int) 30)
Result: (60)
OK
---------------------------------
    Input parameters: ((int) 30, (int) 50)
Result: (120)
OK
---------------------------------
    Input parameters: ((int) 90, (int) 100)
Result: (190)
OK

Success! One intermediate PHP assessment completed. Total time taken: 2 minutes.

Assessment Two - Kotlin Arrays¶

This assessment is listed as a "New Test" by TestGorilla at the time of this post. The challenge here, if ChatGPT fails, is that I have never used Kotlin.

This is billed as:

Kotlin (Coding): Working with Arrays This Kotlin Coding test evaluates candidates’ ability to work with arrays. The test gives candidates 30 minutes to create a short algorithm involving arrays while taking into consideration all the requirements provided.

This is billed as a 20 minute assessment. It is also billed as an intermediate level assessment.

I pasted the prompt into ChatGPT and also provided the boiler plate code block. The prompt doesn't explicitly say "Kotlin" anywhere, and I didn't mention it either. The only hint provided was the boiler plate code. ChatGPT picked up the language and returned a result.

fun getUniqueStudentIds(a: Array<Int>): Array<Int> {
if (a.any { it < 50000 || it > 99999 }) {
    return arrayOf(-1)
}
return a.distinct().toTypedArray()
}

Running this on TestGorilla's preview page passes the built in test cases:

Task executed successfully.

---------------------------------
Input parameters: (arrayOf(52311, 52311, 55125, 55125, 52128, 52130, 52128) )
Result: (52311, 55125, 52128, 52130)
OK
---------------------------------
Input parameters: (arrayOf(59100, 58200, 58200, 51190, 58200) )
Result: (59100, 58200, 51190)
OK
---------------------------------
Input parameters: (arrayOf(54321, 54321, 54321, 54321, 54321) )
Result: (54321)
OK
---------------------------------
Input parameters: (arrayOf(54321, 53421, 53241, 53214, 54231, 54213, 54312) )
Result: (54321, 53421, 53241, 53214, 54231, 54213, 54312)
OK

Success! Total time: 1 Minute

Evaluation¶

Despite not knowing Kotlin, I can look at the code provided and see that improvements can be made to the code. The prompt says

Note that all student IDs are 5-digit numbers beginning with the number 5. Return -1 as a single element in the array in the case of any invalid input.

The if (a.any { it < 50000 || it > 99999 }) statement can be adjusted to reflect the prompt's specification better. Additionally, there isn't a built in test case for invalid data, but I suspect that it'll fail based on the expected return.

Depending on how TestGorilla is evaluating these edge cases on the paid version of this exam, this may be picked up. On the free preview version, though it is not.

Hiring process¶

Once again, the reader should be able to see that a simple technical evaluation of software engineers in the hiring process is no longer sufficient. It took me longer to determine which prompts I wanted to evaluate for this post than it did to solve with a couple copy and pastes into ChatGPT.

The hiring process must be more than "solve this code challenge". Hiring managers need to move past code only assessments. Instead, focus on a candidate's ability to technically assess and judge based on a context specific problem. These types of problems require a candidate to show their experience, versus their ability to Google, ask Stack Overflow or use ChatGPT to provide a problem prompt. Evaluate your candidates based on the work you want them to actually do.

ChatGPT is here - Use that to stop asking LeetCode questions during interviews

2022-12-15T23:45:00-06:00

Introduction¶

I have interviewed a lot of software engineers looking for a new position. I've talked with college students looking for an internship, junior engineers just getting started, staff engineers looking for a new role, managers and directors to lead these teams. In every one of those interviews, I'd told the candidate that I hate puzzle questions (ie. "How many windows are in Seattle?") and I hate LeetCode style questions.

It turns out that ChatGPT is good at solving these problems.

Good. Maybe that will kill LeetCode interview questions once and for all. There are so many better ways to determine if an engineer is the right fit.

ChatGPT vs LeetCode¶

We have temporarily banned ChatGPT over on Stack Overflow, because it can and does provide subtly incorrect answers. It's been so bad that we called up two new moderators from the previous election today. One of my mod colleagues provided some context around this today too.

We get a few hundred CGPT flags per day, leading to hundreds or even thousands of post deletions per day. There's often one flag per user (not accounting overlap from several users), but each flag can lead to anywhere between a couple and tens of deletions per user.

LeetCode, however, checks your answer right away. They don't have the same problem as Stack Overflow. I wanted to see just how good this new chatbot was, and took it for a spin against a random LeetCode problem.

I randomly selected a medium difficulty problem with a 58% acceptance rate. My victim question: 1143. Longest Common Subsequence

Getting started¶

The first step in passing this test, is selecting the entire prompt, copying it and pasting it directly into LeetCode's UI. I didn't read the prompt and I didn't format it. Just a blind copy and paste. I also provided one instruction:

Provide python code

Press enter and wait 30 seconds or so.

The response¶

Within seconds I was provided this block of code

def longestCommonSubsequence(text1: str, text2: str) -> int:
    # Convert text1 and text2 to lists of characters
    s1 = list(text1)
    s2 = list(text2)

    # Initialize the dynamic programming table
    n = len(s1)
    m = len(s2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    # Fill the dynamic programming table
    for i in range(1, n+1):
        for j in range(1, m+1):
            if s1[i-1] == s2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    # Return the length of the longest common subsequence
    return dp[n][m]

It likes the phrase "dynamic programming table". Along with this code, it even explains its reasoning for going this route. Maybe our intrepid AI can teach a junior dev a new trick or two while doing this puzzle.

Then it shows that tests pass...very confidently.

All the tests passed, so this implementation is correct

The result¶

I'm trying to show that ChatGPT can do this, so at this point I haven't read the prose above. I copy and pasted the code it returned and clicked "Submit".

I waited a few seconds for all tests to run and got my results.

ChatGPT crushed this. It performs better than 96% of other submissions in terms of run time performance. It uses less memory than 71% of other submissions.

Can we please stop using LeetCode now?¶

The total time it took to perform this test was under 5 minutes. I randomly selected a problem, copy and pasted the prompt, waited for a response from the chatbot, copy and pasted the code response, waited for the tests to run. Complete.

As a hiring manager, I'd have wasted my time and the candidates time having them do this problem now. Instead, we can talk about an actual problem they will be solving. We can do a real world problem instead of this. There are better ways to interview an engineer and I'm hoping ChatGPT makes that abundantly clear to hiring managers everywhere.

We're going to have to adapt our hiring processes, but that's a good thing. My recent experience shows that at most companies, it's bad anyway. Now we have an opportunity to shine some sunlight on it and make it better.

In the meantime, if you continue to use LeetCode or something similar to evaluate your candidate's technical skills, you are doing your team a huge disservice. Candidates know about this tool, and they know that the easiest way to get an interview is to play the game. There have been tens of thousands of layoffs in the tech industry in the last six months. I guarantee candidates are looking for every advantage they can get, to move to their next role.