Taking Codility's sample interview exam using ChatGPT

Introduction¶

It was announced on January 3, 2023 that New York City has banned students and teachers from using ChatGPT, because of "concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content". This continues the daily news stories about ChatGPT that have come out over the past several weeks. I suspect that 2023 is going to be a massive year for public awareness of "AI". Especially with GPT-4's expected release in a month or so. In the meantime, ChatGPT remains banned over on Stack Overflow. The accuracy of ChatGPT was also mentioned by New York City in their banning of the tool in the school system.

My perspective is that this is a tool that software engineers can utilize, but they have to understand that it is just that - a tool. It is only as good as the engineer that wields it. A good engineer will be able to utilize it very effectively. Hiring managers need to be aware of what this tool is capable of, though. I've already shown that it can easily solve common interview questions that the software industry uses in their hiring process. I showed that questions from LeetCode, TestGorilla, and CodeSignal are easily solved by this tool in previous articles in this series.

Today, I'm going to look at another service, similar to these: Codility. Much like CodeSignal, Codility sits on both sides of the interview process. It has a developer facing set of challenges and learning material. It also sells services to companies seeking to streamline a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of Codility.

The question in this post is taken from Codility's sample test. I will not be sharing the exact prompt from the sample test, as Codility has a disclaimer in the prompt clearly stating that disclousure is prohibited. Fair enough.

It'd be a shame if they hid all of their technical exam questions behind such a disclaimer though. Especially when this sample question sits on Stack Overflow with over 400k views and has over 100 answers. It also generated it's own meta question.

This is just one reason that these types of questions aren't great for technical interview questions. Exam questions will always escape the exam environment. A student, a job candidate, or someone interested in the content will be able to find it and the answers after it's out there.

The problem¶

If you want to see the full prompt, take a look at the Stack Overflow question. But, briefly, the problem to solve is to find the smallest positve integer that does not appear in an array of numbers. We are provided three sample arrays with positive, negative and duplicate entires.

I copy and pasted the prompt into ChatGPT and got a short function back.

def solution(A):
    # Create a set of all the positive integers in the list
    positive_int_set = set([x for x in A if x > 0])

    # Iterate through the positive integers, starting at 1
    i = 1
    while i in positive_int_set:
        i += 1

    # Return the first integer that is not in the set
    return i

I pasted this back into the testing environment and ran the code.

Compilation successful.

Example test:   [1, 3, 6, 4, 1, 2]
OK

Example test:   [1, 2, 3]
OK

Example test:   [-1, -3]
OK

Your code is syntactically correct and works properly on the example test.
Note that the example tests are not part of your score. On submission at least 8 test cases not shown here will assess your solution.

Results¶

The built in test cases worked as expected. After I submitted the code, I need to wait about 5 minutes for the hidden test cases to be processed and receive my results.

I passed the 3 example test cases, 5 additional "correctness" test cases and 4 performance test cases. 100%

Hooray! ChatGPT wins again. Total time taken - 4 minutes (plus 5 minutes for Codility to do it's evaluation).

Change your interview processes¶

Engineers are smart individuals. They, generally, enjoy new technologies too. ChatGPT is a new technology that is going to be a valuable tool in their future. GitHub has had Copilot for a while now. ChatGPT brings a new tool to the toolbelt. It should be expected by hiring managers that your current engineering team and engineers you hire will use such tools.

Interview processes need to adapt to the existance of these tools. If you are asking a question that a simple tool can solve, are you really determining an engineer's ability? Do your counterparts ask someone in finanace to calculate a percentage and not expect the candidate to use, at least, a calculator?

Hiring processes much adapt. Evaluate your candidates, especially the more experienced candidates, with something other than an exam question that their newest tool can solve. Your engineer is going to be doing more than writing code and implementing an algorithm. Evaluate those skills. Those are the ones you really need on your team.

If you are curious about the other common interview systems and how they perform against ChatGPT, read one of the articles in the series below.