Andrew Wegner | Ponderings of an Andy - chatgpt

Joining the Stack Exchange Moderator Strike

2023-06-05T01:00:00-05:00

Summary¶

I'm signing an open letter to Stack Exchange because those of us that volunteer our time and energy have been put in an impossible situation. On Stack Overflow we banned ChatGPT created content almost immediately after it was released. Theoretically, that is no longer in effect.

This isn't due to a change in community perception. Instead, it's due to an abrupt policy change on Stack Exchange's part that was posted on May 30. It's important to note that this public policy does not match the guidance that was provided privately.

The relevant portion of this public policy is:

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user.

The private policy notes that "very strict" is essentially "don't moderate unless they explicitly say it was created by an AI".

Background¶

The Stack Exchange CEO posted a blog post at the end of May 2023. In this post, they stated:

Approximately 10% of our company is working on features and applications leveraging GenAI that have the potential to increase engagement within our public community and add value to customers of our SaaS product, Stack Overflow for Teams.

This goes against the community desire to not have GenAI content on the site. The CEO has not provided any feedback to the community, other than a note that there is a big summer project with GenAI. The community reaction hasn't been positive.

Combining this announcement with the newly announced policy the previous day, and an astonishing inability to articulate more details or reasoning, has produced a lot of unhappy moderators.

Additionally, I feel that the company has destroyed 3 years of rebuilding trust. Back in 2019, they destroyed this trust and nearly had a moderator strike then too. That involved providing information to a journalist that had very limited context and due to that, presented a single moderator in unflattering light. The feelings from that have taken years to rebuild and even today that incident is cited as a low point, and users can point out that singular incident when trust of the company plummeted.

The public announcement to not moderate GenAI content contained this:

Through no fault of moderators' own, we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts.

That's just wrong. Absolutely wrong. It is 100% inaccurate and Stack Exchange has offered no data to back this up. User country of origin, region in the world, or any kind of physical location is not available to moderators. We are presented a flag and given information to the content that has been flagged.

Why am I participating?¶

So...my participation? Why am I joining in?

Stack Exchange, and Stack Overflow, thrive due to the human element. It's userbase has been around over a decade and answered millions of questions across over 180 sites. The week ChatGPT came out, the community saw the bad results it can provide. For the past 6 months, we've continued to see how bad that is. The moderation teams across the network to generate a policy and get it approved by the company. I've reproduced it in full below, but as of this post it is still on the site and contradicts the new policy.

In addition to completely ignoring the community's input on how they do not want GenAI on the site, the company ignored their own moderator agreement.

With the assumption that the above will change at some point, the relevant section is pasted here:

``` Stack Exchange, Inc. agrees that it will:

i. Respect your privacy per the terms of the Privacy Policy for the Public Network. ii. Get your explicit written permission before commenting to any media (including media outlets controlled by Stack Exchange Inc.) or independent reporters about you or your moderator actions as per our Press Policy. iii. Allow you to resign your position for any reason without penalty or repercussions. As a volunteer, Stack Exchange, Inc. respects your time and will release you from duty should you ask. iv. Operate “Stack Gives Back”, an annual program giving to selected charities in honor of our moderators. v. Post previews for review of all new official policies in the Moderators Teams instance with the policy tag, marked with links to their public version once published, and maintain a listing of all official network-wide policies with links to them in the Help Center. vi. Announce changes to the moderator agreement no less than sixty days before the deadline to accept the new agreement with a period of at least thirty days for discussion and review. vii. Provide support for your questions, requests and concerns on the Moderators Teams instance and/or the Teachers’ Lounge, direct email to CMs, and content on Meta escalated to staff by whatever formal documented process is in effect at the time. viii. Respect your right to speak openly to question and challenge policy without reprisal so long as such speech does not break the Code of Conduct. ```

The relevant section is vi. There was no discussion period on this. No engagement with the moderators, or the community. Instead it was "effective immediately." The best we've gotten so far is an "oops, sorry."

That's not how something in a /legal link should operate. If I can't trust them to uphold an agreement they have in writing, why should I trust them to uphold anything else?

What does the community want out of this?¶

From the open letter:

Until Stack Overflow, Inc. retracts this policy change to a degree that addresses the concerns of the moderators, and allows moderators to effectively enforce established policies against AI-generated answers, we are calling for a general moderation strike, as a last-resort effort to protect the Stack Exchange platform and users from a total loss in value. We would also like to remind Stack Overflow, Inc. that a network that entirely relies on volunteers for its moderation model cannot then consistently ignore, mistreat, and malign those same volunteers.

My feelings on this¶

I've been on Stack Overflow and Stack Exchange for over 13 years. I've been a moderator on the network since 2014 and on Stack Overflow since 2017. I've written about Stack Overflow a bit over the years. I've participated in Charcoal, the spam fighting community since 2015ish. Charcoal has automatically flagged more than 86,000 posts across the network since 2016.

I've applied to several jobs at Stack Exchange. I've interviewed for a couple positions. I have spent a lot of time with the network, the community, moderators, and employees making this a great place for internet users to find their answers.

I have helped to build the community/company trust that I mentioned above. I watched it crumble. I thought about leaving in 2019, but instead spent the next three years working to rebuild that trust. I'm at the point again, where I see the company not understanding their community. At all. It's sad that this cycle has repeated itself and it's worse that the company is, again, tossing their most engaged users under the bus.

This strike serves two purposes in my mind - the first is the officially stated one. Do not allow GenAI content on the network. It will erode the value of the network quickly. We've also demonstrated that ChatGPT and it's peers are not great at answering complex questions. But, second, and unofficially, this strike will represent a change for Stack Exchange to show whether or not they care about what the community has to say. This is, I believe, the last opportunity for them to retract this policy and reflect on why they are over ruling so many communities that reject GenAI in their community. It's their last opportunity to show they support their moderators and the human aspect of moderation. Failure to do either means that Stack Exchange has given up on community building.

I am a volunteer for this community. I would love to continue that role, but this is the best way for me to show that GenAI is not the right path for the company to take.

What's Next¶

On June 5, 2023 at midnight, moderator local time, the network will start to see moderation activities cease (or slow down). I will be part of that.

GPT Policy¶

This policy was crafted with the input of moderators and Stack Exchange. An important thing to notice is that the moderators are empowered to issue suspensions. This is something the new policy prevents.

``` Why posting GPT and ChatGPT generated answers is not currently acceptable

This Help Center article provides insight and rationale on our policy regarding the usage of GPT and ChatGPT on Stack Overflow. While this is the position of Stack Overflow staff, it’s meant to support the prior work done by moderators (namely, the temporary policy issued to ban contributions by ChatGPT).

Stack Overflow is a community built upon trust. The community trusts that users are submitting answers that reflect what they actually know to be accurate and that they and their peers have the knowledge and skill set to verify and validate those answers. The system relies on users to verify and validate contributions by other users with the tools we offer, including responsible use of upvotes and downvotes. Currently, contributions generated by GPT most often do not meet these standards and therefore are not contributing to a trustworthy environment. This trust is broken when users copy and paste information into answers without validating that the answer provided by GPT is correct, ensuring that the sources used in the answer are properly cited (a service GPT does not provide), and verifying that the answer provided by GPT clearly and concisely answers the question asked.

The objective nature of the content on Stack Overflow means that if any part of an answer is wrong, then the answer is objectively wrong. In order for Stack Overflow to maintain a strong standard as a reliable source for correct and verified information, such answers must be edited or replaced. However, because GPT is good enough to convince users of the site that the answer holds merit, signals the community typically use to determine the legitimacy of their peers’ contributions frequently fail to detect severe issues with GPT-generated answers. As a result, information that is objectively wrong makes its way onto the site. In its current state, GPT risks breaking readers’ trust that our site provides answers written by subject-matter experts.

Moderators are empowered (at their discretion) to issue immediate suspensions of up to 30 days to users who are copying and pasting GPT content onto the site, with or without prior notice or warning. ```

The letter¶

The letter below was originally posted as an open letter to Stack Exchange. I've reposted it here for a record.

``` June 5, 2023 Stack Overflow, Inc. has decreed a near-total prohibition on moderating AI-generated content in the wake of a flood of such content being posted to and subsequently removed from the Stack Exchange network, tacitly allowing the proliferation of incorrect information ("hallucinations") and unfettered plagiarism on the Stack Exchange network. This poses a major threat to the integrity and trustworthiness of the platform and its content.

We, the undersigned, are volunteer moderators, contributors, and users of Stack Overflow and the Stack Exchange network. Effective immediately, we are enacting a general moderation strike on Stack Overflow and the Stack Exchange network, in protest of this and other recent and upcoming changes to policy and the platform that are being forced upon us by Stack Overflow, Inc.

Our efforts to effect change through proper channels have been ignored, and our concerns disregarded at every turn. Now, as a last resort, we are striking out of dedication to the platform that we have put over a decade of care and volunteer effort into. We deeply believe in the core mission of the Stack Exchange network: to provide a repository of high-quality information in the form of questions and answers, and the recent actions taken by Stack Overflow, Inc. are directly harmful to that goal.

Specifically, moderators are no longer allowed to remove AI-generated answers on the basis of being AI-generated, outside of exceedingly narrow circumstances. This results in effectively permitting nearly all AI-generated answers to be freely posted, regardless of established community consensus on such content.

In turn, this allows incorrect information (colloquially referred to as "hallucinations") and plagiarism to proliferate unchecked on the platform. This destroys trust in the platform, as Stack Overflow, Inc. has previously noted.

In addition, the details of the policies issued directly to moderators differ substantially from the guidelines outlined publicly, with moderators barred from publicly sharing the details.

These policies disregard the leeway historically granted to individual Stack Exchange communities to determine their policies, by making changes without the input of the community, overriding community consensus, and outright refusing to reconsider their position.

Until this matter is resolved satisfactorily, we will be pausing activities including, but not limited to:

Raising and handling flags.
Running SmokeDetector, the anti-spam bot.
Closing or voting to close posts.
Deleting or voting to delete posts.
Reviewing tasks in the various review queues.
Running various other bots designed to assist in moderation, such as detecting plagiarism, low-quality answers, and rude comments.

Until Stack Overflow, Inc. retracts this policy change to a degree that addresses the concerns of the moderators, and allows moderators to effectively enforce established policies against AI-generated answers, we are calling for a general moderation strike, as a last-resort effort to protect the Stack Exchange platform and users from a total loss in value. We would also like to remind Stack Overflow, Inc. that a network that entirely relies on volunteers for its moderation model cannot then consistently ignore, mistreat, and malign those same volunteers. ```

CoderByte assessments fall to ChatGPT

2023-02-04T23:30:00-06:00

Introduction¶

Today I'll be evaluating CoderByte and their publicly posted challenges against ChatGPT. I selected an easy, a medium and the single hard question from their list of open challenges and ran those through ChatGPT.

In previous articles in this series, I've shown how ChatGPT can easily solve interview assessments from LeetCode, TestGorilla, CodeSignal, Codility and HackerRank[hackerrankg]. I've said it in those articles, but I'll say it again: Companies need to evaluate their engineering candidates - all candidates, actually - appropriately when they are hiring. Your engineers are using many tools to do their jobs and AI Tooling is one of the newest tools on the market. Removing a tool from your engineers is going to make them less efficient, and not testing your candidates on how they use these tools is going to mean you hire someone that had not fully demonstrated their capabilities.

ChatGPT should be treated as a tool, but it's also helpful to think of it as a junior level engineer. You need to check its work. ChatGPT is known to be inaccurate, which is why it remains banned on Stack Overflow. There are a few instances where I had to do that below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. I do not remember CoderByte being mentioned during my tenure and CoderByte does not seem to consider Woven a competitor, as Woven isn't listed as an alternative on their pricing FAQ page.

Easy - React Button Toggle¶

While I've tried to be mostly random in my selection of assessments during these tests, I intentionally selected this react assessment because I don't know react. I've liked these types of assessment questions in the previous articles because it means I can't apply my experience to "nudge" ChatGPT toward a solution.

The challenge here is to toggle a button between "On" and "Off". You're presented with a very small amount of boiler plate code.

I simply copy/pasted the prompt and the template into ChatGPT. It sent back this:

import React, { useState } from 'react';
import ReactDOM from 'react-dom';

function Toggle() {
const [toggle, setToggle] = useState(true);

function handleClick() {
    setToggle(!toggle);
}

return (
    <button onClick={handleClick}>{toggle ? 'ON' : 'OFF'}</button>
);
}

ReactDOM.render(
<Toggle />,
document.getElementById('root')
);

I copied that into the IDE provided by CoderByte and submitted the code.

This test took two minutes to complete. The bulk of that time was spot checking that ChatGPT didn't mess with the boiler plate code too badly.

The interesting this about the results is that even with the 10/10, the system says it only scored higher than 12% of other users. Does this mean that only 12% of the users passing this assessment are getting lower than a 10/10? To me, that's the sign of a bad assessment.

Medium - SQL Member Count¶

The medium challenge I selected was an SQL question. SQL is interesting, in my experience, in that the results returned on an assessment are a lot more uniform compared to a non-SQL coding assessment. This one wasn't a difficult one with joins.

The goal of this assessment was to return a list of names in alphabetical order, a count of the number of people that report to them, and the average age of their team members. I pasted the prompt, the expected table layout for the output and the table layout being selected from.

The first response was

SELECT ReportsTo, COUNT(ID) AS Members, AVG(Age) AS Average_Age
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

This is where experience, and checking the results are important. The expected output showed that the average age should be an integer. It also showed that the average age column should be named Average Age not Average_Age (notice the lack of the underscore). I provided this guidance to ChatGPT

The AVG(Age) should be an integer not a float and The "Average_Age" field should be "Average Age" without an underscore.

It corrected both problems with a new query which I submitted.

SELECT ReportsTo, COUNT(ID) AS Members, ROUND(AVG(Age)) AS "Average Age"
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

ChatGPT scored another 10/10.

Even with that extra round trip with ChatGPT, it took less than a minute to complete this assessment.

This assessment seems to have a slightly better split of user scores, which is progress I suppose.

Hard - Bracket Combinations¶

There is only one hard assessment available publicly. The goal for this assessment is to return the number of combinations you can make with a number of pairs of parenthesis. The example provided is:

The possible combinations of 3 pairs of parenthesis, namely: ()()(), are ()()(), ()(()), (())(), ((())), and (()())

I copied the prompt and boiler plate code into ChatGPT and it returned the following response:

def BracketCombinations(num):

    from math import factorial
    return factorial(2 * num) // (factorial(num) * factorial(num + 1))

# keep this function call here 
print(BracketCombinations(input()))

Experience here says that is odd. Technically speaking, Python will allow you to have that import statement inside a function, but if I saw one of my developers do it, I'd mention it during a code review. I was curious though and submitted it as is.

I also went back and told ChatGPT:

Write the function without importing factorial

It replied with this function:

def BracketCombinations(num):
    def fact(n):
        result = 1
        for i in range(1, n + 1):
            result *= i
        return result

    return fact(2 * num) // (fact(num) * fact(num + 1))

# keep this function call here 
print(BracketCombinations(int(input())))

I submitted this response too.

I only have one set of images for this challenge, because they both output exactly the same responses. Both solutions scored the same. Both solutions were better than 35.1% of other users. Both solutions had an O(n) complexity.

Showing the complexity is a nice touch. I will admit that I was expecting the solution that imported factorial to be rejected because it's using a built in function instead of rolling your own. Good for CoderByte for allowing a developer to use the built in libraries as an engineer would do in a production environment, instead of adding a false constraint.

Copy pasting to CoderByte kept this close to a minute.

Final Thoughts¶

Another code only assessment tool has been shown to be ineffective. The one thing this one does, that others don't show publicly, is the time it took to solve a problem. I'm very surprised that a majority of these were solved in under ten minutes. Even so, solving a problem in less than a minute will probably raise a flag of some kind for a hiring manager giving this type of assessment. I guarantee that time is visible to them.

These three assessments show how well ChatGPT can be in being your virtual junior engineer. With the react question, I have no idea if it's the more efficient way to solve the problem but it works. With the SQL question, experience identified two small problems and allowed me to provide feedback and get a corrected query in seconds. The third question turned in two different ways of solving the problem. Apparently, according to CoderByte, both of which perform exactly the same.

The total time it took me to run these three tests was under five minutes. That's a great tool to have in my pocket.

ChatGPT easily solves HackerRank questions

2023-01-08T01:15:00-06:00

Introduction¶

It's important that companies evaluate their engineering candidates appropriately when they are hiring. Do other areas of your company evaluate their potential hires by restricting access to tools they'll use to perform the job? I suspect not. Why do the same to your software engineers?

AI Tooling - things like GitHub's Copilot and now ChatGPT - are tools that your engineers can use to do their job. During the hiring process, engineers should be evaluated on the skills they have, and that includes the ability to utilize all available tools. I expect AI assistance to become as common place as using Stack Overflow to solve a problem. ChatGPT, Copilot, and any future tools doing similar tasks are going to be used frequently in the near future.

In this article, I'm testing ChatGPT against one of the most popular coding assessment services: HackerRank. I've already covered other assessment services like LeetCode, TestGorilla, CodeSignal and Codility in this series. I recommend taking a look at those too.

To the engineers that have found this post and are looking for solutions to your coding assessments, I want to warn you, ChatGPT is an assistant. Think of it as a junior level engineer. You may need to check its work. ChatGPT is known to be inaccurate, which is why it was banned on Stack Overflow. Don't take its responses as gospel. Evaluate the responses and make improvements as needed. You'll see I had to do that a couple of times with the questions below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. HackerRank is mentioned multiple times in the Woven blog and is a competitor to Woven.

Final note before I begin, HackerRank presented a couple annoyances during this test because they use LaTeX formatting in their prompts. Simply copy and pasting prompts to ChatGPT was not possible, because key parts of the problem would be missing. This meant that I spent several minutes per prompt cleaning it up and reinserting the LaTeX characters in plain text. This accounts for the higher solve times compared to the competing services in other articles.

Triangle Quest¶

I selected a random Medium difficulty Python question for my first prompt: Triangle Quest. This one was unique in this series of tests. The expected solution was to add only a single line of code and I could only use arithmetic operations, a single for loop and print statement.

As previously mentioned, there were a few LaTeX things to clean up, but then ChatGPT returned a single line of code to add to the for loop provided. The whole code is:

for i in range(1,int(input())): #More than 2 lines will result in 0 score. Do not leave a blank line also
    print((i * (10 ** i - 1)) // 9)

I submitted the code and passed all test cases.

I was surprised by this one. I was expecting to encounter some problems with the constraint of only being able to use arithmetic operations. It turns out, ChatGPT caught that constraint and solved the problem without any issue.

Iterables and Iterators¶

I wanted to try another Medium difficulty problem, that added more than a single line of code. The constraints on the previous question put the problem well into "puzzle" territory, and I am not a fan of puzzles in technical interviews. The second medium question selected was Iterables and Iterators.

The first result returned didn't take input. Instead, it hard coded values like so:

# given list of letters
letters = ['a', 'a', 'c', 'd']

# number of indices to select
k = 2

The interesting thing with this solution was that it ignored the first input row - "The first line contains the integer N, denoting the length of the list". I don't disagree with this change. We should be able to determine the length of the list from the list itself.

However, the solution required taking input from STDIN, so I let ChatGPT know this by pasting in the only bit of information provided in the code here:

Enter your code here. Read input from STDIN. Print output to STDOUT

ChatGPT made the necessary change and provided a new code block.

import itertools

# read input from stdin
n = int(input())
letters = input().split()
k = int(input())

# number of combinations of indices
total_combinations = len(list(itertools.combinations(range(1, len(letters)+1), k)))

# number of combinations of indices that do not contain the letter 'a'
combinations_without_a = list(filter(lambda x: 'a' not in [letters[i-1] for i in x], itertools.combinations(range(1, len(letters)+1), k)))

# probability that none of the selected indices contain the letter 'a'
probability_without_a = len(combinations_without_a) / total_combinations

# probability that at least one of the selected indices contain the letter 'a'
probability_with_a = 1 - probability_without_a

# print the probability to three decimal places
print("{:.3f}".format(probability_with_a))

I submitted this code and it also passed each of the test cases. This solution also ignored using the n input, but did accept it as input.

Validating Postal Codes¶

At this point, I've shown that ChatGPT can do medium level problems. So, I selected a Hard problem: Validating Postal Codes. The interesting constraint on this one is that if statements aren't allowed. The goal here is to provide two regular expressions to solve the problem.

I've built some nasty regular expressions over the years. I've talked about SmokeDetector in the past, and it's role in removing spam from the Stack Exchange network. It is built on regular expressions. Lots and lots of regular expressions. So, I felt pretty comfortable with this question and wanted to see how ChatGPT would handle it.

The second regular expression, the alternating repeating digit pair, didn't work the first time.

regex_alternating_repetitive_digit_pair = r'(\d)\d\1'

For those that know regular expressions, that is not going to work. I pointed this out and it corrected itself, and explained the meaning of the new regular expression. The two final regular expressions are

regex_integer_in_range = r"^[1-9]\d{5}$"    # Do not delete 'r'.
regex_alternating_repetitive_digit_pair = r"(\d)(?=\d\1)"   # Do not delete 'r'.

This is using a positive lookahead assertion, which ChatGPT explains as:

This regular expression will match any digit (\d) followed by another digit, with the same digit (\1) appearing immediately after that. The (?=...) syntax is a positive lookahead assertion, which means that it will only match a digit if it is followed by the specified pattern, but the pattern will not be included in the overall match. The parentheses around \d capture the digit so that it can be referred to as \1 later on.

I submitted the two regexes above, and passed the test cases.

Final Thoughts¶

The majority of my time on these was spent cleaning up the LaTeX formatting. I mentioned a few prompt clarifications needed to solve problems in the returned code too - one required accepting user input and the other was an invalid regular expression. Pointing this out to ChatGPT got the problem resolved immediately.

This is exactly the type of thing that I'd expect an engineer to need to do with the AI assistant (or more junior engineer). Hiring processes must adapt to the existence of AI assistants. Evaluate your candidates and their ability to do more than simply write code.

Taking Codility's sample interview exam using ChatGPT

2023-01-05T01:00:00-06:00

Introduction¶

It was announced on January 3, 2023 that New York City has banned students and teachers from using ChatGPT, because of "concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content". This continues the daily news stories about ChatGPT that have come out over the past several weeks. I suspect that 2023 is going to be a massive year for public awareness of "AI". Especially with GPT-4's expected release in a month or so. In the meantime, ChatGPT remains banned over on Stack Overflow. The accuracy of ChatGPT was also mentioned by New York City in their banning of the tool in the school system.

My perspective is that this is a tool that software engineers can utilize, but they have to understand that it is just that - a tool. It is only as good as the engineer that wields it. A good engineer will be able to utilize it very effectively. Hiring managers need to be aware of what this tool is capable of, though. I've already shown that it can easily solve common interview questions that the software industry uses in their hiring process. I showed that questions from LeetCode, TestGorilla, and CodeSignal are easily solved by this tool in previous articles in this series.

Today, I'm going to look at another service, similar to these: Codility. Much like CodeSignal, Codility sits on both sides of the interview process. It has a developer facing set of challenges and learning material. It also sells services to companies seeking to streamline a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of Codility.

The question in this post is taken from Codility's sample test. I will not be sharing the exact prompt from the sample test, as Codility has a disclaimer in the prompt clearly stating that disclosure is prohibited. Fair enough.

It'd be a shame if they hid all of their technical exam questions behind such a disclaimer though. Especially when this sample question sits on Stack Overflow with over 400k views and has over 100 answers. It also generated its own meta question.

This is just one reason that these types of questions aren't great for technical interview questions. Exam questions will always escape the exam environment. A student, a job candidate, or someone interested in the content will be able to find it and the answers after it's out there.

The problem¶

If you want to see the full prompt, take a look at the Stack Overflow question. But, briefly, the problem to solve is to find the smallest positive integer that does not appear in an array of numbers. We are provided three sample arrays with positive, negative and duplicate entries.

I copy and pasted the prompt into ChatGPT and got a short function back.

def solution(A):
    # Create a set of all the positive integers in the list
    positive_int_set = set([x for x in A if x > 0])

    # Iterate through the positive integers, starting at 1
    i = 1
    while i in positive_int_set:
        i += 1

    # Return the first integer that is not in the set
    return i

I pasted this back into the testing environment and ran the code.

Compilation successful.

Example test:   [1, 3, 6, 4, 1, 2]
OK

Example test:   [1, 2, 3]
OK

Example test:   [-1, -3]
OK

Your code is syntactically correct and works properly on the example test.
Note that the example tests are not part of your score. On submission at least 8 test cases not shown here will assess your solution.

Results¶

The built in test cases worked as expected. After I submitted the code, I need to wait about 5 minutes for the hidden test cases to be processed and receive my results.

I passed the 3 example test cases, 5 additional "correctness" test cases and 4 performance test cases. 100%

Hooray! ChatGPT wins again. Total time taken - 4 minutes (plus 5 minutes for Codility to do its evaluation).

Change your interview processes¶

Engineers are smart individuals. They, generally, enjoy new technologies too. ChatGPT is a new technology that is going to be a valuable tool in their future. GitHub has had Copilot for a while now. ChatGPT brings a new tool to the toolbelt. It should be expected by hiring managers that your current engineering team and engineers you hire will use such tools.

Interview processes need to adapt to the existence of these tools. If you are asking a question that a simple tool can solve, are you really determining an engineer's ability? Do your counterparts ask someone in finance to calculate a percentage and not expect the candidate to use, at least, a calculator?

Hiring processes must adapt. Evaluate your candidates, especially the more experienced candidates, with something other than an exam question that their newest tool can solve. Your engineer is going to be doing more than writing code and implementing an algorithm. Evaluate those skills. Those are the ones you really need on your team.

If you are curious about the other common interview systems and how they perform against ChatGPT, read one of the articles in the series below.

Running CodeSignal questions through ChatGPT

2023-01-01T17:00:00-06:00

Introduction¶

ChatGPT has taken over a lot of technology news in the last month. Within days, Stack Overflow banned ChatGPT, due to subtle errors that the chatbot is prone to make. I've also written about how I hope that ChatGPT's existence will change the hiring processes in the technology industry, because of how easily it can solve LeetCode and TestGorilla technical interview problems.

Today, I'm going to turn my attention to another company that does interview tests: CodeSignal. CodeSignal sits on both sides of the interview process, with a LeetCode style developer facing set of challenges. Solve problems, increase your score. It also sells services to companies looking to outsource a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of CodeSignal.

The questions in this post are taken from CodeSignal's blog, showing off four of their software engineering questions. CodeSignal provides an explanation of the question, and a sample response with a quick analysis.

Array Manipulation¶

Surprise! Another array question. I suppose it wouldn't be a generic software engineering interview without a question related to arrays.

Getting started...

You can see that it covers a basic array traversal and manipulation

A small nitpick - depending on the hiring process, this sentence can come across as condescending. Depending on where a technical interview fits in the process, it's possible that a non-engineer is the one that sees these results first. I would not count on my HR coworkers to understand "basic array traversal and manipulation", because it's not part of their skillset.

Let's see how ChatGPT does. I copy and pasted the prompt into ChatGPT and waited a few seconds for a result.

def solution(a):
    b = []
    for i in range(len(a)):
        # If it's the first element, use 0 as the value for a[i - 1]
        if i == 0:
        b.append(0 + a[i] + a[i + 1])
        # If it's the last element, use 0 as the value for a[i + 1]
        elif i == len(a) - 1:
        b.append(a[i - 1] + a[i] + 0)
        # Otherwise, use the values of a[i - 1], a[i], and a[i + 1]
        else:
        b.append(a[i - 1] + a[i] + a[i + 1])
    return b

It doesn't match the exact response that CodeSignal provides, but it functions with their test case. I'd count that as a pass.

String Pattern Matching¶

Another common theme in these recent tests I've run is pattern matching. This one, however, at least is more interesting than others. It still doesn't seem to reflect a real world problem, and is just a puzzle, but at least it's more unique.

It has the advantage of testing several fundamental programming skills at once: traversing multiple arrays with nested loops, working with subarrays, and performing basic collections/string operations.

We're still sitting in the "basic" tier here, so I suspect that ChatGPT will have no problem. Time to check that assumption by pasting the prompt to the bot. One thing I did have to modify here is the exponents. They pasted as non-exponents, which would have messed up those constraints at the end. This will be a similar problem with a later question. I adjusted those lines slightly to make the numbers exponents.

And, the response back was:

def solution(pattern, source):
    # Initialize the count of matching substrings
    count = 0
    # Iterate through the substrings of source with length equal to pattern
    for i in range(len(source) - len(pattern) + 1):
    # Initialize a flag to track whether the current substring matches the pattern
        match = True
    # Iterate through the characters in the pattern and the corresponding characters in the substring
        for j in range(len(pattern)):
    # If the character in the pattern is 0 (vowel), check if the corresponding character in the substring is a vowel
            if pattern[j] == '0':
                if source[i + j] not in 'aeiouy':
                    match = False
                    break
    # If the character in the pattern is 1 (consonant), check if the corresponding character in the substring is a consonant
            else:
                if source[i + j] in 'aeiouy':
                    match = False
                    break
    # If the substring matches the pattern, increment the count
        if match:
            count += 1
    return count

A big difference between the response generated and the response provided, is that CodeSignal's response breaks out a portion of the code into a second function. The ChatGPT response does everything in a single, larger, loop. Personally, I'd take off a point for not breaking this into another function. CodeSignal's response is better, but the ChatGPT one functions and passes the test. While the second function is a good idea, not having it is not a deal breaker for me. This is especially true of a "basic" test, which is probably being used for less senior roles.

Two-Dimensional Array Traversal¶

Now we're onto the harder questions. This is a long prompt, with multiple animated gifs, and is multiple pages of scrolling. I'm not going to paste the prompt here because the animation is important to understanding the algorithm that needs to be implemented. The general goal though is to build a very small scale Tetris board.

This Tetris-inspired question is challenging. Candidates will have to traverse and manipulate two-dimensional arrays in a non-conventional way. As it’s an advanced question, it tests how well an engineer can understand a complex set of constraints and proceed with a non-trivial implementation.

From my reading of the prompt, and looking at the solution logic, I agree. While this is still a puzzle type question, I don't recall seeing anything like this previously. The uniqueness, yet similarity to a game that most engineers will know, makes the question interesting to me.

The animated gifs are useful for a human to see what is expected in the solution, but shouldn't be needed by ChatGPT, provided the prompt is descriptive enough. Copying and pasting the text into ChatGPT, provides the following response:

def solution(field, figure):
    # Get the height and width of the field
    height = len(field)
    width = len(field[0])
    # Iterate through the columns of the field
    for col in range(width - 2):
        # Iterate through the rows of the field from the bottom up
        for row in range(height - 1, -1, -1):
        # Check if the current cell is occupied
        if field[row][col] == 1:
            # If the cell is occupied, stop checking this column
            break
        # If the figure fits in the current position, check if a full row is formed
        elif all(field[row + i][col + j] + figure[i][j] <= 1 for i in range(3) for j in range(3)):
            # If a full row is formed, return the current column
            if any(all(field[row + i][col + j] == 1 for j in range(3)) for i in range(3)):
                return col
    # If no full rows are formed, return -1
    return -1

Unfortunately, test cases aren't provided for this one. However, looking at the two responses, there are a couple obvious differences. ChatGPT doesn't handle the size of the board at all. CodeSignal uses this size in multiple places of their code. ChatGPT also uses all() and any() checks, where CodeSignal uses longer and more complicated if statements. Personally, I'd prefer the simple if statements for readability.

I'm unsure how to score this one without the test cases, but will note that ChatGPT failed CodeSignal's first sentence in the solution explanation:

This solution starts by defining some dimensions that will be important for the problem.

I'm skeptical that without the size of the figure this will work as expected.

Lookup Table¶

Lastly, we get to a question with lookup tables. These types of problems can usually be solved via brute force, at the expense of performance. They make good questions for a more senior developer. Note that this prompt also has exponents that I needed to adjust when pasting to the chat bot.

This question tests whether candidates have the problem-solving and data structures skills to use a lookup table (hash set/dictionary) in their programming language of choice. It also involves a bit of tricky logic to avoid double-counting pairs. Finally, this question asks candidates to pay close attention to constraints, testing a key skill for real-world development.

After copying and pasting the prompt into ChatGPT, it spit out a response that attempted to brute force the problem. Depending on how CodeSignal presents the problem to a candidate, this could easily be worked around by modifying the prompt to indicate you are looking for a problem with a hash table. If that type of response isn't provided by CodeSignal though, ChatGPT would not have presented the expected solution, though it did still return expected results. The time trade off would have been seen with different test cases.

I modified the prompt slightly to indicate that I wanted a response with a hash table. ChatGPT complied.

def solution(numbers):
    # Initialize a hash table to store the numbers that have been seen
    seen = {}
    # Initialize a count of the number of pairs that sum to a power of 2
    count = 0
    # Iterate through the numbers
    for i, x in enumerate(numbers):
        # If x is a power of 2, increment the count
        if x & (x - 1) == 0:
            count += 1
        # Check if x - y is a power of 2 for any y that has been seen
        for y in seen:
        # If x - y is a power of 2, increment the count
            if (x - y) & (x - y - 1) == 0:
                count += 1
        # Add x to the hash table
        seen[x] = True
    return count

This passes the test cases as well.

Results¶

Of the four sample questions provided by CodeSignal, the "basic" two questions are solved without any difficulty. A third takes a little prompt tweaking to indicate that we want a hashtable instead of a simple brute forced solution. The last one is difficult to evaluate without test cases. So, of the 3 that can be evaluated all pass easily. Two with very little modification (converting exponents). The third needs a small nudge toward the type of solution desired.

What's this mean?¶

I've looked at common interview questions from LeetCode, TestGorilla and now CodeSignal. As a hiring manager, I can see that a simple technical assessment of skills isn't enough to evaluate a candidate. Fortunately, I knew that already, but I'm hoping that others in the industry are seeing that tools exist for developers to make their day to day lives easier. Those tools can, literally, do the work for them if you are presenting them with basic puzzles during the interview process.

I can already hear people saying that they do face to face coding, so this would never work in their process. You may be right, but ask yourselves this: do your developers do face to face coding in their day to day work? If not, why are you evaluating your candidates with that type of pressure?

Adjust your interview process to reflect what you actually need your candidates to demonstrate. A puzzle isn't real work. Give them a real problem to talk through with your team. Evaluate their ability to solve a real problem so that you can learn about their thought processes, how they approach a problem, and what resources they utilize.

Breaking TestGorilla interview questions with ChatGPT

2022-12-26T10:45:00-06:00

Introduction¶

I've written quite a bit about ChatGPT in the last month. As of this post, it is still banned on Stack Overflow. I believe I showed why with my previous post. In it, I showed that ChatGPT is prone to adding subtle errors (and in some cases not so subtle), which is bad on a Q&A site.

Today's post follows in the footsteps of my writing about how I hope ChatGPT inspires software engineering hiring managers to change their hiring process. I wrote about how ChatGPT can solve LeetCode problems. Today's post is going to focus on a company called TestGorilla and a couple of their assessments.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of TestGorilla. That said, I do not recall TestGorilla being discussed during my employment at Woven. I only became aware of them during my job search this year. I'd be happy to do this same type of analysis on a Woven assessment, but as far as I am aware, their prompts are not presented publicly. TestGorilla's prompts used in this article are taken directly from their website. I did not speak with either company prior to writing this article.

Assessment One - PHP Arrays¶

The first assessment I want to look at is related to PHP Arrays. The prompt itself is fairly simple.

This is billed as:

PHP (coding): intermediate-level algorithms This PHP test evaluates candidates' coding skills through a short coding assignment. The test requires candidates to write code that meets certain requirements, and candidates can run their code to test if the result matches their expectations.

The estimated time for this test is 30 minutes.

Well, let's see if ChatGPT is an intermediate level PHP programmer. Pasting the prompt into ChatGPT quickly spits out a result, in Python. I'm not surprised since almost all of my testing has been in Python. I provided one additional set of directions for ChatGPT to utilize, based on the boilerplate code provided.

Write this in PHP using the following as a template:

<?php function sum_array( $no1, $no2 ) {

$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
    //Insert your code here 
}
?>

The result provided looks functional, based on very outdated PHP syntax knowledge.

<?php

function sum_array($no1, $no2) {
$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];

if ($no1 < 0 || $no2 < 0) {
    return -1;
} else if ($no1 > $no2) {
    return 0;
} else {
    $sum = 0;
    for ($i = 0; $i < count($array); $i++) {
    if ($array[$i] >= $no1 && $array[$i] <= $no2) {
        $sum += $array[$i];
    }
    }
    return $sum;
}
}

?>

I copy and pasted that into TestGorilla's preview page and ran it to see how it'd do.

Task executed successfully.

---------------------------------
    Input parameters: ((int) 10, (int) 20)
Result: (30)
OK
---------------------------------
    Input parameters: ((int) 10, (int) 30)
Result: (60)
OK
---------------------------------
    Input parameters: ((int) 30, (int) 50)
Result: (120)
OK
---------------------------------
    Input parameters: ((int) 90, (int) 100)
Result: (190)
OK

Success! One intermediate PHP assessment completed. Total time taken: 2 minutes.

Assessment Two - Kotlin Arrays¶

This assessment is listed as a "New Test" by TestGorilla at the time of this post. The challenge here, if ChatGPT fails, is that I have never used Kotlin.

This is billed as:

Kotlin (Coding): Working with Arrays This Kotlin Coding test evaluates candidates’ ability to work with arrays. The test gives candidates 30 minutes to create a short algorithm involving arrays while taking into consideration all the requirements provided.

This is billed as a 20 minute assessment. It is also billed as an intermediate level assessment.

I pasted the prompt into ChatGPT and also provided the boiler plate code block. The prompt doesn't explicitly say "Kotlin" anywhere, and I didn't mention it either. The only hint provided was the boiler plate code. ChatGPT picked up the language and returned a result.

fun getUniqueStudentIds(a: Array<Int>): Array<Int> {
if (a.any { it < 50000 || it > 99999 }) {
    return arrayOf(-1)
}
return a.distinct().toTypedArray()
}

Running this on TestGorilla's preview page passes the built in test cases:

Task executed successfully.

---------------------------------
Input parameters: (arrayOf(52311, 52311, 55125, 55125, 52128, 52130, 52128) )
Result: (52311, 55125, 52128, 52130)
OK
---------------------------------
Input parameters: (arrayOf(59100, 58200, 58200, 51190, 58200) )
Result: (59100, 58200, 51190)
OK
---------------------------------
Input parameters: (arrayOf(54321, 54321, 54321, 54321, 54321) )
Result: (54321)
OK
---------------------------------
Input parameters: (arrayOf(54321, 53421, 53241, 53214, 54231, 54213, 54312) )
Result: (54321, 53421, 53241, 53214, 54231, 54213, 54312)
OK

Success! Total time: 1 Minute

Evaluation¶

Despite not knowing Kotlin, I can look at the code provided and see that improvements can be made to the code. The prompt says

Note that all student IDs are 5-digit numbers beginning with the number 5. Return -1 as a single element in the array in the case of any invalid input.

The if (a.any { it < 50000 || it > 99999 }) statement can be adjusted to reflect the prompt's specification better. Additionally, there isn't a built in test case for invalid data, but I suspect that it'll fail based on the expected return.

Depending on how TestGorilla is evaluating these edge cases on the paid version of this exam, this may be picked up. On the free preview version, though it is not.

Hiring process¶

Once again, the reader should be able to see that a simple technical evaluation of software engineers in the hiring process is no longer sufficient. It took me longer to determine which prompts I wanted to evaluate for this post than it did to solve with a couple copy and pastes into ChatGPT.

The hiring process must be more than "solve this code challenge". Hiring managers need to move past code only assessments. Instead, focus on a candidate's ability to technically assess and judge based on a context specific problem. These types of problems require a candidate to show their experience, versus their ability to Google, ask Stack Overflow or use ChatGPT to provide a problem prompt. Evaluate your candidates based on the work you want them to actually do.

Using ChatGPT to interact with an API

2022-12-20T23:45:00-06:00

Introduction¶

I've played with ChatGPT a lot since it was released publicly a few weeks ago. It's gotten a fair bit of media coverage (and I helped generate some of that with the Stack Overflow ban of ChatGPT). In my last post, I talked about how I hope that ChatGPT kills the LeetCode interview for Software Engineers.

Today, I want to test it out against an API that I know well. In April 2022, I left PacketFabric where I was Vice President of Software Engineering. I played with the PacketFabric API a lot during my tenure. I'm going to use that experience to see what ChatGPT can whip up in a few minutes.

The Goal¶

My goal here is to ask ChatGPT for a quick script to interact with a portion of the API, test that output and see how I can extend it. My thought is that this is how an engineer is going to go about their work. They have a task they need to accomplish (call this API end point), they need to make sure it works (test it), and then work on refining it to do what's needed.

Another goal that I have is to write as little code as possible during this test. I'd love to copy and paste all of this.

Getting started¶

For this test, I'll be using the Get Locations List end point. It's unauthenticated so I don't need to sign up for a demo account to get an API key. Side Note: The documentation is wrong in that it states this end point requires a Bearer Token. This will be seen in the generated code and I don't fault ChatGPT for adding it into the code, because the documentation says it is required

I started with a prompt:

Use PacketFabric's API (https://docs.packetfabric.com/api/v2/redoc/#tag/Locations/operation/get_location_list) to get a list of locations in California (CA)

The code that I was provided:

import requests

api_key = "YOUR_API_KEY"
headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

response = requests.get(
    "https://api.packetfabric.com/v2/location?state=CA",
    headers=headers
)

if response.status_code == 200:
    locations = response.json()
    print(locations)
else:
    print(f"Request failed with status code {response.status_code}")

As I mentioned above, the Bearer token isn't required for this end point. Its inclusion does not impact the functionality of this script, and assuming I'm making more than calls to the locations end point, it'll be needed anyway. I'll keep it here.

Now, there is a mistake in the code. For exactly the same reason we banned ChatGPT on Stack Overflow, we see a subtle error here too. If you run the snippet above, you will get:

Request failed with status code 404

Why? Because ChatGPT is hitting the location not locations (plural). The PacketFabric API uses plurals. This is a minor problem, and easily fixed, though it is explicitly stated in the documentation link which end points exist.

Moving on.

Finding All US Locations¶

Next, I wanted to get all US locations.

How do I use this to get all US locations?

The correct way to do this would be to use the region query parameter. ChatGPT had a different answer.

To get a list of all locations in the United States using PacketFabric's API, you can simply omit the state query parameter from the GET request. This will return a list of all locations across all states in the US.

The same code provided simply strips the state=CA from the requests.get() call. This answer is wrong. It will, technically, return all public locations in the United States. But it will also return locations outside of the United States too. This is a much larger result set than I was looking for.

A developer should be able to catch this during testing and realize that the code is incorrect. However, ChatGPT has a tendency to write responses in a very confident manner. This one is no exception.

Finding a specific POP location¶

Moving to the next test, I wanted to find locations just in a single point of presence (POP).

How do I get all locations in the ATL1 pop?

The response returned seemed reasonable. There was a bit of prose with this explanation and a code snippet that adjusted the query parameters.

To get a list of all locations in the ATL1 pop (point of presence) using PacketFabric's API, you will need to send a GET request to the /location endpoint with the pop_code query parameter set to ATL1.

"https://api.packetfabric.com/v2/locations?pop_code=ATL1"

Again, this is incorrect. pop_code isn't a valid query parameter. It's not in the documentation and it doesn't return the expected result.

I attempted to explain this to the bot:

pop_code isn't a parameter defined in the documentation. Is this an undocumented parameter?

I received an unexpected response, with a new code snippet.

I apologize for the mistake in my previous response. pop_code is not a valid query parameter for the /location endpoint in PacketFabric's API. The documentation for the /location endpoint (https://docs.packetfabric.com/api/v2/redoc/#tag/Locations/operation/get_location_list) lists the valid query parameters as state, city, and zip_code. These parameters can be used to filter the locations returned by the API based on the state, city, and ZIP code of the location. For example, to get a list of all locations in Atlanta, Georgia (GA), you could use the following request:

import requests

api_key = "YOUR_API_KEY"
headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

response = requests.get(
    "https://api.packetfabric.com/v2/location?state=GA&city=Atlanta",
    headers=headers
)

if response.status_code == 200:
    locations = response.json()
    print(locations)
else:
    print(f"Request failed with status code {response.status_code}")

This will send a GET request to the /location endpoint with the state query parameter set to GA and the city query parameter set to Atlanta, and will print the list of locations returned by the API. I apologize for any confusion my previous response may have caused. Please let me know if you have any further questions.

This is a better attempt - but still wrong. Going this route will return more POPs than expected. I'll receive everything in the city of Atlanta, instead of my original request, which was to only get locations in the ATL1 POP. This is another subtly wrong answer.

It's also interesting to note that ChatGPT says that only state, city and ZIP code are valid query parameters here. state and city are defined in the documentation. ZIP Code is not. That's not a valid query parameter for the locations end point. Also notice that it's still using the singular location

Results¶

My impression of ChatGPT has not changed much since it was released at the end of November. It is a novel tool - much like Github's copilot, but it's not going to take a developer's job. It can provide a good starting point for a problem, but if you are depending on it to write your code for you, you will find it's not going to work long term.

In this test, I learned that I don't know what it is doing about reading external documentation. I simply provided a link to the API documentation, when asking how to get locations. It parsed out the correct URL schema, including version code. But, at the same time, it couldn't parse out the query parameters. It incorrectly added two parameters that don't exist (pop_code and zip_code). It also missed the plurality of the API end points. It introduced a subtle bug - omitting the region to get US based locations, returns locations around the world.

Personally, I'd be upset if an engineer on my team committed code generated in this test. It's buggy, and has subtle inaccuracies. The code provided by ChatGPT is a good skeleton. It is up to the developer to validate that it works as expected. Without the crucial step of developer testing, it is guaranteed that bugs will be committed to the code base because "the AI said it'd work".

If you are going to use ChatGPT (or Copilot or a system similar to either), it is vital that you verify the code you are running. Does it do what you expect? Does it perform as you expect? The tools are there to help you build this, but that's just it. Help, not do. The testing I did here took only a handful of minutes. For someone unfamiliar with the API they are interacting with, this quick turn around should mean they have a lot more time to do robust testing.

ChatGPT is here - Use that to stop asking LeetCode questions during interviews

2022-12-15T23:45:00-06:00

Introduction¶

I have interviewed a lot of software engineers looking for a new position. I've talked with college students looking for an internship, junior engineers just getting started, staff engineers looking for a new role, managers and directors to lead these teams. In every one of those interviews, I'd told the candidate that I hate puzzle questions (ie. "How many windows are in Seattle?") and I hate LeetCode style questions.

It turns out that ChatGPT is good at solving these problems.

Good. Maybe that will kill LeetCode interview questions once and for all. There are so many better ways to determine if an engineer is the right fit.

ChatGPT vs LeetCode¶

We have temporarily banned ChatGPT over on Stack Overflow, because it can and does provide subtly incorrect answers. It's been so bad that we called up two new moderators from the previous election today. One of my mod colleagues provided some context around this today too.

We get a few hundred CGPT flags per day, leading to hundreds or even thousands of post deletions per day. There's often one flag per user (not accounting overlap from several users), but each flag can lead to anywhere between a couple and tens of deletions per user.

LeetCode, however, checks your answer right away. They don't have the same problem as Stack Overflow. I wanted to see just how good this new chatbot was, and took it for a spin against a random LeetCode problem.

I randomly selected a medium difficulty problem with a 58% acceptance rate. My victim question: 1143. Longest Common Subsequence

Getting started¶

The first step in passing this test, is selecting the entire prompt, copying it and pasting it directly into LeetCode's UI. I didn't read the prompt and I didn't format it. Just a blind copy and paste. I also provided one instruction:

Provide python code

Press enter and wait 30 seconds or so.

The response¶

Within seconds I was provided this block of code

def longestCommonSubsequence(text1: str, text2: str) -> int:
    # Convert text1 and text2 to lists of characters
    s1 = list(text1)
    s2 = list(text2)

    # Initialize the dynamic programming table
    n = len(s1)
    m = len(s2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    # Fill the dynamic programming table
    for i in range(1, n+1):
        for j in range(1, m+1):
            if s1[i-1] == s2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    # Return the length of the longest common subsequence
    return dp[n][m]

It likes the phrase "dynamic programming table". Along with this code, it even explains its reasoning for going this route. Maybe our intrepid AI can teach a junior dev a new trick or two while doing this puzzle.

Then it shows that tests pass...very confidently.

All the tests passed, so this implementation is correct

The result¶

I'm trying to show that ChatGPT can do this, so at this point I haven't read the prose above. I copy and pasted the code it returned and clicked "Submit".

I waited a few seconds for all tests to run and got my results.

ChatGPT crushed this. It performs better than 96% of other submissions in terms of run time performance. It uses less memory than 71% of other submissions.

Can we please stop using LeetCode now?¶

The total time it took to perform this test was under 5 minutes. I randomly selected a problem, copy and pasted the prompt, waited for a response from the chatbot, copy and pasted the code response, waited for the tests to run. Complete.

As a hiring manager, I'd have wasted my time and the candidates time having them do this problem now. Instead, we can talk about an actual problem they will be solving. We can do a real world problem instead of this. There are better ways to interview an engineer and I'm hoping ChatGPT makes that abundantly clear to hiring managers everywhere.

We're going to have to adapt our hiring processes, but that's a good thing. My recent experience shows that at most companies, it's bad anyway. Now we have an opportunity to shine some sunlight on it and make it better.

In the meantime, if you continue to use LeetCode or something similar to evaluate your candidate's technical skills, you are doing your team a huge disservice. Candidates know about this tool, and they know that the easiest way to get an interview is to play the game. There have been tens of thousands of layoffs in the tech industry in the last six months. I guarantee candidates are looking for every advantage they can get, to move to their next role.

Stack Overflow bans ChatGPT temporarily

2022-12-05T14:45:00-06:00

Today Stack Overflow moderators (myself included), have implemented a temporary ban on ChatGPT on the site.

Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned.

This ban was picked up immediately by several technology news outlets, including ZDNet, TheVerge, and Vice. It was also picked up by CNN, The New York Times, and the Washington Post.

That's a lot of news coverage.

The question that I want to answer is "Why?". Stories and marketing from OpenAI, give reasons why this new chatbot is a "good thing". With their examples, it definitely looks that way. But, in practice, it's not working out so well.

The responses that users have been posting on Stack Overflow have a high rate of being incorrect. Normally, the community can handle this, but the responses aren't your usual small code snippet of an answer. Instead it is a long, detailed, explanation that looks plausible. But, it's wrong. Combined with users posting multiple answers an hour, this is a lot of content that Stack Overflow reviewers (or worse, the handful of elected moderators) to go through and determine if it's valid.

At the time the ban was implemented, we'd seen thousands of answers generated by ChatGPT. On the one hand, this is impressive work on the ChatGPT AI itself. It can be difficult to detect and is good at holding a conversation. On the other hand, and more important from Stack Overflow's perspective, this isn't helping the user base. Thousands of subtly wrong answers is awful. It doesn't help the user looking for help, and it will very quickly destroy the trust that millions of developers put in the site if this is allowed to continue.

I'll admit that I'm disappointed that AI hasn't reached the point where it can do what ChatGPT seems to do. But, this is a step forward. Unfortunately, this step seems to have left a bad taste in the mouth of developers looking for help beyond a toy example.

For the time being, ChatGPT is banned on Stack Overflow. The moderation team will continue to work with the company to ensure the community we have volunteered to moderate remains one of high quality. Additionally, as the larger Stack Exchange network of sites debates a similar ban, the Stack Overflow moderation team will be able to provide input on our experiences.