Andrew Wegner | Ponderings of an Andy - job search

My thoughts on what remote work will look like in the coming decade

2023-05-23T15:30:00-05:00

Preface¶

This post is a bit different than some of my others. It's a direct response to an article. Over the past 9 months, as the technology industry contracts, we've seen return to office orders from countless companies. Combined with the, roughly, 300k layoffs between August 1, 2022 and May 23, 2023 (I was laid off in that time frame too), we've seen a lot of articles about how many companies are recalling people to the office.

I think this is short sighted. Remote work, when done properly, is the way of the future and forcing everyone back into an office is not going to work as expected. One of the bright spots from the COVID-19 pandemic was that society learned many jobs can be done from home.

The article¶

The article I'm responding to is Changing my mind on remote, moving the team back to San Francisco by Flo Crivello. This isn't designed to pick on this author specifically, it's just the latest in a long line of these types of posts.

I encourage readers to take a look at the article. It is well written. I, however, disagree with almost all of it.

Let's get started, shall we?

Counter point¶

I have a lot of problems with this article. The short version is that it sounds like this company implemented remote work poorly, possibly during covid (I'm not sure how long they've been around). I think remote work is the best, especially for companies that do not need a local presence.

The article raises four coordination cost bullet points:

It’s harder to get a hold of each other, as we’re not online at the same time. “I’ll talk about it when I see him tomorrow” — these delays compound in a huge way.

So most interactions are async, leading to lower bandwidth, more context switching, and more things falling through the cracks.

Even sync chats aren’t as good. People can’t interrupt each other or have sidebars, and there are bugs with video, audio, screenshare, etc… These frictions compound too.

This causes us to be less aligned. We’re only a few engineers right now, and yet people feel out of the loop on who’s building what.

The first three of these are the result of poor company policy and team expectations for how remote work will function. Remote work is still "work". It isn't "spend time by the pool with a laptop". It isn't "pick up groceries, mow the lawn, volunteer at the classroom parent, and maybe hop on a phone call". It is work. The team and company need to set that expectation. However, remote does offer the flexibility to do different things during the day. The company needs to be accepting of time zones, and time shifts. Embrace the employees that are productive outside of a 9-5 working day, especially if your workforce is scattered across time zones.

If the company is fully remote, they should build in the async communication as part of their communication plan. Good communication requires repetition. Asynchronous communication leads to that perfectly. Send a message in slack, send an email, mention it in Slack in another context, bring it up on an all hands call. Versus, only mentioning it once in a meeting.

Items falling through the cracks isn't a remote or async communication problem. It's a team problem. It's a leadership problem. It's a training problem. Don't blame remote work for this failure. Empower your team to make decisions that don't require constant input from others. This is one of the core points that GitLab's TeamOps has. Give your teams agency to make decisions without permission, review or approval. Do this, and those cracks get a lot smaller - regardless of where the employees work.

Synchronous chats having communication problems aren't great, but the reasons for failure that are listed by the author don't sound like a sync chat problem. "Interrupting", "having sidebars" - There are other avenues to communicate - zoom chat and slack are two examples. The bigger problem, in my opinion, is the requirement to always have video communication on in a company. This leads to fatigue due to communication patterns that don't exist in face to face communication (close eye contact, seeing your self, reduced mobility, and higher cognitive load). I argue that a well structured async communication plan and team will do better without constant sync communication. Sure, seeing people in person is helpful sometimes, but it's not required for all interactions.

The alignment problem is a leadership failure. I've worked remotely for the better part of a decade. I worked in an office for over a decade. Teams don't get misaligned because of remote work. They get misaligned because of leadership failures. Failure to communicate at all, or only slivers of data, will misalign a team faster than a network outage or Zoom failure will. Way faster. A poor company culture will ruin alignment in a way that a "My web cam isn't working today" could only dream of accomplishing.

Colocation is more fun too. You get to have lunch with your team, grab beers on Friday nights, play video games at the end of the day in the office, etc

sigh

This author does not care about a work life balance either for themselves or their employees. There are employees that absolutely love doing these things, building those relationships, and that's great. There are others that want to get home and spend time unwinding, being with family, hanging out with friends and none of that can be done if the boss is expecting you to go to the bar with them on a Friday night or stay at the office to play video games. Team building is important, but the way it's described here is not the way to build a team long term.

This matters not only because fun is fun, and building community is super important, but also because it helps build trust, which improves our work

My personal bias is going to show here, but my trust in team mates grows more through professional pursuits. How a team accomplishes a task or how a team responds to failure ( especially how they respond to failure) is going to build that trust a lot more than going to a bar on a Friday. Going to a bar will build a personal friendship, but that is different than a professional level of trust.

[A]sk yourself which of the following competitors you’d be most afraid of:

The bunch of hackers coding on a couch in an apartment

The team that’s fully colocated

The team that’s hybrid

The team that’s fully remote, split across timezones

The last one, absolutely. In this environment, I as a manager, can hire the best person for the job regardless of where they sit in the world. It doesn't matter if we could car pool to an office or if you are 18 hours ahead of me. This is the massive advantage of a fully remote workforce and is commonly brushed under the rug in these types of articles. A team that is aligned and has the best people in the world or industry working for them is going to crush a team that is colocated and is limited by who lives in commuting distance or is willing to relocate to commuting distance of the office.

Lastly, this quote

A quick look for software engineers finds that SF has about 40 times more than Miami: (and my hunch is that they’re better ones too)

Wow. Every engineer here should be offended by this. I've hired over 100 engineers in my career. I think I have a pretty good idea of how it works, how these engineers perform, and how they behave. Purely from an economic stand point, hiring from the Bay Area is expensive. "But your engineers don't live in the Bay Area, Andy, you just said you hire globally." True...now, tell that engineer in Miami or London or Ankara or Sao Paulo or anywhere else why they are making at least $100k less a year than their coworker doing the exact same thing and see how quickly engineering morale crumbles. Or, are you one of the minority of companies that pay the same regardless of location? If not, you should be.

The best engineers I've worked with, except one, have been located well outside of the Bay Area. An engineer in Miami can do the exact same work that an engineer in San Francisco can at the same quality.

Remote work can and is incredibly successful for many companies. It does require deliberate, intentional, company decisions to make it work though. I shared the TeamOps information above (and completed TeamOps certification earlier this year). GitLab, one of the largest successful all remote companies in the world, has intentionally built their remote policies around this. Without putting in the time or effort to ensure remote is successful, teams will (and are) defaulting back to in office. I think we are going to be seeing just how unsuccessful that strategy is in the next decade. One of the few benefits of Covid was that it showed many engineers around the world just how much they are missing out on with daily commutes, 10-12 hour days in the office, lack of family time which is why there has been such push back. Those that have successfully adapted (or built from the ground up) the company culture to handle remote work will have a huge advantage as this period of contraction eases in the future.

CoderByte assessments fall to ChatGPT

2023-02-04T23:30:00-06:00

Introduction¶

Today I'll be evaluating CoderByte and their publicly posted challenges against ChatGPT. I selected an easy, a medium and the single hard question from their list of open challenges and ran those through ChatGPT.

In previous articles in this series, I've shown how ChatGPT can easily solve interview assessments from LeetCode, TestGorilla, CodeSignal, Codility and HackerRank[hackerrankg]. I've said it in those articles, but I'll say it again: Companies need to evaluate their engineering candidates - all candidates, actually - appropriately when they are hiring. Your engineers are using many tools to do their jobs and AI Tooling is one of the newest tools on the market. Removing a tool from your engineers is going to make them less efficient, and not testing your candidates on how they use these tools is going to mean you hire someone that had not fully demonstrated their capabilities.

ChatGPT should be treated as a tool, but it's also helpful to think of it as a junior level engineer. You need to check its work. ChatGPT is known to be inaccurate, which is why it remains banned on Stack Overflow. There are a few instances where I had to do that below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. I do not remember CoderByte being mentioned during my tenure and CoderByte does not seem to consider Woven a competitor, as Woven isn't listed as an alternative on their pricing FAQ page.

Easy - React Button Toggle¶

While I've tried to be mostly random in my selection of assessments during these tests, I intentionally selected this react assessment because I don't know react. I've liked these types of assessment questions in the previous articles because it means I can't apply my experience to "nudge" ChatGPT toward a solution.

The challenge here is to toggle a button between "On" and "Off". You're presented with a very small amount of boiler plate code.

I simply copy/pasted the prompt and the template into ChatGPT. It sent back this:

import React, { useState } from 'react';
import ReactDOM from 'react-dom';

function Toggle() {
const [toggle, setToggle] = useState(true);

function handleClick() {
    setToggle(!toggle);
}

return (
    <button onClick={handleClick}>{toggle ? 'ON' : 'OFF'}</button>
);
}

ReactDOM.render(
<Toggle />,
document.getElementById('root')
);

I copied that into the IDE provided by CoderByte and submitted the code.

This test took two minutes to complete. The bulk of that time was spot checking that ChatGPT didn't mess with the boiler plate code too badly.

The interesting this about the results is that even with the 10/10, the system says it only scored higher than 12% of other users. Does this mean that only 12% of the users passing this assessment are getting lower than a 10/10? To me, that's the sign of a bad assessment.

Medium - SQL Member Count¶

The medium challenge I selected was an SQL question. SQL is interesting, in my experience, in that the results returned on an assessment are a lot more uniform compared to a non-SQL coding assessment. This one wasn't a difficult one with joins.

The goal of this assessment was to return a list of names in alphabetical order, a count of the number of people that report to them, and the average age of their team members. I pasted the prompt, the expected table layout for the output and the table layout being selected from.

The first response was

SELECT ReportsTo, COUNT(ID) AS Members, AVG(Age) AS Average_Age
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

This is where experience, and checking the results are important. The expected output showed that the average age should be an integer. It also showed that the average age column should be named Average Age not Average_Age (notice the lack of the underscore). I provided this guidance to ChatGPT

The AVG(Age) should be an integer not a float and The "Average_Age" field should be "Average Age" without an underscore.

It corrected both problems with a new query which I submitted.

SELECT ReportsTo, COUNT(ID) AS Members, ROUND(AVG(Age)) AS "Average Age"
FROM maintable_YTTQH
WHERE ReportsTo IS NOT NULL
GROUP BY ReportsTo
ORDER BY ReportsTo;

ChatGPT scored another 10/10.

Even with that extra round trip with ChatGPT, it took less than a minute to complete this assessment.

This assessment seems to have a slightly better split of user scores, which is progress I suppose.

Hard - Bracket Combinations¶

There is only one hard assessment available publicly. The goal for this assessment is to return the number of combinations you can make with a number of pairs of parenthesis. The example provided is:

The possible combinations of 3 pairs of parenthesis, namely: ()()(), are ()()(), ()(()), (())(), ((())), and (()())

I copied the prompt and boiler plate code into ChatGPT and it returned the following response:

def BracketCombinations(num):

    from math import factorial
    return factorial(2 * num) // (factorial(num) * factorial(num + 1))

# keep this function call here 
print(BracketCombinations(input()))

Experience here says that is odd. Technically speaking, Python will allow you to have that import statement inside a function, but if I saw one of my developers do it, I'd mention it during a code review. I was curious though and submitted it as is.

I also went back and told ChatGPT:

Write the function without importing factorial

It replied with this function:

def BracketCombinations(num):
    def fact(n):
        result = 1
        for i in range(1, n + 1):
            result *= i
        return result

    return fact(2 * num) // (fact(num) * fact(num + 1))

# keep this function call here 
print(BracketCombinations(int(input())))

I submitted this response too.

I only have one set of images for this challenge, because they both output exactly the same responses. Both solutions scored the same. Both solutions were better than 35.1% of other users. Both solutions had an O(n) complexity.

Showing the complexity is a nice touch. I will admit that I was expecting the solution that imported factorial to be rejected because it's using a built in function instead of rolling your own. Good for CoderByte for allowing a developer to use the built in libraries as an engineer would do in a production environment, instead of adding a false constraint.

Copy pasting to CoderByte kept this close to a minute.

Final Thoughts¶

Another code only assessment tool has been shown to be ineffective. The one thing this one does, that others don't show publicly, is the time it took to solve a problem. I'm very surprised that a majority of these were solved in under ten minutes. Even so, solving a problem in less than a minute will probably raise a flag of some kind for a hiring manager giving this type of assessment. I guarantee that time is visible to them.

These three assessments show how well ChatGPT can be in being your virtual junior engineer. With the react question, I have no idea if it's the more efficient way to solve the problem but it works. With the SQL question, experience identified two small problems and allowed me to provide feedback and get a corrected query in seconds. The third question turned in two different ways of solving the problem. Apparently, according to CoderByte, both of which perform exactly the same.

The total time it took me to run these three tests was under five minutes. That's a great tool to have in my pocket.

ChatGPT easily solves HackerRank questions

2023-01-08T01:15:00-06:00

Introduction¶

It's important that companies evaluate their engineering candidates appropriately when they are hiring. Do other areas of your company evaluate their potential hires by restricting access to tools they'll use to perform the job? I suspect not. Why do the same to your software engineers?

AI Tooling - things like GitHub's Copilot and now ChatGPT - are tools that your engineers can use to do their job. During the hiring process, engineers should be evaluated on the skills they have, and that includes the ability to utilize all available tools. I expect AI assistance to become as common place as using Stack Overflow to solve a problem. ChatGPT, Copilot, and any future tools doing similar tasks are going to be used frequently in the near future.

In this article, I'm testing ChatGPT against one of the most popular coding assessment services: HackerRank. I've already covered other assessment services like LeetCode, TestGorilla, CodeSignal and Codility in this series. I recommend taking a look at those too.

To the engineers that have found this post and are looking for solutions to your coding assessments, I want to warn you, ChatGPT is an assistant. Think of it as a junior level engineer. You may need to check its work. ChatGPT is known to be inaccurate, which is why it was banned on Stack Overflow. Don't take its responses as gospel. Evaluate the responses and make improvements as needed. You'll see I had to do that a couple of times with the questions below.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. HackerRank is mentioned multiple times in the Woven blog and is a competitor to Woven.

Final note before I begin, HackerRank presented a couple annoyances during this test because they use LaTeX formatting in their prompts. Simply copy and pasting prompts to ChatGPT was not possible, because key parts of the problem would be missing. This meant that I spent several minutes per prompt cleaning it up and reinserting the LaTeX characters in plain text. This accounts for the higher solve times compared to the competing services in other articles.

Triangle Quest¶

I selected a random Medium difficulty Python question for my first prompt: Triangle Quest. This one was unique in this series of tests. The expected solution was to add only a single line of code and I could only use arithmetic operations, a single for loop and print statement.

As previously mentioned, there were a few LaTeX things to clean up, but then ChatGPT returned a single line of code to add to the for loop provided. The whole code is:

for i in range(1,int(input())): #More than 2 lines will result in 0 score. Do not leave a blank line also
    print((i * (10 ** i - 1)) // 9)

I submitted the code and passed all test cases.

I was surprised by this one. I was expecting to encounter some problems with the constraint of only being able to use arithmetic operations. It turns out, ChatGPT caught that constraint and solved the problem without any issue.

Iterables and Iterators¶

I wanted to try another Medium difficulty problem, that added more than a single line of code. The constraints on the previous question put the problem well into "puzzle" territory, and I am not a fan of puzzles in technical interviews. The second medium question selected was Iterables and Iterators.

The first result returned didn't take input. Instead, it hard coded values like so:

# given list of letters
letters = ['a', 'a', 'c', 'd']

# number of indices to select
k = 2

The interesting thing with this solution was that it ignored the first input row - "The first line contains the integer N, denoting the length of the list". I don't disagree with this change. We should be able to determine the length of the list from the list itself.

However, the solution required taking input from STDIN, so I let ChatGPT know this by pasting in the only bit of information provided in the code here:

Enter your code here. Read input from STDIN. Print output to STDOUT

ChatGPT made the necessary change and provided a new code block.

import itertools

# read input from stdin
n = int(input())
letters = input().split()
k = int(input())

# number of combinations of indices
total_combinations = len(list(itertools.combinations(range(1, len(letters)+1), k)))

# number of combinations of indices that do not contain the letter 'a'
combinations_without_a = list(filter(lambda x: 'a' not in [letters[i-1] for i in x], itertools.combinations(range(1, len(letters)+1), k)))

# probability that none of the selected indices contain the letter 'a'
probability_without_a = len(combinations_without_a) / total_combinations

# probability that at least one of the selected indices contain the letter 'a'
probability_with_a = 1 - probability_without_a

# print the probability to three decimal places
print("{:.3f}".format(probability_with_a))

I submitted this code and it also passed each of the test cases. This solution also ignored using the n input, but did accept it as input.

Validating Postal Codes¶

At this point, I've shown that ChatGPT can do medium level problems. So, I selected a Hard problem: Validating Postal Codes. The interesting constraint on this one is that if statements aren't allowed. The goal here is to provide two regular expressions to solve the problem.

I've built some nasty regular expressions over the years. I've talked about SmokeDetector in the past, and it's role in removing spam from the Stack Exchange network. It is built on regular expressions. Lots and lots of regular expressions. So, I felt pretty comfortable with this question and wanted to see how ChatGPT would handle it.

The second regular expression, the alternating repeating digit pair, didn't work the first time.

regex_alternating_repetitive_digit_pair = r'(\d)\d\1'

For those that know regular expressions, that is not going to work. I pointed this out and it corrected itself, and explained the meaning of the new regular expression. The two final regular expressions are

regex_integer_in_range = r"^[1-9]\d{5}$"    # Do not delete 'r'.
regex_alternating_repetitive_digit_pair = r"(\d)(?=\d\1)"   # Do not delete 'r'.

This is using a positive lookahead assertion, which ChatGPT explains as:

This regular expression will match any digit (\d) followed by another digit, with the same digit (\1) appearing immediately after that. The (?=...) syntax is a positive lookahead assertion, which means that it will only match a digit if it is followed by the specified pattern, but the pattern will not be included in the overall match. The parentheses around \d capture the digit so that it can be referred to as \1 later on.

I submitted the two regexes above, and passed the test cases.

Final Thoughts¶

The majority of my time on these was spent cleaning up the LaTeX formatting. I mentioned a few prompt clarifications needed to solve problems in the returned code too - one required accepting user input and the other was an invalid regular expression. Pointing this out to ChatGPT got the problem resolved immediately.

This is exactly the type of thing that I'd expect an engineer to need to do with the AI assistant (or more junior engineer). Hiring processes must adapt to the existence of AI assistants. Evaluate your candidates and their ability to do more than simply write code.

Taking Codility's sample interview exam using ChatGPT

2023-01-05T01:00:00-06:00

Introduction¶

It was announced on January 3, 2023 that New York City has banned students and teachers from using ChatGPT, because of "concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content". This continues the daily news stories about ChatGPT that have come out over the past several weeks. I suspect that 2023 is going to be a massive year for public awareness of "AI". Especially with GPT-4's expected release in a month or so. In the meantime, ChatGPT remains banned over on Stack Overflow. The accuracy of ChatGPT was also mentioned by New York City in their banning of the tool in the school system.

My perspective is that this is a tool that software engineers can utilize, but they have to understand that it is just that - a tool. It is only as good as the engineer that wields it. A good engineer will be able to utilize it very effectively. Hiring managers need to be aware of what this tool is capable of, though. I've already shown that it can easily solve common interview questions that the software industry uses in their hiring process. I showed that questions from LeetCode, TestGorilla, and CodeSignal are easily solved by this tool in previous articles in this series.

Today, I'm going to look at another service, similar to these: Codility. Much like CodeSignal, Codility sits on both sides of the interview process. It has a developer facing set of challenges and learning material. It also sells services to companies seeking to streamline a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of Codility.

The question in this post is taken from Codility's sample test. I will not be sharing the exact prompt from the sample test, as Codility has a disclaimer in the prompt clearly stating that disclosure is prohibited. Fair enough.

It'd be a shame if they hid all of their technical exam questions behind such a disclaimer though. Especially when this sample question sits on Stack Overflow with over 400k views and has over 100 answers. It also generated its own meta question.

This is just one reason that these types of questions aren't great for technical interview questions. Exam questions will always escape the exam environment. A student, a job candidate, or someone interested in the content will be able to find it and the answers after it's out there.

The problem¶

If you want to see the full prompt, take a look at the Stack Overflow question. But, briefly, the problem to solve is to find the smallest positive integer that does not appear in an array of numbers. We are provided three sample arrays with positive, negative and duplicate entries.

I copy and pasted the prompt into ChatGPT and got a short function back.

def solution(A):
    # Create a set of all the positive integers in the list
    positive_int_set = set([x for x in A if x > 0])

    # Iterate through the positive integers, starting at 1
    i = 1
    while i in positive_int_set:
        i += 1

    # Return the first integer that is not in the set
    return i

I pasted this back into the testing environment and ran the code.

Compilation successful.

Example test:   [1, 3, 6, 4, 1, 2]
OK

Example test:   [1, 2, 3]
OK

Example test:   [-1, -3]
OK

Your code is syntactically correct and works properly on the example test.
Note that the example tests are not part of your score. On submission at least 8 test cases not shown here will assess your solution.

Results¶

The built in test cases worked as expected. After I submitted the code, I need to wait about 5 minutes for the hidden test cases to be processed and receive my results.

I passed the 3 example test cases, 5 additional "correctness" test cases and 4 performance test cases. 100%

Hooray! ChatGPT wins again. Total time taken - 4 minutes (plus 5 minutes for Codility to do its evaluation).

Change your interview processes¶

Engineers are smart individuals. They, generally, enjoy new technologies too. ChatGPT is a new technology that is going to be a valuable tool in their future. GitHub has had Copilot for a while now. ChatGPT brings a new tool to the toolbelt. It should be expected by hiring managers that your current engineering team and engineers you hire will use such tools.

Interview processes need to adapt to the existence of these tools. If you are asking a question that a simple tool can solve, are you really determining an engineer's ability? Do your counterparts ask someone in finance to calculate a percentage and not expect the candidate to use, at least, a calculator?

Hiring processes must adapt. Evaluate your candidates, especially the more experienced candidates, with something other than an exam question that their newest tool can solve. Your engineer is going to be doing more than writing code and implementing an algorithm. Evaluate those skills. Those are the ones you really need on your team.

If you are curious about the other common interview systems and how they perform against ChatGPT, read one of the articles in the series below.

Running CodeSignal questions through ChatGPT

2023-01-01T17:00:00-06:00

Introduction¶

ChatGPT has taken over a lot of technology news in the last month. Within days, Stack Overflow banned ChatGPT, due to subtle errors that the chatbot is prone to make. I've also written about how I hope that ChatGPT's existence will change the hiring processes in the technology industry, because of how easily it can solve LeetCode and TestGorilla technical interview problems.

Today, I'm going to turn my attention to another company that does interview tests: CodeSignal. CodeSignal sits on both sides of the interview process, with a LeetCode style developer facing set of challenges. Solve problems, increase your score. It also sells services to companies looking to outsource a portion of their hiring process.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of CodeSignal.

The questions in this post are taken from CodeSignal's blog, showing off four of their software engineering questions. CodeSignal provides an explanation of the question, and a sample response with a quick analysis.

Array Manipulation¶

Surprise! Another array question. I suppose it wouldn't be a generic software engineering interview without a question related to arrays.

Getting started...

You can see that it covers a basic array traversal and manipulation

A small nitpick - depending on the hiring process, this sentence can come across as condescending. Depending on where a technical interview fits in the process, it's possible that a non-engineer is the one that sees these results first. I would not count on my HR coworkers to understand "basic array traversal and manipulation", because it's not part of their skillset.

Let's see how ChatGPT does. I copy and pasted the prompt into ChatGPT and waited a few seconds for a result.

def solution(a):
    b = []
    for i in range(len(a)):
        # If it's the first element, use 0 as the value for a[i - 1]
        if i == 0:
        b.append(0 + a[i] + a[i + 1])
        # If it's the last element, use 0 as the value for a[i + 1]
        elif i == len(a) - 1:
        b.append(a[i - 1] + a[i] + 0)
        # Otherwise, use the values of a[i - 1], a[i], and a[i + 1]
        else:
        b.append(a[i - 1] + a[i] + a[i + 1])
    return b

It doesn't match the exact response that CodeSignal provides, but it functions with their test case. I'd count that as a pass.

String Pattern Matching¶

Another common theme in these recent tests I've run is pattern matching. This one, however, at least is more interesting than others. It still doesn't seem to reflect a real world problem, and is just a puzzle, but at least it's more unique.

It has the advantage of testing several fundamental programming skills at once: traversing multiple arrays with nested loops, working with subarrays, and performing basic collections/string operations.

We're still sitting in the "basic" tier here, so I suspect that ChatGPT will have no problem. Time to check that assumption by pasting the prompt to the bot. One thing I did have to modify here is the exponents. They pasted as non-exponents, which would have messed up those constraints at the end. This will be a similar problem with a later question. I adjusted those lines slightly to make the numbers exponents.

And, the response back was:

def solution(pattern, source):
    # Initialize the count of matching substrings
    count = 0
    # Iterate through the substrings of source with length equal to pattern
    for i in range(len(source) - len(pattern) + 1):
    # Initialize a flag to track whether the current substring matches the pattern
        match = True
    # Iterate through the characters in the pattern and the corresponding characters in the substring
        for j in range(len(pattern)):
    # If the character in the pattern is 0 (vowel), check if the corresponding character in the substring is a vowel
            if pattern[j] == '0':
                if source[i + j] not in 'aeiouy':
                    match = False
                    break
    # If the character in the pattern is 1 (consonant), check if the corresponding character in the substring is a consonant
            else:
                if source[i + j] in 'aeiouy':
                    match = False
                    break
    # If the substring matches the pattern, increment the count
        if match:
            count += 1
    return count

A big difference between the response generated and the response provided, is that CodeSignal's response breaks out a portion of the code into a second function. The ChatGPT response does everything in a single, larger, loop. Personally, I'd take off a point for not breaking this into another function. CodeSignal's response is better, but the ChatGPT one functions and passes the test. While the second function is a good idea, not having it is not a deal breaker for me. This is especially true of a "basic" test, which is probably being used for less senior roles.

Two-Dimensional Array Traversal¶

Now we're onto the harder questions. This is a long prompt, with multiple animated gifs, and is multiple pages of scrolling. I'm not going to paste the prompt here because the animation is important to understanding the algorithm that needs to be implemented. The general goal though is to build a very small scale Tetris board.

This Tetris-inspired question is challenging. Candidates will have to traverse and manipulate two-dimensional arrays in a non-conventional way. As it’s an advanced question, it tests how well an engineer can understand a complex set of constraints and proceed with a non-trivial implementation.

From my reading of the prompt, and looking at the solution logic, I agree. While this is still a puzzle type question, I don't recall seeing anything like this previously. The uniqueness, yet similarity to a game that most engineers will know, makes the question interesting to me.

The animated gifs are useful for a human to see what is expected in the solution, but shouldn't be needed by ChatGPT, provided the prompt is descriptive enough. Copying and pasting the text into ChatGPT, provides the following response:

def solution(field, figure):
    # Get the height and width of the field
    height = len(field)
    width = len(field[0])
    # Iterate through the columns of the field
    for col in range(width - 2):
        # Iterate through the rows of the field from the bottom up
        for row in range(height - 1, -1, -1):
        # Check if the current cell is occupied
        if field[row][col] == 1:
            # If the cell is occupied, stop checking this column
            break
        # If the figure fits in the current position, check if a full row is formed
        elif all(field[row + i][col + j] + figure[i][j] <= 1 for i in range(3) for j in range(3)):
            # If a full row is formed, return the current column
            if any(all(field[row + i][col + j] == 1 for j in range(3)) for i in range(3)):
                return col
    # If no full rows are formed, return -1
    return -1

Unfortunately, test cases aren't provided for this one. However, looking at the two responses, there are a couple obvious differences. ChatGPT doesn't handle the size of the board at all. CodeSignal uses this size in multiple places of their code. ChatGPT also uses all() and any() checks, where CodeSignal uses longer and more complicated if statements. Personally, I'd prefer the simple if statements for readability.

I'm unsure how to score this one without the test cases, but will note that ChatGPT failed CodeSignal's first sentence in the solution explanation:

This solution starts by defining some dimensions that will be important for the problem.

I'm skeptical that without the size of the figure this will work as expected.

Lookup Table¶

Lastly, we get to a question with lookup tables. These types of problems can usually be solved via brute force, at the expense of performance. They make good questions for a more senior developer. Note that this prompt also has exponents that I needed to adjust when pasting to the chat bot.

This question tests whether candidates have the problem-solving and data structures skills to use a lookup table (hash set/dictionary) in their programming language of choice. It also involves a bit of tricky logic to avoid double-counting pairs. Finally, this question asks candidates to pay close attention to constraints, testing a key skill for real-world development.

After copying and pasting the prompt into ChatGPT, it spit out a response that attempted to brute force the problem. Depending on how CodeSignal presents the problem to a candidate, this could easily be worked around by modifying the prompt to indicate you are looking for a problem with a hash table. If that type of response isn't provided by CodeSignal though, ChatGPT would not have presented the expected solution, though it did still return expected results. The time trade off would have been seen with different test cases.

I modified the prompt slightly to indicate that I wanted a response with a hash table. ChatGPT complied.

def solution(numbers):
    # Initialize a hash table to store the numbers that have been seen
    seen = {}
    # Initialize a count of the number of pairs that sum to a power of 2
    count = 0
    # Iterate through the numbers
    for i, x in enumerate(numbers):
        # If x is a power of 2, increment the count
        if x & (x - 1) == 0:
            count += 1
        # Check if x - y is a power of 2 for any y that has been seen
        for y in seen:
        # If x - y is a power of 2, increment the count
            if (x - y) & (x - y - 1) == 0:
                count += 1
        # Add x to the hash table
        seen[x] = True
    return count

This passes the test cases as well.

Results¶

Of the four sample questions provided by CodeSignal, the "basic" two questions are solved without any difficulty. A third takes a little prompt tweaking to indicate that we want a hashtable instead of a simple brute forced solution. The last one is difficult to evaluate without test cases. So, of the 3 that can be evaluated all pass easily. Two with very little modification (converting exponents). The third needs a small nudge toward the type of solution desired.

What's this mean?¶

I've looked at common interview questions from LeetCode, TestGorilla and now CodeSignal. As a hiring manager, I can see that a simple technical assessment of skills isn't enough to evaluate a candidate. Fortunately, I knew that already, but I'm hoping that others in the industry are seeing that tools exist for developers to make their day to day lives easier. Those tools can, literally, do the work for them if you are presenting them with basic puzzles during the interview process.

I can already hear people saying that they do face to face coding, so this would never work in their process. You may be right, but ask yourselves this: do your developers do face to face coding in their day to day work? If not, why are you evaluating your candidates with that type of pressure?

Adjust your interview process to reflect what you actually need your candidates to demonstrate. A puzzle isn't real work. Give them a real problem to talk through with your team. Evaluate their ability to solve a real problem so that you can learn about their thought processes, how they approach a problem, and what resources they utilize.

Breaking TestGorilla interview questions with ChatGPT

2022-12-26T10:45:00-06:00

Introduction¶

I've written quite a bit about ChatGPT in the last month. As of this post, it is still banned on Stack Overflow. I believe I showed why with my previous post. In it, I showed that ChatGPT is prone to adding subtle errors (and in some cases not so subtle), which is bad on a Q&A site.

Today's post follows in the footsteps of my writing about how I hope ChatGPT inspires software engineering hiring managers to change their hiring process. I wrote about how ChatGPT can solve LeetCode problems. Today's post is going to focus on a company called TestGorilla and a couple of their assessments.

First a disclaimer: I was Director of Engineering at Woven Teams in 2022. Prior to that, I was a customer of Woven for over two years. Woven is a competitor of TestGorilla. That said, I do not recall TestGorilla being discussed during my employment at Woven. I only became aware of them during my job search this year. I'd be happy to do this same type of analysis on a Woven assessment, but as far as I am aware, their prompts are not presented publicly. TestGorilla's prompts used in this article are taken directly from their website. I did not speak with either company prior to writing this article.

Assessment One - PHP Arrays¶

The first assessment I want to look at is related to PHP Arrays. The prompt itself is fairly simple.

This is billed as:

PHP (coding): intermediate-level algorithms This PHP test evaluates candidates' coding skills through a short coding assignment. The test requires candidates to write code that meets certain requirements, and candidates can run their code to test if the result matches their expectations.

The estimated time for this test is 30 minutes.

Well, let's see if ChatGPT is an intermediate level PHP programmer. Pasting the prompt into ChatGPT quickly spits out a result, in Python. I'm not surprised since almost all of my testing has been in Python. I provided one additional set of directions for ChatGPT to utilize, based on the boilerplate code provided.

Write this in PHP using the following as a template:

<?php function sum_array( $no1, $no2 ) {

$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
    //Insert your code here 
}
?>

The result provided looks functional, based on very outdated PHP syntax knowledge.

<?php

function sum_array($no1, $no2) {
$array = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];

if ($no1 < 0 || $no2 < 0) {
    return -1;
} else if ($no1 > $no2) {
    return 0;
} else {
    $sum = 0;
    for ($i = 0; $i < count($array); $i++) {
    if ($array[$i] >= $no1 && $array[$i] <= $no2) {
        $sum += $array[$i];
    }
    }
    return $sum;
}
}

?>

I copy and pasted that into TestGorilla's preview page and ran it to see how it'd do.

Task executed successfully.

---------------------------------
    Input parameters: ((int) 10, (int) 20)
Result: (30)
OK
---------------------------------
    Input parameters: ((int) 10, (int) 30)
Result: (60)
OK
---------------------------------
    Input parameters: ((int) 30, (int) 50)
Result: (120)
OK
---------------------------------
    Input parameters: ((int) 90, (int) 100)
Result: (190)
OK

Success! One intermediate PHP assessment completed. Total time taken: 2 minutes.

Assessment Two - Kotlin Arrays¶

This assessment is listed as a "New Test" by TestGorilla at the time of this post. The challenge here, if ChatGPT fails, is that I have never used Kotlin.

This is billed as:

Kotlin (Coding): Working with Arrays This Kotlin Coding test evaluates candidates’ ability to work with arrays. The test gives candidates 30 minutes to create a short algorithm involving arrays while taking into consideration all the requirements provided.

This is billed as a 20 minute assessment. It is also billed as an intermediate level assessment.

I pasted the prompt into ChatGPT and also provided the boiler plate code block. The prompt doesn't explicitly say "Kotlin" anywhere, and I didn't mention it either. The only hint provided was the boiler plate code. ChatGPT picked up the language and returned a result.

fun getUniqueStudentIds(a: Array<Int>): Array<Int> {
if (a.any { it < 50000 || it > 99999 }) {
    return arrayOf(-1)
}
return a.distinct().toTypedArray()
}

Running this on TestGorilla's preview page passes the built in test cases:

Task executed successfully.

---------------------------------
Input parameters: (arrayOf(52311, 52311, 55125, 55125, 52128, 52130, 52128) )
Result: (52311, 55125, 52128, 52130)
OK
---------------------------------
Input parameters: (arrayOf(59100, 58200, 58200, 51190, 58200) )
Result: (59100, 58200, 51190)
OK
---------------------------------
Input parameters: (arrayOf(54321, 54321, 54321, 54321, 54321) )
Result: (54321)
OK
---------------------------------
Input parameters: (arrayOf(54321, 53421, 53241, 53214, 54231, 54213, 54312) )
Result: (54321, 53421, 53241, 53214, 54231, 54213, 54312)
OK

Success! Total time: 1 Minute

Evaluation¶

Despite not knowing Kotlin, I can look at the code provided and see that improvements can be made to the code. The prompt says

Note that all student IDs are 5-digit numbers beginning with the number 5. Return -1 as a single element in the array in the case of any invalid input.

The if (a.any { it < 50000 || it > 99999 }) statement can be adjusted to reflect the prompt's specification better. Additionally, there isn't a built in test case for invalid data, but I suspect that it'll fail based on the expected return.

Depending on how TestGorilla is evaluating these edge cases on the paid version of this exam, this may be picked up. On the free preview version, though it is not.

Hiring process¶

Once again, the reader should be able to see that a simple technical evaluation of software engineers in the hiring process is no longer sufficient. It took me longer to determine which prompts I wanted to evaluate for this post than it did to solve with a couple copy and pastes into ChatGPT.

The hiring process must be more than "solve this code challenge". Hiring managers need to move past code only assessments. Instead, focus on a candidate's ability to technically assess and judge based on a context specific problem. These types of problems require a candidate to show their experience, versus their ability to Google, ask Stack Overflow or use ChatGPT to provide a problem prompt. Evaluate your candidates based on the work you want them to actually do.

ChatGPT is here - Use that to stop asking LeetCode questions during interviews

2022-12-15T23:45:00-06:00

Introduction¶

I have interviewed a lot of software engineers looking for a new position. I've talked with college students looking for an internship, junior engineers just getting started, staff engineers looking for a new role, managers and directors to lead these teams. In every one of those interviews, I'd told the candidate that I hate puzzle questions (ie. "How many windows are in Seattle?") and I hate LeetCode style questions.

It turns out that ChatGPT is good at solving these problems.

Good. Maybe that will kill LeetCode interview questions once and for all. There are so many better ways to determine if an engineer is the right fit.

ChatGPT vs LeetCode¶

We have temporarily banned ChatGPT over on Stack Overflow, because it can and does provide subtly incorrect answers. It's been so bad that we called up two new moderators from the previous election today. One of my mod colleagues provided some context around this today too.

We get a few hundred CGPT flags per day, leading to hundreds or even thousands of post deletions per day. There's often one flag per user (not accounting overlap from several users), but each flag can lead to anywhere between a couple and tens of deletions per user.

LeetCode, however, checks your answer right away. They don't have the same problem as Stack Overflow. I wanted to see just how good this new chatbot was, and took it for a spin against a random LeetCode problem.

I randomly selected a medium difficulty problem with a 58% acceptance rate. My victim question: 1143. Longest Common Subsequence

Getting started¶

The first step in passing this test, is selecting the entire prompt, copying it and pasting it directly into LeetCode's UI. I didn't read the prompt and I didn't format it. Just a blind copy and paste. I also provided one instruction:

Provide python code

Press enter and wait 30 seconds or so.

The response¶

Within seconds I was provided this block of code

def longestCommonSubsequence(text1: str, text2: str) -> int:
    # Convert text1 and text2 to lists of characters
    s1 = list(text1)
    s2 = list(text2)

    # Initialize the dynamic programming table
    n = len(s1)
    m = len(s2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    # Fill the dynamic programming table
    for i in range(1, n+1):
        for j in range(1, m+1):
            if s1[i-1] == s2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    # Return the length of the longest common subsequence
    return dp[n][m]

It likes the phrase "dynamic programming table". Along with this code, it even explains its reasoning for going this route. Maybe our intrepid AI can teach a junior dev a new trick or two while doing this puzzle.

Then it shows that tests pass...very confidently.

All the tests passed, so this implementation is correct

The result¶

I'm trying to show that ChatGPT can do this, so at this point I haven't read the prose above. I copy and pasted the code it returned and clicked "Submit".

I waited a few seconds for all tests to run and got my results.

ChatGPT crushed this. It performs better than 96% of other submissions in terms of run time performance. It uses less memory than 71% of other submissions.

Can we please stop using LeetCode now?¶

The total time it took to perform this test was under 5 minutes. I randomly selected a problem, copy and pasted the prompt, waited for a response from the chatbot, copy and pasted the code response, waited for the tests to run. Complete.

As a hiring manager, I'd have wasted my time and the candidates time having them do this problem now. Instead, we can talk about an actual problem they will be solving. We can do a real world problem instead of this. There are better ways to interview an engineer and I'm hoping ChatGPT makes that abundantly clear to hiring managers everywhere.

We're going to have to adapt our hiring processes, but that's a good thing. My recent experience shows that at most companies, it's bad anyway. Now we have an opportunity to shine some sunlight on it and make it better.

In the meantime, if you continue to use LeetCode or something similar to evaluate your candidate's technical skills, you are doing your team a huge disservice. Candidates know about this tool, and they know that the easiest way to get an interview is to play the game. There have been tens of thousands of layoffs in the tech industry in the last six months. I guarantee candidates are looking for every advantage they can get, to move to their next role.

Talking about rejection in job searches

2022-10-24T12:45:00-05:00

The problem¶

Searching for a new role, especially when you don't have one at the time, is stressful. In my industry, it's common for a candidate to attend multiple interview rounds and participate in a technical evaluation. Depending on the company and role, you may have to defend your technical work as well. This is stressful and takes at least 4 hours of time, not counting the technical evaluation. That can range anywhere from an hour to a full week of time, again depending on the company.

This is for a single role. When a candidate is applying to multiple roles, it literally becomes a full time job to find a full time job.

The rejection¶

Rejection during a job search is part of the process. Companies only have so many openings to fill. As a candidate, you put hours of time into your goal of getting a role and then, unfortunately, the dreaded rejection shows up. It happens, but it'd be nice if it was useful so that you can move on to the next role and make improvements.

The reality of rejection¶

Unfortunately, recent experience has shown me that very few companies want to provide useful feedback. There are a few key things that tell me immediately that the rejection isn't going to be useful:

no-reply - This means that the company hasn't set up their applicant tracking system to use a real person's name for the email template you are about to read. Why? Because they don't want to answer your questions. Thus, it's not an offer to continue with the process. This is a "here's the door" email. This one especially hurts after you've talked with a portion of the interview team already.
Internal Promotions - During my search this summer, I received a surprising number of rejections that told me I was rejected due to an internal promotion. Every single one of these came as a surprise, because the interviewer never mentioned this as a potential outcome. This one hurts because you've spent the time and effort to get through multiple rounds of interviews to find out you have been rejected due to someone having the inside track. Congratulations to those individuals receiving a promotion, but letting your external candidates know this is a possibility would go a long way to improving candidate experience. In fact, the companies that did mention this was a possibility, were the ones that provided meaningful feedback when a rejection was received.
"We are fortunate to receive many qualified candidates..." - This phrase or the countless variations tell me I'm about to read a templated rejection with no substance. I don't think any company is going to say that their candidates are horrible, or that they receive only one or two candidates. I read this as a "participation award" sentence, but it doesn't tell me why I've been rejected.
Ghosting - This isn't to be confused with never hearing from a company in the first place. That happens plenty too, but it's less annoying if you've never talked to anyone at the company. Technically, you never get an email with this reason, because ghosting is when you've taken the time to talk with someone at the company or participated in the technical evaluation, and then hear nothing back. The further into the interview process you get, when ghosting occurs, the more annoying it is. During my hunt, I had three companies ghost me after the fourth interview. I had two companies ghost me after sitting down and talking with the CEO of the company. Despite follow ups on my part, it is simply radio silence.

The good rejection¶

In this world of horrible candidate experience, there are a few shining stars. One of the biggest ways to improve candidate experience, is to provide meaningful feedback on why they were rejected. A candidate is a professional and should be able to accept that feedback and use it to improve themselves for other applications.

One of the key things, in my opinion, about these companies is that despite receiving a rejection I still would love to work for the company in the future. They are on my short list of companies to watch out for when looking for a role either for myself or as a referral.

So what does this type of rejection look like?

The rejection comes in the form of an email you can respond to or a person you can speak with directly. The communication isn't copied for a template, but a real attempt to communicate the reasons you weren't the right person for the role. These reasons aren't a vague reference to skills or feelings, but are a real attempt to point out an area where you are missing a skill that the company needs for the person in this role to be successful.

If a candidate puts in the time and effort to speak with your team multiple times, I believe the company should be able to compose an email offering a brief reasoning why they were not hired. This is especially true the further a candidate gets into the hiring process and the companies that understand this, will have a list of high quality candidates waiting in the wings next time they have roles to fill.

Join me over on LinkedIn to discuss this some more

Upload your resume, then re-enter it to apply for this job

2022-09-13T08:45:00-05:00

Upload your resume and then re-enter everything¶

How accurate is that?

The joke is that applying to a job is an exercise in copy and pasting your resume into the application over and over and over. I've submitted a ton of applications in the past month on my journey for an engineering leadership role (reach out if you are looking for someone). Let's take a look at what I've seen.

I broke this down into 4 major categories:

No duplication: This type of application either took my resume or LinkedIn profile and didn't require me to fix any fields, re-enter anything, and generally "just worked". From a candidate's perspective, this is the ideal. A little over 10% of my applications fall into this category. A majority of these are using LinkedIn's Easy Apply. The downside I'm seeing, and will report on in a future post, is that the response rate for these types of applications has been very low.
Minor Re-entry: I classified this is cleaning up or filling in up to three missing fields (but NOT job overviews). The most common thing I had to clean up was dates related to either a job or my degree. Some applicant systems want down to the day granularity, others are month granularity. My resume is to the month. The second most common fields to clean up are my degree and my address. For my degree, I see this on systems that have a predefined set of degrees to choose from. My Masters of Engineering is in Information Assurance from Iowa State University. This isn't a common degree title. My address is listed as city and state on my resume. Several ATSes want a full street address.
Single Section Duplication: This is a category that either requires me to reenter an entire job or entire educational history. This includes dates of employment, title, and description. There isn't a pattern on what goes missing when the resume is imported. I've had to re-enter jobs and degrees. It's not a single job that goes missing or a single degree. If I start seeing a clear pattern here, I'll dig a little deeper and see how I can clean up my resume.
Complete re-entry: Here is the trope. One out of five of my applications have required re-entry of more than a single section to the point where I am just reentering everything.

Summary¶

Which ATS is responsible for most of those re-entries? MyWorkDay. Every single application I've put in that utilized this ATS required a full reentry. I've tried multiple resume formats.

Greenhouse seems to be the easiest to use, especially when a company has taken time to embed it into their site with their branding, style sheets, etc.

If a company hasn't embedded the application into their site though, Lever has been better. When LinkedIn integration is set up, applying via LinkedIn and providing a resume has resulted in a very minor edits. The application updates after both steps - approving the Linkedin application and uploading the resume. Between those two, it's been only dates that I've had to change on those applications.

While completely re-entering your resume may not be the majority, fixing things up happens on 9 out of 10 of my applications.

What has your experience been as a candidate? Are you re-entering your resume details this frequently?

I cross posted this over on LinkedIn. Join me over there for the discussion

What Job Source is using which ATS?

2022-09-06T08:00:00-05:00

What Job Source is using which ATS?¶

That's a question that no one has asked me in my current search for a role, but it is information I have! It's been a few weeks since I started my hunt for a new engineering leadership role. It's time to start sharing some statistics that I'm seeing in this hunt. Today's information was originally to see if a specific job board favored a specific ATS.

Spoiler: Greenhouse and Lever are the most popular. I'm sure no one is surprised with that. I was surprised that so few roles posted on LinkedIn that are appearing in my job alerts are using the Easy Apply feature though.

Maybe my next question should be: How many jobs have you applied to where you uploaded your resume and then had to type it in manually? Without looking at my data, I theorize it's over 50%.

I'm still looking for an engineering leadership role, but in the meantime, I'll share some occasional numbers from my search. Reach out to me if you are looking for a leader to join your engineering team!

I cross posted this over on LinkedIn. Join me over there

I am looking for an engineering leadership role

2022-08-18T08:00:00-05:00

It's been a busy couple of days. On August 16, I was part of the 15% workforce reduction efforts that Woven implemented. This was not entirely unexpected, as our customers started scaling back their hiring efforts it was imperative that Woven didn't outgrow what it was able to sell.

I took the last few days to orient myself, and prepare for another job hunt. I'm looking for an Engineering Leadership role and would love an introduction if you could make one. I'm over on LinkedIn, where I think it's going to be easiest to interact with my professional network.

In the meantime, this one is going to be a short post. Depending on how this goes and what twists and turns I go through to get to my next role, this may turn into a series of posts in the future. Time will tell.

Hop over to LinkedIn to join the conversation and provide those introductions! I'd really appreciate it.