Stack Overflow still has issues and it's getting worse
Posted on Fri 29 March 2019 in Side Activities
Last time on this blog¶
A little over a year and a half ago, I wrote an article about Stack Overflow's problems from my perspective as an experienced user. This was before I was elected as a moderator on Stack Overflow. I ended the previous article with this:
I continue to invest my time and effort into the community, but even as an active user who really wants the company and community to succeed, it's getting harder and harder to ignore that those of us that have been around for years are not being listened to any more. We're being treated as the grumpy old person that grumbles about the way things used to be. Our experiences on the site are brushed aside as being unhelpful to new users. That completely ignores that fact that we are still trying to reach the goal on which Stack Overflow was created: "With your help, we're working together to build a library of detailed answers to every question about programming." To do this, we need high quality questions and answers so that we can actually provide help to all users. I think this is the biggest challenge that Stack Overflow is going to face in the next 18 months.
So, what's happened in the last 18 months?
Documentation¶
After years of development (being announced in 2015), Documentation was shuttered in August of 2017. Stack Overflow wasn't drawing users to the Documentation feature. Their own metrics and analysis showed that fixing Documentation to be useful to users - both new and experienced - would require a significantly larger team.
What went wrong?¶
In my opinion, and as I mentioned in 2017, Stack Overflow has ignored it's user base. This is going to be a recurring theme in this post. For years, users provided feedback on meta, in dedicated user experience interviews and in chatrooms. This resulted in superficial changes and major rewrites. Yet, complaints still existed. These complaints turned off the experienced users that could produce the high quality documentation. Instead, Documentation became a reputation farming operation in all but name. This turned off even more users.
By Stack Overflow's own admission when sun setting the feature, Documentation was built to solve a problem that wasn't really a problem.
Finally, our research showed that while a lot of developers were dissatisfied, the current state of programming documentation is not universally broken the way Q&A was when Stack Overflow started. In particular, we heard over and over that Stack Overflow has become de facto documentation for many technologies. As many of you pointed out, Stack Overflow is already good enough at providing documentation of obscure features. Even when considering just the company's mission of helping programmers “learn, share their knowledge and build their careers”, Documentation isn't the most efficient use of resources.
Two years of major development, focusing on a problem that the community had not been enthusiastic about, and intentionally ignoring other feature requests and other improvements angered a lot of users.
Teams¶
In my last post, I mentioned that Teams had been launched and shut down in less than a year. Teams is back! At least the name is. Initially launched as "Channels", and later re-branded to "Stack Overflow for Teams", this is a money generating route for Stack Overflow. It uses the old URL.
Now, generating money is good. It's good for both the community and the company. Without money, the company can't survive. Without the company there is not Stack Overflow or community. My problem isn't with money generation. My problem is that, once again, community feature requests for higher quality and moderation tooling to cultivate that higher quality was ignored.
By all accounts, Teams seems to be doing well and bringing in revenue. I am hopeful that this translates into development time to build out the features the community still clamors for.
Meta hatred¶
Meta. It's murder. Until it's not. Meta is how Stack Overflow communicates with the community. It's how the community communicates with itself. It's where governing principals/thoughts/guidance/sticky notes comes from. In short, meta is a large part of how Stack Overflow the company and Stack Overflow the community talks with one another. Decisions are questioned here, announcements are posted here, and little by little the site is made better.
That is, until nothing happens. Stack Overflow's response time has become a meme.
"6 to 8 weeks" is a joke. It's used to indicate that something isn't going to be built or changed. It's so prevalent that this comment crops up over and over on feature request posts. It's used by the community to say that nothing is going to happen.
When something does happen, it's a "big deal". There have been a few examples in the past year. Unfortunately, these changes happened due to feedback from Twitter, not Meta. For years we've been told to post on Meta. For years we've been told that Meta is where the company will engage with us. Then two massive changes happened.
The Welcoming¶
The first change was to make Stack Overflow more "welcoming". This isn't bad. As both an experienced user and as a moderator, I've seen my fair share of users not being welcomed. I've seen hostility to poorly asked questions.
Unfortunately, this whole blog post and resulting meta-drama appears to have cropped up because of a post on Twitter from someone who felt unwelcome. That's fair. I believe they felt that way. However, from my point of view, Stack Overflow ignored their own users (some of whom had been saying the exact same thing for years) because it was suddenly posted on Twitter where the entire world could comment on things that may have been out of context. Instead of listening to their own users and the experiences those users had, Stack Overflow went into damage control mode and rapidly updated it's "Be Nice" policy.
Whether this is actually what happened or not is really beside the point. Many long time users had this perception. Meta was ignored. User feedback was ignored. Instead, the person that could shout the loudest and had made the most noise appeared to be the one that was listened to.
A few months after the welcoming blog was posted and a month after the update, another post was made about how the company was attempting to classify comments. The idea behind this was good, the execution of the blog post was not. In the initial version of the post, exact comments were posted to show "bad comments". I disagreed that a few of them were rude. I'd have removed them as no longer needed without a problem. Honestly, I'd probably have removed them as rude too, because comments don't need to stick around and it's easier to accept the rude flag than it is to decline and manually delete.
My problem was that the exact comment content was posted as a "wall of shame". Then, despite only employees being involved, none of these comments were removed or even flagged for moderators to remove. In short, it really was a "wall of shame".
I believe I covered my disappointment in both this failure and in the technical aspect in my comment on the blog.
I am a huge fan of automatically removing unwanted comments. I did so for several years. That said, I’m disappointed in how this is playing out here. I’m disappointed on both a community level and a technical level.
On the community level, I am very disappointed that 57 Stack Exchange employees were able to evaluate bad comments, determine they were bad enough to put in the hall of shame post here, and then do nothing about them. It took users less than 15 minutes to find those comments on Stack Overflow and identify the “rude” users. Users who are rude because they asked why a certain tag was on a question. Did none of your 57 users have a diamond where you could remove the comment from the site? Even if that’s the case, all of you have the option to flag a comment. Even that wasn't done.
On a technical level, you evaluated less than 4000 comments. That is a few hours worth of comments on a single week day. (source: http://data.stackexchange.com/stackoverflow/query/872382) Is that really representative? How did you determine which comments to use in your evaluation?
The good news is that the comment samples were edited to be "representative" of the problem later.
Welcoming users is great. Helping users is the purpose of the site. I fully support all of that. What I don't support is ignoring the feedback mechanism you've built and told everyone to use for years because someone else with a lot of Twitter followers put Stack Overflow in a bad light. Yes, it should be fixed and should have been fixed sooner, but the perception of "listen to the loudest shout" is not a good look.
Which brings me to...
Removal from Network Questions list¶
In October, the entire "Twitter shouted, Stack Overflow reacted" repeated itself. This time, a user was offended (while on Stack Overflow) over the Hot Network Questions list for two questions on another Stack Exchange site. In under an hour, Stack Overflow (the company) removed that question from the hot network questions list.
The community in question was shocked by the result. A community manager explained the decision on that site's meta
It was the solution we chose - without consulting IPS - because it was effective and easy to implement since it would fix the perceived problem immediately and there was already a technical solution in place for doing it.
Notice a couple things here that stand out to me:
- "perceived problem"
- "without consulting IPS"
The company knee-capped an entire community and a large source of their traffic (the Hot Network Questions list) because of a single Twitter comment. Understandably, the community was upset.
Behind the scenes was even worse. On Twitter, the original user posting their complaint was engaged by community moderators. It didn't go well. Then they complained about that. A Stack Overflow employee jumped into the thread with the following:
If the DM trolls claimed to be moderators on any of the sites then I'd like to follow up with the community team and see about getting removed - they take this very seriously.
Turns out that Stack Overflow doesn't value their community moderators. One employee might be misguided, but this Twitter reply remained active and moderators across the network clamored for an official response. A moderator reached out to the Twitter user in good faith and was threatened with removal by a Stack Overflow employee.
One of Stack Exchange's most respected moderators posted their frustrations on Medium. I highly recommend you read it. One of the community managers posted a response on their own blog too.
The "super-official almost response" was posted even later. This was more than 10 days after the original incident. It took half a month for a first draft of a "moderator social media guidelines" post to be made in the private Stack Moderators Team. That post consisted of bullet points on how a moderator should behave on social media. I replied to that post with this
I am underwhelmed by this response. The event that led to this post and recent discussions around Stack Exchange (and the broader internet) wasn't due to a moderator's bad behavior. Moderators engaged a user on Twitter following the bullets in this post, and yet stuff still exploded in everyone's face. From my point of view, this post is so far down the list of responses that I was hoping to see from Stack Exchange that I'm feeling insulted.
I was asked to hold my judgment until the final draft was posted. That took place in December - two months after the incident. It was changed from "Social media guidelines" to a "community emergency process". These four bullets were provided:
- Introduce yourself and if necessary, your role as moderator of a SO/SE site.
- Offer to help with the situation, and be very respectful if someone declines your assistance. Sometimes, people just want to vent, and the best thing we can do to help is to give them space.
- Be aware of the volatile nature of online discussions; if the path to constructive discourse becomes blurred, it's often best to disengage.
- Keep your interactions with others, concerning SO/SE, as clear and as kind as possible. If things begin to get out of hand, please disengage and let us know about it.
In short, do exactly what the moderator did initially on Twitter which resulted in the threat of being removed.
Communication¶
Stack Overflow is slowly isolating itself from the community. There have been multiple comments scattered around the network saying the employees don't want to engage on any meta. There are community managers that are feeling hated because of complaints users have made. Users are taking out their anger of being ignored on posts talking about new or unrelated features. In turn, the employees engage just a little bit less. Lines are being drawn. I see it as a moderator. I see it as a user. Very slowly the community is trusting the company less and less.
Everything is becoming "us" vs. "them". There is "the company" vs. "the users". Blog posts, comments, meta discussions also appear to be driving a wedge between "the users" and making it "new users" vs. "established users". In the blog post announcing the search for a new Stack Overflow CEO, this comment was made by the current CEO:
One thing I’m very concerned about, as we try to educate the next generation of developers, and, importantly, get more diversity and inclusiveness in that new generation, is what obstacles we’re putting up for people as they try to learn programming. In many ways Stack Overflow’s specific rules for what is permitted and what is not are obstacles, but an even bigger problem is rudeness, snark, or condescension that newcomers often see.
The underlying sentiment - improving inclusiveness and diversity - is great. I'm all for that. The rest of it, though, is a dig at the established community in the same way that the Welcoming blog post was. Stack Overflow's high quality standards are the problem. It makes the community seem rude and abusive. You should stop closing those questions, stop down voting new users, and just be nice. It doesn't say that directly, but that's how existing members are seeing it. Read under hairboat's answer to see some of the simmering feelings of high reputation users.
The idea of trust between users and the company is brought up in the comments. This is just another example, in a long list, where the community and the company are butting heads. Something happens that the community doesn't like - reacting to incidents off site, focusing on features no one asked for, not explaining why these new features need to be done, comments are made by one side that makes the other seem unflattering - and another round of not trusting the company starts again.
The company has had a decade of experience with this community. It's grown, shrunk, and grown again. For most of that time, there has been fairly open communication and trust. I am afraid that trust has eroded over the last few years and can't be recovered.
What can be done?¶
The company wants to focus on areas that can bring in more money. In my previous post I quoted the President and Chief Technology Officer of the company.
I appreciate that there are a lot of issues on Stack Overflow that need to be addressed, and maybe we haven't been responding to them as quickly as we should. But Stack Overflow Q&A is a big, established product, most of the problems left are hard, and we can't let maintenance become the only thing we work on or we'll just slowly run out of money and go out of business. We are trying to both maintain Q&A and solve new problems for developers and reach new audiences. The latter is hard, and maybe we'll fail on a lot of our ideas, but we're not going to stop trying. – David Fullerton May 17 at 21:10
I bemoaned that this sounded that Q&A was feature frozen. It's been nearly two years since that time. I can't remember a new feature that was introduced into Q&A that helped the community maintain high quality posts. There was a new wizard introduced for new users that is supposed to help. A quick look at the review queue numbers on Stack Overflow shows that they are still stable at the same point it was two years ago.
My suggestion as a user, a moderator and someone interested in see Stack Overflow remain successful, is to focus on helping to manage the quality of your content. Users have been asking for years to be able to better handle poor content. They've asked for tools (both system tools and moderator tools). There have been projects started, stopped, restarted, and stopped again that are supposed to improve quality. Community tools have been built to help deal with quality problems. Use some of this!
Stack Overflow has a data science team. Work with the community directly to help figure out ways to prevent low quality content from ever getting posted. Force users - all users - to post higher quality content. Work with the communities that have developed automated tools. Run it with larger data sets. Even if Stack Overflow has to be more conservative than the community tool, if you can prevent some of the low quality content from making it to the site you have a victory.
Obviously the company can't ignore the areas that bring in revenue, but it's becoming increasingly clear that the community is much less forgiving than they used to be. Continued communication blunders will not help with anything.
Where to from here?¶
I ended my last post with this:
I want Stack Overflow to continue to grow. I also want Stack Overflow to have high quality content. I think my experience and the experience of others can help build the features to accomplish this. We just need Stack Overflow to refocus on the Q&A portion of their network again.
I think that holds true today, just as it did 18 months ago. The aspect of the site that draws users in is Q&A. Make it better. Make the content better. Give users tools to make it better. With all of this, I believe, the "welcoming" aspect will improve. Let the system handle the low quality stuff automatically. Eliminate the need for users to ask basic questions or remind users to post their code. Let the system be "the bad guy", and let users interact and help one another.
We'll see how everything looks in 18 months. In the meantime, I'll be here, cleaning up the low quality content and prodding the company to provide improvements to Q&A.