Stack Overflow's Problem - Feedback from an experienced user
Posted on Mon 22 May 2017 in Side Activities
Introduction¶
Stack Overflow launched in 2008. As it nears it's 9th year of operation, I am afraid the resource that I depend on is losing it's way. Stack Overflow launched after I graduated college. I can't imagine how helpful it would have been during that time period, but it's been invaluable in my professional career. I joined the site about a year after it's public launch, in October 2009.
In that time, I've gone from lurker to participant to moderator candidate (several times). I know Stack Overflow and Meta Stack Overflow. I am a moderator on another Stack Exchange site and have a good understanding of how the network operates. I also am one of the most prolific reviewers in the Stack Overflow Low Quality Posts review queue and have built several applications that work with the Stack Exchange API. I am a power user and know the network and the community.
With those credentials out of the way, I want you to understand that I am active on the network. I am in good standing on Stack Overflow and am not a disgruntled user. I am a concerned user. I am getting more and more concerned that Stack Overflow - the company - is losing it's way.
This post isn't another "Stack Overflow sucks" post (Google if you're curious). I'm going to present a few areas that I'm concerned about and hopefully provide either my suggestions for improvement or acknowledge that I don't know the solution but want the team to be aware of in the future. I still believe Stack Overflow is an incredible resource. I'd just like it to fix some of the perceived missteps that have occurred over the past two years.
What's going wrong?¶
In the past two years, Stack Overflow has made several changes that the established community hasn't liked. Some of these changes still are not liked. These changes include the Teams feature, the new top bar, the Stack Overflow (versus existing Stack Exchange) mobile app, and Documentation. There have also been minor missteps that have caused a rift between portions of the community and the company. These areas include multiple political stances, and a number of post quality improvements that haven't been made.
Each of these, separately, is a minor problem that could be worked through and moved on from. The problem I'm seeing is that taken together, all of these are causing a rift between users, power users and the community.
Let's work through each of these items.
Teams¶
Teams was announced in October 2015 and clarified a week later. It was then shut down after nine months. The page it used to go to now has the following blurb (emphasis is mine):
Teams was in private beta for almost a year with 295 teams created and while we believe in its potential value, after a lot of consideration we’ve decided to un-ship the idea for the time being. We’ve realized that making a successful version of the Team page, as we originally proposed would ultimately take more time and resources than we want to devote to it. Our resources are currently allocated on projects to enhance and improve quality on Q&A, Documentation, and Jobs on Stack Overflow, as a result we don’t have the dedicated developers to get Teams to its fullest potential. The intention was to add more features to Teams, but we never expanded it to anything beyond a team description.
The emphasized section sounds good, except that the one section that is taking up a majority of time (Documentation) has it's own major issues. The area that many power users want developers to focus on is Q&A.
The problem with Teams, and many of the projects mentioned in this post, is that this was a feature that removed focus on areas the community wanted improved. Meta Stack Overflow has been asking for improvements to reduce the number of low quality posts for years. Moderators have been asking for better tooling. The review queues are overflowing with tasks and the number of users performing reviews isn't high enough to keep up. Teams was built without a true end goal and users weren't entirely sure what to do with it. This was the first in a series of mis-steps that continue to plague community interactions when new features are announced.
Top Bar¶
The new top bar was announced in November 2016. It went through a handful of iterations before being released in mid-February 2017. During the iterations users provided feedback. When initially released, though, much of this feedback felt ignored. Things like notification overload, stickiness of the top bar, and hidden review counts were all mentioned during the three months of testing but not implemented until the change was live to millions of users.
After three months of usage, a larger problem was noticed. One of the review queues was constantly full. One of the changes that was made with this top bar was that the "Review" button no longer linked directly to the "Suggested Edits" review queue. Now it went to the page showing all review queues. Users that used to click once to get to a review queue were now presented with a list of queues to work in. Some of these queues are much more time consuming that others. It turns out the number of reviews being done has decreased significantly since the top bar was implemented.
The spike in reviews in February 2017 is when the new top by was released. Since that release, the number of reviewers has plummeted. This has been attributed to notification fatigue and not linking users directly to the Suggested Edits queue.
Three months after implementation, it took the community asking for results (disclaimer: I asked the question), to find out how the top bar has been performing. It turns out that the top bar is performing decently well compared to what the developers were expecting, with the exception of fewer review tasks being performed.
The problem with this project, is that it's felt unneeded and has materially impacted one of the quality control features of the site. There is still a vocal group of users that don't like it because it doesn't match the rest of the network. Several are concerned about the review queue problem. Experienced users felt that they were ignored during the beta tests. Users provided feedback and examples of problems and it was only after implementation when millions of other users experienced the same thing that these changes were made.
Mobile App¶
A recent announcement (as in last week, at the time of this post) announced a new Stack Overflow mobile application. The community response was not positive. Users asked why a new application was being built when one already existed (the response was "branding"). Users asked why the new app was less functional than the existing one (it's limited to Stack Overflow versus the entire Stack Exchange network). Users asked why it took a year to develop and why the existing application hasn't received bug fixes in that year.
I think one of the most disappointing things about this is a response I received in the comments from the VP of Engineering:
@Andy You're right, it wasn't worth a year. There's a long, sad story here, but it was originally expected to only take a few months and... well, here we are a year later. We decided to go ahead and launch and see what we can learn, and we'll reassess from here. – David Fullerton? May 17 at 16:26
Another user expressed the dissatisfaction in a very pointed way. They provided a list of features that the community has asked for over the years that many feel have been ignored. The VP's response to this wasn't encouraging either:
I appreciate that there are a lot of issues on Stack Overflow that need to be addressed, and maybe we haven't been responding to them as quickly as we should. But Stack Overflow Q&A is a big, established product, most of the problems left are hard, and we can't let maintenance become the only thing we work on or we'll just slowly run out of money and go out of business. We are trying to both maintain Q&A and solve new problems for developers and reach new audiences. The latter is hard, and maybe we'll fail on a lot of our ideas, but we're not going to stop trying. – David Fullerton? May 17 at 21:10
This sounds like work on the Q&A side is feature frozen at this point. They are done innovating in this area and instead are focused on drawing in users via other features - like Jobs or Documentation. Multiple times in the comments the new app was promoted as being able to use the Dev Story or Jobs features in the future. Perhaps it's just me, but I don't apply for jobs via my phone. That doesn't seem like a good way to really put the effort needed into a cover letter or application.
Documentation¶
Now we've reached Documentation. This is the project that's sucked up development time over the past two years. This is the project that Stack Overflow developers are defending tooth and nail and the community has all but given up on.
Documentation was announced back in August 2015. It's had a ton of updates since then. It was met with initial enthusiasm but that quickly turned around. When the system launched for all users, one of the first complaints was that the reputation generated via documentation was doing back things to the main Q&A site. This resulted in a massive recalculation of reputation and resulted in many users losing a lot of their internet points.
Another change that was announced with the introduction of a new review queue for documentation. Initially, developers didn't expect the low quality to begin immediately, it seems. Long time users weren't surprised. Now we've reached the point where the company is realizing that the users knew what they were talking about. Documentation is undergoing a massive change, to the point that much of it is being completely redone - not fixed - scraped and redone.
This project has years worth of feedback from the community that has been ignored. It is the black sheep of Stack Overflow and many community users feel that quality of the content is lacking so badly that they don't participate any longer. This feeling isn't helped that many users have been explaining why things aren't working for a while and it's only after two years the developers are starting to realize the private beta testers, public beta testers and experienced community users mentioned many of these problems. In this particular instance, the company took Jeff Atwood's advice (co-founder or Stack Overflow) to not let the community tell you what to do to heart. To the company's surprise, a community of developers that live in programming documentation had decent thoughts on what does and does not work in programming documentation.
Politics¶
For many users, the lack of true social features on Stack Overflow and across the Stack Exchange network has been a good thing. You can't easily follow a single user, you can't send private messages to a user, and you can't really do anything on the site that isn't public to everyone. The focus is on content, not opinions or social interactions.
This breaks down once and a while though when a big political thing occurs. The two most frequently mentioned instances are the response to the Obergefell v. Hodges Supreme Court decision and the response to President Trump's initial immigration executive order.
Both of these caused huge uproars within the community when the company took a stand. These stands caused problems due to users holding opposite political views, users not wanting politics on their programming site, users not wanting to deal with the drama caused by the vocal members of the other groups. This led to an apology. The community wasn't pleased with this apology. Users mentioned in multiple answers to this apology that they don't want the company to post such political agendas on the site. It's out of place for a programmer community. Both of these instances are still brought up on Meta when the community feels that the company is imposing on them.
I don't really have advice or suggestions on this problem other than "I don't want to see this on Stack Overflow, because these hot button issues cause so much drama that nothing gets accomplished". These posts grind Meta and chatrooms to a halt while everyone expresses their opinion on the post, on the post's existance, on one another and on related issues.
Quality Improvements¶
Finally, the community has been asking for years about ways to improve the quality of posts on the site. Stack Exchange started a project to improve the quality back in October 2016. This generated 80 different suggestions on how the community sees "quality improvement" taking place. Since then there haven't been any updates on the status of this project or even subprojects.
This was brought up during all of the projects listed above by long time users. The hope was that this quality project would help. Being ignored hasn't brought any good feelings. The lower quality has been measurable and seen less participation from experienced users.
The Fix¶
Above I've pointed out several issues that I've seen over the past two years. These issues are part of a bigger problem though. It seems that Stack Overflow doesn't know to how handle it's community size any longer. It's in the top 300 sites visited in the US and receives half a billion views globally per month. Couple this with the fact that they don't have a sustainable business model yet and have a sizable team with good benefits and they are getting concerned.
Q&A is what built Stack Overflow, but it isn't enough to sustain them. Thus, the other projects are being created. Unfortunately, in this process, it seems the company is forgetting it's existing user base at the expense of expanding to new users. Existing users are getting frustrated with the lack of quality improvements, being ignored and not having changes that benefit their use cases.
Documentation has taken up a giant chunk of time and developer effort and it's all been wasted. The announcement that it is being redone has been met with "thanks" from the community, along with warnings to consider that "quality" problem. We'll see how it plays out, or if that quality issue is ignored like their own Quality Project.
Which brings us to the final point I want to make. I think the feeling of Q&A being "done" is the biggest problem I've had with Stack Overflow over the last year. New features aren't being built in that space. Instead of focusing on some of the "hard" problems, the company is throwing stuff at the wall and hoping something will stick. Unfortunately, the four biggest projects in the last year have either failed completely (Teams, Documentation, Mobile App...perhaps) or have significant unintended consequences that isn't helping the quality issue users have been reporting for years.
Power users, the underlying community that has put time and effort into growing Stack Overflow to that it is today, is feeling ignored. It is only after months or years long experiments fail that community opinions are finally validated or considered. Users have expressed concerns in each of the above projects repeatedly. Yet, those opinions were not addressed. The silos that the developers have built around themselves are causing the company to lose touch with it's community. This is being done at the expense of alienating the users that care and the cost of developer time.
Users want a high quality site with answers to their questions. Even new or potentially new users want this. Stack Overflow continues to avoid dealing with that problem because "it's hard". The unfortunate thing is, this is costing the site users that return to provide more than one answer.
This chart is showing the number of answers provided per month by different types of users. Users that have provided more than 100 answers, between 11 and 100 answers, between 2 and 10 answers and only a single answer. The furthest data point on the right is an artifact of being an incomplete month. From this chart, we can see that the only group that has continued to rise are users that provide a single answer over time. The other groups took a steep drop in April 2014 and haven't recovered since then. The number of experienced users that are participating has dropped.
What happened in April 2014? That's been answered by a Stack Overflow community manager. The theory is that users aren't getting answers to their questions and due to being ignored they never return to participate further in the site. Another community manager also provided an answer:
Starting around 2013 and peaking around March, 2014, people began asking fewer interesting questions. That lead to a decrease in voting on questions and fewer answers being given. Since the feedback on these uninteresting questions was discouraging, people began asking fewer questions on the whole. Meanwhile, truly poor questions continued being asked with little regard to negative feedback.
Stack Overflow users began noticing increasing numbers of truly awful questions and decided, rightly, that downvoting and refusing to answer them is the best remedy. These questions fit broad categories of awful and users began withholding votes from questions that were not themselves awful, but bore some of the markers of awful. Fewer of these questions got answered and askers of mediocre questions did not see any point in trying to improve.
Thus began a slow spiral downward. Not all is lost though, because there are the upticks. I hope it's enough to break the cycle, but I really fear that something needs to be done about this quality issue. This is the issue that is brought up by the experienced community.
Where to from here?¶
I continue to invest my time and effort into the community, but even as an active user who really wants the company and community to succeed, it's getting harder and harder to ignore that those of us that have been around for years are not being listened to any more. We're being treated as the grumpy old person that grumbles about the way things used to be. Our experiences on the site are brushed aside as being unhelpful to new users. That completely ignores that fact that we are still trying to reach the goal on which Stack Overflow was created: "With your help, we're working together to build a library of detailed answers to every question about programming." To do this, we need high quality questions and answers so that we can actually provide help to all users. I think this is the biggest challenge that Stack Overflow is going to face in the next 18 months.
I want Stack Overflow to continue to grow. I also want Stack Overflow to have high quality content. I think my experience and the experience of others can help build the features to accomplish this. We just need Stack Overflow to refocus on the Q&A portion of their network again.