Andrew Wegner | Ponderings of an Andy - Stack Exchange Strike

Stack Exchange Strike - The strike is over

2023-08-07T12:00:00-05:00

Introduction¶

The Stack Exchange strike is over. It took two months and two days of inaction on behalf of community members and moderators to resolve the strike. The results of the negotiations were posted last week. Over the weekend the community voted that the goals of the strike have been achieved and called the end of the strike.

So, what was achieved? What did more than two months of community upheaval get the users of Stack Exchange?

AI Generated Posts¶

Effective immediately, Stack Exchange has agreed to allow the removal of content based on a combination of strong and weak indicators of GPT usage. Additionally, the original policy that was the straw that started all of this has been both publicly released and declared invalid.

This release took two months and was the cause of the strike beginning. It drove a wedge between the moderators of Stack Exchange and the company and its employees. I do not understand why it took so long to release this. The public answer is that there was initial moderator pushback on releasing it but that was rescinded within days. Like much of the public communications during this, Stack Exchange clung to outdated information.

Something that I think is amazing is that this private policy was never leaked in full during the strike. Despite what the company was saying about moderators and the community, the community abided by its belief that agreements - private moderator space - should remain just that: private. I applaud the moderation community for this commitment to their ideals.

Data Dumps¶

The Data Dumps were turned off back in March. They were re-enabled several weeks into the strike as one of the first concessions. However, it's important to note that a former employee has presented that this was not an unwavering commitment. They had been contacted by the Stack Exchange CEO in March to disable the scheduled data dump.

The company officially states that it's committed to the long-term (foreseeable future) survival of the data dumps, the API, and SEDE [Stack Exchange Data Explorer].

This is good. It is concerned that this action was done without informing the community and only discovered after the fact though. This type of "ask for forgiveness" behavior from Stack Exchange is common and is a concern for me.

Moderation Agreement Changes¶

Stack Exchange agreed to a review period for binding policy changes and policies must be made public. This is another big one that caused the strike to move forward. I'm happy to see this has been resolved in a way that benefits the community and transparency. Time will tell if this holds true, as it's only something that we can say is effective until it is no longer effective.

Assuming that the agreement holds though, a review period will be good for the moderators that are expected to enforce changes. It will give them time to get clarifications and be prepared for the discussions on meta.

The company also agreed to update their press policy. New statements to the press must get at least one member of the community management team to sign off on the statement and statements must be as general as possible. This works in tandem with the existing policy put in place in 2019, where statements won't discuss an individual moderator without written permission.

This is another one that we can't judge effectiveness of until it's been broken. Unfortunately, the press policy from 2019 wasn't enough to prevent some statements to the press this time that presented moderators in very unflattering light. Even though none were identified by name, the broad statements were taken out of context in a couple instances.

That said, there was an apology for that from the Vice President of Community:

I would also like to take this opportunity to extend my most sincere personal apologies to mods who felt that in our previous text we were accusing them of racism. While that was not the intent of the text that I wrote (nor did that sentiment reflect the feelings of anyone involved in drafting the text), I can understand how it could be read that way, and I regret that we allowed it to be published like that. You have my sincere apologies, which I will also deliver in person at the upcoming mod/staff meetup.

Violations of the Moderator Agreement¶

The current moderator agreement does not lay out a process for determining if the company violates the agreement. The strike representatives and the company agreed to an outline for such a process. If Stack Exchange is found to have violated the agreement, actions taken and comments made during the violation must be retracted and nullified and a public apology must be made detailing the violation.

This is one that I hope we don't have to see utilized. Again, time will tell if it is something that will occur.

There is also a process where the moderation team can vote on if a violation occurred. Exact numbers are still being determined, but essentially a minimum number of moderators must vote on if a violation was committed and from that a minimum amount must vote that a violation occurred.

While I'm unhappy with the percentages used as placeholders, the discussions here continue. I am happy with the process and the proposed actions if a violation is determined.

Stack Exchange Processes¶

Several internal changes were negotiated as well. Each of these were around how the company communicates. These include being transparent that the strike occurred, collaboration with the community instead of fighting it, public policies, and clear communication.

All of these are positives changes and can only be evaluated over time. There are already signs that some are taking place with public releases of policies, acknowledgement of the agreement itself and discussions around the final details.

What do I think?¶

I've been pretty pessimistic of the last two months. Stack Exchange appeared to be following reddit's footsteps in some cases. I was afraid that the company would start replacing moderators in a few months - either when the automated systems started flagging inactivity or when the products announced at the developer conference last month started appearing on the site.

I'm still pessimistic about the future of the platform, if I'm being honest with myself. I do not think generative AI will be a benefit to the community, as it exists right now. By the time it's to a point where generative AI doesn't make things up, it'll be too late. The bad data and information will have already been on the site and trust will be gone.

The policies that can't be measured until a violation occurs also has me concerned. We went through this cycle back in 2019 with a strike being very narrowly averted then. Unfortunately, the same things that occurred then triggered the issues now: a private policy and talking to the press. Both were supposed to be resolved then. Both are supposed to be resolved now. Unfortunately, we won't know if that's the case until the policies cause another problem.

So...am I sticking around as a moderator? There is a virtual moderator meet up later this month with the CEO of Stack Exchange. I think that will be when I make my final determination. However, two months of not moderating was a welcome break.

Stack Exchange Strike - Community and AI Talk Reaction

2023-07-27T12:00:00-05:00

Introduction¶

I really was trying not to post anything about the strike again until it was resolved. There was a long period of no action in July so my assumption was that it was being worked on behind the scenes with the company and the designated representatives. Turns out, that it was just quiet for a couple weeks with no movement. One important development was the release (finally) of the private policy on GPT Generators. This is significantly different from the one that was released publicly and was part of the cause of the current moderator strike. The community's response to this policy was as expected - suprise at how bad the policy was even with moderators saying it was bad.

Today, Prashanth Chandrasekar - CEO of Stack Overflow - presented the next vision for Stack Exchange. The presentation is available on YouTube. It starts around the 9 minute mark.

Reaction¶

Welcoming¶

You will continue to be the focus. 100%. We are here to serve you. To fight on your behalf. To make sure you are recognized for the work you are doing and that you are able to responsibly harness the power of AI for your needs to be the best developers you can be. All of this is centered around the collective community and knowledge. That is irreplaceable.

This is a good start. It rings hollow to me right now though. These last two months in particular have shown that the community is easily tossed aside. 2019, and its literal legal settlement with a community member for tossing them aside, shows that current time isn't a one time thing. We'll see what the rest of this talk covers, but right now this feels like pandering to the audience.

Guiding Principles¶

Find new ways to give technologists more time to create amazing things.

Good. I'm all about efficiency for myself and my teams. If we can make developers' and engineers' lives easier, that's a win for me.

Accuracy is fundamental. That comes from attributed, peer-reviewed sources that provide transparency.

Spoiler - I received a sneak peak of a portion of this presentation two days before the official one in Berlin. The information that will be shown later in the presentation will cover some of this. It doesn't address all of it though. I'm pleased with the attribution aspect of this. Stack Overflow content is licensed under CC-by-SA. Attribution is required and I approve of any effort to improve that.

I'm concerned about accuracy. Their own test of the GenAI powered formatting assistant last month showed this is a problem. Current GenAI - Chat GPT, Bard, etc - are well known for making information up instead of saying "I don't know".

The technology field should be accessible to all, including beginners to advanced users.

Weird ending phrase, but I agree. Technology fields are not limited to engineers, software developers and people with the desire to learn how to program. Technological improvements should come from any one who wants to make life easier. AI can and will help with that.

Humans should always be included in the application of any new technology.

Again, agreed. My concern here is whether or not Stack Exchange has the capability to do that. Technically, they've had a human involved, but it's only been from a single perspective - that of the company. They have ignored the other side of this: the community. The people that provided the data for their business to thrive.

Overflow AI¶

insert applause

This has been in development for "the past three months". It also covers 6 items being talked about today and 6 additional items. That time frame is concerning. Three months is not a lot of time to build a lot of these larger items.

Search: Summarized version of multiple answers with citations from where the answers came from. It also allows you to continue the conversation with the results - including code. It also allows the user to post the question to Stack Overflow if the generative AI portion gets stuck.

My concerns lie with how this impacts the community. Hand waving away that the system is able to synthesize an answer and properly attribute it, what happens next? Stack Overflow has a reputation - deserved or not is up to the reader - for being harsh on new users that don't "try". If a new user comes and gets an answer without posting the question that's wonderful for the user. They move on and complete what they need to complete. For the community, though, we have a problem.

I find it unlikely that a new user is going to post a question to Stack Overflow when they received an answer. Why would they? They already got the answer they need. If this repeats constantly, the knowledge base that Stack Overflow represents becomes significantly less useful. Fewer questions mean fewer answers. With fewer answers, the platform becomes more dependent on the trained AI model which is receiving less data to feed it. The system spirals.

Does this only matter to prevent duplicates though? If the system is only summarizing other topics, wouldn't that mean the question being asked is a duplicate? I don't think that's the case. I think with the "continue discussion" portion of this, Stack Overflow will be losing out on better written and described problems. Problems with more detail that the community could benefit from seeing and answering.

The Visual Studio plugin looks interesting. I don't think it's adding much that other plugins can't do already, other than providing a first party integration to Stack Overflow. The same is true for the Slack / Stack Overflow integration.

The Enterprise knowledge injestion is an interesting product. While we don't use Stack Overflow for teams at my current company, this would be a great way to start using it. The ability to load initial data into a new system is important. I'd love to see this in practice and how well it does at building initial Q&A pairs from the data it ingests.

I haven't looked into Stack Overflow for Teams for my professional work. A couple comments that were made do raise a few concerns about how Stack Exchange is using company data. Before making a recommendation to utilize it for my company, I'd need to investigate that data from my company is not made available outside of my organization.

Finally, discussions. Stack Exchange is adding a forum to their collectives product. The community has spent over a decade to not be a forum. This change is not great in my opinion. Forums, social media, are a different beast than Stack Overflow.

Final Thoughts¶

I am concerned about the direction Stack Exchange is taking and the employee resources they are using to go that route. Engagement, page views, and traffic has been a huge underlying concern at Stack Exchange for a while. It was particularly noticed when ChatGPT was released and with the comments about driving users away. However, this downward trend has been ongoing for years.

I don't think GenAI is going to solve this problem. I think it will improve new user experience, but the way this has been presented it will be to the detriment of the larger community. Fewer questions will exist to answer and that will slowly cause users to disengage from the site.

This is why the opening of the talk feels like lip service to the community to me. These changes are designed to put a barrier between users of the community. Even the discussions product has the barrier. Discussions will only be available in Collectives, not to the general community.

Stack Exchange Strike - Now AI is bad? Does Stack Exchange know what it is doing?

2023-07-04T23:30:00-05:00

Introduction¶

My previous posts about the ongoing moderator and curator strike on the Stack Exchange network can be found linked at the bottom of this post, or by visiting the Stack Exchange Strike category on this site. I'd post a summary about what's happened in the last ten days, but there is nothing to report. There are discussions, but no agreements. The appointed Stack Exchange employee empowered to talk with moderators stepped back and is not participating any longer.

Tomorrow marks the one month point. We are hours away from 10,000 pending moderator flags on Stack Overflow. This is up from 78 (yes, two digits, in mid-May). The way this has gone down, the lack of progress, and the continued mischaracterization of moderators to the press hasn't motivated me to spend my free time to volunteer in the last long though. I still have this feeling that Stack Exchange is looking at the reddit protests recently with their demand that moderators return to the community and wondering if they can replicate that here.

New confusion¶

On July 3, 2023 Stack Overflow published a blog post entitled: "Do large language models know what they are talking about?". Spoiler: the conclusion of the article is "Nope."

But that's not the interesting thing. The interesting thing is how this answer is presented. The very last paragraph of the post cuts to the heart of the matter that moderators on Stack Overflow raised in December when we banned ChatGPT.

Treating AI-generated information as purely actionable might be the biggest danger of LLMs, especially as more and more web content gets generated by GPT and others: we’ll be awash in information that no one understands. The original knowledge will have been vacuumed up by deep learning models, processed into vectors, and spat out as statistically accurate answers. We’re already in a golden age of misinformation as anyone can use their sites to publish anything that they please, true or otherwise, and none of it gets vetted. Imagine when the material doesn’t even have to pass through a human editor.

We saw this in action with ChatGPT. We still see it in action with ChatGPT and it's still a problem users are becoming more aware of as the strike continues. We saw it when Stack Exchange tried their formatting assistant on Stack Overflow. What I see here is Stack Overflow admitting that the moderators are correct, in public.

The other interesting thing about that paragraph is that it links to an article from The Verge that quotes the Stack Overflow moderators and the decision to ban AI. It also has this dig at Stack Exchange executives:

The mods say AI output can’t be trusted, but execs say it’s worth the risk.

Their own post is explaining why it's not worth the risk.

What's this mean?¶

I see this as more communication failure on Stack Exchange's part. In an update I posted weeks ago, I linked to internal emails that were leaked.

How are we messaging this? Who is allowed to post and respond to questions and comments on Meta, chat, social media, etc?

The Community Leadership Team ([redacted]) are working together in close coordination with Marketing ([redacted]) on comms. They will post and respond to questions on-site. Unless you are specifically tapped to respond to something please do not engage. It is best to avoid commenting on anything related to this action on site, even if you think you have something helpful to add. Please get review and approval from Philippe prior to posting on site, or from [redacted] if you are approached off-site.

Someone, somewhere, didn't realize what this blog post was about or what it linked to.

But, nothing changes with this. The company has dug in so hard on forcing GenAI to be on the sites and is marching toward an announcement of some kind in late July 2023 about AI. In the meantime, I can only see blog posts like this one as an indication that Stack Exchange doesn't know what they are attempting to build toward and at the same time have come to the conclusion (or at least a team within Stack Exchange has) that GenAI isn't to be trusted.

Just like the community said back in December and continues to say now.

Stack Exchange Strike - How does the company regain my trust?

2023-06-23T23:00:00-05:00

Introduction¶

This post was originally posted on Meta Stack Exchange, the network wide "Meta" for the Stack Exchange network. This meta site and the "child meta" sites mentioned in the post are utilized by community members to discuss the network itself. This is where questions about how the individual site or the network as a whole are posted, where policies are determined, where questions about questions are discussed. This is the location that the community has to make their voice heard.

This post is my answer to the question "What is needed for users to trust the Stack Exchange company?" It's been edited slightly to fit the format of this blog.

How will my trust be regained?¶

Short version¶

TL;DR: I'm not sure and that's a bad thing for me and for the community.

Before I begin, I'm not going to segment the company into various groups. I've gotten the impression from moderator representatives that this is a bad thing and they are offended by this segmentation. I have no desire to further that, so "Stack Exchange" in this case refers to both the company as a whole and all employees.

Author note: This complaint was relayed by the moderator representatives from Stack Exchange during discussions. It seems that using phrases like "management" and "leadership" is being interpreted as "good staff" vs "bad staff" at Stack Exchange. While I disagree with this, to me it's not worth the argument thus it's just "Stack Exchange" for me from now on.

Who am I?¶

For context, I've been on the network for nearly 14 years. I'm a moderator on Stack Overflow, Hardware Recommendations and Community Building. I have built automated tooling to flag comments (at one point accounting for 15% of the comment flags raised on Stack Overflow in a year), and I am an admin on the community led Smoke Detector (spam detection) project. In short, I know this network, the tooling it does and does not have, and various communities across the network. My time here is voluntary. Time that I, until recently, was happy to provide without much of a thought. I've had very interesting discussions with fellow moderators and Stack Exchange employees throughout my time here.

Why don't I trust Stack Exchange?¶

What is needed for users to trust the Stack Exchange company

Stack Exchange has gone through this cycle before. I've written about it in those past cycles for anyone who wishes to go through my profile and find previous thoughts. Each time, less of my energy comes back as we - community and company - reconcile and bury the problem in the sand.

The last major cycle ended with a lot of lawyer language, including the new moderator agreement that every mod had to accept to retain their diamond. This cycle started with a violation of one of the provisions of that agreement by Stack Exchange. It received a, in my opinion, flippant "Oops, that was my fault" by the Vice President of Community at Stack Exchange.

This tells me that the legal agreement is completely one sided and Stack Exchange feels comfortable violating it without repercussion. If I, on the other hand, had violated a term in the agreement I'd be forced to hand in my diamond. This has eroded a ton of trust I have with the company.

In the announcement regarding how Generative AI can and should be moderated and in statements to the press, there has been disparagement against the moderators of the network. To me, the subtext of all of that reads as "we don't trust you to moderate correctly". If the company does not trust us to perform activities we've either been elected or appointed to do for our community, why are we still here?

Combining this with the incredible way this cycle all started and the fact that none of this mistrust was known by the moderation team, my trust of the company took a hit. This policy was announced at the end of May. Data was shared several weeks ago. In all of that, there are allusions to improper moderator activity and hints that moderators are banning so many people that engagement across the platform is down. It wasn't until yesterday (nearly a month) that moderators saw any discussion of these "improper bans". It was just...silent. This big, massive problem that could have been talked about back in February or March was just tossed into the public eye with the implication that moderators are doing the wrong thing. Then it took nearly a month for a conversation to begin.

The Stack Exchange network has lost at least four months of time where this "moderation problem" could have been discussed, policies adjusted, and moderators who deal with generative AI on their sites on the daily basis educating the company on how it's actually being detected. Instead, an easily disproved lie about using ChatGPT detectors has been blamed and shared repeatedly with the press for the reason for their sudden policy change.

My trust level of the company takes several hits here too. I dislike being lied to and I really dislike being lied about.

Finally, the method of communication through out the last month. The company has a team dedicated to managing the community. There have been many questions on this site and on child metas during the moderator strike. I have seen very little coming from the community management team to answer these questions. The community has questions and the company is not providing answers to them. Instead, we see announcements on topics that the community is against being announced. Long discussions, in public, are not occurring though. Which erodes my trust even further.

What's this all mean?¶

Where am I today then? How does the company rebuild my trust in them? My answer to that is that I don't know. This past month has eroded so much of my faith in the company to be the trusted repository of knowledge that it was in the past. It's also removed much faith that the company actually cares about the community. Much like the previous cycle we've seen details come out that reflect poorly on the company and employees attempt to respond to that only for more details to come out that make the response look like lies.

14 years is a lot of time to spend some place and not have strong feelings about. It makes simply accepting negative changes impossible and it makes walking away difficult. That's part of why I'm still here. The other part is the communities I mentioned in my introduction. I have built friends and acquaintances across the network and the sense of community that used to exist is a strong desire to remain. But, this isn't something that will hold the community as a whole together. I am an outlier in terms of a user on the network. Honestly, everyone reading this on Meta is an outlier.

The company's goal is engagement and traffic. I guarantee the moderation team has not banned enough users to bring down the traffic the network has seen since December. But, we are the scapegoat at least right now. We're nearing a month and there are users with access to site analytics. The number of bans has been close to 0. Theoretically, traffic should be recovering if we were the problem.

Stack Exchange Strike - Personal Frustrations

2023-06-21T11:00:00-05:00

Recap¶

The Moderation and curator strike started on June 5.
Stack Exchange has downplayed the effect of this to the press, while at the same time straight lied to the press about causes of the strike.
Stack Exchange removed access to the data dump back in March but never told anyone until they were called out on it in early June.
The moderation team has elected three representatives to engage with Stack Exchange to solve these problems and end the strike.
Stack Exchange launches the GenAI powered formatting assistant. The community quickly shows that it's very bad at its job and it is shut down temporarily.

What's new?¶

Yesterday Stack Exchange announced the upcoming launch of a new Prompt Design site. To say the community disliked the idea would be an understatement. The community has pointed out that this will be incredibly niche, with very short term answers because models update constantly. There are also concerns that this will essentially become a "write my prompt" (instead of "write my code") site.

I agree with all of those concerns.

Also hidden in this announcement is another announcement. Stack Exchange is changing their method of launching new sites. This is something that should have it's own announcement. This is a big deal and the community has been asking for improvements to the Area 51 incubator site for a decade. Unfortunately, the new method is to completely do away with all of the work that Area 51 does - building a community, setting initial questions, helping to set scope, ensuring there is an audience for the topic - and instead launching with a "Community Stakeholders" group that will do the work in either a private Stack Overflow for Teams instance or a read only chatroom.

Both of those options entirely exclude people that may want to participate but don't have access. It adds barriers that don't exist on Area 51.

My thoughts¶

The announcement sets a launch date of July 26 for this new site. This is the day before the CEO talks at WeAreDevelopers World Congress. A venue where the company has been promising a major AI announcement for months. This new site, combined with the formatting assistant failure from last week, is starting to show clearly what the company wants to do here.

I mentioned in my last post that I was becoming more pessimistic that this strike doesn't end with resignations. That continues to hold true. I also mentioned that Stack Exchanges seems to be keeping an eye on the Reddit strike - first with the blackouts, then with the John Oliver protests and currently with the NSFW toggles to prevent ads. Last night, Reddit started removing moderators as a result.

With the lack of updates and communication from Stack Exchange to the moderators and curators in the last week, I can't help but think that something similar is being discussed at Stack Exchange. Time will tell, but my feeling is that Stack Exchange is going to plow ahead with GenAI content on their platform. They are going to burn 15 years of trust and quality content and they are going to do it regardless of what the community wants. If the community protests, they will be shown the door.

Obviously, I'm still pretty pessimistic about all of this.

Stack Exchange Strike - Does Stack Exchange Care?

2023-06-19T14:00:00-05:00

Recap¶

The Moderation and curator strike started on June 5.
Stack Exchange has downplayed the effect of this to the press, while at the same time straight lied to the press about causes of the strike.
Stack Exchange removed access to the data dump back in March but never told anyone until they were called out on it in early June.
The moderation team has elected three representatives to engage with Stack Exchange to solve these problems and end the strike.

Where are we now?¶

Last week, I ended with a note that the data dumps should be restored by June 16. Good news. That has been completed and the dump was uploaded by June 16. It's progress, but after two weeks we have only accomplished one out of four tasks on the list of conditions to end the strike.

A retraction of the prohibition of moderating GPT content.
The private policy on GPT content that was issued to moderators must be revealed publicly.
The data dumps must be re-enabled and SEDE and API access guaranteed.
Stack Exchange, Inc. must communicate, gather feedback, and act on that feedback before making major policy or software changes to the public platform.

I've heard rumors that the second bullet may occur, but nothing has been done publicly. Thus, nothing outside of feedback from representatives to go on here.

GPT Content¶

On Thursday, June 15, Stack Exchange enabled their "formatting assistant". To say it went poorly, is an understatement. There are currently 52 answers to that question showing how it doesn't work. It's a thin wrapper around a version of ChatGPT or GPT4. It's also very, very bad at being a "formatting assistant". Instead, it's rewriting content, butchering code being asked about, making stuff up, answering questions and everything other than making formatting better. One user, Mithical, found the prompt the formatting assistant is using.

The one small positive that came out of this is that Stack Exchange did communicate with the community a few days before this was released. This isn't enough though.

Communication¶

Why isn't one post enough?

Behind the scenes it's become clearer to me that several staff members of Stack Exchange don't wish to engage with the community. I briefly touched on one of these people in my [last update][strikeweek1update]. The public press has been out of the loop sounding, the internal reactions on moderation channels has been complaints that moderators are too negative.

Of course moderators are negative right now. There is a strike going on because they are unhappy. The feedback from representatives continues to be filled with road blocks.

Where do I sit today?¶

I think it's becoming more clear that Stack Exchange is not interested in removing GenerativeAI content from it's site. It's actively building and promoting a tool that utilizes ChatGPT under the hood. I am very surprised that they pulled the plug on the formatting assistant after two days. Previous negative feedback has been ignored and I fully expected this one to be as well. The problem with pulling the plug on this, is that CEO has committed to exciting announcements about AI this summer. If the community just showed that one of those AI projects was a flop, they are going to go even harder at getting the next one to succeed.

As this drags on into it's third week, I've become more pessimistic that this strike doesn't end with resignations. Reddit had a two day strike during this time period, and they are already threatening to replace community moderators. The Stack Exchange CEO has expressed their fondness for Reddit on occasion, so I suspect that the action being taken over there is being considered here. Of course, the difference here is that Stack Exchange wasn't effectively shut down by the strike like Reddit was. Sites didn't go dark. Instead, curators and moderators stopped curating and moderating. Everything is still available, the sites are still working, it's just less tidy than usual.

Stack Exchange Strike Update 2

2023-06-13T23:00:00-05:00

Summary¶

Yesterday's update summarized the first week of the Stack Exchange Moderator strike. The strike began last Monday with an open letter to Stack Exchange.

Over the weekend the moderators, and community curators elected three representatives to talk with Stack Exchange. Those talks started this week. They went into these discussions to reiterate the four conditions to end the strike.

A retraction of the prohibition of moderating GPT content.
The private policy on GPT content that was issued to moderators must be revealed publicly.
The data dumps must be re-enabled and SEDE and API access guaranteed.
Stack Exchange, Inc. must communicate, gather feedback, and act on that feedback before making major policy or software changes to the public platform.

So, where are we now?

The update¶

Representatives¶

Discussions with Stack Exchange started off less than stellar, in my opinion. Quoting from the Vice President of Community at Stack Exchange, as relayed to those of us not in the discussions:

"So in summary: Cesar is my delegate for issues here, while I reserve final decision making to myself I"ve vested him with broad discretionary authority and we're meeting on a frequent (daily or multiple times daily) basis to clear any differences between us."

My take away here is that the Vice President of Community - a person who's job should involve dealing with the community doesn't think this is important enough to attend. While I've worked with Cesar in my role as a moderator, this is just making Cesar the car salesman that has to "talk with the manager about your offer". It's a way for the company to drag this out and a way to make the real decisions without moderation input. I fully expect to hear from the community representatives that Cesar liked a proposal but the VP did not but that the VP wasn't around to discuss why not. I'd love to be proven wrong on that though.

Data Dumps¶

Data dumps are the third bullet on our list of things that must be restored. Good news! Stack Exchange will have those restored by June 16, 2023.

This was posted (twice) by the VP of Community - the one not attending the talks above.

[...] Our intention was never to stop posting the data dump, only to begin to collect more information on how it was being used and by whom - especially in light of the rise of LLMs and questions around how genAI models are handling attribution. However, it’s clear that many individual users (academics, researchers, etc) have an immediate need to access updated versions of the dumps. So we are re-enabling the automatic data dumps (and uploading the one that’s about a week overdue). We believe that this can happen by end of the day Friday. We will continue to work toward the creation of certain guardrails (for large AI/LLM companies) for both the dumps and the API, but again - we have no intention of restricting/charging community members or other responsible users of the dumps or the API from accessing them. [...] In the meantime, the data dumps will be re-enabled by end of day Friday. We will communicate here when that has been completed or if there are any delays. We will also post here prior to making any future changes to the dumps or distribution of the dumps.

I suppose now we wait to see if there are any "delays" before Friday.

This message was confirmed by one of the Co-Founders that has since left Stack Exchange and originally committed to these data dumps back in June 2009.

I have confirmation via email from Prashanth that this is, indeed, the new official policy. I'm glad to see it. Creative Commons is part of our contract with the community, and it should never be broken -- however, CC does need to address the AI issue in an updated license, in my personal opinion. [...] - Jeff Atwood

I am happy with this concession and confirmation of the concession from our representatives, Stack Exchange and a Co-founder.

However, it's telling that once again it's Philippe making statements that are lies.

He's done it with posts to the press that I mentioned yesterday (moderators are depending on GPT detectors!). He's done it with the internal emails to his own coworkers. He's doing it again here.

Our intention was never to stop posting the data dump...

This directly contradicts that statement provided by the Stack Exchange Chief Technology Officer last week.

Stack Overflow senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs. While working on this strategy, we decided to stop the dump until we could put guardrails in place.

For being a VP of Community, the ability to communicate with the community is greatly lacking.

AI Policy¶

Not much more progress has been provided by the community representatives. From what I have seen, Stack Exchange is pushing to call an end of this with the promise of a new policy. But it's not done yet. They'll work on it with the moderators and once that's done, that will replace the current policy that started the strike. The representative mentioned they were pushing for a deadline on how quickly moderators would be able to commit to this change.

In my opinion, this is a way to end the community's moderation strike and agree to essentially nothing. It's another promise that something will happen. It gets the community back to moderation (which Stack Exchange employees have been doing for the week), and if they break that promise the effort to re-organize action has to start all over again.

Right now, I'm not agreeing to go back to utilizing my free time to perform moderation duties without knowing what the new policy is.

Where do I sit?¶

Much like yesterday, I continue to re-evaluate my relationship with Stack Exchange. I'm really happy that the data dump has been restored. The messaging around it though continues to erode my trust in the company's actions. This was also one of the easier items for Stack Exchange to agree to, even though it looks like a co-founder may have had a hand in resolving this as well. I don't know if that's true and I appreciate the work the representatives have done to resolve our first point of contention.

It's also very telling that the messaging doesn't mention the restoration in the context of the strike at all. If the company was attempting to build goodwill in this environment, I'd think they would point to the conditions in the open letter and tie the enablement of the data dumps directly to that. Instead, we got a statement that says they didn't intend to stop posting the data dump, directly contradicting a previous statement saying senior leadership decided to stop the dump.

Amazing.

Stack Exchange Strike - Week One

2023-06-12T09:00:00-05:00

Summary¶

A week ago, many Stack Exchange diamond moderators began a moderation strike. They have been joined by power users and curators. At the time of this post, there are over 1200 users that have signed the open letter to Stack Exchange.

This strike has been covered in a few news articles, but I suspect this week's actions against Reddit and their new API pricing changes will overshadow Stack Overflow for a little while. That's fine with me. Perhaps cooler heads will prevail when there is less public focus.

The Stack Exchange strike has been covered:

On Gizmodo
On Vice
On DevClass (with an interview from a fellow Charcoal power user and Stack Overflow moderator)

The important thing in this set of articles is the public statement that was released by Philippe Beaudette, Vice President of Community (taken from the Vice article above).

A small number of moderators (11%) across the Stack Overflow network have stopped engaging in several activities, including moderating content. The primary reason for this action is dissatisfaction with our position on detection tools regarding AI-generated content. Stack Overflow ran an analysis and the ChatGPT detection tools that moderators were previously using have an alarmingly high rate of false positives.

We stand by our decision to require that moderators stop using the tools previously used. We are confident that we will find a path forward. We regret that actions have progressed to this point, and the Community Management team is evaluating the current situation as we work hard to stabilize things in the short term.

They doubled down on this explanation in two meta posts. The initial statement and a post with "data". I encourage readers to spend a while reading through that second link and the answers. The community is skeptical of the conclusions drawn and have counter arguments and data scattered in the answers.

Finally, during the week it was discovered that Stack Exchange stopped their quarterly data dump of all content. This was announced after a former employee stated that the data dumps were turned off in March.

Stack Overflow senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs. While working on this strategy, we decided to stop the dump until we could put guardrails in place.

We are working on setting up the infrastructure to do this correctly in the age of LLMs --- where we continue to be open and share the data with our developer community but work to set up a formal framework for large AI companies that want to leverage the data.

We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines. We will provide regular updates on our progress to this group.

Where are we now?¶

With the summary out of the way, where do we sit now?

As of midnight today, the users that have stopped moderation activities have selected three representatives to be our voice in conversations with Stack Exchange and listed the conditions for ending the strike.

A retraction of the prohibition of moderating GPT content.
The private policy on GPT content that was issued to moderators must be revealed publicly.
The data dumps must be re-enabled and SEDE and API access guaranteed.
Stack Exchange, Inc. must communicate, gather feedback, and act on that feedback before making major policy or software changes to the public platform.

I want to point out that there is nothing here about GPT detection tools. That's because this isn't the reason for the strike. Despite what Stack Exchange has said in their messages to the press, this isn't about detection tools. (I also have an answer on that question about the policy's origination.)

The discovery that the data dumps were turned off has angered many people - those both already involved and others that learned of it this week. The fact that these were turned off over two months ago and nothing was said to the community has made the situation even worse.

What's Stack Saying?¶

This section was added after original publication

Shortly after publishing my thoughts with this article, internal Stack Exchange emails were published. These show how Stack Exchange is communicating this with their employees.

I highly encourage everyone to read those too. There is a lot of reading scattered around to get the full scope of how unhappy the users of Stack Exchange are.

I think it's telling that the company managed to copy and paste from the strike letter, but at the same time managed to completely ignore that this isn't about GPT detectors. Even worse, the company is spreading that falsehood to its employees, and the press. On top of that, has two teams - Community Leadership and Marketing - working on communications, yet no progress has been made.

Where do I sit?¶

I mentioned last week that I've been here for over 13 years. I've applied and interviewed at the company. I've made friends with the employees and gotten recommendations during those interviews. I've built tooling to help moderate comments and eliminate spam on the network. I've been here a long time.

In 2019, I reevaluated my role on the network during Stack Exchange's last screw up. In that one, they managed to libel a moderator, by name, to the press. This event still reverberates through the network today and serves as a brick that this current situation is built with. Stack Exchange destroyed 10 years worth of trust and community relationships in that event. They've tried to rebuild it over these last three years and have been marginally successful. That's gone again.

Now I'm reevaluating how I utilize my free time again. It's not constructive to say I'll hand in my diamonds and walk away if the conditions above aren't hit. But, it's worth noting that I agree with all four of those conditions. This is my free time I'm donating and if the organization I'm donating that time to has changed their philosophy, I will take that into consideration as I reevaluate.

I think we are long past the point of "how it used to be" at Stack Exchange. The question I am asking my self is whether or not I agree with the new direction the platform is going.

Updated after the email publication, above

Reading the email and FAQ that Stack Exchange sent to their employees, I am struck by how out of touch members of the company are. Not the people that don't interact with the community. The people that should know the pulse of the moderation teams, the community opinions, and users in general - like the Vice President of Community. While I'm not surprised that they are down playing the strike, I am surprised that they are flat out lying to their employees.

Stack Exchange should be called out on that and their employees should know that it's happening. Stack Exchange, Stack Overflow, Inc., is lying to their employees. The email presented was written after the strike started and contains information about all of the FAQ items mentioned.

Joining the Stack Exchange Moderator Strike

2023-06-05T01:00:00-05:00

Summary¶

I'm signing an open letter to Stack Exchange because those of us that volunteer our time and energy have been put in an impossible situation. On Stack Overflow we banned ChatGPT created content almost immediately after it was released. Theoretically, that is no longer in effect.

This isn't due to a change in community perception. Instead, it's due to an abrupt policy change on Stack Exchange's part that was posted on May 30. It's important to note that this public policy does not match the guidance that was provided privately.

The relevant portion of this public policy is:

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user.

The private policy notes that "very strict" is essentially "don't moderate unless they explicitly say it was created by an AI".

Background¶

The Stack Exchange CEO posted a blog post at the end of May 2023. In this post, they stated:

Approximately 10% of our company is working on features and applications leveraging GenAI that have the potential to increase engagement within our public community and add value to customers of our SaaS product, Stack Overflow for Teams.

This goes against the community desire to not have GenAI content on the site. The CEO has not provided any feedback to the community, other than a note that there is a big summer project with GenAI. The community reaction hasn't been positive.

Combining this announcement with the newly announced policy the previous day, and an astonishing inability to articulate more details or reasoning, has produced a lot of unhappy moderators.

Additionally, I feel that the company has destroyed 3 years of rebuilding trust. Back in 2019, they destroyed this trust and nearly had a moderator strike then too. That involved providing information to a journalist that had very limited context and due to that, presented a single moderator in unflattering light. The feelings from that have taken years to rebuild and even today that incident is cited as a low point, and users can point out that singular incident when trust of the company plummeted.

The public announcement to not moderate GenAI content contained this:

Through no fault of moderators' own, we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts.

That's just wrong. Absolutely wrong. It is 100% inaccurate and Stack Exchange has offered no data to back this up. User country of origin, region in the world, or any kind of physical location is not available to moderators. We are presented a flag and given information to the content that has been flagged.

Why am I participating?¶

So...my participation? Why am I joining in?

Stack Exchange, and Stack Overflow, thrive due to the human element. It's userbase has been around over a decade and answered millions of questions across over 180 sites. The week ChatGPT came out, the community saw the bad results it can provide. For the past 6 months, we've continued to see how bad that is. The moderation teams across the network to generate a policy and get it approved by the company. I've reproduced it in full below, but as of this post it is still on the site and contradicts the new policy.

In addition to completely ignoring the community's input on how they do not want GenAI on the site, the company ignored their own moderator agreement.

With the assumption that the above will change at some point, the relevant section is pasted here:

``` Stack Exchange, Inc. agrees that it will:

i. Respect your privacy per the terms of the Privacy Policy for the Public Network. ii. Get your explicit written permission before commenting to any media (including media outlets controlled by Stack Exchange Inc.) or independent reporters about you or your moderator actions as per our Press Policy. iii. Allow you to resign your position for any reason without penalty or repercussions. As a volunteer, Stack Exchange, Inc. respects your time and will release you from duty should you ask. iv. Operate “Stack Gives Back”, an annual program giving to selected charities in honor of our moderators. v. Post previews for review of all new official policies in the Moderators Teams instance with the policy tag, marked with links to their public version once published, and maintain a listing of all official network-wide policies with links to them in the Help Center. vi. Announce changes to the moderator agreement no less than sixty days before the deadline to accept the new agreement with a period of at least thirty days for discussion and review. vii. Provide support for your questions, requests and concerns on the Moderators Teams instance and/or the Teachers’ Lounge, direct email to CMs, and content on Meta escalated to staff by whatever formal documented process is in effect at the time. viii. Respect your right to speak openly to question and challenge policy without reprisal so long as such speech does not break the Code of Conduct. ```

The relevant section is vi. There was no discussion period on this. No engagement with the moderators, or the community. Instead it was "effective immediately." The best we've gotten so far is an "oops, sorry."

That's not how something in a /legal link should operate. If I can't trust them to uphold an agreement they have in writing, why should I trust them to uphold anything else?

What does the community want out of this?¶

From the open letter:

Until Stack Overflow, Inc. retracts this policy change to a degree that addresses the concerns of the moderators, and allows moderators to effectively enforce established policies against AI-generated answers, we are calling for a general moderation strike, as a last-resort effort to protect the Stack Exchange platform and users from a total loss in value. We would also like to remind Stack Overflow, Inc. that a network that entirely relies on volunteers for its moderation model cannot then consistently ignore, mistreat, and malign those same volunteers.

My feelings on this¶

I've been on Stack Overflow and Stack Exchange for over 13 years. I've been a moderator on the network since 2014 and on Stack Overflow since 2017. I've written about Stack Overflow a bit over the years. I've participated in Charcoal, the spam fighting community since 2015ish. Charcoal has automatically flagged more than 86,000 posts across the network since 2016.

I've applied to several jobs at Stack Exchange. I've interviewed for a couple positions. I have spent a lot of time with the network, the community, moderators, and employees making this a great place for internet users to find their answers.

I have helped to build the community/company trust that I mentioned above. I watched it crumble. I thought about leaving in 2019, but instead spent the next three years working to rebuild that trust. I'm at the point again, where I see the company not understanding their community. At all. It's sad that this cycle has repeated itself and it's worse that the company is, again, tossing their most engaged users under the bus.

This strike serves two purposes in my mind - the first is the officially stated one. Do not allow GenAI content on the network. It will erode the value of the network quickly. We've also demonstrated that ChatGPT and it's peers are not great at answering complex questions. But, second, and unofficially, this strike will represent a change for Stack Exchange to show whether or not they care about what the community has to say. This is, I believe, the last opportunity for them to retract this policy and reflect on why they are over ruling so many communities that reject GenAI in their community. It's their last opportunity to show they support their moderators and the human aspect of moderation. Failure to do either means that Stack Exchange has given up on community building.

I am a volunteer for this community. I would love to continue that role, but this is the best way for me to show that GenAI is not the right path for the company to take.

What's Next¶

On June 5, 2023 at midnight, moderator local time, the network will start to see moderation activities cease (or slow down). I will be part of that.

GPT Policy¶

This policy was crafted with the input of moderators and Stack Exchange. An important thing to notice is that the moderators are empowered to issue suspensions. This is something the new policy prevents.

``` Why posting GPT and ChatGPT generated answers is not currently acceptable

This Help Center article provides insight and rationale on our policy regarding the usage of GPT and ChatGPT on Stack Overflow. While this is the position of Stack Overflow staff, it’s meant to support the prior work done by moderators (namely, the temporary policy issued to ban contributions by ChatGPT).

Stack Overflow is a community built upon trust. The community trusts that users are submitting answers that reflect what they actually know to be accurate and that they and their peers have the knowledge and skill set to verify and validate those answers. The system relies on users to verify and validate contributions by other users with the tools we offer, including responsible use of upvotes and downvotes. Currently, contributions generated by GPT most often do not meet these standards and therefore are not contributing to a trustworthy environment. This trust is broken when users copy and paste information into answers without validating that the answer provided by GPT is correct, ensuring that the sources used in the answer are properly cited (a service GPT does not provide), and verifying that the answer provided by GPT clearly and concisely answers the question asked.

The objective nature of the content on Stack Overflow means that if any part of an answer is wrong, then the answer is objectively wrong. In order for Stack Overflow to maintain a strong standard as a reliable source for correct and verified information, such answers must be edited or replaced. However, because GPT is good enough to convince users of the site that the answer holds merit, signals the community typically use to determine the legitimacy of their peers’ contributions frequently fail to detect severe issues with GPT-generated answers. As a result, information that is objectively wrong makes its way onto the site. In its current state, GPT risks breaking readers’ trust that our site provides answers written by subject-matter experts.

Moderators are empowered (at their discretion) to issue immediate suspensions of up to 30 days to users who are copying and pasting GPT content onto the site, with or without prior notice or warning. ```

The letter¶

The letter below was originally posted as an open letter to Stack Exchange. I've reposted it here for a record.

``` June 5, 2023 Stack Overflow, Inc. has decreed a near-total prohibition on moderating AI-generated content in the wake of a flood of such content being posted to and subsequently removed from the Stack Exchange network, tacitly allowing the proliferation of incorrect information ("hallucinations") and unfettered plagiarism on the Stack Exchange network. This poses a major threat to the integrity and trustworthiness of the platform and its content.

We, the undersigned, are volunteer moderators, contributors, and users of Stack Overflow and the Stack Exchange network. Effective immediately, we are enacting a general moderation strike on Stack Overflow and the Stack Exchange network, in protest of this and other recent and upcoming changes to policy and the platform that are being forced upon us by Stack Overflow, Inc.

Our efforts to effect change through proper channels have been ignored, and our concerns disregarded at every turn. Now, as a last resort, we are striking out of dedication to the platform that we have put over a decade of care and volunteer effort into. We deeply believe in the core mission of the Stack Exchange network: to provide a repository of high-quality information in the form of questions and answers, and the recent actions taken by Stack Overflow, Inc. are directly harmful to that goal.

Specifically, moderators are no longer allowed to remove AI-generated answers on the basis of being AI-generated, outside of exceedingly narrow circumstances. This results in effectively permitting nearly all AI-generated answers to be freely posted, regardless of established community consensus on such content.

In turn, this allows incorrect information (colloquially referred to as "hallucinations") and plagiarism to proliferate unchecked on the platform. This destroys trust in the platform, as Stack Overflow, Inc. has previously noted.

In addition, the details of the policies issued directly to moderators differ substantially from the guidelines outlined publicly, with moderators barred from publicly sharing the details.

These policies disregard the leeway historically granted to individual Stack Exchange communities to determine their policies, by making changes without the input of the community, overriding community consensus, and outright refusing to reconsider their position.

Until this matter is resolved satisfactorily, we will be pausing activities including, but not limited to:

Raising and handling flags.
Running SmokeDetector, the anti-spam bot.
Closing or voting to close posts.
Deleting or voting to delete posts.
Reviewing tasks in the various review queues.
Running various other bots designed to assist in moderation, such as detecting plagiarism, low-quality answers, and rude comments.

Until Stack Overflow, Inc. retracts this policy change to a degree that addresses the concerns of the moderators, and allows moderators to effectively enforce established policies against AI-generated answers, we are calling for a general moderation strike, as a last-resort effort to protect the Stack Exchange platform and users from a total loss in value. We would also like to remind Stack Overflow, Inc. that a network that entirely relies on volunteers for its moderation model cannot then consistently ignore, mistreat, and malign those same volunteers. ```