Andrew Wegner | Ponderings of an Andy - Side Activities

My first experiences with a laser cutter

2025-01-02T10:15:00-06:00

Introduction¶

My local library spent a lot of time and money remodeling the last few years. Part of this remodel was adding an Innovation Lab with different tools that the public can utilize. These include a Cricut cutter, a Glowforge, a couple 3D printers, sewing machines, a T-shirt screen printer, and a few others I can't recall off hand. I am excited to try out several of these, but started with the laser cutter, because I have a home project that needs a few small things cut and engraved that I wasn't sure how I was going to do.

The end goal is to get a set of two inch by one inch rectangles with various years engraved, scored or cut so that I can use these as labels.

A simple start¶

After a bit of research, I found that what I wanted was relatively simple - at least compared to what the machine is able to do. To do more advanced things, I'd have to learn a lot more about how some software tools operate. For my project though, I used Inkscape. I started by setting up the rectangle I wanted to use, rounding the corners, and adding a year to engrave. This became my basic template, to ensure that everything was the same size.

In all of my SVG files, I only used three colors:

Red for cuts
Black for engrave
Blue for scoring

While I read the Glowforge can handle different colors, I noticed that other tooling and software could not and used these three colors. If I ever get access to another laser cutter - or buy one of my own - I don't want to have to redo all of my work.

The important thing at this step was to select the text, then Text->Object to path. Without this step, the Glowforge application didn't see the text, just the rectangle.

Getting Fancy¶

Once I had a basic cut done on basswood proof grade material from Glowforge, I wanted to try a few other designs. All of my examples below are using 2009, because that's the set I grabbed when taking pictures.

With this, I engraved the background, instead of the text, so that the year would pop out. I didn't end up liking this one very much, because it makes the material so much thinner. This is obvious when I had multiple tiles of years that had not all been engraved this way. The thinness made these specific tiles feel flimsy. Since I didn't want to use this pattern for all years for the project, it wasn't going to work for only a couple.

I tried a couple acrylic colors - Black and Blue - to see how it'd look. In this test, I wanted to deal with the thinness I mentioned above too. I did this by adding a small border around the outside, so that when the tiles were side by side, the outer edges were all the same thickness. I liked how this looked and would end up using a portion of this in some final designs. The black acrylic turned out well.

Unfortunately, I didn't like the blue acrylic. This is how blue looks with the masking tape still in place. This tape is is to prevent scorching marks. It looks ok with the tape in place, but once removed, the blue text just kind of got lost.

This test didn't have the outer border, but I don't think that would have helped me like it any better. The blue text just gets lost from any angle other than straight on.

I cut out individual numbers as well. These look really good. Unfortunately, I didn't plan beyond the cut out, and dealing with four individual numerals on each item I wanted to label very quickly became a ton of work to ensure they were unmasked, aligned, and glued into place. They also didn't end up looking as good as the tiled items once I had them in place.

Something I should have learned from this experiment that I was vaguely aware of, but didn't pay attention to because I was focused on the numbers themselves, was that the inner part of the numbers would come out. This was more important in the next two experiments.

At this point I had decided that I liked how the black acrylic looked, but wanted to highlight a few years. I attempted one last test to see if I could utilize the narrower engraved look and stack it on top of another. The original goal had been to end up with the same thickness as the other tiles, but that didn't work as planned.

That said, for the few years that I needed to highlight, this worked very well. To build this, I kept the lower layer - the wood - the same size as other tiles. I engraved like I had done for the acrylic tests above. Then on the acrylic itself, I made it the size of the inner border and reversed the numerals. I did this because I originally tested by engraving the acrylic down to size too, but it didn't look good. Instead, I kept the acrylic the original width and just cut out the numbers.

Once stacked and glued together, I realized that I hadn't kept the inner pieces of the zeros. It is easy enough to go cut out a couple more numbers at the library, but at the same time, not bothering me enough to do so.

Conclusion¶

Over the holidays, I completed the project and got everything labeled. I'm pleased with how it turned out. I was not surprised at how easy the Glowforge itself is to use. Learning how to design the SVG cut files took longer than I expected. There are likely other software tools that could do the job better, but for this small project and for some experiments, Inkscape worked just fine.

For my next project, I'd like to finish off the WLED work I did last summer. The original project didn't work as expected, but the WLED portion worked wonderfully. I could build a couple stand lights and laser cut out the base. I'd have to figure out how to model the aluminium rail to cut it correctly, but that sound like a fun task.

A Decade of Fighting Spam

2023-11-27T15:45:00-06:00

Introduction¶

Charcoal is nearing a decade of existance. In January of 2024, the Stack Exchange community will have been fighting the good fight of keeping spam off the platform. I've written about a machine being able to flag spam in the past. I've also posted the original and it's follow up on being able to spam flag even better on Stack Exchange itself.

Recently, I was asked to talk a bit about a hobby of mine. I put together this presentation.

What is Stack Exchange?¶

To set a bit of context for those need it. Stack Exchange is a network of over 180 sites covering almost any topic you can think of. It's a question and answer network. The slide you are seeing here are just a handful of the sites of more interesting logos - but you can see they cover a range of topics from professional work place questions, to the intricacies of the English Language, to Data Science and gaming.

But by far the largest and most popular is Stack Overflow. With 24 million questions covering any programming language or framework you have used. It consistently ranks in the top 500 most visited sites on the internet - depending on what service is doing the measuring. Basically, it gets a lot of eyeballs looking at it daily.

Which means it's a target for spam.

What is spam?¶

The network and the community within it settled on a fairly standard definition of spam:

A post exists only to promote a product or service and doesn't disclose author's affiliation.

The images here show what the site looks like when the community systems aren't operational. This is the front page of two sites and if you look closely at the time stamps, you'll see that these posts occurred within about 10 minutes.

If users - new or experienced - come to the site and see this, they start to turn away.

Back in 2013/2014 this was common. Spam posts would stick around for hours and a group of users decided they could help out across the network by flagging these posts more quickly.

What is flagging?¶

The final bit of context that is needed is: flagging. It's exactly what you'd think it is. The goal of a flag to bring attention to the post by forcing it in the community review queues. This gets more people to look at it. Stack Exchange is built around community moderation. There is very little that elected "Diamond Moderators" need to handle that the community can't handle.

If enough people flag a post as spam, it's automatically deleted. The community and company decided that getting 6 people to agree a post is spam is an appropriate number.

Once a post is removed as spam, the post is locked, deleted and the author has 100 reputation points removed. These are the visible actions. The reputation hit is to prevent - or slow - a spammer from getting more privileges within the network.

Behind the scenes, a spam post also triggers company checks against future posts matching similar information to the user. These aren't publicly disclosed. But, the company is fairly conservative in terms of blocking users.

What is Charcoal?¶

The community hates spam. It's a bad user experience at best to have a page filled with spammy posts. It also makes the site, and community at large, look rather seedy. This isn't great for a community and a company that has built its reputation on accuracy and trust.

Charcoal was created to watch for spam across the 180+ sites. Actually, when we started it was less than half of that, but over the past decade the network has grown and the anti-spam systems have grown with it.

The community has a two phase process to dealing with spam. First is alerting the spam fighting community of potential spam. Users can go cast their flags across the network and deal with it. Second, for the truly egregious spam, the system can utilize the community's flags and automatically cast those flags.

How's this work?

SmokeDetector and Metasmoke¶

There are two systems behind this community effort - SmokeDetector and Metasmoke.

SmokeDetector, affectionatly named "Smokey", is designed to be the early warning system. It quickly provides a yes/no decision on whether a post is spam and alerts users for manual action. It passes off the more intense confidence checks and automatic flags to MetaSmoke.

Every post on the network goes through the process below. A user clicks the submit button and Stack Exchange does their few checks - remember these are black boxed - and if the post makes it through these it gets published to a real time web socket.

SmokeDetector does a quick "Is this spam?" check. If it is, it's posted to chat rooms around the network - to the network wide Charcoal room and usually to a site specific room if the room is utilized enough. Users than go and investigate and if they agree that it's spam, cast a flag. After 6 of these are cast, the post is removed. Hooray! Another victory again spam.

When Smokey posts that spam is found, it also sends a message to MetaSmoke. This system is checking how confident we are that this is spam. If there is high confidence, it will start utilizing community member flagging privileges to cast spam flags on the post as well. If there isn't high confidence, no automatic flags will be cast.

The goal is to remove the spammy posts as quickly as possible - and by utilizing automatic flags the number of people that have to go do this manually is reduced. Due to larger community and company discussions and outcomes, the system will not cast all 6 flags except in very very rare circumstances. Someone has to agree with the machines here.

How to detect spam¶

What's spam detection look like? Over the last decade we've tried things like classification schemes, machine learning algorithms, and a handful of AI attempts. But, by far the most reliable has been...

Regular expressions.

(Take a deep breath fellow engineers)

Each post - goes through thousands of regular expressions. Each expression is weighted based on how likely matching that particular expression means the post is spam. The higher weights are posted into the chatroom kicking off this entire process.

The community has built watchlists and blacklists over the decade to help find these posts.

Watchlists are experimental checks. Spam evolves over time. It's actually pretty interesting to watch a dedicated spammer craft their posts to get it to last on the network more than a few minutes. These watchlists are designed to allow the team to test regular expressions without fear of automatically flagging something during testing.

Blacklists are finalized regular expressions that catch spam with a high number of true positives and very low false positives. These the weight spam checkers.

Like Stack Exchange itself, the spam fighting community has built tooling that allows work to be done without a high level user to be around. Users can watch for a new regular expression.

Users that aren't trusted just yet, will have their request created as a pull request in GitHub that needs to be approved. Trusted users will get their watchlist automatically added to the system. The same holds true for blacklisted items.

But, watchlists and blacklists are only half the problem. The other half is validating that these are accurate. As posts are detected as spam, users provide a signal back to the system on whether a post is a tp - True Positive - Spam

Or a false positive - fp - not spam. These feedback to the watchlists and will prevent elevating watchlists that are inaccurate to a full blacklist.

Sometimes, a post has features that the system doesn't detect as spam. In those cases, the community can manually report the post. This triggers the alerts through out the chatrooms so that others can flag it and get it removed. It also allows the community to find potential patterns to watch for in the future.

Users that aren't trusted yet get pull requests created for their patterns. All of this can be handled and approved within the chatrooms. A lot of this system is built on top of, and keeps most users within, the Stack Exchange ecosystem.

I mentioned the weighted reasons on a detected post. When these are posted in chat, the reasons are also posted as well as the weight of the post. The one on the slide below is particularly bad. Generally anything over about ~225-250 is spam with higher numbers becoming more and more certain.

These weights shift over time and as a regular expression is utilized more. This keeps the system flexible.

For this particular post, the system determined it was spam and cast three automatic flags from our users. Each user that grants permissions for the system to utilize their flags - because they are responsible for the usage of the flags - can set their threshold for when to allow their name to be used.

The 4th flag here came in via a user script the community built, but was not automatically cast. The remaining two flags would have come from the users of the site or from someone that saw the Smoke Detector alert and manually flagged it. Metasmoke doesn't have a record of that because it didn't go through Metasmoke.

By the numbers¶

Let's look at some numbers.

SmokeDetector has been running since January of 2014. We didn't start recording stats until about 18 months later though, so the dates in the graph start in August 2015. Initially, the system didn't have watch lists, which is why you see the blue and orange lines are pretty close together.

Around mid 2018/early 2019 we introduced watchlists. This was done because we started seeing persistent spammers. These were spammers that noticed their posts were being deleted quickly and worked to find ways to change the message to stick around longer.

The chatrooms are open and based on some messages we have removed, it's obvious the room is watched by the bored spammers. The watchlists reduced the true positives. But because we didn't ever separate the data between blacklists and watchlists the lines began to separate.

In early 2017, autoflagging was introduced. With autoflagging the system can reduce the time on site for nearly half of the true positives.

You'll notice a major spike in the summer of 2022 and a dip in the summer of 2023. The spike was for a massive spam wave. This was the work of a spammer that had access to a lot of geographically distributed systems - which bypassed Stack Exchange's built in protections - and was a persistent spammer or team of spammers that watched the public chatrooms for changes the spam fighting community made to detect their posts. This went on for 2-3 weeks with thousands of posts being made, adjusted, and deleted. Ultimately, the spammer was blocked at the Stack Exchange level based on heuristics the Charcoal team presented.

This past summer, in 2023, the dip you see is because Stack Exchange experienced a crisis of confidence from the community at large. Moderation work stopped for the months of June and July in protest of the company's policies toward generative AI on the platform. Charcoal participated in that. While not fully resolved, some of the worst policies were reworked with input from the larger community and work resumed.

The goal of the Charcoal project is to remove spam quickly from the site. Flags that are cast by the system are tracked and we can clearly see that more automatic flags mean the post is active for less time.

When there is no system cast flags, an average spam post lives for 21 minutes on the site. If the system casts all 6 - which is only utilized during a spam wave like in the summer of 2022 and with company permission - a post lives for 16 seconds. During day to day operations, the system is configured to cast 3 automatic flags. This was determined by a lot of conversations with individual sites around the network and what they felt comfortable with.

SmokeDetector has over 103 thousand commits to its repostory over the last 10 years with 90 different code contributors. In the slide below, the top two graphs show that it's rulesets are updated daily - except for this past summer.

Over the course of a 24 hour period, flags are automatically cast from nearly 420 different users around the network.

Finally, the entire goal: over 450,000 spam posts have been identified and deleted by the system and the community in the last decade.

Conclusion¶

You now have an idea of how one of the largest sites on the internet handles spam. I do want to point out that StackExchange operates very differently from sites like reddit or YouTube or Facebook which spent a lot of company time building their anti-spam systems. Stack Exchange built basic protections themselves and then saw the technical community members step up and take on the challenge.

The Launch of Viva Printworks

2023-05-21T01:30:00-05:00

Viva Printworks has launched! It's a small - for now - business producing unique designs for various products. We are launching with 30 designs on t-shirt and hope to expand to other product types in the coming months.

Dana and I have talked about building a side business off and on for years. During my job search last year (and all the fun that entailed), these discussions became most ernest. After determining what we wanted to do, figuring out the technicalities of starting a business, and figuring out how we were going to accomplish this - we did it!

Viva Printworks launched our first wave of designs on May 18, 2023 on Etsy. As we go through launch, growth, expansion and general business lifecycles, we hope to expand beyond a single platform.

This is exciting for us! Check us out at Viva Printworks on Etsy. I suspect that I'll have posts in the future here, as well, discussing some of the technical aspects of the business.

Automatically checking for broken links using Github Actions

2023-02-13T16:30:00-06:00

Introduction¶

Over the past six months or so, I've been making changes to the site. This culminated in November with a theme update and site relaunch. I have a couple more articles planned about things I've learned from that relaunch, started with today's article. The site is over a decade old, with the initial article in 2009, with over 100 posts since then.

A decade plus is a long time to assume that links will remain operational. I wrote a short series of articles about how broken links impacted Stack Overflow over 7 years ago, including a proposal to fix it and how I performed the link checking from a technical perspective. Now, I want to make sure that I don't have dead links all over my site, especially because this is the site I give out in a professional context.

GitHub Action¶

The Github Action script, is available in the repository where the content of this blog is kept. It's in the .github/workflow/ directory.

Scheduling¶

My goal for this project was to check links for the entire site on a regular basis. I was hoping roughly once a week. Fortunately, GitHub Actions allow you to schedule your actions, using syntax like this:

on:
    schedule:
        - cron:  '12 1 * * 5'

The important thing to note, which is called out in the docs, is that * is a special character in YAML, so the string has to be quoted. Using this schedule, my link checks will run every Friday at 1:12am. For me, timezone doesn't matter, but the documentation says this is going to be UTC time.

Checking links¶

The next thing that has to happen is the actual checking of the links. I use broken-link-checker for now. I picked this because I'm familiar with it. It hasn't been updated since 2018, though, so in the future I may spend some time either looking for an alternative or building an alternative. For the time being though, it works just fine.

I put this snippet into my actions file.

steps:
    - name: Run Broken Links Checker
    run: npx broken-link-checker $WEBSITE_URL --ordered --recursive \
    --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0" \
    --exclude linkedin \
    --exclude "udemy" \
    --exclude "ude.my" \
    --exclude "eia.gov" \
    --exclude "backpack.tf" \
    --exclude hlsw \
    --exclude dell \
    --exclude supermicro \
    --exclude mysql

The quick break down:

$WEBSITE_URL is an env variable that I define as https://andrewwegner.com/
I set up a user-agent because several sites block the default one. I did something similar in my Stack Overflow analysis 7 years ago.
All of those excludes are due to those specific domains still not allowing automatic scraping. It's a little disappointing that I need these, but excluding these few to ensure the rest are functional is more important to me than trying to figure out a work around.

Create an issue¶

If broken links are found, it doesn't do me any good to have that buried in a log somewhere without alerts. Perhaps unsurprisingly, I don't check Github every day. My goal was to create an issue on the repository if broken links were found. I did that using the following, and the Create an issue action.

- uses: actions/checkout@v3
  if: failure()

- uses: JasonEtco/create-an-issue@v2
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  with:
    filename: ${{ env.ISSUE_TEMPLATE }}
  if: failure()

The important line here is the filename: line. Using another environment variable, I defined a file with my issue template. Everytime an issue is created by the action, it will have the same format.

Issue Template¶

My template is defined in the same directory as my workflow and currently looks like this:

---
title: Website Contains Broken Links
labels: ''
assignees: ''
---

## Broken Links Detected

Broken Link Checker found broken links on https://andrewwegner.com

[View Results](https://github.com/AWegnerGitHub/awegnergithub.github.io-content/actions/workflows/check-broken-links.yml)

_Use search filter `─BROKEN─` to highlight failures_

The top metadata defines the issue title, and any labels or assignees I want to set up. I'll probably add myself as an assignee once I'm happy with the stability and reliablity of the checks.

You can see what an issue created with this template looks like in the repository.

Environment Variables¶

The last section, I've alluded to already - the environment variables. I have two currently defined.

env:
    WEBSITE_URL: "https://andrewwegner.com/"
    ISSUE_TEMPLATE: ".github/workflows/check-broken-links.md"

The first is to set the domain I'm scanning. Mine, obviously. The second is a link to the issue template that is utilized if broken links are discovered.

Things I learned¶

While setting this up there where a few things that I learned or desired that were not immediately obvious while reading documentation. The first was that GitHub actions aren't triggerable without setting up the appropriate on event type. In this case, I want to occassionally run the link checker on command, instead of once a week. That means I need the workflow_dispatch event and it needs to be empty.

on:
  schedule:
    - cron:  '12 1 * * 5'
  workflow_dispatch:
env:
  WEBSITE_URL: "https://andrewwegner.com/"
  ISSUE_TEMPLATE: ".github/workflows/check-broken-links.md"

Notice that the empty workflow_dispatch has no other parameters. It can accept some, if I wanted to provide input to the script at run time, but I have no need for that right now. So, mine remains empty.

A couple nice to haves, that I wasn't able to immediately figure out were how to retry a failed link "later" in the script. While testing the functionality, I had a run that failed because one image, in one article failed. Checking that article showed that it worked. Checking the log showed that the link being checked worked. It was just the internet being the internet with a temporary blip. I'd love to be able to retry failed linked a little later in the run or X seconds later, etc. If it fails both times, assume it's broken and report it as normal. Without that retry, I'm a little worried this is going to be fragile, which is part of why I haven't populated the assignee metadata in the template yet.

Another nice to have would be a way to embed the broken links (and page they appear on) in the issue itself. I couldn't find a way to accomplish that with the current tooling, so the template links to the log and you have to search for the appropriate string.

Like I mentioned earlier, I will likely look for a more up to date tool. Or build one my own. I'm interested in learning more about how GitHub actions work and this may be a good usecase for myself.

Stack Overflow bans ChatGPT temporarily

2022-12-05T14:45:00-06:00

Today Stack Overflow moderators (myself included), have implemented a temporary ban on ChatGPT on the site.

Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned.

This ban was picked up immediately by several technology news outlets, including ZDNet, TheVerge, and Vice. It was also picked up by CNN, The New York Times, and the Washington Post.

That's a lot of news coverage.

The question that I want to answer is "Why?". Stories and marketing from OpenAI, give reasons why this new chatbot is a "good thing". With their examples, it definately looks that way. But, in practice, it's not working out so well.

The responses that users have been posting on Stack Overflow have a high rate of being incorrect. Normally, the community can handle this, but the responses aren't your usual small code snippet of an answer. Instead it is a long, detailed, explaination that looks plausible. But, it's wrong. Combined with users posting multiple answers an hour, this is a lot of content that Stack Overflow reviewers (or worse, the handful of elected moderators) to go through and determine if it's valid.

At the time the ban was implemented, we'd seen thousands of answers generated by ChatGPT. On the one hand, this is impressive work on the ChatGPT AI itself. It can be difficult to detect and is good at holding a conversation. On the other hand, and more important from Stack Overflow's perspective, this isn't helping the user base. Thousands of subtly wrong answers is awful. It doesn't help the user looking for help, and it will very quickly destroy the trust that millions of developers put in the site if this is allowed to continue.

I'll admit that I'm disappointed that AI hasn't reached the point where it can do what ChatGPT seems to do. But, this is a step forward. Unfortunately, this step seems to have left a bad taste in the mouth of developers looking for help beyond a toy example.

For the time being, ChatGPT is banned on Stack Overflow. The moderation team will continue to work with the company to ensure the community we have volunteered to moderatate remains one of high quality. Additionally, as the larger Stack Exchange network of sites debates a similar ban, the Stack Overflow moderation team will be able to provide input on our experiences.

Updating the design of the site

2022-11-17T10:00:00-06:00

Welcome to the update!¶

I have a new theme for the site, and I think it looks great! This is a heavily modified Flex theme. I talk about the changes I made to the theme below, as well as my goals for the site going forward. A new theme isn't going be responsible for hitting all of those goals, but I think it will help. The site was pretty bland before. It did what it needed to do, but it didn't look the best doing it. My skillset is not front end development, but I am rather proud of the new theme and the changes I put in place.

If you're curious about what the site used to look like, jump down to the Where the site was section.

The default version of the Flex theme can be seen on it's demo site, to give you an idea of the changes I've made to my instance.

Look & Feel¶

The logo is perhaps the most subtle change. It's designed, intentionally, to not be a circle (to common on other sites )or a square (to boring). It's an off shape blob. Take a look at it. It very slowly shifts too. You can see it along the bottom of the shirt most clearly. It's designed to move only a little and pretty slowly, so that it's not distracting. It's just active enough to catch your eye if you are looking but not designed to pull focus from the article you are reading.

The default flex theme is very...neat. There are nice crisp lines. It's very clinical. I changed this just a little with splash of a curved red border between the side bar and the main article. It's not a major change, but it adds a little personality. Combined with the blob/animated head shot, it makes it feel unique among the flex theme'd blogs I looked at to evaluate if I liked the theme.

A favicon is such a little thing, but I didn't have one with the old theme. Having one now adds just a little bit more of a professional touch and it was so easy to do. The color of the favicon, the side bar, and the responsive bars from the tag and category pages all match.

ToC¶

The table on contents on the side bar is an addition I made, because I dislike when the table of contents scrolls off the screen. It's the same reason I froze the table of contents to the side bar on the previous iteration of the site. When I'm reading, I like to know how far I am in and article and be able to easily jump to another section. Since I am also a reader of my site (especially on the technical posts), I try to make it easy for myself too.

Tags/Categories¶

The tags and category pages on my old site where bad. They were a giant list of meta-data and a link to a post. It was messy. I didn't like it, but I also didn't use it so I spent very little time trying to make it work better. With a new theme, I wanted to make the pages more useful, at least to me. I found an excellent post by John Paton on how to make a responsive bar chart for these pages.

Take look at my tags or categories pages. To me, that looks wonderful. I can get a quick look at how many articles are in each section and clicking on one brings me to a page specifically for articles in that slice of meta-data.

SEO¶

Before you run away...this is a good thing, and I've been doing it for a while. All of my reviews, have been sporting Schema.org Review microdata for years. My blog posts have been sporting Article microdata. This was all hard coded into my template. It was beyond time to update to utilize JSON+LD and provide even more context to the topics I write about.

Initially, I selected Flex because it claims to support Rich Snippets. My results on that are fuzzy, at best. The snippets provided are not full rich snippets. However, they provided a great base. I've updated my instance to support JSON+LD for Person (click view source on my About Me to see it), Article, Review and Breadcrumbs (see the navigation bar at the top of this page). Search engines use all of that to get a better idea of what's on the site. I have a few more I want to implement too, but this gets me beyond parity with my old theme.

I've also made sure that canonical links are included in my articles. I have a couple that Google sees as duplicates due to UTM tags I stuck on links a while ago. Having cononical links in place should resolve some annoying errors and warnings I see in my search console.

Goals of the update¶

I have a few goals with this newly updated site. A couple relate directly to the theme, and a few are general site goals.

Updated Tool set - I was running an old version of Pelican and missed out on some new features. By updating, I'm hoping that I can remain closer to the current release. I like to get the newest features, and that alone feels better.
A useful tags and categories page. Like I mentioned above, they used to be useless. Now, there is more than just a list of 1-2 work phrases sitting on my site across two pages. If I start another series - I believe I have two right now total, I may add a similar series page.
Site face lift. As you can see below, the old version was pretty bland. This update adds some color, makes a few things more accessible, and generally feels less like reading a sheet of paper on a screen. It's not the flashiest thing on the internet, but you can see by my social media links to the left, that I'm not that type of person anyway.
Better SEO. Obviously a template doesn't immediately solve this problem, but it will help. Plus some other plans to add more content (gasp!) and it should help. Right now, though, "Andrew Wegner" returns this page 9th on Google. It's behind someone with a different last name spelling, someone with a different first name, and a few other "Andrew Wegners". While search for a job, I couldn't depend on my site being the one that was clicked.
Lastly, a vanity goal, is to get a Knowledge Graph when searching for my name. It'll take a while before I can get there, but some of the SEO work I'm putting in now will, hopefully, pay off.

Where the site was¶

One last trip down memory lane, as the template that has served as the base of the site since the intial migration from Wordpress in 2015 is retired. The old template was using Elegant. I am the #7 contributor to the theme at the time of publishing this. That's not saying a whole lot, with my brief spurt of activity in 2019, but it was a contribution that I hope others using the theme are able to utilize.

I was stuck on the Pelican 3.0 branch. I really don't remember why. I remember attempting to upgrade to Pelican 4.0 and everything broke. At the time I was unable to spend a significant time to fix it, so I rolled back and stuck with the latest version that ran.

Going through this revamp, I upgraded to the latest Pelican, the newest Markdown and go all the new fancy things. We'll see if they actually improve my experience. From a user stand point, not much should change other than a visual overhual. It's a static site and is designed to be pretty non-interactive. On my side though, I'm hoping things are a little less brittle.

On to the last pictures of the old template.

The Old Home Page:

A few other screenshots of the old site sit here:

Stack Overflow still has issues and it's getting worse

2019-03-29T10:34:00-05:00

Last time on this blog¶

A little over a year and a half ago, I wrote an article about Stack Overflow's problems from my perspective as an experienced user. This was before I was elected as a moderator on Stack Overflow. I ended the previous article with this:

I continue to invest my time and effort into the community, but even as an active user who really wants the company and community to succeed, it's getting harder and harder to ignore that those of us that have been around for years are not being listened to any more. We're being treated as the grumpy old person that grumbles about the way things used to be. Our experiences on the site are brushed aside as being unhelpful to new users. That completely ignores that fact that we are still trying to reach the goal on which Stack Overflow was created: "With your help, we're working together to build a library of detailed answers to every question about programming." To do this, we need high quality questions and answers so that we can actually provide help to all users. I think this is the biggest challenge that Stack Overflow is going to face in the next 18 months.

So, what's happened in the last 18 months?

Documentation¶

After years of development (being announced in 2015), Documentation was shuttered in August of 2017. Stack Overflow wasn't drawing users to the Documentation feature. Their own metrics and analysis showed that fixing Documentation to be useful to users - both new and experienced - would require a significantly larger team.

What went wrong?¶

In my opinion, and as I mentioned in 2017, Stack Overflow has ignored it's user base. This is going to be a recurring theme in this post. For years, users provided feedback on meta, in dedicated user experience interviews and in chatrooms. This resulted in superficial changes and major rewrites. Yet, complaints still existed. These complaints turned off the experienced users that could produce the high quality documentation. Instead, Documentation became a reputation farming operation in all but name. This turned off even more users.

By Stack Overflow's own admission when sun setting the feature, Documentation was built to solve a problem that wasn't really a problem.

Finally, our research showed that while a lot of developers were dissatisfied, the current state of programming documentation is not universally broken the way Q&A was when Stack Overflow started. In particular, we heard over and over that Stack Overflow has become de facto documentation for many technologies. As many of you pointed out, Stack Overflow is already good enough at providing documentation of obscure features. Even when considering just the company's mission of helping programmers “learn, share their knowledge and build their careers”, Documentation isn't the most efficient use of resources.

Two years of major development, focusing on a problem that the community had not been enthusiastic about, and intentionally ignoring other feature requests and other improvements angered a lot of users.

Teams¶

In my last post, I mentioned that Teams had been launched and shut down in less than a year. Teams is back! At least the name is. Initially launched as "Channels", and later re-branded to "Stack Overflow for Teams", this is a money generating route for Stack Overflow. It uses the old URL.

Now, generating money is good. It's good for both the community and the company. Without money, the company can't survive. Without the company there is not Stack Overflow or community. My problem isn't with money generation. My problem is that, once again, community feature requests for higher quality and moderation tooling to cultivate that higher quality was ignored.

By all accounts, Teams seems to be doing well and bringing in revenue. I am hopeful that this translates into development time to build out the features the community still clamors for.

Meta hatred¶

Meta. It's murder. Until it's not. Meta is how Stack Overflow communicates with the community. It's how the community communicates with itself. It's where governing principals/thoughts/guidance/sticky notes comes from. In short, meta is a large part of how Stack Overflow the company and Stack Overflow the community talks with one another. Decisions are questioned here, announcements are posted here, and little by little the site is made better.

That is, until nothing happens. Stack Overflow's response time has become a meme.

"6 to 8 weeks" is a joke. It's used to indicate that something isn't going to be built or changed. It's so prevalent that this comment crops up over and over on feature request posts. It's used by the community to say that nothing is going to happen.

When something does happen, it's a "big deal". There have been a few examples in the past year. Unfortunately, these changes happened due to feedback from Twitter, not Meta. For years we've been told to post on Meta. For years we've been told that Meta is where the company will engage with us. Then two massive changes happened.

The Welcoming¶

The first change was to make Stack Overflow more "welcoming". This isn't bad. As both an experienced user and as a moderator, I've seen my fair share of users not being welcomed. I've seen hostility to poorly asked questions.

Unfortunately, this whole blog post and resulting meta-drama appears to have cropped up because of a post on Twitter from someone who felt unwelcome. That's fair. I believe they felt that way. However, from my point of view, Stack Overflow ignored their own users (some of whom had been saying the exact same thing for years) because it was suddenly posted on Twitter where the entire world could comment on things that may have been out of context. Instead of listening to their own users and the experiences those users had, Stack Overflow went into damage control mode and rapidly updated it's "Be Nice" policy.

Whether this is actually what happened or not is really beside the point. Many long time users had this perception. Meta was ignored. User feedback was ignored. Instead, the person that could shout the loudest and had made the most noise appeared to be the one that was listened to.

A few months after the welcoming blog was posted and a month after the update, another post was made about how the company was attempting to classify comments. The idea behind this was good, the execution of the blog post was not. In the initial version of the post, exact comments were posted to show "bad comments". I disagreed that a few of them were rude. I'd have removed them as no longer needed without a problem. Honestly, I'd probably have removed them as rude too, because comments don't need to stick around and it's easier to accept the rude flag than it is to decline and manually delete.

My problem was that the exact comment content was posted as a "wall of shame". Then, despite only employees being involved, none of these comments were removed or even flagged for moderators to remove. In short, it really was a "wall of shame".

I believe I covered my disappointment in both this failure and in the technical aspect in my comment on the blog.

I am a huge fan of automatically removing unwanted comments. I did so for several years. That said, I’m disappointed in how this is playing out here. I’m disappointed on both a community level and a technical level.

On the community level, I am very disappointed that 57 Stack Exchange employees were able to evaluate bad comments, determine they were bad enough to put in the hall of shame post here, and then do nothing about them. It took users less than 15 minutes to find those comments on Stack Overflow and identify the “rude” users. Users who are rude because they asked why a certain tag was on a question. Did none of your 57 users have a diamond where you could remove the comment from the site? Even if that’s the case, all of you have the option to flag a comment. Even that wasn't done.

On a technical level, you evaluated less than 4000 comments. That is a few hours worth of comments on a single week day. (source: http://data.stackexchange.com/stackoverflow/query/872382) Is that really representative? How did you determine which comments to use in your evaluation?

The good news is that the comment samples were edited to be "representative" of the problem later.

Welcoming users is great. Helping users is the purpose of the site. I fully support all of that. What I don't support is ignoring the feedback mechanism you've built and told everyone to use for years because someone else with a lot of Twitter followers put Stack Overflow in a bad light. Yes, it should be fixed and should have been fixed sooner, but the perception of "listen to the loudest shout" is not a good look.

Which brings me to...

Removal from Network Questions list¶

In October, the entire "Twitter shouted, Stack Overflow reacted" repeated itself. This time, a user was offended (while on Stack Overflow) over the Hot Network Questions list for two questions on another Stack Exchange site. In under an hour, Stack Overflow (the company) removed that question from the hot network questions list.

The community in question was shocked by the result. A community manager explained the decision on that site's meta

It was the solution we chose - without consulting IPS - because it was effective and easy to implement since it would fix the perceived problem immediately and there was already a technical solution in place for doing it.

Notice a couple things here that stand out to me:

"perceived problem"
"without consulting IPS"

The company knee-capped an entire community and a large source of their traffic (the Hot Network Questions list) because of a single Twitter comment. Understandably, the community was upset.

Behind the scenes was even worse. On Twitter, the original user posting their complaint was engaged by community moderators. It didn't go well. Then they complained about that. A Stack Overflow employee jumped into the thread with the following:

If the DM trolls claimed to be moderators on any of the sites then I'd like to follow up with the community team and see about getting removed - they take this very seriously.

Turns out that Stack Overflow doesn't value their community moderators. One employee might be misguided, but this Twitter reply remained active and moderators across the network clamored for an official response. A moderator reached out to the Twitter user in good faith and was threatened with removal by a Stack Overflow employee.

One of Stack Exchange's most respected moderators posted their frustrations on Medium. I highly recommend you read it. One of the community managers posted a response on their own blog too.

The "super-official almost response" was posted even later. This was more than 10 days after the original incident. It took half a month for a first draft of a "moderator social media guidelines" post to be made in the private Stack Moderators Team. That post consisted of bullet points on how a moderator should behave on social media. I replied to that post with this

I am underwhelmed by this response. The event that led to this post and recent discussions around Stack Exchange (and the broader internet) wasn't due to a moderator's bad behavior. Moderators engaged a user on Twitter following the bullets in this post, and yet stuff still exploded in everyone's face. From my point of view, this post is so far down the list of responses that I was hoping to see from Stack Exchange that I'm feeling insulted.

I was asked to hold my judgment until the final draft was posted. That took place in December - two months after the incident. It was changed from "Social media guidelines" to a "community emergency process". These four bullets were provided:

Introduce yourself and if necessary, your role as moderator of a SO/SE site.

Offer to help with the situation, and be very respectful if someone declines your assistance. Sometimes, people just want to vent, and the best thing we can do to help is to give them space.

Be aware of the volatile nature of online discussions; if the path to constructive discourse becomes blurred, it's often best to disengage.

Keep your interactions with others, concerning SO/SE, as clear and as kind as possible. If things begin to get out of hand, please disengage and let us know about it.

In short, do exactly what the moderator did initially on Twitter which resulted in the threat of being removed.

Communication¶

Stack Overflow is slowly isolating itself from the community. There have been multiple comments scattered around the network saying the employees don't want to engage on any meta. There are community managers that are feeling hated because of complaints users have made. Users are taking out their anger of being ignored on posts talking about new or unrelated features. In turn, the employees engage just a little bit less. Lines are being drawn. I see it as a moderator. I see it as a user. Very slowly the community is trusting the company less and less.

Everything is becoming "us" vs. "them". There is "the company" vs. "the users". Blog posts, comments, meta discussions also appear to be driving a wedge between "the users" and making it "new users" vs. "established users". In the blog post announcing the search for a new Stack Overflow CEO, this comment was made by the current CEO:

One thing I’m very concerned about, as we try to educate the next generation of developers, and, importantly, get more diversity and inclusiveness in that new generation, is what obstacles we’re putting up for people as they try to learn programming. In many ways Stack Overflow’s specific rules for what is permitted and what is not are obstacles, but an even bigger problem is rudeness, snark, or condescension that newcomers often see.

The underlying sentiment - improving inclusiveness and diversity - is great. I'm all for that. The rest of it, though, is a dig at the established community in the same way that the Welcoming blog post was. Stack Overflow's high quality standards are the problem. It makes the community seem rude and abusive. You should stop closing those questions, stop down voting new users, and just be nice. It doesn't say that directly, but that's how existing members are seeing it. Read under hairboat's answer to see some of the simmering feelings of high reputation users.

The idea of trust between users and the company is brought up in the comments. This is just another example, in a long list, where the community and the company are butting heads. Something happens that the community doesn't like - reacting to incidents off site, focusing on features no one asked for, not explaining why these new features need to be done, comments are made by one side that makes the other seem unflattering - and another round of not trusting the company starts again.

The company has had a decade of experience with this community. It's grown, shrunk, and grown again. For most of that time, there has been fairly open communication and trust. I am afraid that trust has eroded over the last few years and can't be recovered.

What can be done?¶

The company wants to focus on areas that can bring in more money. In my previous post I quoted the President and Chief Technology Officer of the company.

I appreciate that there are a lot of issues on Stack Overflow that need to be addressed, and maybe we haven't been responding to them as quickly as we should. But Stack Overflow Q&A is a big, established product, most of the problems left are hard, and we can't let maintenance become the only thing we work on or we'll just slowly run out of money and go out of business. We are trying to both maintain Q&A and solve new problems for developers and reach new audiences. The latter is hard, and maybe we'll fail on a lot of our ideas, but we're not going to stop trying. – David Fullerton May 17 at 21:10

I bemoaned that this sounded that Q&A was feature frozen. It's been nearly two years since that time. I can't remember a new feature that was introduced into Q&A that helped the community maintain high quality posts. There was a new wizard introduced for new users that is supposed to help. A quick look at the review queue numbers on Stack Overflow shows that they are still stable at the same point it was two years ago.

My suggestion as a user, a moderator and someone interested in see Stack Overflow remain successful, is to focus on helping to manage the quality of your content. Users have been asking for years to be able to better handle poor content. They've asked for tools (both system tools and moderator tools). There have been projects started, stopped, restarted, and stopped again that are supposed to improve quality. Community tools have been built to help deal with quality problems. Use some of this!

Stack Overflow has a data science team. Work with the community directly to help figure out ways to prevent low quality content from ever getting posted. Force users - all users - to post higher quality content. Work with the communities that have developed automated tools. Run it with larger data sets. Even if Stack Overflow has to be more conservative than the community tool, if you can prevent some of the low quality content from making it to the site you have a victory.

Obviously the company can't ignore the areas that bring in revenue, but it's becoming increasingly clear that the community is much less forgiving than they used to be. Continued communication blunders will not help with anything.

Where to from here?¶

I ended my last post with this:

I want Stack Overflow to continue to grow. I also want Stack Overflow to have high quality content. I think my experience and the experience of others can help build the features to accomplish this. We just need Stack Overflow to refocus on the Q&A portion of their network again.

I think that holds true today, just as it did 18 months ago. The aspect of the site that draws users in is Q&A. Make it better. Make the content better. Give users tools to make it better. With all of this, I believe, the "welcoming" aspect will improve. Let the system handle the low quality stuff automatically. Eliminate the need for users to ask basic questions or remind users to post their code. Let the system be "the bad guy", and let users interact and help one another.

We'll see how everything looks in 18 months. In the meantime, I'll be here, cleaning up the low quality content and prodding the company to provide improvements to Q&A.

...and then there was a backup server

2018-02-12T15:15:00-06:00

Introduction¶

In my last post, I covered the events that lead to my data loss scare. Faulty, untested, backups will bite you every time. The question is just, "when will it happen?". By mid-to-late January (three months later), I'd gotten everything back from SERT Data Recovery and was happy that everything was recovered. It was time to finally build that huge NAS.

Server Goals¶

A NAS for a home backup solution could be something as simple as a prebuilt device with a couple hard drives. I'm a bit of a geek and have a lot of digital data, not to mention family pictures, years worth of programs I've written, and a digital music and movie collection. I'm a digital pack rat, but a well organized digital pack rat. I also wanted to get more out of this server than "plug the device into the router". I ran game servers for Vipers for five years. I did a little bit of server work at Caterpillar. I've toyed with server virtual machines off and on for testing various packages and software over the years. Through all of this, though, I've never had a server in the house that I could use to run some of my scripts on. Those always ran on my desktop because it was always on. This server was going to change that.

I had several goals when building this thing:

Have more storage space than I needed for several years. I didn't want to rebuild this in 18 months because I was bad at planning. Years ago when I built my computer for college, I stuck two 120 gigabyte hard drives in the machine and thought I'd never fill that. When I came home that first summer, I already had to upgrade hard drives because I was low on space.
Run a server version of Linux. I don't want to buy a license for a Microsoft server product and my Microsoft Academic Licenses expired a while ago. During my time with Vipers, I used a Red Hat variant of Linux. At Caterpillar we used Ubuntu.
Utilize ZFS for protection against data corruption. This combined with my more recent usage of Ubuntu lead me to decide on using Ubuntu Server for the operating system. At the time of this post, I'll be using the 16.04 LTS version. I'll continue to upgrade to future LTS versions.
Back up data from all devices in the house automatically. As camera phones have gotten better, we've found that we carry our bulky digital camera less and less. The problem with the phone camera is that we need to get the pictures to the computer. I don't want to hunt down a data cable or email the pictures to myself. I'm also not a fan of posting everything to social media. I want my phone to send the pictures to a backup location automatically.
Host my personal git repositories and personal projects.
Be able to stream music and movies to other devices on the network.

Hardware¶

Now that I've decided my goals, it was time to pick out hardware. The biggest decision was to determine how much storage space I'd be getting. The idea was that hard drives would be the majority of the cost of this machine. In the end, I went with the following hardware:

Rosewell 4U server chasis. It's rack mountable for the future when I can convince myself that a server rack in the basement is a thing I want to spend money on and haul around.
Supermicro MBD-X11SSM-F-O Micro AT server motherboard (LGA 1151)
Intel Xeon E3-1230 V5 3.4 Ghz processor
2x Supermicro certified MEM-DR416L-SL01-EU21 16 GB DDR4-2133 ECC server memory. Take careful note of that model. I originally ordered MEM-DR416L-SL01-ER21 (notice the single "R" to "U" character difference). The motherboard did not like the ER21 at all.
EVGA 650 power supply (I've been really happy with EVGA power supplies on my last 4 machine builds).
1x Western Digital Blue 1 terabyte SSD (for the operating system and other applications)
7x Western Digital Red 4 terabyte hard drives (for all the data)
Enough SATA cables for all 8 drives

I'll be running ZFS in RAIDZ2 (dual parity). This means with 7 drives, two will be effectively parity drives. I'll have a total of 20 terabytes, minus formatting, for data. After formatting this comes down to a little over 16 terabytes of usable space. Considering that the rest of the household has a combined 5 terabytes, if I use up every available bit, I'm hoping that 16 will last me a while.

That RAM...¶

I went with the Supermicro board based on a recommendation from a friend. Supermicro's site is really good. It has tested compatible hardware lists and, it turns out, a knowledgeable person behind their online store's chat feature. The problem that I ran into when building this machine is that the compatible RAM was really hard to find. I didn't realize that and ordered the mother board in my first batch of components. When I finally went to look for RAM, I failed to notice a single character difference between the EU21 version that I needed and the ER21 version that I ordered first.

I assembled the machine, plugged everything in, and turned on the new server. Then it beeped at me. A lot. After some troubleshooting, re-seating the RAM and finally realizing that I ordered the wrong stuff, I exchanged what I ordered with what I needed. The EU21 RAM worked perfectly.

What's next¶

The hardware is assembled. Ubuntu 16.04 Server has been installed. The next step is configuring the server to be the backup solution for the entire house and meeting my other goals. I'll have a few more posts in this series on how I accomplished those goals. Stay tuned!

Note: The link to SERT Data Recovery was removed in March of 2023 because it failed multiple link checks with multiple errors. While I am still very happy with the work they performed in 2018, as of 2023 I don't feel comfortable linking to a site that continues to fail basic SSL certificate checks and forces browsers to warn users against visiting the site.

Well, there goes my data...

2018-01-27T00:04:00-06:00

Introduction¶

October 2017 started nicely. The weather was still good. Fall hadn't really arrived yet, though I was still raking some leaves. Halloween was approaching and soccer season had just ended so I had time of weekends to do a few more projects. Then in mid-October one of my hard drives died. It was sudden. There were no SMART warnings. There wasn't any weird sounds.

In fact, the only indication that there was a problem initially was that one of my running applications couldn't access a log file. I didn't think any of it. I took a few minutes to install a Windows update that had been nagging me for a day or so and rebooted the machine. A reboot on my desktop normally takes twenty-ish seconds.

As the reboot stretched into the fifth minute, I realized there was a problem. However, I was initially getting ready to blame that Windows update.

A Data Disaster¶

Finally, the reboot finished and I logged into the machine. Immediately, I received alerts that files couldn't be accessed and programs couldn't start. I tried to launch Firefox and was told it couldn't be found. "Well, this is bad," I thought. Turns out that was an understatement. After a short investigation, I found that one of my hard drives wasn't being detected by Windows.

On my desktop, I have three hard drives. The first is, a decently sided solid state machine that hosts the operating system. The second is a large solid state drive that holds my game installs. The third is a large spinning hard drive (Seagate), which is where I install software and hold my data. The missing drive was the large one that holds my data.

Panic set in. However, I had just backed stuff up to my trusty external hard drive. "At most I'll lose a week's worth of stuff. And, I still have the pictures on phones and camera SD cards," I said, trying to comfort myself. I also new I had at least two brand new and unused drives identical to the one that had just had a problem. Those were the result of cheap hard drives when I build my desktop and the promise that one day I'd build a true backup solution for the house. Yeah, hadn't done that yet.

After messing with the missing drive by unplugging it and rebooting (bringing back normal boot times), swapping it to another SATA port ("hey, maybe it's the motherboard"), trying to access it in a Linux LiveCD and plugging it into an external drive, I concluded that it was dead. I pulled it from the desktop, resisted smashing it to pieces in frustration and plugged one of it's unused siblings in.

Windows booted right up. I had an empty hard drive. I'd be spending a day reinstalling software, but I could handle that. I plugged in my backup external drive and started copying data back into place. I breathed a sign of relief when all the family pictures were restored. I lost a weekend, but everything looked good. I went to bed Sunday night happy and planning on finally building that backup solution. "That was almost a disaster!"

The Problem Appears¶

On Monday morning, I saw the family off to school and work and then settled in for my day. I went to pull up a personal project that I'd work on for an hour or so before the work day began. Well, it turns out that one of the things I stored on that drive that hosts all my data was my personal projects. It also turns out that I never actually backed up my personal projects.

More than decade worth of personal programming projects, horrible attempts at graphic work (seriously...graphics aren't my thing), notes for games that I'd love to design if I ever get the time were gone. I dug through my most immediate backup. I found old drives - because "you never know when you'll need it" - and dug through those. Nothing. I have no idea how I missed this rather important directory in all my various manual and semi-automated backup routines over the years.

I'd either have to suck it up and lose that history or get serious about data recovery.

Data Recovery Attempt¶

Back in college, I was manager of the university's help desk. One of the tasks we performed was data recovery for students. Usually this was on flash drives that had "suddenly deleted a paper", but occasionally we'd have to work on a hard drive too. Our recovery efforts were limited to a quick run of a handful of applications that attempted to recovery deleted files. In the world of stressed students, this was all that was usually needed. If you could recover that term paper, thesis paper or dissertation, you were a hero.

That's what I tried. I hoped it'd work. Windows still couldn't see the back drive. A Linux LiveCD could see that a drive existed, but couldn't find any data. That was still progress. I ran software for almost two days as it tried to find anything for me to recovery.

Nothing. Nothing at all. It looked like some how erased everything from the drive. I was back to my choices - lose the history or find a professional.

I went with a professional. I couldn't lose that much of my history. I still those projects for both personal and professional work.

SERT Data Recovery¶

I went with SERT Data Recovery. The technician over the phone explained their process. They asked questions that made me confident in their ability to at least diagnose the problem, and most importantly were thousands of dollars cheaper than a competitor I spoke with. They explained their pricing structure (they provide a quote and it won't be a giant range), and pricing for replacement parts and allowed me to either send in an empty drive or pay for one that my data would be returned to me on. They answered my questions.

They also didn't scold me for doing the single most damaging thing to a drive. I had done it repeatedly. I had turned the bad drive back on. I had turned it on and let it spin for days as I attempted to recover data.

I shipped them the drive. After the initial diagnosis came back - one of the platter heads had stopped working - I paid for the replacement part and waited. Eventually the data was recovered with a 100% recovery rate. I was ecstatic. My data was shipped back to me and I quickly transferred my projects to the new drive.

My only complaint in the entire process was that the recovery process took almost seven weeks. I shipped the drive at the end of October and received the data back the week before Christmas. However, all of my data was recovered, so it's a pretty minor complaint.

What's next?¶

It's been a little over a month since I've transferred my data to a the new drive in my desktop. I've manually run my back up process two or three times. I'm pretty much where I was at the beginning of October. Crossing my fingers that when I get on the computer in the morning, all my data will be there. That's not acceptable any more.

I'm finally building the backup solution. It'll be a huge (for a family) NAS. The goal is to be able to back up from Windows, Ubuntu and a couple Android phones automatically.

The next entry will cover the device itself and my choices for certain hardware. I'll follow that up with an article on the software set up to get the entire thing working.

Collecting Diamonds on Stack Exchange

2017-08-18T10:06:00-05:00

Introduction¶

It's been over two years since I first ran for moderator on Stack Overflow. I've run for moderator three times, previously on Stack Overflow. In each election I've done better, coming in fifth in the third election. Well, it's been a little over 8 months since the last one and new moderators are needed again. I decided to run once more with the knowledge that if I lost, I probably wouldn't run again in the next election.

Nomination Phase¶

The nomination phase started off as usual, with a handful of users posting their nomination. This time there were 12 candidates, meaning there would be a primary to narrow it down to 10 before the final election. My nomination was the following:

I'm Andy. I've answered the questions posted by the community here I encourage you to take a look.

Why should you vote for me?¶

I've been a moderator on Community Building for several years. I was appointed to a moderator position on Hardware Recommendations. I know the moderator tools and have worked with the current moderators.

I'm active in the review queues (I am the top reviewer in the Low Quality Post reviewers, with over 26,500 reviews in this queue). I also enjoy the other moderation aspects of Stack Exchange.

I believe that moderation can be tool assisted. I've helped to flag a sizable percentage of comments on Stack Overflow. I've helped build the community spam detection bot. These types of tools help eliminate the obvious bad stuff so that moderation time can be spent on the less obvious stuff.

I have a history of good community moderation, am here consistently, and believe I can help the current team.

Nomination Reflections¶

An astute reader may have noticed this is pretty similar to previous nomination posts. There are a couple major differences though. The first thing is that I am the the number one Low Quality Posts reviewer on the site. I am pretty proud of this particular statistic. It shows just how much work I've done during my tenure at Stack Overflow to improve the quality of the site. The unfortunate thing is that I'd probably lose this position as a moderator because they don't sit in the review queues.

The other major change was that I had picked up a moderator position on Hardware Recommendations. That happened at the end of June. Hardware Recommendations is about ten times the size of Community Building (a site I've moderated for several years). It's also a couple orders of magnitude smaller than Stack Overflow.

Primary Phase¶

The most exciting part of the election season is the primary phase. The community can see the scores of users over time and have built tools to watch those numbers change over time. It turns out that this time, my numbers were really high.

There were plenty of good people in the election this time. One interesting thing that I found was that a lot of candidates, like me, were supportive of automation. Several users utilized bots that posted low quality content to various chat rooms. This is a big change from previous elections. It was a welcome change. I think that automated quality content checking can help a lot.

Election¶

The election ended on August 1st. (A busy day for me, apparently). It was a close election. Most surprisingly, no one won in the first round with everyone else picking up carry over votes to get second. I think that speaks to the quality of the candidates. After 8 rounds in OpaVote, both Cody Gray and I were elected the two newest moderators on Stack Overflow!

Post Election¶

The election ended a few weeks ago. I handled more moderator flags in my first hour as a Stack Overflow moderator than I had at both Community Building and Hardware Recommendations combined. What I'm saying...Stack Overflow has a ton of flags that need to be handled. In good news, since the election, we have gotten the moderator queue down from about 1,100 flags to about 75 at any given time. I doubt it will stay that low, but it's still nice to see that I was immediately helpful.

Finally, since the election I turned off the comment flagging bot. It had been used for just over 3 years. The community is currently debating whether or not it should run under a moderator account. The thing that I am finding more interesting about this discussion is that the community seems to agree it's helpful, respects the 99+% accuracy, would love for Stack Exchange themselves to run this tool, but doesn't want the bot to run with moderator privileges. There is, however, a very sizable portion of the community that does want this done under my account. We'll see how this plays out, but I'm hoping to be able to use the bot again soon.

Stack Overflow's Problem - Feedback from an experienced user

2017-05-22T23:45:00-05:00

Introduction¶

Stack Overflow launched in 2008. As it nears it's 9th year of operation, I am afraid the resource that I depend on is losing it's way. Stack Overflow launched after I graduated college. I can't imagine how helpful it would have been during that time period, but it's been invaluable in my professional career. I joined the site about a year after it's public launch, in October 2009.

In that time, I've gone from lurker to participant to moderator candidate (several times). I know Stack Overflow and Meta Stack Overflow. I am a moderator on another Stack Exchange site and have a good understanding of how the network operates. I also am one of the most prolific reviewers in the Stack Overflow Low Quality Posts review queue and have built several applications that work with the Stack Exchange API. I am a power user and know the network and the community.

With those credentials out of the way, I want you to understand that I am active on the network. I am in good standing on Stack Overflow and am not a disgruntled user. I am a concerned user. I am getting more and more concerned that Stack Overflow - the company - is losing it's way.

This post isn't another "Stack Overflow sucks" post (Google if you're curious). I'm going to present a few areas that I'm concerned about and hopefully provide either my suggestions for improvement or acknowledge that I don't know the solution but want the team to be aware of in the future. I still believe Stack Overflow is an incredible resource. I'd just like it to fix some of the perceived missteps that have occurred over the past two years.

What's going wrong?¶

In the past two years, Stack Overflow has made several changes that the established community hasn't liked. Some of these changes still are not liked. These changes include the Teams feature, the new top bar, the Stack Overflow (versus existing Stack Exchange) mobile app, and Documentation. There have also been minor missteps that have caused a rift between portions of the community and the company. These areas include multiple political stances, and a number of post quality improvements that haven't been made.

Each of these, separately, is a minor problem that could be worked through and moved on from. The problem I'm seeing is that taken together, all of these are causing a rift between users, power users and the community.

Let's work through each of these items.

Teams¶

Teams was announced in October 2015 and clarified a week later. It was then shut down after nine months. The page it used to go to now has the following blurb (emphasis is mine):

Teams was in private beta for almost a year with 295 teams created and while we believe in its potential value, after a lot of consideration we’ve decided to un-ship the idea for the time being. We’ve realized that making a successful version of the Team page, as we originally proposed would ultimately take more time and resources than we want to devote to it. Our resources are currently allocated on projects to enhance and improve quality on Q&A, Documentation, and Jobs on Stack Overflow, as a result we don’t have the dedicated developers to get Teams to its fullest potential. The intention was to add more features to Teams, but we never expanded it to anything beyond a team description.

The emphasized section sounds good, except that the one section that is taking up a majority of time (Documentation) has it's own major issues. The area that many power users want developers to focus on is Q&A.

The problem with Teams, and many of the projects mentioned in this post, is that this was a feature that removed focus on areas the community wanted improved. Meta Stack Overflow has been asking for improvements to reduce the number of low quality posts for years. Moderators have been asking for better tooling. The review queues are overflowing with tasks and the number of users performing reviews isn't high enough to keep up. Teams was built without a true end goal and users weren't entirely sure what to do with it. This was the first in a series of mis-steps that continue to plague community interactions when new features are announced.

Top Bar¶

The new top bar was announced in November 2016. It went through a handful of iterations before being released in mid-February 2017. During the iterations users provided feedback. When initially released, though, much of this feedback felt ignored. Things like notification overload, stickiness of the top bar, and hidden review counts were all mentioned during the three months of testing but not implemented until the change was live to millions of users.

After three months of usage, a larger problem was noticed. One of the review queues was constantly full. One of the changes that was made with this top bar was that the "Review" button no longer linked directly to the "Suggested Edits" review queue. Now it went to the page showing all review queues. Users that used to click once to get to a review queue were now presented with a list of queues to work in. Some of these queues are much more time consuming that others. It turns out the number of reviews being done has decreased significantly since the top bar was implemented.

The spike in reviews in February 2017 is when the new top by was released. Since that release, the number of reviewers has plummeted. This has been attributed to notification fatigue and not linking users directly to the Suggested Edits queue.

Three months after implementation, it took the community asking for results (disclaimer: I asked the question), to find out how the top bar has been performing. It turns out that the top bar is performing decently well compared to what the developers were expecting, with the exception of fewer review tasks being performed.

The problem with this project, is that it's felt unneeded and has materially impacted one of the quality control features of the site. There is still a vocal group of users that don't like it because it doesn't match the rest of the network. Several are concerned about the review queue problem. Experienced users felt that they were ignored during the beta tests. Users provided feedback and examples of problems and it was only after implementation when millions of other users experienced the same thing that these changes were made.

Mobile App¶

A recent announcement (as in last week, at the time of this post) announced a new Stack Overflow mobile application. The community response was not positive. Users asked why a new application was being built when one already existed (the response was "branding"). Users asked why the new app was less functional than the existing one (it's limited to Stack Overflow versus the entire Stack Exchange network). Users asked why it took a year to develop and why the existing application hasn't received bug fixes in that year.

I think one of the most disappointing things about this is a response I received in the comments from the VP of Engineering:

@Andy You're right, it wasn't worth a year. There's a long, sad story here, but it was originally expected to only take a few months and... well, here we are a year later. We decided to go ahead and launch and see what we can learn, and we'll reassess from here. – David Fullerton? May 17 at 16:26

Another user expressed the dissatisfaction in a very pointed way. They provided a list of features that the community has asked for over the years that many feel have been ignored. The VP's response to this wasn't encouraging either:

I appreciate that there are a lot of issues on Stack Overflow that need to be addressed, and maybe we haven't been responding to them as quickly as we should. But Stack Overflow Q&A is a big, established product, most of the problems left are hard, and we can't let maintenance become the only thing we work on or we'll just slowly run out of money and go out of business. We are trying to both maintain Q&A and solve new problems for developers and reach new audiences. The latter is hard, and maybe we'll fail on a lot of our ideas, but we're not going to stop trying. – David Fullerton? May 17 at 21:10

This sounds like work on the Q&A side is feature frozen at this point. They are done innovating in this area and instead are focused on drawing in users via other features - like Jobs or Documentation. Multiple times in the comments the new app was promoted as being able to use the Dev Story or Jobs features in the future. Perhaps it's just me, but I don't apply for jobs via my phone. That doesn't seem like a good way to really put the effort needed into a cover letter or application.

Documentation¶

Now we've reached Documentation. This is the project that's sucked up development time over the past two years. This is the project that Stack Overflow developers are defending tooth and nail and the community has all but given up on.

Documentation was announced back in August 2015. It's had a ton of updates since then. It was met with initial enthusiasm but that quickly turned around. When the system launched for all users, one of the first complaints was that the reputation generated via documentation was doing back things to the main Q&A site. This resulted in a massive recalculation of reputation and resulted in many users losing a lot of their internet points.

Another change that was announced with the introduction of a new review queue for documentation. Initially, developers didn't expect the low quality to begin immediately, it seems. Long time users weren't surprised. Now we've reached the point where the company is realizing that the users knew what they were talking about. Documentation is undergoing a massive change, to the point that much of it is being completely redone - not fixed - scraped and redone.

This project has years worth of feedback from the community that has been ignored. It is the black sheep of Stack Overflow and many community users feel that quality of the content is lacking so badly that they don't participate any longer. This feeling isn't helped that many users have been explaining why things aren't working for a while and it's only after two years the developers are starting to realize the private beta testers, public beta testers and experienced community users mentioned many of these problems. In this particular instance, the company took Jeff Atwood's advice (co-founder or Stack Overflow) to not let the community tell you what to do to heart. To the company's surprise, a community of developers that live in programming documentation had decent thoughts on what does and does not work in programming documentation.

Politics¶

For many users, the lack of true social features on Stack Overflow and across the Stack Exchange network has been a good thing. You can't easily follow a single user, you can't send private messages to a user, and you can't really do anything on the site that isn't public to everyone. The focus is on content, not opinions or social interactions.

This breaks down once and a while though when a big political thing occurs. The two most frequently mentioned instances are the response to the Obergefell v. Hodges Supreme Court decision and the response to President Trump's initial immigration executive order.

Both of these caused huge uproars within the community when the company took a stand. These stands caused problems due to users holding opposite political views, users not wanting politics on their programming site, users not wanting to deal with the drama caused by the vocal members of the other groups. This led to an apology. The community wasn't pleased with this apology. Users mentioned in multiple answers to this apology that they don't want the company to post such political agendas on the site. It's out of place for a programmer community. Both of these instances are still brought up on Meta when the community feels that the company is imposing on them.

I don't really have advice or suggestions on this problem other than "I don't want to see this on Stack Overflow, because these hot button issues cause so much drama that nothing gets accomplished". These posts grind Meta and chatrooms to a halt while everyone expresses their opinion on the post, on the post's existance, on one another and on related issues.

Quality Improvements¶

Finally, the community has been asking for years about ways to improve the quality of posts on the site. Stack Exchange started a project to improve the quality back in October 2016. This generated 80 different suggestions on how the community sees "quality improvement" taking place. Since then there haven't been any updates on the status of this project or even subprojects.

This was brought up during all of the projects listed above by long time users. The hope was that this quality project would help. Being ignored hasn't brought any good feelings. The lower quality has been measurable and seen less participation from experienced users.

The Fix¶

Above I've pointed out several issues that I've seen over the past two years. These issues are part of a bigger problem though. It seems that Stack Overflow doesn't know to how handle it's community size any longer. It's in the top 300 sites visited in the US and receives half a billion views globally per month. Couple this with the fact that they don't have a sustainable business model yet and have a sizable team with good benefits and they are getting concerned.

Q&A is what built Stack Overflow, but it isn't enough to sustain them. Thus, the other projects are being created. Unfortunately, in this process, it seems the company is forgetting it's existing user base at the expense of expanding to new users. Existing users are getting frustrated with the lack of quality improvements, being ignored and not having changes that benefit their use cases.

Documentation has taken up a giant chunk of time and developer effort and it's all been wasted. The announcement that it is being redone has been met with "thanks" from the community, along with warnings to consider that "quality" problem. We'll see how it plays out, or if that quality issue is ignored like their own Quality Project.

Which brings us to the final point I want to make. I think the feeling of Q&A being "done" is the biggest problem I've had with Stack Overflow over the last year. New features aren't being built in that space. Instead of focusing on some of the "hard" problems, the company is throwing stuff at the wall and hoping something will stick. Unfortunately, the four biggest projects in the last year have either failed completely (Teams, Documentation, Mobile App...perhaps) or have significant unintended consequences that isn't helping the quality issue users have been reporting for years.

Power users, the underlying community that has put time and effort into growing Stack Overflow to that it is today, is feeling ignored. It is only after months or years long experiments fail that community opinions are finally validated or considered. Users have expressed concerns in each of the above projects repeatedly. Yet, those opinions were not addressed. The silos that the developers have built around themselves are causing the company to lose touch with it's community. This is being done at the expense of alienating the users that care and the cost of developer time.

Users want a high quality site with answers to their questions. Even new or potentially new users want this. Stack Overflow continues to avoid dealing with that problem because "it's hard". The unfortunate thing is, this is costing the site users that return to provide more than one answer.

This chart is showing the number of answers provided per month by different types of users. Users that have provided more than 100 answers, between 11 and 100 answers, between 2 and 10 answers and only a single answer. The furthest data point on the right is an artifact of being an incomplete month. From this chart, we can see that the only group that has continued to rise are users that provide a single answer over time. The other groups took a steep drop in April 2014 and haven't recovered since then. The number of experienced users that are participating has dropped.

What happened in April 2014? That's been answered by a Stack Overflow community manager. The theory is that users aren't getting answers to their questions and due to being ignored they never return to participate further in the site. Another community manager also provided an answer:

Starting around 2013 and peaking around March, 2014, people began asking fewer interesting questions. That lead to a decrease in voting on questions and fewer answers being given. Since the feedback on these uninteresting questions was discouraging, people began asking fewer questions on the whole. Meanwhile, truly poor questions continued being asked with little regard to negative feedback.

Stack Overflow users began noticing increasing numbers of truly awful questions and decided, rightly, that downvoting and refusing to answer them is the best remedy. These questions fit broad categories of awful and users began withholding votes from questions that were not themselves awful, but bore some of the markers of awful. Fewer of these questions got answered and askers of mediocre questions did not see any point in trying to improve.

Thus began a slow spiral downward. Not all is lost though, because there are the upticks. I hope it's enough to break the cycle, but I really fear that something needs to be done about this quality issue. This is the issue that is brought up by the experienced community.

Where to from here?¶

I continue to invest my time and effort into the community, but even as an active user who really wants the company and community to succeed, it's getting harder and harder to ignore that those of us that have been around for years are not being listened to any more. We're being treated as the grumpy old person that grumbles about the way things used to be. Our experiences on the site are brushed aside as being unhelpful to new users. That completely ignores that fact that we are still trying to reach the goal on which Stack Overflow was created: "With your help, we're working together to build a library of detailed answers to every question about programming." To do this, we need high quality questions and answers so that we can actually provide help to all users. I think this is the biggest challenge that Stack Overflow is going to face in the next 18 months.

I want Stack Overflow to continue to grow. I also want Stack Overflow to have high quality content. I think my experience and the experience of others can help build the features to accomplish this. We just need Stack Overflow to refocus on the Q&A portion of their network again.

Third time's the charm?

2016-11-06T22:54:00-06:00

Introduction¶

Last year, I ran for moderator (twice) on Stack Overflow and didn't make it through the primaries. I came close on that second run. Now, a year later, and a year more experienced, I'm going to try again. This post will document my progress through the election cycle.

Spoiler Alert: I didn't win. The rest of this post details my thoughts as the election occured though.

Nomination Phase¶

The election this year took a slightly different route than last time. In previous years, the election was announced at the same time as the call for nominations began. Users had a week to nominate themselves, then we answered a series of community provided questions during the primaries, then the final election.

This year, the election was announced a week in advance of nominations. During the week, a call was put out for Community questions. When a nomination was posted, the answers would be posted as well. This change was made due to how much the community needed to read during the primaries. The primaries were only a few days long and the Q&As were usually ten questions for each user. When a primary has 20-30 nominees, that is a lot of reading that was expected in a short period of time. By bringing this phase forward, now the community has the entire election cycle to read and interact with the nominees.

I provided one question that was used in the final selection of questions. I mentioned last time that I thought it was a great question, so I suggested it again:

Do you have any Meta posts that you're particularly proud of, or that you feel best demonstrate your moderation style?

My nomination¶

My platform isn't all the different than the last two times.

Hi Everyone, I'm Andy and I'd like to be a moderator for you and Stack Overflow. I've answered the questions posted by the community here. I encourage you to take a look.

Why should you vote for me?¶

I've been a moderator on Community Building for over two years. I know the moderator tools and have worked with many of the current moderators. This interaction will continue as a new moderator here.

I have a lot of helpful flags. A decent percentage of these are on comments, but not all. I'd like to help keep the site clean without adding to the current moderators' work load.

I'm active in the review queues (currently holding 5th in Low Quality Post reviewers of all time), provide edits to posts, answers and enjoy the moderation aspect of Stack Exchange.

I have a history on Meta.SO that shows I'm involved in the meta aspect of the site as well.

I enjoy the moderation aspect on Stack Overflow (and Stack Exchange in general). I have a history of good community moderation, am here all the time and believe I can help the current team.

During the first full day, I'm gotten positive responses to this post. My two favorite, so far, are:

Andy's work around comment flags has been very impressive. I'm definitely curious to see what his thoughts on the mod queue are and if we could incorporate some of his work permanently on the site. Better identification of flags is something that would be very nice to have permanently. - bluefeet Stack Overflow Community Manager

and

There are always some nominees for this position who are very active, some who have good judgment and cool heads, and some who innovate with their approach to community moderation. Andy is the rare candidate who very clearly checks all three boxes. As a user on SO for 3.5 years, a moderator pro tempore on Engineering SE for 1.7 years and an early participant in the Community Building SE beta, I strongly support this nomination. - Air Moderator on Engineering.SE

My candidate score this time is an impressive 39/40. This is up six from a year ago, and up ten from my first run. The one missing point is due to missing the Refiner badge. I believe the reason for this is because of my workflow. I, generally, don't edit and answer questions at the same time. If I'm answering, I'm not in "edit" mode. If I'm editting, I'm usually in "moderation" mode. It's something I'll work on. I'm 38 out of 50 questions there, so I'll get it soon enough.

Candidate questions¶

None of the questions were that surprising. With the added benefit of a week to prepare answers prior to nominating, I am very pleased with my answers. Two answers have generated a bit of discussion though.

A 10k+ user regularly has their comments flagged as "rude or offensive" or "not constructive", to the tune of 4-5 flags a day. No comment by itself is particularly offensive, but their general tone causes them to be flagged by multiple users. You've contacted them privately about this, but they believe that they aren't doing anything wrong and that people are being too sensitive. The flags keep coming in on their comments. What, if anything, do you do next?

My response is:

No one has an exemption from the Be Nice policy. I think the first step is to understand why nothing has already been done about the user. 4-5 a day seems like the user has moved beyond the "nuisance" stage. I think a temporary ban is appropriate, with another explanation as to what is expected when interacting with others. While some users are more sensitive than others, a stream of this many flags across an extended period of time doesn't lead me to believe the problem is with the community users.

The point raised in the comments was that I was rushing into banning the user without commuicating first. I disagree with that, and explained that they've already been contacted privately and ignored those warnings. A ban is the next step in getting the user's attention. I was told this would be "humiliating" for a high rep user. Again, I disagree and believe it's not humiliating, but educating the user.

The second question that generated some discussion was:

You impose a temporary ban (say 1 week) on a user for what you judged as reasonable and valid reasons (the user gets notified by email of your action and the reason). The user replies to your email acknowledging the transgression, says they won't do it again and asks for the ban to be lifted. The user sounds genuine. Do you remove the ban? Do you even reply at all? Explain your reasoning. The context of this question applies to longer bans too. If it helps get the juices flowing, consider the situation of a second offence for the same behaviour, which has a default ban period of 1 month.

My response:

I have two answers for this question, based on the user's history. If this is a first offense, up to this point the user hasn't been pushing limits and attempting to disrupt others, and the ban isn't related to voting fraud, then I'd be willing to remove the ban. Sometimes a ban is put in place to get the user's attention. Once the situation has been resolved, the ban is no longer appropriate and should be removed.

On the other hand, if the user has a history of crossing the line and looking for a reaction, or if the ban is related to vote fraud, I'd simply not reply and the user will return in a week. Stack Overflow has enough "voting irregularity" bans that I imagine the responses to such bans are all similar (and invalid). I see no reason to change that policy.

The push back I recieved on this was that I was letting a user off the hook by unbanning them. I argued that unbanning has been done in the past. Sometimes the ban is needed simply to get the user's attention and start the conversation and explain that why they are doing is wrong. If the user abuses the trust at that point and repeats the behavior, then the longer ban is completely justified. A bit of compassion isn't a bad thing.

Primary Phase¶

There are 12 nominees, so a primary will occur. Once again, the primary phase will reduce the number of candidates in the final phase to 10. With so few being eliminated this time around, it feels a little unneeded. The primary will last for a few days and during that time users can vote candidates up or down depending whether they believe the nominee should be a moderator. I'll return in a few days...

Primary Results¶

The Primary phase has ended and the final election has begun. I ended the primary in 5th place, securing a position in the final election. I have a sizable margin between my position and sixth place as well. One other stat that I'm rather proud of: I received the fewest number of down votes of any candidate.

On to the election!

Election Phase¶

The election lasts for several days and covers a weekend. We'll see how it turns out in a few days.

Election Results¶

Well, the election has concluded. I didn't secure on of the three positions for moderator. I finished in 5th place, with my elimination propelling second and third place to a victory. I was eliminated in the 10th round of the Meek STV process.

Good luck to the new moderators!

Post Election thoughts¶

This election started differently than the previous two I've run in. This election was announced a week in advance and solicited community input for questions for the candidates. I think this was a good change. The element of surprise in the previous two made it much more stressful. Additionally, by having the questions available at the start of the election - instead of at the start of the primary phase - I was able to better answer the questions. Previously, the questions would be available at the start of the primary phase. With the amount of reading needed to get through one candidate's answers, let alone all of them, I imagine that many people didn't read all of the responses.

The other nice thing about this lead time, is that I had time to get my answers read for when I posted my nomination. By posting the questions and answers at the same time, I was able to have my responses available the entire time. Score-wise, on the questionnaire, I did much better than my opponents. I think a big reason for this is that I have my responses posted as soon as my nomination was posted.

One question this time, though, seemed to split the candidates. I mentioned it previously, but it was regarding potentially removing a temporary ban.

You impose a temporary ban (say 1 week) on a user for what you judged as reasonable and valid reasons (the user gets notified by email of your action and the reason). The user replies to your email acknowledging the transgression, says they won't do it again and asks for the ban to be lifted. The user sounds genuine. Do you remove the ban? Do you even reply at all? Explain your reasoning. The context of this question applies to longer bans too. If it helps get the juices flowing, consider the situation of a second offence for the same behaviour, which has a default ban period of 1 month.

I was one of two candidates that explicitly stated we'd consider removing the ban. A third user didn't state it explicitly, but did say they'd consider it. I was surprised by the harsh tone the others took, especially since there is a lot of previous discussions on Meta where the outcome is the moderators or community managers removing the ban. I was happy to see that the other candidate who said they'd consider removing the ban get elected though.

I still believe that removing the ban is a valid option. Especially because their next ban would be much longer if they broke my trust.

We'll see when the next election on Stack Overflow is, but with three new moderators and no resignations, I suspect it'll be a while. I'll consider running again then.

I'm running for moderator on Stack Overflow again

2015-11-18T09:38:00-06:00

Introduction¶

In April, I ran for moderator on Stack Overflow and didn't make it through the primaries. That's ok though, there were several very good users that did get elected. In a surprise announcement, though, Stack Overflow is running a second election this year. This is the first time this has happened since 2011. I'm still interested in a position and I'm still active in the community, so I'm going to run again. This post will follow the process.

Nomination Phase¶

Like last time, the nomination phase began with users throwing their hat into the ring. Nominations were slower and fewer this time. Only 19 nominees, so no one was eliminated due to low reputation. Several users from the last election are rerunning too.

My Platform¶

My platform hasn't changed much since the previous run. Below is my nomination post. This time, I tried to pull emphasis off the automated script by putting it lower on the list of things I've done and instead focused on the moderation tasks I do on Stack Overflow and the work I've done on Community Building. We'll see if it works.

Hi Everyone, I'm Andy and I'd make a great moderator for Stack Overflow.

Why vote for me?¶

I'm active in the review queues (currently holding 10th in Low Quality Post reviewers of all time), provide edits to posts, answers and enjoy the moderation aspect of Stack Exchange

I've been a moderator on CommunityBuilding for nearly a year and a half. I know the moderator tools and have worked with several of the current moderators. This interaction will continue as a new moderator here.

I've built an automated script that continues to handle noisy comments very accurately.

I have a history on Meta.SO that shows I'm involved in the meta aspect of the site as well.

I have a history of good community moderation already. I enjoy the moderation aspect on Stack Overflow (and Stack Exchange in general). I deal with users with respect, even if our opinions on an issue differ.

With this, I received my "candidate score". It was 33/40. Not the highest, but better than last time. The score wasn't mentioned in April. I am not expecting it to be an issue this time either.

Primary Phase¶

Updated November 21, 2015

The primary phase is in the third day. In day 1, I was hovering around 9th/10th place. Overnight, between days 1 and 2 though, I dropped down to 11th. I've been sitting here consistantly for a full day now and, while still gaining votes, I'm not gaining as fast as 10th position. It appears I may not make the cutoff by Friday's deadline. While disappointing, there are a few things that I came away with that I'm very happy about.

In the last primary, I revieved 1,492 positive votes. I've surpassed that already. I've over 2,100 currently. I'm pleased with that upswing. I was also more prepared for the questionnaire portion of the primaries this time. I've gotten the second highest number of upvotes on my responses. Several of the questions were similar to last time, but there are a few that I think should be included in the future elections.

Questionnaire¶

This first question is a great post for candidates. It allows them to show off their involvement in Meta and show their best work. For users, it gives them a sense of how a candidate interacts with the community. I am very surprised that several candidates list only one or two posts. This seems to be doing a disservice to themselves.

Do you have any Meta posts that you're particularly proud of, or that you feel best demonstrate your moderation style?

My response to this question:

I'm proud of several of my posts both here on Meta.SO and on other network sites I participate in. Here on MSO, I have two questions that I am proud of:

Can a machine be taught to flag comments automatically?

I estimate 10% of the links posted here are dead. How do we deal with them?

In both of these, you can see that I care about quality on Stack Overflow. I've spent time analyzing the problem, as I see it, and present my findings to the community. I participated in the discussions that both posts generated and continue to run the bot to this day.

Elsewhere on the network, my participation in meta has helped to shape communities. For example, on Hardware Recommendations, my meta post about "What type of hardware is allowed" helped to set the scope of what the community accepts as on topic hardware. I've also helped to set up the high quality guidelines for questions and argued against certain types of tags and hardware.

With all of these, I've presented my arguments and logic and strived to remain professional. I believe the community on HardwareRecs has seen that as well.

As a moderator on Community Building, I've been involved in many discussions. I was involved in the discussions to rename the community from Moderators.SE to CommunityBuilding.SE. I've been involved in discussions about slow growth of the community. I've also presented arguments that go against other moderators, and walked away still feeling like a moderation team. (Go communication!)

Finally, on OpenSource, I made a post about how moderators had implemented a policy to watch the reviewers. It was similar to the long removed "flag weight" option that used to exist. I believe the post was presented in a way that questioned the decisions of the moderators, yet remained professional.

With all of these meta posts, across the network, I think you can pick up on my moderation style and personality. I like data and I try to present my thoughts in a way that is understandable to all. I'm also willing to speak my mind.

This second question I struggled with for a bit. I've had ideas on how Stack Overflow/Stack Exchange could improve, but what did I want to present in this response.

If you could add/revise one Stack Overflow policy/guideline, what would you change? Why would you change it, and what would it mean for the community?

My response to this question:

At the risk of talking myself out of a position, I think more community moderation would help the problem that Stack Overflow has with scaling moderators. There are a couple areas that I think would work well in opening this to the higher reputation users

Comment flagging: Comments can be removed if enough users flag a comment. If not, a moderator needs to handle the flag. Instead, opening this as a review queue can remove a lot of this burden from the moderators. Users could handle all but the "Other..." flag. There may be guidance needed on the "Obsolete" one due to the difference between "obsolete comment" and "obsolete code block" differences.

Audit Review reviews: On Stack Overflow, we get a decent number of disputed audit review posts on meta. There may be a way to get users with a history of passing both audits and good reviews involved in dealing with these disputed audits. The idea would be to say whether an audit is good or not.

These changes, and other areas where the community could be leveraged for moderation tasks, helps to remove the burden on moderators. Handling 2,000 (and growing) flags a day means that something needs to change. Moderators are exception handlers. They should be handling the cases that are exceptional - not comments that are no longer relevant.

For the community, this would be more involvement with the moderation aspect. Users would be able to more quickly clean up a comment thread. Flag it and it appears in the review queue. From here, the moderators don't need to be involved. The downside of this is that it adds another queue for users to be involved with.

Primary Results¶

With the primaries over, I ended with 2483 positive votes. This put me in 11th place. Sadly, this was not enough to get into the election. I was 185 votes shy of over taking 10th. Good luck to the candidates that made it.

One of the tools that came out of this election was a way to visualize various data points to compare candidates. I provided a couple notes about outliers various candidates show regarding aspects on the site. I found it interesting to see what each user had "specialized" in.

Election¶

Updated December 8, 2015

The election is over and the new moderators have settled in. We've had our first bout of public drama over one of these moderators actually moderating a chat room too. gasp

Final thoughts¶

I was closer to the top 10 this time, but still missed it. Even more surprising was that the user that ended up in 3rd in the primaries didn't even come close to getting elected. He was eliminated in the 5th round of final STV votes. I still think I'd make a great moderator for Stack Overflow, but I need to figure out the best way to promote myself in the next election.

Link Analysis - Technical Explanation

2015-08-10T23:41:00-05:00

Introduction¶

In my last two posts, I've discussed the number of rotten links on Stack Overflow and a proposal to fix said links. In this post, I'm going to discuss how I performed this analysis.

Set up¶

The database¶

The process began by downloading the March 2013 data dump. I loaded the posts into a MariaDB instance on my local machine. This was accomplished with a very simple script and patience, as the script took a while to run.

load xml local infile '/path/to/posts.xml'
into table posts
rows identified by '<row>';

The data¶

Once this was done, the next step was selecting my random sample of data. I did this by randomly selecting 25% of the days in a year and then pulling all posts for those days across all years of Stack Overflow's existence. The Python script I used to do this was fairly simple:

from datetime import timedelta, datetime
from random import randint
from math import ceil

def random_date(start, end):
    return start + timedelta(
        seconds=randint(0, int((end - start).total_seconds())))

percentage = 0.25
days = 366

dayslist = []
for d in xrange(int(ceil(days*percentage))):
    dayslist.append(random_date(datetime(2008,1,1), datetime(2008,12,31)))

At the end of this run, the days that I cared about are in the dayslist variable. I used that to pull questions and answers from the database that were created on that month/day combination. In the end, this resulted in just over 25% of the total posts being selected. To ensure that I could replicate the results, I also saved the dates that were selected.

Parsing the data¶

The next step was to parse out links from the data. I used the following script to extract anchor text and links from a post.

def links_in_post(post):
    """
    Returns a list of all links found
    :param posts: A list of dictionaries with a 'body' key containing HTML strings
     [
        {
            'body': "<b>This is HTML</b>"
        },
    ]
    :return: A list of tuples containing anchor text and URL
        [
            ('Display Text', 'http://example.com')
        ]
    """
    logging.debug("Extracting links...")
    links = []
    images = []
    regexp = "&.+?;"
    list_of_html = re.findall(regexp, post)
    for e in list_of_html:
        if e in invalid_entities:
            h = HTMLParser.HTMLParser()
            unescaped = h.unescape(e) 
            post = post.replace(e, unescaped)

    doc = html.fromstring(post)
    for link in doc.xpath('//a'):
        links.append(Link(text=link.text_content(), link=link.get('href')))
    for image in doc.xpath('//img'):
        images.append(Link(text=image.get('alt'), link=image.get('src')))
    all_items = links + images
    seen = set()
    unique_items = [item for item in all_items if item[1] not in seen and not seen.add(item[1])]
    return unique_items

The regular expression being utilized, is to strip out HTML entities. This was needed due to weird parsing issues with non-ASCII characters. Fortunately, I wasn't the first to encounter oddities like this. The list comprehension at the end of the function is returning only unique tuples of anchor text/link. I was surprised how often I'd end up with tuples such as ('this', 'http://google.com') in the same post. This uniqueness saved a lot of processing time later.

After all links in a post had been extracted, this information and information about the post itself, was saved to the database. If a post had no links, it was not saved. The database consisted of three tables.

Links - This table contains the base URLs seen in all posts. URLs are distinct. It also contains an ID that will be utilized for linking to the other tables.
Post Links - This table contains information about links in a post. This includes the specific anchor text/link combinations
Link Results - This table contains the results of link status checks

Processing the posts was fairly time consuming, but was able to be parallelized easily. That significantly cut down on processing time.

Checking the links¶

The most time consuming portion of this entire project was actually checking link status. Each link that appeared in the Links table was checked. As I mentioned in my first post, the original idea was to simply send a HEAD request to each URL. The idea was to save myself and the end point a tiny amount of bandwidth. I had over a million links to process. I figured a little saved bandwidth wouldn't hurt.

I turns out this isn't a good idea. When I started seeing larger sites as not being accessible, I go suspicious that something was wrong. These sites were returning status 405 errors. This indicates that the method is not allowed. So, I switched to GET for every link. The next problem I ran into was that many sites didn't like the default user agent of the spider. They rejected requests with 404 and 401 errors. In the end, I got around this by mimicking Firefox on every request.

With those kinks worked out, every link was sent a GET request that looked to be from a Firefox browser. The process would allow 20 seconds per link. If the link didn't respond in that time limit, it was declared inaccessible.

A week later, I repeated the process with anything that hadn't returned a status code less than 400. Once more, on the third week, I repeated this with the failed links. At the end of three weeks, I had a list of sites that were inaccessible to me - on a residential connection - three times over a period of three weeks.

Results¶

The SVG image that I created for the write up was generated with Pygal. The tables were the result of various break downs of the data via queries to the status results.

Wrap up¶

I am rather proud of how the results turned out for this project. I went into it expecting about 15% of links to be broken, but I didn't really realize what the meant. Fifteen percent of 21 million total posts is over 3 million. That's a large number. BUT, it also ignored that a large percentage of posts don't contain links. I failed to consider that in my original hypothesis.

Less than half of my sample had links (2.3M out of 5.6M). Of the 2.3M with links, only 1.5M were unique links. The final result of 10% failed links makes much more sense in this context. Ten percent of 1.5M links means that there are 150K links that are bad.

A proposal to fix broken links on Stack Overflow

2015-08-07T07:34:00-05:00

This post was published by me on Meta Stack Overflow on August 7th, 2015. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

This is a follow up to yesterday's post about how many links on Stack Overflow are starting to rot.

The proposal¶

I propose another hybrid of the previous broken link queue (as was mentioned above in comments and other answers) and an automated process to fix broken links with an archived version (which has also been suggested).

The broken link queue should focus on editing and fixing the links in a post (as opposed to closing it). It'd be similar to the suggested edits queue, but with the focus intended to correct links not spelling and grammar. This could be done by only allowing a user to edit the links.

One possibility, I envision is presenting the user with the links in the post and a status on whether or not the link is available. If it's not available, give the user a way to change that specific link. Utilizing this post, I have a quick mock up of what I propose such a review task looks like:

The Queue¶

All the links that appear in the post are on the right hand side of the screen. The links that are accessible have a green check mark. The ones that are broken (and the reason for being in this queue) have a red X. When a user elects to fix a post, they are presented with a modal showing only the broken URLs.

The Automation¶

With this queue, though, I think an automated process would be helpful as well. The idea is that this would operate similarly to the Low Quality queue, where the system can automatically add a post to the queue if certain criteria are met or a user can flag a post as having broken links. I've based my idea on what Tim Post outlined in the comments to a previous post.

Automated process performs a "Today in History" type check. This keeps the fixes limited to a small subset of posts per day. It also focuses on older posts, which were more likely to have a broken link than something posted recently. Example: On July 31, 2015, the only posts being checked for bad links would be anything posted on July 31 in any year 2008 through current year - 1.
Utilizing the Wayback Machine API, or similar service, the system attempts to change broken links into an archived version of the URL. This archived version should probably be from "close" to the time the post was originally made. If the automated process isn't able to find an archived version of the link, the post should be tossed into the Broken Link queue
When the Community edits a post to fix a link, a new Post History event is utilized to show that a link was changed. This would allow anyone looking at revision history to easily see that a specific change was only to fix links.
Actions performed in the previous bullets are exposed to 10K users in the moderator tools. Much like recent close/delete posts show up, these do as well. This allows higher rep users to spot check (if they so desire). I think this portion is important when the automated process fixes a link. For community edits in the queue, the history tab in /review seems sufficient.
If a post consists of a large percentage of a link (or links) and these links were changed by Community, the post should have further action taken on it in some queue.

Example: - A post where X+% of the text is hyperlinks is very dependent on the links being active. If one or more of the links are broken, the post may no longer be relevant (or may be a link only post). One example I found while doing this was this answer.

I don't think that this type of edit from the Community user should bump a post to the front page. Edits done in the broken link queue, though, should bump the post just like a suggested edit does today. By preventing the automated Community posts from being bumped, we prevent the the front page from being flooded, daily, with old posts and these edits. I think that the exposure in the 10K tools and the broken link queue will provide the visibility needed to check the process is working correctly.

Process flows¶

Queue Flow:

Automated process flow:

Potential pitfalls¶

The automated link checking will likely run into several of the problems I did. Mainly:

Sites modify the HEAD request to send a 404 instead of a 405. My solution to this was to issue GET requests for everything.
Sites don't like certain user agents. My solution to this was to mimic the Firefox user agent. To be a good internet citizen, Stack Exchange probably shouldn't go that far, but providing a unique user agent that is easily identifiable as "StackExchangeBot" (think "GoogleBot"), should be helpful in identifying where traffic is coming from.
Sites that are down one week and up another. I solved this by spreading my tests over a period of 3 weeks. With the queue and automatic linking to an archived version of the site, this may not be necessary. However, immediately converting a link to an archived copy should be discussed by the community. Do we convert the broken link immediately? Or do we try again in X days. If it's still down then convert it? It was suggested in another answer that we first offer the poster the chance to make changes before an automatic process takes place.
The need to throttle requests so that you don't flood a site with requests. I solved this by only querying unique URLs. This still issues a lot of requests to certain, popular, domains. This could be solved by staggering the checks over a period of minutes/hours versus spewing 100s - 1000s of GET requests at midnight daily.

With the broken link queue, I feel the first two would be acceptable. Much like posts in the Low Quality queue appear because of a heuristic, despite not being low quality, links will be the same way. The system will flag them as broken and the queue will determine if that is true (if an archived version of the site can't be found by the automated process). The bullet about throttling requests is an implementation detail that I'm sure the developers would be able to figure out.

Analysis of links posted to Stack Overflow

2015-08-06T07:35:00-05:00

This post was published by me on Meta Stack Overflow on August 6th, 2015. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

TL;DR: Approximately 10% of 1.5M randomly selected unique links in the March 2015 data dump are unavailable. To be more precise, that is approximately 150K dead links.

Motivation¶

I've been running into more and more links that are dead on Stack Overflow and it's bothering me. In some cases, I've spent the time hunting down a replacement, in others I've notified the owner of the post that a link is dead, and (more shamefully), in others I've simply ignored it and left just a down vote. Obviously that's not good.

Before making sweeping generalizations that there are dead links everywhere, though, I wanted to make sure I wasn't just finding bad posts because I was wandering through the review queues. Utilizing the March 2015 data dump, I randomly selected about 25% of the posts (both questions and answers) and then parsed out the links. This works out to 5.6M posts out of 21.7M total.

Of these 5.6M posts, 2.3M contained links and 1.5M of these were unique links. I sent each unique URL a HEAD request, with a user agent mimicking Firefox¹. I then retested everything that didn't return a successful response a week later. Finally, anything that failed from that batch, I resent a final test a week later. If a site was down in all three tests, I considered it down for this test.

Results²¶

By status code¶

Good news/Bad News: A majority of the links returned a valid response, but there are still roughly 10% that failed.

(This image is showing the top status codes returned)

The three largest slices of the pie are the status 200s (site working!), status 404 (page not found, but server responded saying the page isn't found) and Connection Errors. Connection errors are sites that had no proper server response. The request to access the page timed out. I was generous in the time out and allowed a request to live for 20 seconds before failing a link with this status. The 4xx and 5xx errors are status codes that fall in the 400 and 500 range of HTTP responses. These are client and server error ranges, thus counted as a failure. 2xx errors (of which was are in the low triple) are pages that responded with a success message in the 200 range, but it wasn't a 200 code. Finally, there were just over a hundred sites that hit a redirect loop that didn't seem to end. These are the 3xx errors. I failed a site with this range if it redirected more than 30 times. There are a negligible number of sites that returned status codes in the 600 and 700 range⁴

By most common¶

There are, expectedly, many URLs that failed that appeared frequently in the sample set. Below is a list of the top 50³ URLs that are in posts most often, but failed three times over the course of three weeks.

http://docs.jquery.com/Plugins/validation
http://www.eclipse.org/eclipselink/moxy.php
http://jackson.codehaus.org/
http://xstream.codehaus.org/
http://opencv.willowgarage.com/wiki/
http://developer.android.com/resources/articles/painless-threading.html
http://valums.com/ajax-upload/
http://sqlite.phxsoftware.com/
http://qt.nokia.com/
http://www.oracle.com/technetwork/java/codeconv-138413.html
http://download.java.net/jdk8/docs/api/java/time/package-summary.html
http://docs.oracle.com/javase/1.4.2/docs/api/java/text/SimpleDateFormat.html
http://watin.sourceforge.net/
http://leandrovieira.com/projects/jquery/lightbox/
https://graph.facebook.com/
https://ccrma.stanford.edu/courses/422/projects/WaveFormat/
http://www.postsharp.org/
http://www.erichynds.com/jquery/jquery-ui-multiselect-widget/
http://ha.ckers.org/xss.html
http://jetty.codehaus.org/jetty/
http://cpp-next.com/archive/2009/08/want-speed-pass-by-value/
http://codespeak.net/lxml/
http://www.hpl.hp.com/personal/Hans_Boehm/gc/
http://jquery.com/demo/thickbox/
http://book.git-scm.com/5_submodules.html
http://monotouch.net/
http://developer.android.com/resources/articles/timed-ui-updates.html
http://jquery.bassistance.de/validate/demo/
http://codeigniter.com/user_guide/database/active_record.html
http://www.phantomjs.org/
http://watin.org/
http://www.db4o.com/
http://qt.nokia.com/products/
http://referencesource.microsoft.com/netframework.aspx
https://github.com/facebook/php-sdk/
http://java.decompiler.free.fr/
http://pivotal.github.com/jasmine/
http://api.jquery.com/category/plugins/templates/
http://code.google.com/closure/library
http://www.w3schools.com/tags/ref_entities.asp
http://xstream.codehaus.org/tutorial.html
https://github.com/facebook/php-sdk
http://download.java.net/maven/1/jstl/jars/jstl-1.2.jar
https://developers.facebook.com/docs/offline-access-deprecation/
http://www.parashift.com/c++-faq-lite/pointers-to-members.html
https://developers.facebook.com/docs/mobile/ios/build/
http://downloads.php.net/pierre/
http://fluentnhibernate.org/
http://net.tutsplus.com/tutorials/javascript-ajax/5-ways-to-make-ajax-calls-with-jquery/
http://dev.iceburg.net/jquery/jqModal/

By post score¶

Count of posts by score (top 10) (Covers 94% of all broken links):

| Score | Percentage of Total Broken |
|-------|----------------------------|
| 0     | 36.4087%                   |
| 1     | 25.1674%                   |
| 2     | 13.4089%                   |
| 3     | 7.2806%                    |
| 4     | 4.2971%                    |
| 5     | 2.7065%                    |
| 6     | 1.8068%                    |
| 7     | 1.2854%                    |
| -1    | 1.1935%                    |
| 8     | 0.9415%                    |

By number of views¶

Note, this is number of views at the time the data dump was created, not as of today

Count of posts by number of views (top 10):

| Views        | Total Views |
|--------------|-------------|
| (0, 200]     | 24.4709%    |
| (200, 400]   | 14.2186%    |
| (400, 600]   | 9.5045%     |
| (600, 800]   | 6.9793%     | 
| (800, 1000]  | 5.2574%     |
| (1000, 1200] | 4.1864%     |
| (1200, 1400] | 3.3699%     |
| (1400, 1600] | 2.7766%     |
| (1600, 1800] | 2.3477%     |
| (1800, 2000] | 1.9550%     |

By days since post created¶

Note: This is number of days since creation at the time the data dump was created, not from today

Count of posts by days since creation (top 10) (Covers 64% of broken links):

| Days since Creation | Percentage of Total Broken |
|---------------------|----------------------------|
| (1110, 1140]        | 7.2938%                    |
| (1140, 1170]        | 6.7648%                    |
| (1470, 1500]        | 6.6579%                    |
| (1080, 1110]        | 6.6535%                    | 
| (750, 780]          | 6.5535%                    |
| (720, 750]          | 6.5516%                    |
| (1500, 1530]        | 6.3978%                    |
| (390, 420]          | 5.8508%                    |
| (360, 390]          | 5.8258%                    |
| (780, 810]          | 5.5175%                    |

By Ratio of Views:Days¶

Ratio Views:Days (top 20) (Covers 90% of broken links):

| Views:Days Ratio | Percentage of Total Broken |
|------------------|-------------|
| (0, 0.25]        | 27.2369%    |
| (0.25, 0.5]      | 18.8496%    |
| (0.5, 0.75]      | 11.4321%    |
| (0.75, 1]        | 7.2481%     | 
| (1, 1.25]        | 5.1668%     |
| (1.25, 1.5]      | 3.7907%     |
| (1.5, 1.75]      | 2.9310%     |
| (1.75, 2]        | 2.4033%     |
| (2, 2.25]        | 1.9788%     |
| (2.25, 2.5]      | 1.6850%     |
| (2.5, 2.75]      | 1.4080%     |
| (2.75, 3]        | 1.1879%     |
| (3, 3.25]        | 1.0654%     |
| (3.25, 3.5]      | 0.9391%     |
| (3.5, 3.75]      | 0.8334%     |
| (3.75, 4]        | 0.7165%     |
| (4, 4.25]        | 0.6634%     |
| (4.25, 4.5]      | 0.5789%     |
| (4.5, 4.75]      | 0.5508%     |
| (4.75, 5]        | 0.4833%     |

Discussion¶

What can we do with all of this? How do we, as a community, solve the issue of 10% of our outbound links pointing to places on the internet that no longer exist? Assuming that my sample was indicative of the entire data dump, there are close to 600K (150K broken unique links x 4, because I took 1/4 of the data dump as a sample) broken links posted in questions and answers on Stack Overflow. I assume a large number of links posted in comments would be broken as well, but that's an activity for another month.

We encourage posters to provide snippets from their links just in case a link dies. That definitely helps, but the resources behind the links and the (presumably) expanded explanation behind the links are still gone. How can we properly deal with this?

It looks like there have been a few previous discussions:

Utilize the Wayback API to automatically fix broken links. Development appeared to stall on this due to the large number of edits the Community user would be making. This would also hide posts that depended on said link from being surfaced for the community to fix it.
Link review queue. It was in alpha, but disappeared in early 2014.
Badge proposal for fixing broken links

Footnotes¶

This is how it ultimately played out. Originally I sent HEAD requests, in an effort to save bandwidth. This turned out to waste a whole bunch of time because there are a whole bunch of sites around the internet that return a 405 Method Not Allowed when sending a HEAD request. The next step was to sent GET requests, but utilize the default Python requests user-agent. A lot of sites were returning 401 or 404 responses to this user agent.
Links to Stack Exchange sites were not counted in the above results. The failures seen are almost 100% due to a question/answer/comment being deleted. The process ran as an anonymous user, thus didn't have any reputation and was served a 404. A user with appropriate permissions can still visit the link. I verified a number of 404'd links to Stack Overflow posts and this was the case.
The 4th most common failure was to localhost. The 16th and 17th most common were localhost on ports other than 80. I removed these from the result table with the knowledge that these shouldn't be accessible from the internet.
There where 7 total URLs that returned status codes in the 600 and 700 range. One such site was code.org with a status code of 752. Sadly, this is not even defined the joke RFC.

Follow up¶

I posted a proposal on how I think this could be fixed.

How I set up this site with GitHub Pages and CloudFlare

2015-07-09T11:26:00-05:00

Introduction¶

In a previous post, I described why I moved from Wordpress to Pelican for my blog. This one goes a step further and describes how I eliminated the need for the dedicated server I'd been utilizing as a part of Team Vipers. By eliminating that server, I reduced my costs to zero but kept control over the DNS of my domain (thanks to CloudFlare) and had an easier method of updating the site using GitHub Pages.

GitHub Pages Setup¶

To utilize GitHub Pages, I needed to create a new repository that followed the format GitHubUsername.github.io. This repository would house the content that is this site. I also set up a second repository which contains the source for the blog. This repository includes the templates, plugins and markdown version of the pages. The first repository was set up as submodule.

git submodule add https://github.com/AWegnerGitHub/awegnergithub.github.io.git output

I ignored the output directory in .gitignore on the source repository. Finally, I had to adjust publishconf.py slightly to

DELETE_OUTPUT_DIRECTORY = False

Without this, I was constantly destroying the output repository and had to reinitialize it. This prevents that from occuring.

Now, a new post consists of writing up the Markdown page, generating the page with the command below (or the batch script) and then committing and pushing the changes to the submodule to GitHub.

# Generates HTML files without debugging information
pelican content --output output --settings publishconf.py

The new content is available immediately.

Custom Domain¶

You may notice that the URL for this site isn't awegnergithub.github.io, but instead andrewwegner.com. To accomplish this, I added a directory to the content named extra. In this directory is a single file named CNAME (no extension). In the file is my domain name.

Next, I had to modify pelicanconf.py to add the extra/CNAME to the static path and then on generation move the CNAME file from this subdirectory to the root. I could have put it in the root of content by default, but Pelican provides a way to do this and it keeps content clean. One very important note, the EXTRA_PATH_METADATA is operating system sensitive. Since I am generating the content on a Windows machine, I had to use a backslash instead of the forward slash the documentation shows. I found this after posing a question on Stack Overflow on why it wasn't working as the documentation suggested.

The two important fields to add or edit are:

...
STATIC_PATHS = ['images', 'extra/CNAME']
...
EXTRA_PATH_METADATA = {'extra\CNAME': {'path': 'CNAME'},}

Cloudflare Setup¶

The final thing I needed in order to get rid of my server was control over DNS. I could revert back to GoDaddy, but after a little research found that CloudFlare's additional CDN and security was a "good thing" (because, you know, I'm such a highly traffic'd blog these days). Step one was signing up to CloudFlare. This was a 3-5 minute thing.

Once signed up and signed in, I went to set up DNS. This was as simple as adding my domain name and waiting for CloudFlare to import my existing DNS records. With this, I kept by Google Apps email intact (which is what I was most concerned with). Next, I went and removed the A records. I replaced these with CNAME records pointing to my GitHub Pages URL. I also added a www CNAME pointing to the same location. Since I have Pelican configured to strip it with the setting below, it doesn't matter other than people expect to enter www dot domain dot com in their URL bar.

SITEURL = 'http://andrewwegner.com'

Last, I had to point by name servers to CloudFlare instead of my dedicated server. They provide a list of registrars to choose from. Select your registrar and follow the instructions. My biggest issue here was remembering my GoDaddy password. After I made it into my account, the steps to change name servers were very simple. Once those are saved, you wait for the changes to propagate and enjoy your new GitHub Pages / CloudFlare web page for free.

Why I moved from Wordpress to Pelican

2015-05-03T22:41:00-05:00

Introduction¶

For years, I maintained a Wordpress blog covering various things I've done or created. Most of these revolved around things I created to make administering Team Vipers easier for me and for the rest of the admin team. It was my way of documenting what I'd done (in case I ever needed to do it again) and providing a way to update the Team Vipers community about new plugins or applications that would be deployed to the community.

My Issues with Wordpress¶

The problem I had with Wordpress was that it was just to bulky for the simple posts I was making. I needed a database, a full web server (or a hosting provider), and either time to hunt for the "perfect" plugin(s) or PHP knowledge to do it myself. Early in my development career, I used PHP a lot. That was part of the reason I chose Wordpress.

Oh! I know that language. If I ever need something, I can just whip it up myself.

-Me, before the real world ambushed me and beat me with a stick

Spam¶

I spent time setting up Wordpress. I picked out a theme, plugins, and started saving future me with documentation. Then life happened. For whatever reason, I stopped updating Wordpress. My blog sat out there for weeks or months unvisited by anyone. Then, one morning, my phone vibrated and told me that I had a new comment on my site. Woo!

Except it was spam. Boo!

I marked it spam and moved on with my day. Later that morning, I glanced at my phone again. 32 emails. I am just not that popular. Something was wrong. Turns out, a spam bot found me. I sighed and then removed all the comments and checked the box indicating that users had to be registered to post. That solved my problem for a few months.

Then the bots got smarter. They started registering. They started posting legitimate looking messages, except for that associated URL their name would link to in the comments. They pulled keywords out of the post and formulated a somewhat passable English question using those words. The spam prevention plugins I installed would slow the tide for a few weeks. The bots would adapt and then I'd be awash with spam posts again. Eventually, the solution was to completely disable comments. I'd spent way to much time dealing with spam on a blog that received very little legitimate traffic.

Since I don't utilize the comments, Pelican provides a nice simple page that I can post my thoughts and not worry about getting hit by a spam bot. It also provides plugins so that I can include comments should I ever choose to do so in the future. For the time being, though, I have a nice simple page with no comments. That's exactly what I was looking for.

Security¶

If you watch any technology web sites, you'll notice that there are vulnerabilities found in Wordpress frequently. These require patches, which requires me to do something. It may be as simple as logging in and clicking a button to update, but it is still something I need to remember to do for a relatively minor site. When I'd log in to clear the spam backlog, I'd frequently also install updates for 10-20 plugins, themes or Wordpress itself. It was mostly painless, but I didn't like the idea of the site sitting there vulnerable for weeks at a time because I didn't visit and login.

The dynamic nature of Wordpress and the underlying database exposed a fairly sizable target for a web page so small. Pelican generates static HTML pages. I don't have to worry about SQL injections, unauthorized logins, or anything else. I host a basic set of HTML, CSS and JavaScript files. That's it.

PHP vs Python¶

As I mentioned before, I used to use PHP frequently. It was my go to language. I picked Wordpress with the idea that I'd be able to hack together features I needed. The reality, it turns out, was that I wasn't actually interested in doing that. Instead, I picked out plugins that were close enough to the exact functionality I wanted.

I transferred to a job where I used Python. Instead of having a language I used on the side (PHP) and a job where I was a glorified project manager, without the actual title of "Project Manager", I now had a job where I used a language (Python) for 8 hours a day. My usage of PHP plummeted. I found I could get what I wanted done in my side projects faster and easier with Python. At work I used Python to build tools for engineering problems. At home, I started using it for every day tasks.

Soon, I realized I hadn't used PHP for several versions of the language. My knowledge of the language was outdated. The biggest reason I'd chosen Wordpress was no longer relevant, because I couldn't write anything complicated in PHP without glancing at documentation to do even simple things. It's sad that I lost the intimate knowledge of a language, but I feel that I've been more productive with Python anyway.

Pelican is written in Python. Even more importantly though, it generates HTML files which are hosted. I don't need to run a Python environment on a server. I just need to host HTML files.

Markdown¶

Finally, I've fought with Wordpress's text editor countless times. This happened most often when attempting to add code blocks. It was a pain to do. It was a pain to fix when the blocks broke. Pelican supports Markdown. Markdown is supported by large organizations like GitHub, reddit and Stack Exchange. I use all three of those. I know how to utilize Markdown to create code blocks, headers, insert images, create bulleted lists. All without needing to fight how the text editor is going to actually save the data.

I'm running to be a moderator of Stack Overflow

2015-04-06T22:00:00-05:00

Introduction¶

Stack Exchange has over 130 sites in its network. Each of those sites has at least 3 moderators that are glorified janitors:

A lot of the moderation work is extremely mundane, almost janitorial. It’s deleting obvious spam, closing blatantly off-topic questions, and culling some of the worst rated posts in various dimensions.

The ideal moderator does as little as possible. But those little actions may be powerful and highly concentrated. Judiciously limiting your use of moderator powers to selectively prune and guide the community — now that’s the true art of moderation.

This works so well in the Stack Exchange network because the users are able to handle most moderation tasks. As users gain reputation, they gain privileges. With these privileges, they are better able to maintain and cultivate the content on the site.

Not all activity can be performed by users though. This is where moderators come in.

I already have a diamond (♦) on the Community Building site. I was appointed one of the Pro Tempore moderators during the beta phase of the site. This is my first experience moderating on Stack Exchange, beyond what my reputation on Stack Overflow gets me. It is not, however, my first time moderating. Community Building is focused on various aspects of building communities - both online and off. It caters to users from all aspects of a site (owners, moderators, users, advertisers and any thing else). My experience here led me to pursuing a moderator position on Stack Overflow. This site is orders of magnitude larger than Community Building. It has millions of users and millions of questions. With all this traffic, there are still only 18 diamond moderators. The idea of community moderation shows that it scales well.

Nomination Phase¶

The election cycle begins with the nomination phase. Users nominate themselves for the position. In 1200 characters or less, you lay out your platform. In this phase, nearly anyone can nominate themselves. However, there are only 30 positions available. If more than 30 people nominate themselves, then the users with the lowest reputation get bumped. With only 11K reputation, I am at the lower end of the spectrum of other nominees, but I am well above the lowest. I have the 8th lowest reputation. I've looked at previous election cycles. It seems unlikely that there will be 30 or more nominees. I think that I am safe.

My Platform¶

I posted my platform shortly after the nomination phase opened. I stressed the work that I've already done on Stack Overflow and previous moderation experience on another Stack Exchange site (Community Building).

I'm Andy and I want to be one of your moderators.

Why vote for me?

I've built an automated script to handle noisy comments. It runs daily, probably to the chagrin of the current moderators, but it helps keep the comments noise down and it is incredibly accurate.

In addition to the automated flagging, I participate in the review queues, provide edits and post answers.

As a moderator of "Community Building", I know the tools used by the Stack Exchange sites. I've used this position to interact with the existing moderators for advise and questions. This interaction will not only continue but be helpful as a new moderator to SO.

I have the flagging history and experience to make the appropriate judgment calls. I feel that is a positive attribute that will help me in helping you to have the best experience you can on Stack Overflow.

I'll continue to help improve the site regardless of whether I am elected or not. I will be able to do a much better job at that, though, as one of your moderators.

With this, I received my "candidate score". It was 29/40. Not the highest, but not the lowest either. I expect, if any one is concerned about this score, it will be mentioned it in the comments.

Nomination Phase Comments¶

Updated April 11, 2014

The comments on my nomination post have focused mostly on the pros and cons of the automated script I built to flag comments. I was expecting this discussion. I'd classify most of the comments about the script as "cautious". Users seem concerned about something completely automated used to moderate or that I focus on low priority content. My response to this concern was:

[...] by automating the removal of this lower priority content it is easier to focus on the actively harmful stuff.

I'd like to point out that I have done more than "build a script". I've actively participated in review queues on Stack Overflow for years. I've flagged appropriate content (with a high percentage marked as accepted). I have also been moderator on Community Building. All of this, in combination with the automation, makes up my "platform". I have the experience of moderation, the knowledge of how the system works and the drive to improve the community.

It was also mentioned a few times that I'd be unlikely to utilize such an automated tool as a moderator. I promised to bring this concern up when I moved further into the election process.

Finally, I received a few comments supporting either me or the work that I've done.

The automation is very impressive - thanks for that, it's a real contribution to the quality of the site. That said - as a privileged user I suspect you won't be able to run these sort of scripts on your account for obvious reasons. – Benjamin Gruenbaum Apr 6 at 21:34

I can vouch for Andy's track record as a pro tem mod on Community Building. I want to note as well for the record that many of the main site questions and answers on Community Building are explicitly about moderation, and his contributions there also provide insight into his approach to the job. – Air Apr 7 at 16:54

Huh. I have to say I'm impressed. Putting you on my list of candidates to watch. – Qix Apr 7 at 19:23

I'd just like to point out that your response to my concerns is fair and reasonable, as well as polite, even if it isn't quite exactly how I see things. It certainly alleviates my concerns of misusing moderator power. I will be seriously considering voting for you. – jpmc26 Apr 8 at 6:41

Overall, I'm pleased with how the discussion has gone. It has been a few days since there were comments left though. Hopefully a little interest picks up during the primary phase.

I am also very impressed by the other candidates. There are many very good candidates that I'd be happy to work with (or be unashamed to lose to).

Primary Phase¶

Updated April 17, 2015

The nomination phases has ended. I have advanced to the primary phase. The purpose of this phase is to narrow the list of 30 candidates to 10. Those 10 will be the ones that can be voted on to be the next moderators. In the primary phase, users can vote nominees up or down.

As part of the election process, one of the other candidates created several tools to help users see nominee activity. He also created a live vote monitor. It was exciting to hang out in the Election Chat room and watch votes roll in. The position of 10th was highly contested toward the end of the primary. I was very pleased to see that everyone remained civil in the chat room too.

I ended this phase with 1492 positive votes. This put me in 15th. Sadly, not enough to advance to the election phase. However, there are several excellent candidates that did advance. Good luck to them.

Other Primary Data¶

Part of the primary phase involves answering questions that users have posed during the nomination phase. These questions were voted upon and the highest were included in the questionnaire. My response was removed at the end of this phase because I did not advance. I was prepared with answers about my moderation style and my expectations for the position.

One of the other tools that was created during this cycle showed nominee activity on the site over time. This was a quick way to compare nominees based on the number of activities they have performed on undeleted posts (because data related to deleted posts is removed from public view). It provided, at a glance, a way to see who is and is not active.

Below are the charts of my activity that the tool created. The green vertical bar is account creation date. The orange/brown vertical bars are previous election windows.

This is my activity chart for Stack Overflow itself. It shows that I've had an account since 2009, but didn't really start utilizing the site, more than occassionally, until 2013. Since then, it shows that a large majority of my activity onsite is in the review queues. There are posts, revisions, comments and more along the bottom, but most of my activity is in the review queues.

My Meta Stack Overflow and Meta Stack Exchange charts show a fairly low activity level. Unfortunately, I think this is obscuring the posts that I do make simply because of the large scale on the y-axis.

Finally, the last chart shows the combined activity across all three areas (SO, MSO, and MSE). It looks very similar to the Stack Overflow one because of the lower activity levels on the two Meta sites.

Election Phase¶

Updated April 21, 2015

The election is over. Three great candidates were elected. One thing I am like about Stack Exchange's elections is their usage of OpenSTV and the Meek STV method. The results of each round of the election cycle are available at opavote.org.

Final Thoughts¶

Like most elections, there are known and unknown winners before the election even begins. The response that Martijn Pieters received from the time he nominated through the final moments of the election all but guarenteed a victory for him. It is a well deserved victory. I hope that he is able to keep up his frequent, high quality, answers in the python tag. If not, his drop in activity will be felt. The surprise, to me, was Jeremy Banks. He ended in 9th place in the primaries. That doesn't mean he can't do the job.

I've interacted with all three moderators on site and in chat prior to their victory. I've been very happy with those interactions. With their promotion, I look forward to working with them as a moderator on the Stack Exchange network in the future.

Congratulations to the winners!

Moderation postion on Moderators

2014-07-02T10:51:00-05:00

Introduction¶

I've been the owner of Vipers for almost 5 years. In that time I've help the community grown and get stronger. I've helped members begin their own communities and I've maintained the game servers the players utilize 24 hours a day. I've been concerned about low activity, excited about new events and updates, and watched the community ebb and flow through out the years. I've dealt with trolls, spammers, and new community members. I think I've gained valuable experience in managing a community.

I'm also a member on Stack Overflow and over the years gotten to know others that enjoy the "Meta" aspect of Stack Exchange. That's the "behind the scenes", "how does this site work?" discussions and activities.

Moderators¶

A few days ago, I found a way to merge my experience with Vipers and the knowledge of the Stack Exchange platform. A new community started recently: Moderators. The idea behind this site is for people building, administering, managing or cultivating digital communities to have a place to ask others with similar experiences questions. I love it!

One of the things a new site needs on the Stack Exchange network is a set of pro tempore moderators. While Stack Exchange believe in community moderation, there are still some things that require more access. These pro tempore moderators are those people. I nominated myself for one such position.

The Nomination¶

This is the nomination I presented to the community. In it, I discuss my experience with Vipers and some of our challenges. One in particular that I mention is the automated language filter we have on the game servers.

I am an administrator/site owner of a medium sized gaming community that runs on a PHPBB3 board. We host multiple game servers as well. I've got a team of moderators that help keep the forum and game servers clean. I've run this site for 5 years, after taking it over from the original creator of the community who wanted to move on. In my time as admin, we've seen the number of participants on the forum increase. We've seen our game server population increase as well. I attribute this to getting the community involved in change discussions.

One of our biggest changes occurred several years ago. Community members complained that our game servers would be over run with trolls at hours when moderators weren't available and spewing filth. The community wanted a cleaner game server experience. Users wanted these players gone immediately. Previous community leaders felt that trolling of this kind was part of the game and did nothing. After some discussions regarding what was and wasn't appropriate, we decided to be (for lack of a better term) "family friendlier". Certain 'extreme' phrases were no longer tolerated at all. A technical solution was built to automatically remove players that violated these rules. This solution allowed users to swear, but once it became excessive (again, defined by the community) they, too, were removed.

The tool we have (PHPBB3) may not have the reputation, badges, or increasing privileges used here on Stack Exchange but for my community that has not been a negative. Engaging with the community in discussions and letting the members provide input that me and my team utilize has been extremely beneficial.

I have no experience moderating a Stack Exchange site. I don't feel that's a down side though. I can provide the "outside" perspective in a Stack Exchange heavy group. That does mean, though, that I'd depend on and expect the community to provide feedback on how moderation in being handled. Much like my existing gaming community, input from the community to the moderation team is important and the moderation team should be listening to that input.

Appointment¶

Updated on August 12, 2014

I have been appointed to a moderator position on Moderators. I now "moderator the moderators", as the running joke has been on meta and the chatroom. I am looking forward to helping this community grow and to providing my experience of community management outside of Stack Exchange to this position.

Renaming to Community Building¶

Updated on December 4, 2014

Almost since the beginning of Moderators, there have been conversations about whether the name is limiting our scope. We aren't a community for only moderation questions. We are a community about how to build communities and moderation happens to be a part of community building. (How many times can you say "community" before it sounds weird?)

Discussions began in August about possible name changes. These discussions continued into October as we worked on the scope of our site. Finally, on December 2, we received our new name: Community Building. It's the same great site but with a much more relevant name.