Andrew Wegner | Ponderings of an Andy - programming

Choosing an ORM library for a new project

2017-04-26T14:30:00-05:00

Project History¶

The SmokeDetector project is over three years old at this point. It's grown from a small python script to a decently sized application that integrates with another project. In that time, it's expanded what types of spam and patterns it looks for, what chat rooms it posts to, what external services it integrates with and how permissions to use the system are determined.

A lot has changed under the hood. I was hoping to put a cool chart here showing code change over time, but some early decisions with the project really throw off the chart. Using a Ship of Theseus analogy for code, you can see how much has changed. The basic idea is, if a ship leaves port and replaces every plank along its journey, is it still the same ship when it returns? With code, the idea is to apply this to lines of code in an application.

What happened in 2014?!¶

In late 2014, the project attempted its first machine learning method of detecting spam. In this time period, a commit was added that added about 200,000 lines of code to the project. This was almost all training data for a Bayesian algorithm. It wasn't needed and probably shouldn't have been added to the main repository. Unfortunately, it stayed in the repository for over a year and was finally removed in late 2015. This is the cause of the weird graph above, and why almost everything added in 2014 looks like it's missing in later years.

What has really changed?¶

After eliminating that Bayesian directory from git history, you can get a much better idea of how much has changed.

Very little of the original code, written in 2014, remains untouched. The explosion in code after that is due to new detection patterns, chat commands (and a rewrite), integration with MetaSmoke and the introduction of blacklists.

Even more dramatically, you can see how long a line of code is expected to survive in the code base.

Within one year, the team is removing over 40% of what's been committed to the repository. Looking at these commits, it was determined that a vast majority aren't even code. They are new items to blacklist or new patterns to detect.

Enter the database¶

All of this type of data can be stored in a database and managed outside of code. In early 2017, those discussions started taking place. Several team members come from a Ruby background and were familiar with its ORM method of accessing databases. They wanted something similar when a database was brought into SmokeDetector.

A bit of research was done and it was narrowed down to peewee and SQLAlchemy.

How to choose?¶

Fortunately for the SmokeDetector team, there weren't any strong opinions either way. The biggest reason for choosing one over the other came down to a comment made by the peewee author on reddit. They state:

[...] SQLAlchemy is the gold standard for ORM in the Python world. It has a very active community and a maintainer who is committed to excellence. If you're a glass-half-empty guy, to put it another way, you can't go wrong if you choose SQLAlchemy.

The weaknesses they list for using their own package is the smaller ecosystem, support and number of developers.

Technical differences¶

That's a boring story though. Not to be deterred from such a glowing review from a competitor, I wanted to see what the technical differences were between the two solutions.

To that end, I put together a small Python notebook showing the differences between peewee and SQLAlchemy in a handful of tests. These tests included inserting two settings in an SQLite database, retrieving one, inserting a large list of users and then retrieving a subset of those users.

The results were...unremarkable.

The two libraries each took two tests (out of four) for being faster than the other. In both cases where SQLAlchemy was faster, it was between two and six times faster. Where peewee was faster it was between a fraction faster and twice as fast.

The time scales are so small though, and SmokeDetector doesn't need to have thousands, hundreds or even tens of hits to the database a second. A hundred extra milliseconds isn't going to cripple anything it handles.

Thus, the choice was made based on the recommendation of the author of the peewee library. SQLAlchemy has a larger community and better support.

Can a machine be taught to flag spam automatically

2017-02-19T22:51:00-06:00

Introduction¶

This post was originally published on Meta Stack Exchange on February 20, 2017. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Exchange and post there.

The post was featured across the entire Stack Exchange network for a week, too. This drove a huge amount of traffic to the question and resulted in some valuable feedback:

TL;DR: We did it, so... yes.

What is this?¶

Charcoal is the organization behind the SmokeDetector bot and other nice things. This bot scans new posts across the entire network for spam posts and reports them to various chatrooms where people can act on them. If a post has been created or edited, anywhere on the network, we've probably seen it. The bot utilizes our knowledge of how spammers work and what they have previously posted to come up with common patterns and rules to detect spam in the new and updated posts. You've likely seen the SmokeDetector bot if you visit chatrooms such as Tavern on the Meta, Charcoal HQ, SO Close Vote Reviewers and others across the network. Over time, the bot has become very accurate.

Now we are leveraging the years of data and accuracy to automatically cast spam flags. With approximately 58,000 posts to draw from and over 46,000 true positives, we have a vast trove of data to utilize.

What problem does this address?¶

To put it simply, spam. Stack Exchange is one of the most popular networks of websites on the Internet, and all of it gets spammed at some point. Our statistics show that we see about 100 spam posts per day, on average over the last three months.

A decent chunk of this isn't the type you'd want to see at work (or at all). The faster we can get this off the home page, the better for all involved. Unfortunately, it's not unheard of for spam to last several hours, even on the larger sites such as Graphic Design.

Over the past three years, efforts with Smokey have significantly cut the time it takes for spam to be deleted. This project is an extension of that, and it's now well within reach to delete spam within seconds of it being posted.

What are we doing?¶

For over 3 years, SmokeDetector has reported potential spam across the Stack Exchange network so that users can flag the posts as appropriate. Users have provided feedback to inform the bot on whether the detection was correct or not (referred to as "feedback"). This feedback is stored in our web dashboard, metasmoke (code). Over time, we've used this feedback to evaluate our patterns ("reasons") and improve our accuracy. Several of our reasons are over 99.9% accurate.

Early last year, and after getting a baseline accuracy from jmac (thank you!), we realized we could use the system to automatically cast spam flags. On Stack Overflow the current accuracy of users flagging spam posts is 85.7%. Across the rest of the network users are 95.4% accurate. We determined we can beat those numbers and eliminate spam from Stack Overflow and the rest of the network even faster.

Without going into too much detail (if you really want it, it's available on our website), we leverage the accuracy of each existing reason to come up with a weight indicating how certain the system is that a post is spam. If this value exceeds a specific threshold, the system will cast up to three spam flags on the post. We cast multiple flags utilizing a number of different users' accounts and the Stack Exchange API. Via metasmoke, users are given the opportunity to enable their accounts to be used to flag spam (You can too, if you've made it this far). When a post is eligible for flagging because it exceeded the threshold set by each individual user, accounts are randomly selected from the pool of enabled users to cast a single flag each, up to a maximum of three per post so that we never unilaterally nuke something.

What are our safety checks?¶

We designed the entire system with accuracy and sanity checks in mind. Our design collaborations are available for your browsing pleasure (RFC 1, RFC 2, RFC 3 (no longer available)). The major things that make this system safe and sane are:

We give users a choice as to how accurate they want to be with their automatic flags. Before casting any flags, we check that the preferences the user has set result in a spam detection accuracy of over 99.5% over a sample of at least 1000 posts. Remember, the current accuracy of humans is 85.7% on SO and network wide it is 95.4%.
We do not unilaterally spam nuke a post, regardless of how sure we are it is spam. This means that a human must be involved to finish off a post, even on the few sites with lower spam thresholds.
We’ve designed the system to be tolerant of faults - if there’s a malfunction anywhere in the system, any user with access to SmokeDetector can immediately halt all automatic flagging - this includes all network moderators. If this happens, it needs a system administrator to step in to re-enable flags.
We've discussed this with a community manager and have their blessing on the project.

Results¶

We have been casting an average of 60-70 automatic flags per day for over two months, for a total of just over 4000 flags network wide. These flags were cast by 22 different users. In that time, we've had four false positives. We would like to be able to automatically cancel these particular cases. This isn't possible though, so we've created a feature request to retract flags via the API. In the mean time, the flags are either manually retracted by the user or declined by a moderator.

The above graph plots the weight of the reasons against its overall volume of reports and accuracy. As minimum weight increases, accuracy (yellow line and rightmost Y-axis) and total reports (blue line) on the left-hand scale increase. The green line represents the number of true positives, which are verified by SmokeDetector user feedback.

This shows the number of posts we've automatically flagged per day over the last month. The jump on February 15th, is due to increasing the number of automatic flags from 1 per post to 3 per post. You can see a live version of this graph on metasmoke's autoflagging page.

Spam arrives on Stack Exchange in waves. It is easy to see the time of day that many spam reports come in. The hours, above, are UTC time. The busiest spam times of day are the 8 hour block between 4am and Noon. We have affectionately named this "spam hour" in the chat room.

Our goal is to delete spam quickly and accurately. The graph shows the time it takes for a reported spam post to be removed from the network. This section has three trend lines that show these averages. The first, red section is when we were simply reporting the posts to chatrooms and all flags had to come from users. You can see we are pretty constant in the time it takes to remove spam during this period. It took, on average, just over five minutes to get a post removed.

The green trend line is when we were issuing a single automatic flag. At implementation, we eliminated a full minute from time to deletion and after a month we'd eliminated two full minutes compared to no automatic flags.

The last section, the orange, is when we implemented three automatic flags to most sites. This was rolled out last week, but it's already had a dramatic improvement on the time to deletion. We are seeing between 1 and 2 minutes to time to deletion.

As mentioned above, spam arrives in waves. The dashed and dotted lines on the graph show the average deletion time during these two different time periods. The dashed lines show deletion time during 4am and Noon UTC, the dotted lines show the rest of the 24 hour period. An interesting thing this graph shows is that time to deletion during spam hour was higher when we didn't cast any automatic flags. It was removed faster outside of spam hour. That reversed when we started issuing a single auto-flag. The spam hour time to deletion is slightly lower than the average. Comparing the two time periods though, time to deletion during non-spam hour at the end of the non-flagging time period and the end of the single flag period are roughly the same.

We'll update these in a few weeks too, to better show the trend we are seeing with three automatic flags.

Discussion¶

We are confident in SmokeDetector and the three years of history it has. We've had many talented developers assist us over the years and many more users have provided feedback to improve our detection rules. Let us know what you want us to elaborate on, features you're wondering about or would like to see added, or things we might have missed in the process or the tooling. Take a look at the feature we'd really like Stack Exchange to consider so that we can further improve this system (and some of the other community built systems). We'll have Charcoal members hanging around and answering your questions. Alternatively, feel free to drop into Charcoal HQ and have a chat.

Link Analysis - Technical Explanation

2015-08-10T23:41:00-05:00

Introduction¶

In my last two posts, I've discussed the number of rotten links on Stack Overflow and a proposal to fix said links. In this post, I'm going to discuss how I performed this analysis.

Set up¶

The database¶

The process began by downloading the March 2013 data dump. I loaded the posts into a MariaDB instance on my local machine. This was accomplished with a very simple script and patience, as the script took a while to run.

load xml local infile '/path/to/posts.xml'
into table posts
rows identified by '<row>';

The data¶

Once this was done, the next step was selecting my random sample of data. I did this by randomly selecting 25% of the days in a year and then pulling all posts for those days across all years of Stack Overflow's existence. The Python script I used to do this was fairly simple:

from datetime import timedelta, datetime
from random import randint
from math import ceil

def random_date(start, end):
    return start + timedelta(
        seconds=randint(0, int((end - start).total_seconds())))

percentage = 0.25
days = 366

dayslist = []
for d in xrange(int(ceil(days*percentage))):
    dayslist.append(random_date(datetime(2008,1,1), datetime(2008,12,31)))

At the end of this run, the days that I cared about are in the dayslist variable. I used that to pull questions and answers from the database that were created on that month/day combination. In the end, this resulted in just over 25% of the total posts being selected. To ensure that I could replicate the results, I also saved the dates that were selected.

Parsing the data¶

The next step was to parse out links from the data. I used the following script to extract anchor text and links from a post.

def links_in_post(post):
    """
    Returns a list of all links found
    :param posts: A list of dictionaries with a 'body' key containing HTML strings
     [
        {
            'body': "<b>This is HTML</b>"
        },
    ]
    :return: A list of tuples containing anchor text and URL
        [
            ('Display Text', 'http://example.com')
        ]
    """
    logging.debug("Extracting links...")
    links = []
    images = []
    regexp = "&.+?;"
    list_of_html = re.findall(regexp, post)
    for e in list_of_html:
        if e in invalid_entities:
            h = HTMLParser.HTMLParser()
            unescaped = h.unescape(e) 
            post = post.replace(e, unescaped)

    doc = html.fromstring(post)
    for link in doc.xpath('//a'):
        links.append(Link(text=link.text_content(), link=link.get('href')))
    for image in doc.xpath('//img'):
        images.append(Link(text=image.get('alt'), link=image.get('src')))
    all_items = links + images
    seen = set()
    unique_items = [item for item in all_items if item[1] not in seen and not seen.add(item[1])]
    return unique_items

The regular expression being utilized, is to strip out HTML entities. This was needed due to weird parsing issues with non-ASCII characters. Fortunately, I wasn't the first to encounter oddities like this. The list comprehension at the end of the function is returning only unique tuples of anchor text/link. I was surprised how often I'd end up with tuples such as ('this', 'http://google.com') in the same post. This uniqueness saved a lot of processing time later.

After all links in a post had been extracted, this information and information about the post itself, was saved to the database. If a post had no links, it was not saved. The database consisted of three tables.

Links - This table contains the base URLs seen in all posts. URLs are distinct. It also contains an ID that will be utilized for linking to the other tables.
Post Links - This table contains information about links in a post. This includes the specific anchor text/link combinations
Link Results - This table contains the results of link status checks

Processing the posts was fairly time consuming, but was able to be parallelized easily. That significantly cut down on processing time.

Checking the links¶

The most time consuming portion of this entire project was actually checking link status. Each link that appeared in the Links table was checked. As I mentioned in my first post, the original idea was to simply send a HEAD request to each URL. The idea was to save myself and the end point a tiny amount of bandwidth. I had over a million links to process. I figured a little saved bandwidth wouldn't hurt.

It turns out this isn't a good idea. When I started seeing larger sites as not being accessible, I got suspicious that something was wrong. These sites were returning status 405 errors. This indicates that the method is not allowed. So, I switched to GET for every link. The next problem I ran into was that many sites didn't like the default user agent of the spider. They rejected requests with 404 and 401 errors. In the end, I got around this by mimicking Firefox on every request.

With those kinks worked out, every link was sent a GET request that looked to be from a Firefox browser. The process would allow 20 seconds per link. If the link didn't respond in that time limit, it was declared inaccessible.

A week later, I repeated the process with anything that hadn't returned a status code less than 400. Once more, on the third week, I repeated this with the failed links. At the end of three weeks, I had a list of sites that were inaccessible to me - on a residential connection - three times over a period of three weeks.

Results¶

The SVG image that I created for the write up was generated with Pygal. The tables were the result of various breakdowns of the data via queries to the status results.

Wrap up¶

I am rather proud of how the results turned out for this project. I went into it expecting about 15% of links to be broken, but I didn't really realize what that meant. Fifteen percent of 21 million total posts is over 3 million. That's a large number. BUT, it also ignored that a large percentage of posts don't contain links. I failed to consider that in my original hypothesis.

Less than half of my sample had links (2.3M out of 5.6M). Of the 2.3M with links, only 1.5M were unique links. The final result of 10% failed links makes much more sense in this context. Ten percent of 1.5M links means that there are 150K links that are bad.

A proposal to fix broken links on Stack Overflow

2015-08-07T07:34:00-05:00

This post was published by me on Meta Stack Overflow on August 7th, 2015. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

This is a follow up to yesterday's post about how many links on Stack Overflow are starting to rot.

The proposal¶

I propose another hybrid of the previous broken link queue (as was mentioned above in comments and other answers) and an automated process to fix broken links with an archived version (which has also been suggested).

The broken link queue should focus on editing and fixing the links in a post (as opposed to closing it). It'd be similar to the suggested edits queue, but with the focus intended to correct links not spelling and grammar. This could be done by only allowing a user to edit the links.

One possibility, I envision is presenting the user with the links in the post and a status on whether or not the link is available. If it's not available, give the user a way to change that specific link. Utilizing this post, I have a quick mock up of what I propose such a review task looks like:

The Queue¶

All the links that appear in the post are on the right hand side of the screen. The links that are accessible have a green check mark. The ones that are broken (and the reason for being in this queue) have a red X. When a user elects to fix a post, they are presented with a modal showing only the broken URLs.

The Automation¶

With this queue, though, I think an automated process would be helpful as well. The idea is that this would operate similarly to the Low Quality queue, where the system can automatically add a post to the queue if certain criteria are met or a user can flag a post as having broken links. I've based my idea on what Tim Post outlined in the comments to a previous post.

Automated process performs a "Today in History" type check. This keeps the fixes limited to a small subset of posts per day. It also focuses on older posts, which were more likely to have a broken link than something posted recently. Example: On July 31, 2015, the only posts being checked for bad links would be anything posted on July 31 in any year 2008 through current year - 1.
Utilizing the Wayback Machine API, or similar service, the system attempts to change broken links into an archived version of the URL. This archived version should probably be from "close" to the time the post was originally made. If the automated process isn't able to find an archived version of the link, the post should be tossed into the Broken Link queue
When the Community edits a post to fix a link, a new Post History event is utilized to show that a link was changed. This would allow anyone looking at revision history to easily see that a specific change was only to fix links.
Actions performed in the previous bullets are exposed to 10K users in the moderator tools. Much like recent close/delete posts show up, these do as well. This allows higher rep users to spot check (if they so desire). I think this portion is important when the automated process fixes a link. For community edits in the queue, the history tab in /review seems sufficient.
If a post consists of a large percentage of a link (or links) and these links were changed by Community, the post should have further action taken on it in some queue.

Example: - A post where X+% of the text is hyperlinks is very dependent on the links being active. If one or more of the links are broken, the post may no longer be relevant (or may be a link only post). One example I found while doing this was this answer.

I don't think that this type of edit from the Community user should bump a post to the front page. Edits done in the broken link queue, though, should bump the post just like a suggested edit does today. By preventing the automated Community posts from being bumped, we prevent the front page from being flooded, daily, with old posts and these edits. I think that the exposure in the 10K tools and the broken link queue will provide the visibility needed to check the process is working correctly.

Process flows¶

Queue Flow:

Automated process flow:

Potential pitfalls¶

The automated link checking will likely run into several of the problems I did. Mainly:

Sites modify the HEAD request to send a 404 instead of a 405. My solution to this was to issue GET requests for everything.
Sites don't like certain user agents. My solution to this was to mimic the Firefox user agent. To be a good internet citizen, Stack Exchange probably shouldn't go that far, but providing a unique user agent that is easily identifiable as "StackExchangeBot" (think "GoogleBot"), should be helpful in identifying where traffic is coming from.
Sites that are down one week and up another. I solved this by spreading my tests over a period of 3 weeks. With the queue and automatic linking to an archived version of the site, this may not be necessary. However, immediately converting a link to an archived copy should be discussed by the community. Do we convert the broken link immediately? Or do we try again in X days. If it's still down then convert it? It was suggested in another answer that we first offer the poster the chance to make changes before an automatic process takes place.
The need to throttle requests so that you don't flood a site with requests. I solved this by only querying unique URLs. This still issues a lot of requests to certain, popular, domains. This could be solved by staggering the checks over a period of minutes/hours versus spewing 100s - 1000s of GET requests at midnight daily.

With the broken link queue, I feel the first two would be acceptable. Much like posts in the Low Quality queue appear because of a heuristic, despite not being low quality, links will be the same way. The system will flag them as broken and the queue will determine if that is true (if an archived version of the site can't be found by the automated process). The bullet about throttling requests is an implementation detail that I'm sure the developers would be able to figure out.

Analysis of links posted to Stack Overflow

2015-08-06T07:35:00-05:00

This post was published by me on Meta Stack Overflow on August 6th, 2015. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

TL;DR: Approximately 10% of 1.5M randomly selected unique links in the March 2015 data dump are unavailable. To be more precise, that is approximately 150K dead links.

Motivation¶

I've been running into more and more links that are dead on Stack Overflow and it's bothering me. In some cases, I've spent the time hunting down a replacement, in others I've notified the owner of the post that a link is dead, and (more shamefully), in others I've simply ignored it and left just a down vote. Obviously that's not good.

Before making sweeping generalizations that there are dead links everywhere, though, I wanted to make sure I wasn't just finding bad posts because I was wandering through the review queues. Utilizing the March 2015 data dump, I randomly selected about 25% of the posts (both questions and answers) and then parsed out the links. This works out to 5.6M posts out of 21.7M total.

Of these 5.6M posts, 2.3M contained links and 1.5M of these were unique links. I sent each unique URL a HEAD request, with a user agent mimicking Firefox¹. I then retested everything that didn't return a successful response a week later. Finally, anything that failed from that batch, I resent a final test a week later. If a site was down in all three tests, I considered it down for this test.

Results²¶

By status code¶

Good news/Bad News: A majority of the links returned a valid response, but there are still roughly 10% that failed.

(This image is showing the top status codes returned)

The three largest slices of the pie are the status 200s (site working!), status 404 (page not found, but server responded saying the page isn't found) and Connection Errors. Connection errors are sites that had no proper server response. The request to access the page timed out. I was generous in the time out and allowed a request to live for 20 seconds before failing a link with this status. The 4xx and 5xx errors are status codes that fall in the 400 and 500 range of HTTP responses. These are client and server error ranges, thus counted as a failure. 2xx errors are pages that responded with a success message in the 200 range, but it wasn't a 200 code. Finally, there were just over a hundred sites that hit a redirect loop that didn't seem to end. These are the 3xx errors. I failed a site with this range if it redirected more than 30 times. There are a negligible number of sites that returned status codes in the 600 and 700 range⁴

By most common¶

There are, expectedly, many URLs that failed that appeared frequently in the sample set. Below is a list of the top 50³ URLs that are in posts most often, but failed three times over the course of three weeks.

http://docs.jquery.com/Plugins/validation
http://www.eclipse.org/eclipselink/moxy.php
http://jackson.codehaus.org/
http://xstream.codehaus.org/
http://opencv.willowgarage.com/wiki/
http://developer.android.com/resources/articles/painless-threading.html
http://valums.com/ajax-upload/
http://sqlite.phxsoftware.com/
http://qt.nokia.com/
http://www.oracle.com/technetwork/java/codeconv-138413.html
http://download.java.net/jdk8/docs/api/java/time/package-summary.html
http://docs.oracle.com/javase/1.4.2/docs/api/java/text/SimpleDateFormat.html
http://watin.sourceforge.net/
http://leandrovieira.com/projects/jquery/lightbox/
https://graph.facebook.com/
https://ccrma.stanford.edu/courses/422/projects/WaveFormat/
http://www.postsharp.org/
http://www.erichynds.com/jquery/jquery-ui-multiselect-widget/
http://ha.ckers.org/xss.html
http://jetty.codehaus.org/jetty/
http://cpp-next.com/archive/2009/08/want-speed-pass-by-value/
http://codespeak.net/lxml/
http://www.hpl.hp.com/personal/Hans_Boehm/gc/
http://jquery.com/demo/thickbox/
http://book.git-scm.com/5_submodules.html
http://monotouch.net/
http://developer.android.com/resources/articles/timed-ui-updates.html
http://jquery.bassistance.de/validate/demo/
http://codeigniter.com/user_guide/database/active_record.html
http://www.phantomjs.org/
http://watin.org/
http://www.db4o.com/
http://qt.nokia.com/products/
http://referencesource.microsoft.com/netframework.aspx
https://github.com/facebook/php-sdk/
http://java.decompiler.free.fr/
http://pivotal.github.com/jasmine/
http://api.jquery.com/category/plugins/templates/
http://code.google.com/closure/library
http://www.w3schools.com/tags/ref_entities.asp
http://xstream.codehaus.org/tutorial.html
https://github.com/facebook/php-sdk
http://download.java.net/maven/1/jstl/jars/jstl-1.2.jar
https://developers.facebook.com/docs/offline-access-deprecation/
http://www.parashift.com/c++-faq-lite/pointers-to-members.html
https://developers.facebook.com/docs/mobile/ios/build/
http://downloads.php.net/pierre/
http://fluentnhibernate.org/
http://net.tutsplus.com/tutorials/javascript-ajax/5-ways-to-make-ajax-calls-with-jquery/
http://dev.iceburg.net/jquery/jqModal/

By post score¶

Count of posts by score (top 10) (Covers 94% of all broken links):

| Score | Percentage of Total Broken |
|-------|----------------------------|
| 0     | 36.4087%                   |
| 1     | 25.1674%                   |
| 2     | 13.4089%                   |
| 3     | 7.2806%                    |
| 4     | 4.2971%                    |
| 5     | 2.7065%                    |
| 6     | 1.8068%                    |
| 7     | 1.2854%                    |
| -1    | 1.1935%                    |
| 8     | 0.9415%                    |

By number of views¶

Note, this is number of views at the time the data dump was created, not as of today

Count of posts by number of views (top 10):

| Views        | Total Views |
|--------------|-------------|
| (0, 200]     | 24.4709%    |
| (200, 400]   | 14.2186%    |
| (400, 600]   | 9.5045%     |
| (600, 800]   | 6.9793%     | 
| (800, 1000]  | 5.2574%     |
| (1000, 1200] | 4.1864%     |
| (1200, 1400] | 3.3699%     |
| (1400, 1600] | 2.7766%     |
| (1600, 1800] | 2.3477%     |
| (1800, 2000] | 1.9550%     |

By days since post created¶

Note: This is number of days since creation at the time the data dump was created, not from today

Count of posts by days since creation (top 10) (Covers 64% of broken links):

| Days since Creation | Percentage of Total Broken |
|---------------------|----------------------------|
| (1110, 1140]        | 7.2938%                    |
| (1140, 1170]        | 6.7648%                    |
| (1470, 1500]        | 6.6579%                    |
| (1080, 1110]        | 6.6535%                    | 
| (750, 780]          | 6.5535%                    |
| (720, 750]          | 6.5516%                    |
| (1500, 1530]        | 6.3978%                    |
| (390, 420]          | 5.8508%                    |
| (360, 390]          | 5.8258%                    |
| (780, 810]          | 5.5175%                    |

By Ratio of Views:Days¶

Ratio Views:Days (top 20) (Covers 90% of broken links):

| Views:Days Ratio | Percentage of Total Broken |
|------------------|-------------|
| (0, 0.25]        | 27.2369%    |
| (0.25, 0.5]      | 18.8496%    |
| (0.5, 0.75]      | 11.4321%    |
| (0.75, 1]        | 7.2481%     | 
| (1, 1.25]        | 5.1668%     |
| (1.25, 1.5]      | 3.7907%     |
| (1.5, 1.75]      | 2.9310%     |
| (1.75, 2]        | 2.4033%     |
| (2, 2.25]        | 1.9788%     |
| (2.25, 2.5]      | 1.6850%     |
| (2.5, 2.75]      | 1.4080%     |
| (2.75, 3]        | 1.1879%     |
| (3, 3.25]        | 1.0654%     |
| (3.25, 3.5]      | 0.9391%     |
| (3.5, 3.75]      | 0.8334%     |
| (3.75, 4]        | 0.7165%     |
| (4, 4.25]        | 0.6634%     |
| (4.25, 4.5]      | 0.5789%     |
| (4.5, 4.75]      | 0.5508%     |
| (4.75, 5]        | 0.4833%     |

Discussion¶

What can we do with all of this? How do we, as a community, solve the issue of 10% of our outbound links pointing to places on the internet that no longer exist? Assuming that my sample was indicative of the entire data dump, there are close to 600K (150K broken unique links x 4, because I took 1/4 of the data dump as a sample) broken links posted in questions and answers on Stack Overflow. I assume a large number of links posted in comments would be broken as well, but that's an activity for another month.

We encourage posters to provide snippets from their links just in case a link dies. That definitely helps, but the resources behind the links and the (presumably) expanded explanation behind the links are still gone. How can we properly deal with this?

It looks like there have been a few previous discussions:

Utilize the Wayback API to automatically fix broken links. Development appeared to stall on this due to the large number of edits the Community user would be making. This would also hide posts that depended on said link from being surfaced for the community to fix it.
Link review queue. It was in alpha, but disappeared in early 2014.
Badge proposal for fixing broken links

Footnotes¶

This is how it ultimately played out. Originally I sent HEAD requests, in an effort to save bandwidth. This turned out to waste a whole bunch of time because there are a whole bunch of sites around the internet that return a 405 Method Not Allowed when sending a HEAD request. The next step was to send GET requests, but utilize the default Python requests user-agent. A lot of sites were returning 401 or 404 responses to this user agent.
Links to Stack Exchange sites were not counted in the above results. The failures seen are almost 100% due to a question/answer/comment being deleted. The process ran as an anonymous user, thus didn't have any reputation and was served a 404. A user with appropriate permissions can still visit the link. I verified a number of 404'd links to Stack Overflow posts and this was the case.
The 4th most common failure was to localhost. The 16th and 17th most common were localhost on ports other than 80. I removed these from the result table with the knowledge that these shouldn't be accessible from the internet.
There where 7 total URLs that returned status codes in the 600 and 700 range. One such site was code.org with a status code of 752. Sadly, this is not even defined in the joke RFC.

Follow up¶

I posted a proposal on how I think this could be fixed.

Zephyr - The bot that watches for low quality vote requests

2015-03-12T23:34:00-05:00

Introduction¶

Stack Exchange receives thousands of questions per day across all of their sites. Not all of these are high quality posts. Fortunately, users of the Stack Exchange network are given tools to help keep that low quality stuff to a minimum. One of these tools is the chat network that spans the Stack Exchange sites.

In the chat rooms, a convention has arisen to tag a message as cv-pls for questions that need to be closed for one reason or another. Over time, this evolved to include other tags such as:

del-pls for a deletion request
spam for notification that spam made it through the already impressive spam filters
reopen for a reopen request
a few others to cover specific flag types (eg. Not an answer, Very Low Quality or Offensive)

Introducing Zephyr¶

The problem with these is that the requests are only seen by users active in the specific room where it was posted. Other users across the network miss the request. Zephyr was built to resolve this problem. Zephyr monitors several rooms where these types of requests are frequent. These requests are all posted into a single chat room. This provides users with a single room to monitor to see requests for multiple questions and sites across the network.

Here is an example of what Zephyr's chat activity looks like during a spam wave:

How it works¶

Zephyr utilizes the ChatExchange package to join and read the chat rooms. To do this, Zephyr required a dedicated account. I decided to run Zephyr with a dedicated account to completely separate the bot that would sit and watch multiple chat rooms 24/7 from my account. Zephyr maintains a small SQLite database of all the posts that it records. The idea behind this, is that eventually this data will be utilized to train other systems on unwanted content. This information is pulled via the API.

Zephyr watches the chat rooms for specific string patterns. If these patterns are matched, a message is posted if should_post is True for the matched pattern.

Overall, a nice simple application. It performs some pattern matching and a couple API calls.

Other bots¶

In addition to watching user activity, Zephyr also watches two other quality bots that patrol Stack Exchange for low quality content: SmokeDetector and Phamhilator. If either of these bots detect spam, Zephyr takes note of the information by recording it to the database, but not reposting. Since both of those bots post their reports, it didn't make sense for Zephyr to add a second (or third, if both of the others detected spam) message to the chat room. The information is recorded, though, to help future training for other systems.

Updates¶

Updated May 8, 2015

Over time Zephyr has been updated to include new rooms to monitor or new patterns to match. Those changes are small (and simple). There are, however, a few larger changes that I'd like to note below.

Commands¶

The other bots that Zephyr monitors respond to user input. Zephyr has very little that requires user interaction since all of its posts are generated by user input. However, there have been times where I, as the bot owner, would like to be able to issue certain commands to it. My most common desire is to see a report of how many spam posts Zephyr has seen. Thus, Zephyr now responds to the command spamreport from me. It then prints out a nice summary of information. This information has been utilized in SmokeDetector to watch for commonly spammed domains.

Upgrade from SQLite to MariaDB¶

Zephyr was originally built against an SQLite database. This worked, but was getting slower as more data was being added. This slow down was beginning to affect performance. I started seeing this error more and more frequently:

Traceback (most recent call last):
  File "H:\python-virtualenvs\zephyr-se-voterequests\lib\site-packages\sqlalchemy\pool.py", line 255, in _close_connection
    self._dialect.do_close(connection)
  File "H:\python-virtualenvs\zephyr-se-voterequests\lib\site-packages\sqlalchemy\engine\default.py", line 418, in do_close
    dbapi_connection.close()
ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 4824 and this is thread id 4660

After spending a lot of time troubleshooting and not resolving it to my satisfaction, I decided to upgrade to a more robust database. I'd used MySQL/MariaDB before and I happened to have another application utilizing MariaDB at the moment so that is the solution I picked.

The first step was transferring data. I learned that there isn't a decent utility to do a straight migration. So, I took these steps to transfer the data:

Export table structures and data from SQLite
Convert the SQLite dump to MySQL format. Though both systems use SQL, there are slight differences in dialect. I utilized this Python script as a starting point. It got me most of the way there, but not completely.
Data clean up. Ugh. The dreaded part of the job for anyone who handles data. Fortunately, the script above did most of the work. I ended up fixing a couple stray back ticks that didn't convert properly, escaping a few extra quotation marks, and replacing a few "smart quotes" (of both the left and right variety). I wish data at the office job was this easy to clean...
Import into MariaDB

Since the transfer to MariaDB, I've noticed no performance degradation. The error about threads has been eliminated as well.

Upgrade to utilize web sockets¶

Originally, Zephyr used the watch method when monitoring a room. This method would long poll the room. It turns out that this is pretty unreliable. I'd get multiple errors throughout the week, ranging from Connection Aborted errors to random 404 messages. The solution has been to switch to watch_socket. The only time I've had problems since this switch is when the Stack Exchange web sockets go down. This saves a lot of restarts to get everything up and running again.

Can a machine be taught to flag comments automatically

2015-01-02T08:47:00-06:00

Introduction¶

This post was originally published by me on Meta Stack Overflow on December 14, 2014. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

TL;DR: Yes it can.

Background¶

On June 27, 2014 Skynet awoke. It looked at Stack Overflow and thought "Why are all these people being so chatty and talking about obsolete things? I should nuke them all!" Fortunately, Skynet was a baby and only had access to my 100 comment flags a day.

Prior to this activation date, the system was fed with 10,000 "Good Comments", "Obsolete" comments and "Too Chatty" comments. These comments were taken from the Stack Exchange Data Explorer. The "Obsolete" and "Too Chatty" comment types had to meet the following criteria:

Total comment length of less than 100 characters
Comment has a 0 score
Had variations of the following phrases:

Phrases

'%mark%answer%'
'%mark%accept%'
'%accept%answer%'
'%lease%accept%'
'%mark%answer%'
'%thank%you%'
'%thx%you%'
'%.....'
'+1%'
'-1%'

"Good Comments" were assumed, initially, to be anything that didn't fall into the above criteria

This provided a base of 30,000 comments that were roughly categorized into 3 distinct groups. Manually scanning the classifications took several weeks, and through this some of the groupings were changed to reflect a more appropriate classification. Not all comments less than 100 characters starting with "Thank you" are "too chatty", just as not all comments over 100 characters are good comments. I reclassified these comments as if I had encountered them on Stack Overflow.

My next step was to train a classifier. I had initially assumed that I'd start with a Naive Bayes to get a baseline and then work to something more complicated from there. Perhaps, extract text features, user information, etc. and build a fancy classifier. My initial tests showed that the Naive Bayes was accurate 80-90% of the time with test data.

I combined the classifier's certainty of classification with an acceptable threshold of when I'd allow a flag to be issued in my name. Tuning these thresholds took a few weeks but eventually I determined the following thresholds were appropriate for my use:

Type            | Threshold     | Flagging Enabled
--------------------------------------------------
too chatty      | 0.9997        | True
obsolete        | 0.99          | True
good comment    | 0.9999        | False

When a comment is classified, if it exceeds the threshold for one of the above, it is recorded into my database for future retraining. If flagging is enabled, the API is utilized to issue an appropriate flag. Obviously, I don't want to flag good comments, but I do want to record them so that I can reuse the data in a later training step.

Results¶

What have the results of this experiment been? From my point of view, I'd venture that it's been successful. I have automatically flagged over 17,000 comments. As of December 17, 2014, the process has been running for 173 days. My comment flagging stats are currently:

26885   comments flagged
26714   deemed helpful
171     declined

Started at (approximately):

9885    comments flagged
9847    deemed helpful
38      declined

This gives me an overall accuracy of 99.36%. Down from 99.61% when no automated process was involved.

There are pictures that help tell this story too. In this first one, we see that the rolling 10 day average for the number of declined flags has stayed below two flags a day. In October, there was a two week period where the rolling average was 0 and nearly a month long period where the system did not make any mistakes.

Since November, the number of mistakes has climbed slightly. The biggest number of mistakes it has made was the opening day of Winter Bash 2014. Purely speculation, but I believe this was the moderators being protective of content and not wanting people to farm the Resolution hat. Of course, I don't know this. Another theory I have about this uptick since November is the adjustment to daylight saving time. My process starts 10 minutes after UTC midnight. It is possible that this earlier hour has caused my flags to be processed by a different moderator, or a moderator that is more awake/less hungry/in a different mood than previously at this point in the daily rotation cycle or because they lost their keys that day.

Except for 3 days, since June 27th, the process has flagged 100 comments a day. In this chart, you can see the number of declined comment flags along the bottom.

Finally, this chart shows the number of comments that the system wanted to act on (and a rolling 5 day average). When the system was brought online, it was acting on 700-800 comments a day (saving to my local database). Many of these were being classified as "Good Comments". You can see the day that I adjusted the threshold for "Good Comments" to be acted upon (saved). The drop in the number of comments the system saved is dramatic. Instead of saving 700-800 comments daily, the system now averages about 150 comments to save. Since I don't flag "Good Comments", I feel this is the appropriate action to take.

Flagged but declined¶

As shown above, I've had comments flags declined. Some of these obviously should have been and required a retraining or threshold adjustment on my part. Others, in my opinion, should have been removed as noise. Below is a small sampling of both types of comments.

Lessons and Observations¶

Replication to other sites would depend on site culture

As a (fairly) non-subjective site, Stack Overflow made a good test case for this. On a site like Community Building, Pets, Parenting or other site that accepts subjective answers, "too chatty" would be much harder to classify.

+/-1 has been discouraged

The observation I made on my own that comments with this type of content were distracting has been noticed by others as well. This was actually a very nice validation of my own process and some of the results posted on that thread show many such comments continue to be noise. Of course, this change did also force users to modify their content and may have added new patterns that can be utilized in future training.

Ability to automatically check flags would be great so that automated runs could be paused if it goes crazy

The process of checking that my flagging history remains accurate is time consuming. The status of a flag can't be acquired via the API. I've submitted a feature request for this information to be added to the API. With this information, flagging can be paused or stopped if X number of flags are declined.

Stack Overflow's volume of comments is a crutch.

Due to the high volume of comments and limited number of comment flags my account has available, I can afford to be picky on which comments I want to act on. The classifier itself is about 85% accurate in determining the type of comment. However, I artificially increase my accuracy by only acting on comments that have a very high classifier certainty by forcing this certainty level to meet or surpass my threshold values from above. Smaller sites, with a lower volume, don't have the benefit of having enough comments to be this picky. It is on these sites that a more feature based classifier would be important.

The human element is still unpredictable.

My classifier was trained utilizing my idea of how comments should be flagged. Prior to automating this, I was not 100% accurate. Additionally, moderators are not 100% accurate in their processing of flags. Users disagree on how these rules should be implemented, but are willing to assist in keeping the site clean. With more than 175K comments a week, every little bit helps.

Discussion¶

As my title states, my original question was whether or not I can teach a machine how to flag comments as I would. The answer to that is yes. The next question is whether this type of system would be helpful in cleaning up comments across Stack Overflow. My system works only on new comments created around each new UTC. Once my 100 flags are hit (or the API tells me to stop), it shuts down for the day. Having something automated go through historical comments or that can run all day would be beneficial.

Finally, now that I've admitted that I've been automatically flagging comments, can I continue to do so?

Update¶

This section has been updated multiple times since the original post. Most recently, it was updated May 3, 2015

As I mentioned in the introduction, this was originally published in December 2014. How is the system behaving now? It is performing very well.

Process Changes¶

In January 2015, another user was using a basic query to look for invalid comments. This caused a high number of moderator flags, many of which were declined. My process was caught in this mass decline. This resulted in 49 declined flags for a single day. This is, by far, the largest number of declined flags the process has generated in a day. It did, however, prompt a process change after consultation with the Stack Overflow moderators.

The process will no longer flag comments newer than 48 hours old. This provides users with a two day window to see a comment before the system will flag it. This single change has provided a huge improvement in terms of flag acceptance.

May 2015 (11 Months)¶

After nearly a year of running, these are my flagging statistics:

39938   comments flagged
39659   deemed helpful
279     declined

This provides a helpful rate of 99.3%. This is down just slightly from 99.36% in December. I attribute a large part of the dip to the issue mentioned above.

Here is an updated chart showing the rolling 10 day average for number of declined flags. I've had several stretches of multi-week time frames with no declined flags.

This is a busy chart, so I've narrowed it down to show just the last 90 days. From here you can see that in the past 90 days there have been only 10 declined flags.

Sept 2015 (15 Months)¶

It has been almost 15 months since the process started. In that time, the model has gotten more accurate. Since the last update in May, I've had only 3 declined comment flags:

52351   comments flagged
52069   deemed helpful
282     declined

This provides a helpful rate of 99.46%. Here is an updated chart showing the rolling 10 day average for number of declined flags. The 90 day window is not even worth showing. It has three days where a single flag was declined.

Summary of 2015¶

I processed comments 359 days out of the year. I missed three in January due to stopping it after a mass decline of flags (more later), I can't account for a missed day in July and August. I don't recall stopping it, but I missed July 3rd and August 19. I also missed December 28th due to a power issue. I flagged 35,960 comments. Of that, 111 were declined.

By month, this is the break down of rejected flags.

The blip at the end of November is due to new moderators being elected and adjusting to what other moderators consider "good" versus "bad" comments. I didn't see the spike in the April election which is interesting, but after a couple days in November it's back to normal. The January spike I mentioned above.

Interesting note: The longest stretch in the year with no declined flags was from August 13th through November 24th.

I had a secret addiction to the TF2 idling economy but I'm better now (honest)

2013-08-13T00:01:00-05:00

Introduction¶

On July 11, 2013, Valve released a patch to Team Fortress 2 to limit idling. Idling is the process of launching Team Fortress and then letting it run for a few hours and collecting item drops. It used to be simple to do by utilizing the -textmode game parameter. This would "launch" the game without graphics. From there, you just let it run and come back in a few hours with 8-10 new items (your weekly limit).

The idea behind this was to then trade or craft these into metal. The metal could be used to trade for keys, hats, etc. and allow you to be "rich". The quotes are there because I do realize that being rich in a virtual game does not make one rich in the real world. But, since the Mann-Conomy Update introduced this massive meta game to Team Fortress, I've tried to stay out of it.

I failed. I didn't just idle one account. I idled 17 accounts. That is between 136 and 170 items a week. That's between 68 to 85 scrap metal a week. That's between 7.55 and 9.44 refined metal a week. That is between 3 and 4 keys a week (depending on who I trade with). Keys are the backbone of the economy. Get enough of those and you can trade for anything else.

I honestly don't know what my goal was. I wasn't interested in the fancy hats (ooooh...shiny pixels). I didn't open crates with the keys. I did utilize the keys to get Tour of Duty tickets for me and some friends. That's honorable, right?

How did this happen?¶

This all started with the bot I built for Vipers to handle the raffles. I realized that I could automate much more than trading with players. I could automate trading between bots. I could automate crafting - which is relatively time consuming, especially when you have 136 items to go through. I could automate trading with established scrap bank bots. These bots will take any two weapon drops and give you one scrap, even if the two items can't be crafted together. The idea is that they get enough weapons that eventually they'll be able to craft it down to metal.

The set up¶

To idle 17 accounts I needed to figure out what my computer could handle. I had to figure out how to run multiple instances of Steam and Team Fortress at a time. Enter an application I'd utilized before: Sandboxie. This application provides isolated sandboxes for applications to run in. Normally Steam won't run more than once on a machine. But, if you launch it via Sandboxie, the host OS (and other Sandboxie environments) can't see that Steam is running in another instance.

A bit of experimentation showed that I could handle 6 accounts idling at a time and have the computer remain just barely usable. I created new accounts and split them into which day of the week they'd be run. I had three days out of the week designated for idling.

A batch script was built to launch the Sandboxie environments of the day that was to idle. Then it'd launch Team Fortress in each of those environments. Then I'd go to bed. When I woke up the next morning, I'd shut down the environments. I'd repeat this for two more nights. At the end of the third night, I had all the items I could get for the week.

Accounts could not be free accounts. This meant that I needed to either buy an item for each account, or trade for an "Upgrade to Premium Gift".

Getting items to a single account¶

The next step in the process was to convert all the items to metal. Normally, this would take place by either logging into each account and crafting there and then trading everything to a master account, or trading everything first then crafting. In either case, 17 accounts is a lot to handle and trading/crafting is rather boring.

My solution was to modify the raffle bot. I'd designate one account as a master account. This would be the account that received items. All others would dump items to it. I'd log into the account that was receiving items and initiate a trade with each other bot in turn. I'd issue a command add all and that bot would dump its inventory into trade. Making it through all the bots would take 5 minutes. Previously it would take me an hour or more to log into each account and manually add items to the trade windows and then confirm the trade on both sides. yawn

Crafting¶

The next step was crafting items to metal. The rule was that any two weapons utilized by the same class could be crafted to scrap metal. That sounds simple enough.

After lots of trial and error, I'd built a set of commands that would select compatible weapons and craft them together. This particular aspect wasn't documented by Steam (or the reverse engineered SteamKit2 library I was utilizing). Crafting would take about 15 seconds to get all compatible items to scrap and then combining 9 scrap to get a refined metal. This saved inventory space.

With the simple command craftall, all those new weapons would be crafted into metal and then combined into refined metal. This would be done in less than a minute. Previously, I'd have to initiate each crafting session, add all items per craft, wait for the craft to complete and then repeat. This was another 15-30 minutes saved.

Left overs¶

After crafting, there would still be left over weapons. There were weapons that had run out of same class items to craft with. The solution was to trade these away in groups of two to ScrapBank.me and get scrap metal back. This could be further combined to refined metal again wasting space.

Metal to Keys¶

At this point, human intervention was required again. I had to convert my metal to keys. I did this via trading. The easy way to do this was to find someone running a keybot, that would take metal for a key. Many exist, but all of them overcharge. The other option was to look for a trader that was getting rid of a bulk set of keys and then trade them. In either case, I'd end the week with 3-4 keys. I'd trade these for Tour of Duty tickets and go play a game with friends.

The actual work required on my part was starting the idling for the night, shutting down idling in the morning, starting the automated trade, crafting and trade left overs processes and finally trading metal for keys. What used to take me hours to do each week, I could now do in less than an hour (the bulk of which ended up being trading metal for keys).

What changed?¶

Valve released a patch to limit how effective idling was. Their "Mann-Conomy" was seeing rapid inflation. The price of keys (which they sold for physical money) in metal was rapidly increasing. From the time I started this to the time I ended it, it jumped from 2-3 refined per key to 9-10 per key, with no sign of stopping. Casual players were complaining they couldn't get enough items to trade for this stuff. The patch also required that you click to confirm each new received item.

I'm ok. Honest.¶

I tried for a few days to find an effective work around. I didn't. When I wasn't able to find one, I transferred all items to the master account and went through the process of converting to keys. With that, I purchased the final set of Tour of Duty tickets. The idle bots were shut down.

It's been a month now. The bots are still down. I'm still around. Looking back on this, I'm happy Valve broke this. It wasn't doing anything for me other than providing me with something to do: "Need to idle tonight", "Need to craft and trade today", etc. Now I have that time back.

I am pleased with the technical challenges I overcame to get this done though. Crafting was the biggest, but I think I'm most proud of the automated transfer of items to the master. Since the bots had to communicate via Steam and not via a local application, working out how I was going to do that took some time.

That, my friends, was my addiction to the meta game of a hat simulator in a first person shooter. I'm over it now and looking forward to some other project.

Give some refined, Win some prizes

2013-02-25T09:00:00-06:00

Introduction¶

Financially, Vipers is supported by donations from the community. When the community doesn't cover the cost, I end up covering the difference. This isn't my favorite thing to do in the world, but we've been pretty successful in the past. In recent months, though, we've been coming up short more frequently. This has motivated me (and the rest of the admin team to find ways to cover costs). Now we have one.

Welcome to the new raffle bot¶

I've built a Raffle bot based on SteamBot, which is the base of scrap.tf. Entries to raffles will be one entry per refined metal. You can have an unlimited number of entries. I will convert the refined metal to various prizes (with the goal being keys most of the time).

Then we'll have the system select a set of winners from the entries. A user can only win once per raffle, so even if you have a gigantic number of entries, your odds of winning more than one position are zero. Only one win per steam id.

Using the prices from backpack.tf, the bot will determine the "value" of the items within the trade.

How does this off set costs?¶

Items that are received that are of high value or any additional keys we can get based on raffle entries will be sold on various TF2 trading markets. The profits from these trades will be used to cover some community costs.

Keep high value items for future raffles?¶

This question was added based on feedback from the community. It was added on March 20, 2013

The original plan was to sell any such items. However, due to community feedback, I've changed my mind. We will utilize "high value" items as prizes for future raffles. These future raffles will not be announced until any running raffles are complete. It is also possible that such a raffle will run separately from the planned monthly ones.

Our first winners¶

Updated on April 3, 2013

Our first raffle completed at Midnight on April 1st. I was surprised by the number of entries we received. I'm even more surprised that the second one has begun and is already three quarters of the way to the number of entries it took a month to receive. People want those keys, and I saw mention of those Bill's Hats too.

The number of entries we received allowed us to completely cover the community costs that donations didn't cover. Thank you to all our players that entered!

Our first winners are:

Cashprizes: Winner of 10 keys
Popinfresh: Winner of 7 keys
Iamthebaron: Winner of 4 refined metal
That Guy From That Thing: Winner of 1 refined metal

Homing projectiles are awesome!

2012-06-30T08:02:00-05:00

Stupid soldier spam¶

The appeal of the crit server is fast game play, overpowered shots, and nearly instant death if you aren't paying attention. The down side is the soldier spam. Lots of it. It's not unusual to have a team of soldiers spamming rockets. This is part of the reason we stuck a class limit on Soldiers.

Pyro is a common way to counter a soldier firing at long range. The problem with pyro is that it has limited long range weapons in return. Unless you can sneak up on an enemy (not easy with spam and some of the maps), the pyro is stuck taking pot shots with either the Flare gun or the shotgun.

Two weeks ago, I added a plugin to the server that made Pyro much more effective at helping the team without needing to advance to the front line constantly.

Reflectiles¶

Reflected Projectiles - Reflectiles, if you will - become homing projectiles when a Pyro air blasts them away. These newly tracking projectiles will track an opposing team member and hunt them down. If the player being tracked dies before the projectile hits them, the projectile will select a new target.

Well, that seems unfair. How do you defend against a homing projectile as a soldier?

It is called Team Fortress 2. You have team mates. Utilize them. That homing projectile can be reflected again by a Pyro on your team. Each time a projectile is reflected it gets just a bit faster.

Source Code¶

Updated May 20, 2015 with link to GitHub instead of the old SVN, my apologies for missing that link when migrating to this blog It is important to note that this version hasn't been updated in a LONG time but was still functioning when Vipers shut down. If it doesn't work, the first thing to try is updating SourceMod's gamedata. This was the fix every other time it didn't work

The source code is released on Github.

The repository is: https://github.com/AWegnerGitHub/Vipers-Server-Plugins

Monitoring Language on the game servers

2012-04-22T12:13:00-05:00

Introduction¶

My admin tool of choice for the TF2 servers is HLSW. It's decent at allowing me to manage a server without ever needing to log to the server. My biggest complaint about it is that I can only watch the game chat of one server at a time. Sometimes, it's helpful to see an ongoing conversation to resolve minor problems before they become big ones. For example, claims of "hacking" usually turn out to be completely baseless. But, if multiple users (and more importantly, multiple trusted users) suddenly start mentioning a hacker, I can step in and resolve the problem without entering the server. HLSW is good for this. A hacker is confined to one server.

The biggest problem is when there are reports of lag across all of the game servers. Vipers has a dedicated machine that runs 5 game servers. If all five suddenly report lag, there is a problem somewhere. With HLSW, though, I can't see all of the servers at once. Thus, I've built a tool...

Chat Monitor¶

All chat that occurs on the servers is logged. I've used this to resolve complaints of unfair bans and reports of hackers. I built in a hook to these logs from the application template to quickly pull known aliases of users. It's been invaluable in solving problems of "what happened" on the servers.

I've expanded its usefulness. Now I can load a single page and see all chat activity occurring on all active game servers on a single screen. It provides, at a glance, a quick way to see if there are problems on the servers. It also allows me to step back from picking which server I think will be "bad" and monitor that. Now I can monitor all of them at once.

Inappropriate words¶

As a community, we've chosen to set a higher standard for our players. As such, we have a restriction on a total of 3 words and a few derivatives of those words. This system is in place because the community stepped forward and wanted to clean up the experience on the servers a bit. The problem with these higher standards is that we don't have admins on the game servers (or watch chat logs) 24 hours a day. Thus, while admins sleep, a troll can wander through and spew garbage. Unless a user reports this behavior, we will never be aware of it.

I've built a system to handle this automatically. The system will monitor chat logs across all servers. If a user hits the threshold for banning, they will be removed from the server and banned for a day. The logic to the system is this:

Automated removal logic¶

Inappropriate terms are configured with a "weight". This "weight" will be used to calculate whether or not a user surpasses a threshold set for being banned.
System monitors chat logs for configured terms.
If a term is found, the offending message is saved. The term weight is added to the user's current threshold value. If this is the user's first time saying one of these terms, they start at 0 and this weight is added.
If user exceeds the threshold, the system issues a ban to Sourcebans. The user is then kicked from the game server. The ban length will be 1 day.
The system keeps messages for a total of 5 minutes. If a message is older than that, the system forgets it.

Currently, the three inappropriate terms all have a threshold of 1. This means saying the words results in a ban. Homophobic, racist remarks aren't welcome on the Viper servers. We can't prevent it, but we can deal with offenses swiftly. The 5 minute window is added because the community requested that excessive swearing also be limited. We don't want to outright ban it, but they don't want a swear filled rant to occur after every match.

Thus, I built in the 5 minute window and the thresholds. The system is configured to catch common swear words, but the words have a low weight. It'd take repeated spamming of the words in a 5 minute window to reach the threshold and be removed from the server.

Updated removal logic¶

Updated May 17, 2012

The automated system has been active for almost a month. I'm finding that the system has been removing the same set of players every other day. They aren't learning. This is despite the message they are shown when removed from the server. I've made a change to the logic in how long a ban will last. It provides a 4 strike system:

First offence: Weight of term(s) said times 1. This means, for most cases, they are issued a single day ban.
Second offence: Weight of term(s) said times 3. This means, for most cases, they are issued a three day ban.
Third offence: Weight of terms said times 21. This means, for most cases, they are issued a three week ban.
Fourth offence: Permanent removal from the game servers.

The community has been very enthusiastic about how quickly users of inappropriate terms are removed. I've seen a few minor complaints about the permanent removal of users on the fourth offense. I've told the community that if a user protests the ban and if they can show they've learned our rules, I will provide one additional chance after the user has waited a minimum of a month from the last time they were banned. If they return to their previous activities, they will be re-banned and they will not be able to return in the future.

Update at shutdown¶

Updated January 20, 2015

In January 2015, Team Vipers shut down. With that shutdown, all chat monitoring also shut down. The system was active from April 23, 2012 to January 4, 2015, for a total of 987 days. In that time, 4457 bans were issued for inappropriate language. That is over 4 users a day being removed from our player base because they couldn't maintain a respectful attitude. I consider that a success. I believe Viper community members did too.

A new, more fair, RTD

2011-02-04T20:01:00-06:00

The old RTD¶

Viper's own LinuxLover, aka pheadxdll on the SourceMod forums, wrote the original version of the Roll the Dice plugin. It has provided countless hours of fun for players. After all, who doesn't love getting Toxic while standing near an enemy spawn and hearing the rage as they die immediately. It's usually worth the instant death that follows when the effect wears off ten seconds later.

There were problems though. The biggest was that the chances of getting a Good vs Bad roll were not equal. Instead, you had a roughly equal chance of getting any roll. There were 14 possible rolls. You had a 1 out of 14 chance of getting a specific effect. However, 9 of those effects were negative. Thus, you had a much higher chance of getting a negative effect vs a positive one. The other major problem was that it was very difficult to add new effects. Finally, with the released mod not being updated, and LinuxLover departing Vipers to handle his own community on the Randomizer server, it was next to impossible to get changes made.

NOTE: LinuxLover released a new version of RTD (the 0.4 branch) sometime after we had forked the version we had. The new version on SourceMod contains many of the same features we have. It does not, however, contain all of them.

The new RTD¶

The logic for how rolls are determined has been changed. There is now a list of Good Effects and a list of Bad Effects. When someone rolls, the first thing that is done is determine whether or not we are going to have a Good or Bad effect. That is a 50/50 chance. Then it randomly selects one of the effects that are active and appropriate for the player's current class that falls under the winning category. This should even out the Good vs Bad complaints.

Another change that I've added is that we can now more easily add effects. Some new effects have been added:

Powerplay: When Uber isn't enough...you need Kritz and Uber.
Freeze Bullets: You've been shot! You should run away, but you can't. You're frozen in place for the next ten seconds, if you're lucky.
Fire Bullets: A bullet wound isn't enough. You need to be on fire too.
No crits: Haha! You are on an all crits server and you just lost your crits. Go sit at the little kid table.
Valve Rockets: We've included something that may be slightly overkill. You tell me:
+9900% damage
+9000% clip size
+75% firing speed
+500 health on kill
10 seconds of crits on kill
+200% speed
+60% reload time

Because sometimes overkill is needed.

We've been testing performance of this over the last few months. Today is the day that everyone can get it without an admin being around.

For those that haven't seen it, here is one of our first tests back in November

The Headless Horseless Headmann: That's right, you can be the Halloween nightmare.

Update:

Homing Projectiles: Is that soldier aimbotting?! Nope. His rockets are just following you where ever you go. Boom! Oh, sniper arrows and pyro flares are probably something to avoid too.

Finally, I added in the ability to log rolls to a database so that we can find fancy stats and ensure it's rolling fairly. I'll update this post in a few months with some of our gathered stats.

Source Code¶

Updated May 20, 2015 with link to GitHub instead of the old SVN, my apologies for missing that link when migrating to this blog It is important to note that this version hasn't been updated in a LONG time and all effects may not work any more. Specifically, homing probably doesn't work because the Sidewinder extension had been broken by a Valve update in mid-2014. Other effects should continue to work

The source code is released on Github. It requires several common modules which are all available on the SourceMod forums.

The repository is: https://github.com/AWegnerGitHub/Vipers-Server-Plugins

Statistics¶

This section was updated in January 2015 after the shutdown

Good vs Bad¶

I don't have exact stats on Good vs Bad rolls before the rewrite, but we started logging all rolls with the rewrite. This is the split of Good vs Bad. I am very happy with an almost exactly 50/50 split over nearly a million RTDs.

Class with the most RTDs¶

Soldier was the most popular class on the crit server. The images below show number of times our soldiers got certain rolls. Note that God Mode is low because the community decided having both God Mode and Powerplay was redundant. They decided to keep Power play. God Mode was disabled about a year after this version was released. Homing is low because it wasn't implemented until about 18 months after the initial release.

Class with the least RTDs¶

Medics rolled RTD the least number of times. This makes sense for two reasons. First, they were not a common class on the crit server. When nearly everything is one hit, one kill, it's very hard to build uber. Second, when you do try to build Uber, ruining it with a 50/50 shot at getting a bad roll isn't good for the team. The low God Mode and Homing results are the same here as they were for the soldier.

Automated template for membership applications

2009-10-30T22:30:00-05:00

Introduction¶

Not too long ago, applications for new membership in Team Vipers consisted of someone wandering onto the site, posting a few words and an administrator saying they were accepted as a member. This worked when we were a small community. We aren't small any more. We have 5 servers and hundreds of players a day. Each server has their own sub-community. There are players joining the forums that entire groups of people have never met because they play exclusively on one game server.

Admins were also inconsistent in how (or if) they voted. Some admins didn't realize they could have a say, thinking it was a privilege granted to only the senior administrators. We've built a system to resolve many of these problems and to make the administration side easier.

What's new?¶

The new system presents all users with an application template. They fill out the form and the system handles the rest. The New Users and Applications subforum has been modified so that no one, except the bot, can create topics. The topics will only be created when the form is submitted. When a user applies to join Vipers, we will automatically include relevant information about the user as we know them:

HLStats information: This helps us get a sense of where they play and how often they play. It will help admins identify users they should recognize based on the servers they frequent (because we all know certain servers are better than others ;) cough Vanilla Nest vs Crits cough )
Ban information: This will check if the user has any recorded bans in Sourcebans. It's important to know if the user has been banned previously.
Known aliases: Pulling information from our chat logs and Valve's profile page, we can build a list of known aliases. This helps identify users that frequently change names but have been around a while.

There may be other features we add in the future as well.

Updated January 2012 I've removed HLStats from the servers and removed it from new application information. We have added a check of a user's Steam Reputation instead.

After a user applies, they are put into a two week hold. During this week it is expected they will stick around the forums and learn about the community they just applied to. Even better would be that they had done this before applying. While this two week hold is in place, the administration team will be able to cast their votes in a separate sub-forum. They can hold administration specific discussions - usually details that are important for admins to know, but don't need to be public. Once voting is complete, if they become a member, the system automatically grants appropriate forum and server related permissions.

Voting rules¶

Three admin "Yes" votes and zero "No" votes grants the user membership immediately after the two week window has expired
One or two "No" votes places a message on the user's application that the administrators are still considering the application
Three or more "No" votes rejects the application

Possible Responses¶

Accepted by admin team¶

If you reach a total of three or more admin "Yes" votes, and do not get three or more admin "No" votes, you will be accepted as a member of Vipers and automatically have your forum access modified. You will receive a message similar to this:

Denied by admin team¶

If you reach three or more "No" votes your application will be denied. This will occur even if you receive more "Yes" votes than "No" votes. Vipers is not a majority rule community. The decision has been made that if three admins do not feel comfortable accepting your application, you will not be granted membership. You will receive a message similar to this:

Denied due to lack of votes¶

To be accepted, your application requires a minimum of three "Yes" votes. If this can not be reached (and you also can't reach three "No" votes), your application will be rejected due to lack of votes from the admin team. This means that the admin team does not feel strongly either way about your application. Post on the forums. Play in the servers. Get to know our players and the community at large and then try again in a month. You will receive a message similar to this:

Denied because of age¶

Repeat after me: "Age does not equal maturity." However, it has a very strong correlation. Over time we have learned that younger players tend to bring a lower maturity level that most of the community does not care for. As such, we've decided to set a minimum age requirement of 16. If a user indicates they are less than that, the system will reject their application with a message similar to this:

Updated January 2010 This process has been in place for a few months now. It has gone very well. We've reduced the clutter in the applications forum. We've also seen the number of "forgotten" applications drop dramatically.

Original Announcement¶

The original announcement is posted here for future reference.

Zephyr is an automated robot designed to improve our new member application process. The current process, which involves copying a template to a new thread and the potential for applications to become misplaced, is cumbersome and inefficient. The goal of Zephyr is to remove as much of the manual process as possible.

Now, our new applicants will fill out an actual application online. The same information will be requested, but it will be in a more reliable format and will not require an applicant copy and paste anything between threads. This new application will be posted in the same forum as you're used to, and members will be freely able to comment and discuss the applicant via that thread. It will also display which date admin voting will be open, which is two full weeks after the original application post. This will hopefully cut down on any confusion related to the delay between application and voting.

As another note, from now on Zephyr will be the only user capable of creating new threads in the New Member Application forum. As previously stated, members will be able to post comments on existing threads, but the only new threads will come from the application process. This will keep the forum cleaner and help prevent applications from becoming lost or forgotten about.

Once again, Zephyr is an automated robot. It is not programmed to respond to comments or questions. Doing so will not get your question answered. As always, if you have questions or comments about the application process, Zephyr, or anything else, you're welcome to send them to any admin. We'll be more than happy to help.

You may all bow to your Robotic Overlord now.

Andrew Wegner | Ponderings of an Andy - programming

Choosing an ORM library for a new project

Project History¶

What happened in 2014?!¶

What has really changed?¶

Enter the database¶

How to choose?¶

Technical differences¶

Can a machine be taught to flag spam automatically

Introduction¶

What is this?¶

What problem does this address?¶

What are we doing?¶

What are our safety checks?¶

Results¶

Discussion¶

Link Analysis - Technical Explanation

Introduction¶

Set up¶

The database¶

The data¶

Parsing the data¶

Checking the links¶

Results¶

Wrap up¶

A proposal to fix broken links on Stack Overflow

The proposal¶

The Queue¶

The Automation¶

Process flows¶

Potential pitfalls¶

Analysis of links posted to Stack Overflow

Motivation¶

Results2¶

By status code¶

By most common¶

By post score¶

By number of views¶

By days since post created¶

By Ratio of Views:Days¶

Discussion¶

Footnotes¶

Follow up¶

Zephyr - The bot that watches for low quality vote requests

Introduction¶

Introducing Zephyr¶

How it works¶

Other bots¶

Updates¶

Commands¶

Upgrade from SQLite to MariaDB¶

Upgrade to utilize web sockets¶

Can a machine be taught to flag comments automatically

Introduction¶

Background¶

Results¶

Flagged but declined¶

Lessons and Observations¶

Discussion¶

Update¶

Process Changes¶

May 2015 (11 Months)¶

Sept 2015 (15 Months)¶

Summary of 2015¶

I had a secret addiction to the TF2 idling economy but I'm better now (honest)

Introduction¶

How did this happen?¶

The set up¶

Getting items to a single account¶

Crafting¶

Left overs¶

Metal to Keys¶

What changed?¶

I'm ok. Honest.¶

Give some refined, Win some prizes

Introduction¶

Welcome to the new raffle bot¶

How does this off set costs?¶

Keep high value items for future raffles?¶

Our first winners¶

Results²¶