Andrew Wegner | Ponderings of an Andy - automation

Can a machine be taught to flag spam automatically

2017-02-19T22:51:00-06:00

Introduction¶

This post was originally published on Meta Stack Exchange on February 20, 2017. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Exchange and post there.

The post was featured across the entire Stack Exchange network for a week, too. This drove a huge amount of traffic to the question and resulted in some valuable feedback:

TL;DR: We did it, so... yes.

What is this?¶

Charcoal is the organization behind the SmokeDetector bot and other nice things. This bot scans new posts across the entire network for spam posts and reports them to various chatrooms where people can act on them. If a post has been created or edited, anywhere on the network, we've probably seen it. The bot utilizes our knowledge of how spammers work and what they have previously posted to come up with common patterns and rules to detect spam in the new and updated posts. You've likely seen the SmokeDetector bot if you visit chatrooms such as Tavern on the Meta, Charcoal HQ, SO Close Vote Reviewers and others across the network. Over time, the bot has become very accurate.

Now we are leveraging the years of data and accuracy to automatically cast spam flags. With approximately 58,000 posts to draw from and over 46,000 true positives, we have a vast trove of data to utilize.

What problem does this address?¶

To put it simply, spam. Stack Exchange is one of the most popular networks of websites on the Internet, and all of it gets spammed at some point. Our statistics show that we see about 100 spam posts per day, on average over the last three months.

A decent chunk of this isn't the type you'd want to see at work (or at all). The faster we can get this off the home page, the better for all involved. Unfortunately, it's not unheard of for spam to last several hours, even on the larger sites such as Graphic Design.

Over the past three years, efforts with Smokey have significantly cut the time it takes for spam to be deleted. This project is an extension of that, and it's now well within reach to delete spam within seconds of it being posted.

What are we doing?¶

For over 3 years, SmokeDetector has reported potential spam across the Stack Exchange network so that users can flag the posts as appropriate. Users have provided feedback to inform the bot on whether the detection was correct or not (referred to as "feedback"). This feedback is stored in our web dashboard, metasmoke (code). Over time, we've used this feedback to evaluate our patterns ("reasons") and improve our accuracy. Several of our reasons are over 99.9% accurate.

Early last year, and after getting a baseline accuracy from jmac (thank you!), we realized we could use the system to automatically cast spam flags. On Stack Overflow the current accuracy of users flagging spam posts is 85.7%. Across the rest of the network users are 95.4% accurate. We determined we can beat those numbers and eliminate spam from Stack Overflow and the rest of the network even faster.

Without going into too much detail (if you really want it, it's available on our website), we leverage the accuracy of each existing reason to come up with a weight indicating how certain the system is that a post is spam. If this value exceeds a specific threshold, the system will cast up to three spam flags on the post. We cast multiple flags utilizing a number of different users' accounts and the Stack Exchange API. Via metasmoke, users are given the opportunity to enable their accounts to be used to flag spam (You can too, if you've made it this far). When a post is eligible for flagging because it exceeded the threshold set by each individual user, accounts are randomly selected from the pool of enabled users to cast a single flag each, up to a maximum of three per post so that we never unilaterally nuke something.

What are our safety checks?¶

We designed the entire system with accuracy and sanity checks in mind. Our design collaborations are available for your browsing pleasure (RFC 1, RFC 2, RFC 3 (no longer available)). The major things that make this system safe and sane are:

We give users a choice as to how accurate they want to be with their automatic flags. Before casting any flags, we check that the preferences the user has set result in a spam detection accuracy of over 99.5% over a sample of at least 1000 posts. Remember, the current accuracy of humans is 85.7% on SO and network wide it is 95.4%.
We do not unilaterally spam nuke a post, regardless of how sure we are it is spam. This means that a human must be involved to finish off a post, even on the few sites with lower spam thresholds.
We’ve designed the system to be tolerant of faults - if there’s a malfunction anywhere in the system, any user with access to SmokeDetector can immediately halt all automatic flagging - this includes all network moderators. If this happens, it needs a system administrator to step in to re-enable flags.
We've discussed this with a community manager and have their blessing on the project.

Results¶

We have been casting an average of 60-70 automatic flags per day for over two months, for a total of just over 4000 flags network wide. These flags were cast by 22 different users. In that time, we've had four false positives. We would like to be able to automatically cancel these particular cases. This isn't possible though, so we've created a feature request to retract flags via the API. In the mean time, the flags are either manually retracted by the user or declined by a moderator.

The above graph plots the weight of the reasons against its overall volume of reports and accuracy. As minimum weight increases, accuracy (yellow line and rightmost Y-axis) and total reports (blue line) on the left-hand scale increase. The green line represents the number of true positives, which are verified by SmokeDetector user feedback.

This shows the number of posts we've automatically flagged per day over the last month. The jump on February 15th, is due to increasing the number of automatic flags from 1 per post to 3 per post. You can see a live version of this graph on metasmoke's autoflagging page.

Spam arrives on Stack Exchange in waves. It is easy to see the time of day that many spam reports come in. The hours, above, are UTC time. The busiest spam times of day are the 8 hour block between 4am and Noon. We have affectionately named this "spam hour" in the chat room.

Our goal is to delete spam quickly and accurately. The graph shows the time it takes for a reported spam post to be removed from the network. This section has three trend lines that show these averages. The first, red section is when we were simply reporting the posts to chatrooms and all flags had to come from users. You can see we are pretty constant in the time it takes to remove spam during this period. It took, on average, just over five minutes to get a post removed.

The green trend line is when we were issuing a single automatic flag. At implementation, we eliminated a full minute from time to deletion and after a month we'd eliminated two full minutes compared to no automatic flags.

The last section, the orange, is when we implemented three automatic flags to most sites. This was rolled out last week, but it's already had a dramatic improvement on the time to deletion. We are seeing between 1 and 2 minutes to time to deletion.

As mentioned above, spam arrives in waves. The dashed and dotted lines on the graph show the average deletion time during these two different time periods. The dashed lines show deletion time during 4am and Noon UTC, the dotted lines show the rest of the 24 hour period. An interesting thing this graph shows is that time to deletion during spam hour was higher when we didn't cast any automatic flags. It was removed faster outside of spam hour. That reversed when we started issuing a single auto-flag. The spam hour time to deletion is slightly lower than the average. Comparing the two time periods though, time to deletion during non-spam hour at the end of the non-flagging time period and the end of the single flag period are roughly the same.

We'll update these in a few weeks too, to better show the trend we are seeing with three automatic flags.

Discussion¶

We are confident in SmokeDetector and the three years of history it has. We've had many talented developers assist us over the years and many more users have provided feedback to improve our detection rules. Let us know what you want us to elaborate on, features you're wondering about or would like to see added, or things we might have missed in the process or the tooling. Take a look at the feature we'd really like Stack Exchange to consider so that we can further improve this system (and some of the other community built systems). We'll have Charcoal members hanging around and answering your questions. Alternatively, feel free to drop into Charcoal HQ and have a chat.

Zephyr - The bot that watches for low quality vote requests

2015-03-12T23:34:00-05:00

Introduction¶

Stack Exchange receives thousands of questions per day across all of their sites. Not all of these are high quality posts. Fortunately, users of the Stack Exchange network are given tools to help keep that low quality stuff to a minimum. One of these tools is the chat network that spans the Stack Exchange sites.

In the chat rooms, a convention has arisen to tag a message as cv-pls for questions that need to be closed for one reason or another. Over time, this evolved to include other tags such as:

del-pls for a deletion request
spam for notification that spam made it through the already impressive spam filters
reopen for a reopen request
a few others to cover specific flag types (eg. Not an answer, Very Low Quality or Offensive)

Introducing Zephyr¶

The problem with these is that the requests are only seen by users active in the specific room where it was posted. Other users across the network miss the request. Zephyr was built to resolve this problem. Zephyr monitors several rooms where these types of requests are frequent. These requests all all posted into a single chat room. This provides users with a single room to monitor to see requests for multiple questions and sites across the network.

Here is an example of what Zephyr's chat activity looks like during a spam wave:

How it works¶

Zephyr utilizes the ChatExchange package to join and read the chat rooms. To do this, Zephyr required a dedicated account. I decided to run Zephyr with a dedicated account to completely separate the bot that would sit and watch multiple chat rooms 24/7 from my account. Zephyr maintains a small SQLite database of all the posts that it records. The idea behind this, is that eventually this data will be utilized to train other systems on unwanted content. This information is pulled via the API.

Zephyr watches the chat rooms for specific string patterns. If these patterns are matched, a message is posted if should_post is True for the matched pattern.

Overall, a nice simple application. It performs some pattern matching and a couple API calls.

Other bots¶

In addition to watching user activity, Zephyr also watches two other quality bots that patrol Stack Exchange for low quality content: SmokeDetector and Phamhilator. If either of these bots detect spam, Zephyr takes note of the information by recording it to the database, but not reposting. Since both of those bots post their reports, it didn't make sense for Zephyr to add a second (or third, if both of the others detected spam) message to the chat room. The information is recorded, though, to help future training for other systems.

Updates¶

Updated May 8, 2015

Over time Zephyr has been updated to include new rooms to monitor or new patterns to match. Those changes are small (and simple). There are, however, a few larger changes that I'd like to note below.

Commands¶

The other bots that Zephyr monitors respond to user input. Zephyr has very little that requires user interaction since all of it's posts are generated by user input. However, there have been times where I, as the bot owner, would like to be able to issue certain commands to it. My most common desire is to see a report of how many spam posts Zephyr has seen. Thus, Zephyr now responds to the command spamreport from me. It then prints out a nice summary of information. This information has been utilized in SmokeDetector to watch for commonly spammed domains.

Upgrade from SQLite to MariaDB¶

Zephyr was originally built against an SQLite database. This worked, but was getting slower as more data was being added. This slow down was beginning to affect performance. I started seeing this error more and more frequently:

Traceback (most recent call last):
  File "H:\python-virtualenvs\zephyr-se-voterequests\lib\site-packages\sqlalchemy\pool.py", line 255, in _close_connection
    self._dialect.do_close(connection)
  File "H:\python-virtualenvs\zephyr-se-voterequests\lib\site-packages\sqlalchemy\engine\default.py", line 418, in do_close
    dbapi_connection.close()
ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 4824 and this is thread id 4660

After spending a lot of time troubleshooting and not resolving it to my satisfaction, I decided to upgrade to a more robust database. I'd used MySQL/MariaDB before and I happened to have another application utilizing MariaDB at the moment so that is the solution I picked.

The first step was transferring data. I learned that there isn't a decent utility to do a straight migration. So, I took these steps to transfer the data:

Export table structures and data from SQLite
Convert the SQLite dump to MySQL format. Though both systems use SQL, there are slight differences in dialect. I utilized this Python script as a starting point. It got me most of the way there, but not completely.
Data clean up. Ugh. The dreaded part of the job for anyone who handles data. Fortunately, the script above did most of the work. I ended up fixing a couple stray back ticks that didn't convert properly, escaping a very extra quotation marks, and replacing a few "smart quotes" (of both the left and right variety). I wish data at the office job was this easy to clean...
Import into MariaDB

Since the transfer to MariaDB, I've noticed no performance degradation. The error about threads has been eliminated as well.

Upgrade to utilize web sockets¶

Originally, Zephyr used the watch method when monitoring a room. This method would long poll the room. It turns out that this is pretty unreliable. I'd get multiple errors through out the week, ranging from Connection Aborted errors to random 404 messages. The solution has been to switch to watch_socket. The only time I've had problems since this switch is when the Stack Exchange web sockets go down. This saves a lot of restarts to get everything up and running again.

Can a machine be taught to flag comments automatically

2015-01-02T08:47:00-06:00

Introduction¶

This post was originally published by me on Meta Stack Overflow on December 14, 2014. I've republished it here so that I can easily update information related to recent developments. If you have questions or comments, I highly encourage you to visit the question on Meta Stack Overflow and post there.

TL;DR: Yes it can.

Background¶

On June 27, 2014 Skynet awoke. It looked at Stack Overflow and thought "Why are all these people being so chatty and talking about obsolete things? I should nuke them all!" Fortunately, Skynet was a baby and only had access to my 100 comment flags a day.

Prior to this activation date, the system was fed with 10,000 "Good Comments", "Obsolete" comments and "Too Chatty" comments. These comments were taken from the Stack Exchange Data Explorer. The "Obsolete" and "Too Chatty" comment types had to meet the following criteria:

Total comment length of less than 100 characters
Comment has a 0 score
Had variations of the following phrases:

Phrases

'%mark%answer%'
'%mark%accept%'
'%accept%answer%'
'%lease%accept%'
'%mark%answer%'
'%thank%you%'
'%thx%you%'
'%.....'
'+1%'
'-1%'

"Good Comments" were assumed, initially, to be anything that didn't fall into the above criteria

This provided a base of 30,000 comments that were roughly categorized into 3 distinct groups. Manually scanning the classifications took several weeks, and through this some of the groupings were changed to reflect a more appropriate classification. Not all comments less than 100 characters starting with "Thank you" are "too chatty", just as not all comments over 100 characters are good comments. I reclassified these comments as if I had encountered them on Stack Overflow.

My next step was to train a classifier. I had initially assumed that I'd start with a Naive Bayes to get a baseline and then work to something more complicated from there. Perhaps, extract text features, user information, etc. and build a fancy classifier. My initial tests showed that the Naive Bayes was accurate 80-90% of the time with test data.

I combined the classifier's certainty of classification with an acceptable threshold of when I'd allow a flag to be issued in my name. Tuning these threshold took a few weeks but eventually I determined the following thresholds were appropriate for my use:

Type            | Threshold     | Flagging Enabled
--------------------------------------------------
too chatty      | 0.9997        | True
obsolete        | 0.99          | True
good comment    | 0.9999        | False

When a comment is classified, if it exceeds the threshold for one of the above, it is recorded into my database for future retraining. If flagging is enabled, the API is utilized to issue an appropriate flag. Obviously, I don't want to flag good comments, but I do want to record them so that I can reuse the data in a later training step.

Results¶

What have the results of this experiment been? From my point of view, I'd venture that it's been successful. I have automatically flagged over 17,000 comments. As of December 17, 2014, the process has been running for 173 days. My comment flagging stats are currently:

26885   comments flagged
26714   deemed helpful
171     declined

Started at (approximately):

9885    comments flagged
9847    deemed helpful
38      declined

This gives me an overall accuracy of 99.36%. Down from 99.61% when no automated process was involved.

There are pictures that help tell this story too. In this first one, we see that the rolling 10 day average for the number of declined flags has stayed below two flags a day. In October, there was a two week period where the rolling average was 0 and nearly a month long period where the system did not make any mistakes.

Since November, the number of mistakes has climbed slightly. The biggest number of mistakes it has made was the opening day of Winter Bash 2014. Purely speculation, but I believe this was the moderators being protective of content and not wanting people to farm the Resolution hat. Of course, I don't know this. Another theory I have about this uptick since November is the adjustment to day light saving time. My process starts 10 minutes after UTC. It is possible that this earlier hour has caused my flags to be processed by a different moderator, or a moderator that is more awake/less hungry/in a different mood than previously at this point in the daily rotation cycle or because they lost their keys that day.

Except for 3 days, since June 27th, the process has flagged 100 comments a day. In this chart, you can see the number of declined comment flags along the bottom.

Finally, this chart shows the number of comments that the system wanted to act on (and a rolling 5 day average). When the system was brought online, it was acting on 700-800 comments a day (saving to my local database). Many of these were being classified as "Good Comments". You can see the day that I adjusted the threshold for "Good Comments" to be acted upon (saved). The drop in the number of comments the system saved is dramatic. Instead of saving 700-800 comments daily, the system now averages about 150 comments to save. Since I don't flag "Good Comments", I feel this is the appropriate action to take.

Flagged but declined¶

As shown above, I've had comments flags declined. Some of these obviously should have been and required a retraining or threshold adjustment on my part. Others, in my opinion, should have been removed as noise. Below is a small sampling of both types of comments.

Lessons and Observations¶

Replication to other sites would depend on site culture

As a (fairly) non-subjective site, Stack Overflow made a good test case for this. On a site like Community Building, Pets, Parenting or other site that accepts subjective answers, "too chatty" would be much harder to classify.

+/-1 has been discouraged

The observation I made on my own that comments with this type of content were distracting has been noticed by others as well. This was actually a very nice validation of my own process and some of the results posted on that thread show many such comments continue to be noise. Of course, this change did also force users to modify their content and may have added new patterns that can be utilized in future training.

Ability to automatically check flags would be great so that automated runs could be paused if it goes crazy

The process of checking that my flagging history remains accurate is time consuming. The status of a flag can't be acquired via the API. I've submitted a feature request for this information to be added to the API. With this information, flagging can be paused or stopped if X number of flags are declined.

Stack Overflow's volume of comments is a crutch.

Due to the high volume of comments and limited number of comment flags my account has available, I can afford to be picky on which comments I want to act on. The classifier itself is about 85% accurate in determining the type of comment. However, I artificially increase my accuracy by only acting on comments that have a very high classifier certainty by forcing this certainty level to meet or surpass my threshold values from above. Smaller sites, with a lower volume, don't have the benefit of having enough comments to be this picky. It is on these sites that a more feature based classifier would be important.

The human element is still unpredictable.

My classifier was trained utilizing my idea of how comments should be flagged. Prior to automating this, I was not 100% accurate. Additionally, moderators are not 100% accurate in their processing of flags. Users disagree on how these rules should be implemented, but are willing to assist in keeping the site clean. With more than 175K comments a week, every little bit helps.

Discussion¶

As my title states, my original question was whether or not I can teach a machine how to flag comments as I would. The answer to that is yes. The next question is whether this type of system would be helpful in cleaning up comments across Stack Overflow. My system works only on new comments created around each new UTC. Once my 100 flags are hit (or the API tells me to stop), it shuts down for the day. Having something automated go through historical comments or that can run all day would be beneficial.

Finally, now that I've admitted that I've been automatically flagging comments, can I continue to do so?

Update¶

This section has been updated multiple times since the original post. Most recently, it was updated May 3, 2015

As I mentioned in the introduction, this was originally published in December 2014. How is the system behaving now? It is performing very well.

Process Changes¶

In January 2015, another user was using a basic query to look for invalid comments. This caused a high number of moderator flags, many of which were declined. My process was caught in this mass decline. This resulted in 49 declined flags for a single day. This is, by far, the largest number of declined flags the process has generated in a day. It did, however, prompt a process change after consultation with the Stack Overflow moderators.

The process will no longer flag comments newer than 48 hours old. This provides users with a two day window to see a comment before the system will flag it. This single change has provided a huge improvement in terms of flag acceptance.

May 2015 (11 Months)¶

After nearly a year of running, these are my flagging statistics:

39938   comments flagged
39659   deemed helpful
279     declined

This provides a helpful rate of 99.3%. This is down just slightly from 99.36% in December. I attribute a large part of the dip to the issue mentioned above.

Here is an updated chart showing the rolling 10 day average for number of declined flags. I've had several stretches of multi-week time frames with no declined flags.

This is a busy chart, so I've narrowed it down to show just the last 90 days. From here you can see that in the past 90 days there have been only 10 declined flags.

Sept 2015 (15 Months)¶

It has been almost 15 months since the process started. In that time, the model has gotten more accurate. Since the last update in May, I've had only 3 declined comment flags:

52351   comments flagged
52069   deemed helpful
282     declined

This provides a helpful rate of 99.46%. Here is an updated chart showing the rolling 10 day average for number of declined flags. The 90 day window is not even worth showing. It has three days where a single flag was declined.

Summary of 2015¶

I processed comments 359 days out of the year. I missed three in January due to stopping it after a mass decline of flags (more later), I can't account for a missed day in July and August. I don't recall stopping it, but I missed July 3rd and August 19. I also missed December 28th due to a power issue. I flagged 35,960 comments. Of that, 111 were declined.

By month, this is the break down of rejected flags.

The blip at the end of November is due to new moderators being elected and adjusting to what other moderators consider "good" versus "bad" comments. I didn't see the spike in the April election which is interesting, but after a couple days in November it's back to normal. The January spike I mentioned above.

Interesting note: The longest stretch in the year with no declined flags was from August 13th through November 24th.

I had a secret addiction to the TF2 idling economy but I'm better now (honest)

2013-08-13T00:01:00-05:00

Introduction¶

On July 11, 2013, Valve released a patch to Team Fortress 2 to limit idling. Idling is the process of launching Team Fortress and then letting it run for a few hours and collecting item drops. It used to be simple to do by utilizing the -textmode game parameter. This would "launch" the game without graphics. From there, you just let it run and come back in a few hours with 8-10 new items (your weekly limit).

The idea behind this was to then trade or craft these into metal. The metal could be used to trade for keys, hats, etc. and all you to be "rich". The quotes are there because I do realize that being rich in a virtual game does not make one rich in the real world. But, since the Mann-Conomy Update introduced this massive meta game to Team Fortress, I've tried to stay out of it.

I failed. I didn't just idle one account. I idled 17 accounts. That is between 136 and 170 items a week. That's between 68 to 85 scrap metal a week. That's between 7.55 and 9.44 refined metal a week. That is between 3 and 4 keys a week (depending on who I trade with). Keys are the backbone of the economy. Get enough of those and you can trade for anything else.

I honestly don't know what my goal was. I wasn't interested in the fancy hats (ooooh...shiny pixels). I didn't open crates with the keys. I did utilize the keys to get Tour of Duty tickets for me and some friends. That's honorable, right?

How did this happen?¶

This all started with the bot I built for Vipers to handle the raffles. I realized that I could automate much more than trading with players. I could automate trading between bots. I could automate crafting - which is relatively time consuming, especially when you have 136 items to go through. I could automate trading with established scrap bank bots. These bots will take any two weapon drops and give you one scrap, even if the two items can't be crafted together. The idea is that they get enough weapons that eventually they'll be able to craft it down to metal.

The set up¶

To idle 17 accounts I needed to figure out what my computer could handle. I had to figure out how to run multiple instances of Steam and Team Fortress at a time. Enter an application I'd utilized before: Sandboxie. This application provides isolated sandboxes for applications to run in. Normally Steam won't run more than once on a machine. But, if you launch it via Sandboxie, the host OS (and other Sandboxie environments) can't see that Steam is running in another instance.

A bit of experimentation showed that I could handle 6 accounts idling at a time and have the computer remain just barely usable. I created new accounts and split them into which day of the week they'd be run. I had three days out of the week designated for idling.

A batch script was built to launch the Sandboxie environments of the day that was to idle. Then it'd launch Team Fortress in each of those environments. Then I'd go to bed. When I woke up the next morning, I'd shut down the environments. I'd repeat this for two more nights. At the end of the third night, I had all the items I could get for the week.

Accounts could not be free accounts. This meant that I needed to either buy an item for each account, or trade for an "Upgrade to Premium Gift".

Getting items to a single account¶

The next step in the process was to convert all the items to metal. Normally, this would take place by either logging into each account and crafting there and then trading everything to a master account, or trading everything first then crafting. In either case, 17 accounts is a lot to handle and trading/crafting is rather boring.

My solution was to modify the raffle bot. I'd designate one account as a master account. This would be the account that received items. All others would dump items to it. I'd log into the account that was receiving items and initiate a trade with each other bot in turn. I'd issue a command add all and that bot would dump it's inventory into trade. Making it through all the bots would take 5 minutes. Previously it would take me an hour or more to log into each account and manually add items to the trade windows and then confirm the trade on both sides. yawn

Crafting¶

The next step was crafting items to metal. The rule was that any two weapons utilized by the same class could be crafted to scrap metal. That sounds simple enough.

After lots of trial and error, I'd built a set of commands that would select compatible weapons and craft them together. This particular aspect wasn't documented by Steam (or the reverse engineered SteamKit2 library I was utilizing). Crafting would take about 15 seconds to get all compatible items to scrap and then combining 9 scrap to get a refined metal. This saved inventory space.

With the simple command craftall, all those new weapons would be crafted into metal and then combined into refined metal. This would be done in less than a minute. Previously, I'd have to initiate each crafting session, add all items per craft, wait for the craft to complete and then repeat. This was another 15-30 minutes saved.

Left overs¶

After crafting, there would still be left over weapons. There were weapons that had run out of same class items to craft with. The solution was to trade these away in groups of two to ScrapBank.me and get scrap metal back. This could be further combined to refined metal again wasting space.

Metal to Keys¶

At this point, human intervention was required again. I had to convert my metal to keys. I did this via trading. The easy way to do this was to find someone running a keybot, that would take metal for a key. Many exist, but all of them overcharge. The other option was to look for a trader that was getting rid of a bulk set of keys and then trade them. In either case, I'd end the week with 3-4 keys. I'd trade these for Tour of Duty tickets and go play a game with friends.

The actual work required on my part was starting the idling for the night, shutting down idling in the morning, starting the automated trade, crafting and trade left overs processes and finally trading metal for keys. What used to take me hours to do each week, I could not do in less than an hour (the bulk of which ended up being trading metal for keys).

What changed?¶

Valve released a patch to limit how effective idling was. Their "Mann-Conomy" was seeing rapid inflation. The price of keys (which they sold for physical money) in metal was rapidly increasing. From the time I started this to the time I ended it, it jumped from 2-3 refined per key to 9-10 per key, with no sign of stopping. Casual players were complaining they couldn't get enough items to trade for this stuff. The patch also required that you click to confirm each new received item.

I'm ok. Honest.¶

I tried for a few days to find an effective work around. I didn't. When I wasn't able to find one, I transferred all items to the master account and went through the process of converting to keys. With that, I purchased the final set of Tour of Duty tickets. The idle bots were shut down.

It's been a month now. The bots are still down. I'm still around. Looking back on this, I'm happy Valve broke this. It wasn't doing anything for me other than providing me with something to do: "Need to idle tonight", "Need to craft and trade today", etc. Now I have that time back.

I am pleased with the technical challenges I over came to get this done though. Crafting was the biggest, but I think I'm most proud of the automated transfer of items to the master. Since the bots had to communicate via Steam and not via a local application, working out how I was going to do that took some time.

That, my friends, was my addiction to the meta game of a hat simulator in a first person shooter. I'm over it now and looking forward to some other project.

Give some refined, Win some prizes

2013-02-25T09:00:00-06:00

Introduction¶

Financially, Vipers is supported by donations from the community. When the community doesn't cover the cost, I end up covering the difference. This isn't my favorite thing to do in the world, but we've been pretty successful in the past. In recent months, though, we've been coming up short more frequently. This has motivated me (and the rest of the admin team to find ways to cover costs). Now we have one.

Welcome to the new raffle bot¶

I've built a Raffle bot based on SteamBot, which is the base of scrap.tf. Entries to raffles will be one entry per refined metal. You can have an unlimited number of entries. I will convert the refined metal to various prizes (with the goal being keys most of the time).

Then we'll have the system select a set of winners from the entries. A user can only win once per raffle, so even if you you have a gigantic number of entries, your odds of winning more than one position are zero. Only one win per steam id.

Using the prices from backpack.tf, the bot will determine the "value" of the items within the trade.

How does this off set costs?¶

Items that are received that are of high value or any additional keys we can get based on raffle entries will be sold on various TF2 trading markets. The profits from these trades will be used to cover some community costs.

Keep high value items for future raffles?¶

This question was added based on feed back from the community. It was added on March 20, 2013

The original plan was to sell any such items. However, due to community feedback, I've changed my mind. We will utilize "high value" items as prizes for future raffles. These future raffles will not be announced until any running raffles are complete. It is also possible that such a raffle will run separately from the planned monthly ones.

Our first winners¶

Updated on April 3, 2013

Our first raffle completed at Midnight on April 1st. I was surprised by the number of entries we received. I'm even more surprised that the second one has begun and is already three quarters of the way to the number of entries it took a month to receive. People want those keys, and I saw mention of those Bill's Hats too.

The number of entries we received allowed us to completely cover the community costs that donations didn't cover. Thank you to all our players that entered!

Our first winners are:

Cashprizes: Winner of 10 keys
Popinfresh: Winner of 7 keys
Iamthebaron: Winner of 4 refined metal
That Guy From That Thing: Winner of 1 refined metal

Monitoring Language on the game servers

2012-04-22T12:13:00-05:00

Introduction¶

My admin tool of choice for the TF2 servers is HLSW. It's decent at allowing me to manage a server without ever needing to log to the server. My biggest complaint about it is that I can only watch the game chat of one server at a time. Sometimes, it's helpful to see an ongoing conversation to resolve minor problems before they become big ones. For example, claims of "hacking" usually turn out to be completely baseless. But, if multiple users (and more importantly, multiple trusted users) suddenly start mentioning a hacker, I can step in and resolve the problem without entering the server. HLSW is good for this. A hacker is confined to one server.

The biggest problem is when there are reports of lag across all of the game servers. Vipers has a dedicated machine that runs 5 game servers. If all five suddenly report lag, there is a problem somewhere. With HLSW, though, I can't see all of the servers at once. Thus, I've built a tool...

Chat Monitor¶

All chat that occurs on the servers is logged. I've used this to resolve complaints of unfair bans and reports of hackers. I built in a hook to these logs from the application template to quickly pull known aliases of users. It's been invaluable in solving problems of "what happened" on the servers.

I've expanded it's usefulness. Now I can load a single page and see all chat activity occurring on all active game servers on a single screen. It provides, at a glance, a quick way to see if there are problems on the servers. It also allows me to step back from picking which server I think will be "bad" and monitor that. Now I can monitor all of them at once.

Inappropriate words¶

As a community, we've chosen to set a higher standard for our players. As such, we have a restriction on a total of 3 words and a few derivatives of those words. This system is in place because the community stepped forward and wanted to clean up the experience on the servers a bit. The problem with these higher standards is that we don't have admins on the game servers (or watch chat logs) 24 hours a day. Thus, while admins sleep, a troll can wander through and spew garbage. Unless a user reports this behavior, we will never be aware of it.

I've built a system to handle this automatically. The system will monitor chat logs across all servers. If a user hits the threshold for banning, they will be removed from the server and banned for a day. The logic to the system is this:

Automated removal logic¶

Inappropriate terms are configured with a "weight". This "weight" will be used to calculate whether or not a user surpasses a threshold set for being banned.
System monitors chat logs for configured terms.
If a term is found, the offending message is saved. The term weight is added to the user's current threshold value. If this is the user's first time saying one of these terms, they start at 0 and this weight is added.
If user exceeds the threshold, the system issues a ban to Sourcebans. The user is then kicked from the game server. The ban length will be 1 day.
The system keeps messages for a total of 5 minutes. If a message is older than that, the system forgets it.

Currently, the three inappropriate terms all have a threshold of 1. This means they saying the words results in a ban. Homophobic, racist remarks aren't welcome on the Viper servers. We can't prevent it, but we can deal with offenses swiftly. The 5 minute window is added because the community requested that excessive swearing also be limited. We don't want to outright ban it, but they don't want a swear filled rant to occur after every match.

Thus, I built in the 5 minute window and the thresholds. The system is configured to catch common swear words, but the words have a low weight. It'd take repeated spamming of the words in a 5 minute window to reach the threshold and be removed from the server.

Updated removal logic¶

Updated May 17, 2012

The automated system has been active for almost a month. I'm finding that the system has been removing the same set of players every other day. They aren't learning. This is despite the message they are shown when removed from the server. I've made a change to the logic in how long a ban will last. It provides a 4 strike system:

First offence: Weight of term(s) said times 1. This means, for most cases, they are issued a single day ban.
Second offence: Weight of term(s) said times 3. This means, for most cases, they are issued a three day ban.
Third offence: Weight of terms said times 21. This means, for most cases, they are issued a three week ban.
Forth offence: Permanent removal from the game servers.

The community has been very enthusiastic about how quickly users of inappropriate terms are removed. I've seen a few minor complaints about the permanent removal of users on the forth offense. I've told the community that if a user protests the ban and if they can show they've learned our rules, I will provide one additional chance after the user has waited a minimum of a month from from last time they were banned. If they return to their previous activities, they will be re-banned and they will not be able to return in the future.

Update at shutdown¶

Updated January 20, 2015

In January 2015, Team Vipers shut down. With that shutdown, all chat monitoring also shut down. The system was active from April 23, 2012 to January 4, 2015, for a total of 987 days. In that time, 4457 bans were issued for inappropriate language. That is over 4 users a day being removed from our player base because they couldn't maintain a respectful attitude. I consider that a success. I believe Viper community members did too.

Automated template for membership applications

2009-10-30T22:30:00-05:00

Introduction¶

Not to long ago, applications for new membership in Team Vipers consisted of someone wandering onto the site, posting a few words and an administrator saying they were accepted as a member. This worked when we were a small community. We aren't small any more. We have 5 servers and hundreds of players a day. Each server has their own sub-community. There are players joining the forums than entire groups of people have never met because they play exclusively on one game server.

Admins were also inconsistent in how (or if) they voted. Some admins didn't realize they could have a say, thinking it was a privilege granted to only the senior administrators. We've built a system to resolve many of these problems and to make the administration side easier.

What's new?¶

The new system presents all users with an application template. They fill out the form and the system handles the rest. The New Users and Applications subforum has been modified so that no one, except the bot, can create topics. They topics will only be created when the form is submitted. When a user applies to join Vipers, we will automatically include relevant information about the user as we know them:

HLStats information: This helps us get a sense of where they play and how often they play. It will help admins identify users they should recognize based on the servers they frequent (because we all know certain servers are better than others ;) cough Vanilla Nest vs Crits cough )
Ban information: This will check if the user has any recorded bans in Sourcebans. It's important to know if the user has been banned previously.
Known aliases: Pulling information from our chat logs and Valve's profile page, we can build a list of known aliases. This helps identify users that frequently change names but have been around a while.

There may be other features we add in the future as well.

Updated January 2012 I've removed HLStats from the servers and removed it from new application information. We have added a check of a user's Steam Reputation instead.

After a user applies, they are put into a two week hold. During this week it is expected they will stick around the forums and learn about the community they just applied to. Even better would be that they had done this before applying. While this two week hold is in place, the administration team will be able to cast they votes in a separate sub-forum. They can hold administration specific discussions - usually details that are important for admins to know, but don't need to be public. Once voting is complete, if they become a member, the system automatically grants appropriate forum and server related permissions.

Voting rules¶

Three admin "Yes" votes and zero "No" votes grants the user membership immediately after the two week window has expired
One or two "No" votes places a message on the user's application that the administrators are still considering the application
Three or more "No" votes rejects the application

Possible Responses¶

Accepted by admin team¶

If you reach a total of three or more admin "Yes" votes, and do not get three or more admin "No" votes, you will be accepted as a member of Vipers and automatically have your forum access modified. You will receive a message similar to this:

Denied by admin team¶

If you reach three or more "No" votes your application will be denied. This will occur even if you receive more "Yes" votes than "No" votes. Vipers is not a majority rule community. The decision has been made that if three admins do not feel comfortable accepting your application, you will not be granted membership. You will receive a message similar to this:

Denied due to lack of votes¶

To be accepted, your application requires a minimum of three "Yes" votes. If this can not be reached (and you also can't reach three "No" votes), your application will be rejected due to lack of votes from the admin team. This means that the admin team does not feel strongly either way about your application. Post on the forums. Play in the servers. Get to know our players and the community at large and then try again in a month. You will receive a message similar to this:

Denied because of age¶

Repeat after me: "Age does not equal maturity." However, it has a very strong correlation. Over time we have learned that younger players tend to bring a lower maturity level that most of the community does not care for. As such, we've decided to set a minimum age requirement of 16. If a user indicates they are less than that, the system will reject their application with a message similar to this:

Updated January 2010 This process has been in place for a few months now. It has gone very well. We've reduced the clutter in the applications forum. We've also seen the number of "forgotten" applications drop dramatically.

Original Announcement¶

The original announcement is posted here for future reference.

Zephyr is an automated robot designed to improve our new member application process. The current process, which involves copying a template to a new thread and the potential for applications to become misplaced, is cumbersome and inefficient. The goal of Zephyr is to remove as much of the manual process as possible.

Now, our new applicants will fill out an actual application online. The same information will be requested, but it will be in a more reliable format and will not require an applicant copy and paste anything between threads. This new application will be posted in the same forum as you're used to, and members will be freely able to comment and discuss the applicant via that thread. It will also display which date admin voting will be open, which is two full weeks after the original application post. This will hopefully cut down on any confusion related to the delay between application and voting.

As another note, from now on Zephyr will be the only user capable of creating new threads in the New Member Application forum. As previously stated, members will be able to post comments on existing threads, but the only new threads will come from the application process. This will keep the forum cleaner and help prevent applications from becoming lost or forgotten about.

Once again, Zephyr is an automated robot. It is not programmed to respond to comments or questions. Doing so will not get your question answered. As always, if you have questions or comments about the application process, Zephyr, or anything else, you're welcome to send them to any admin. We'll be more than happy to help.

You may all bow to your Robotic Overlord now.