The Ultimate Server Part 4 - Newsgroups and SABnzbd

This post was last updated on 12/30/2014

Glad to see you stuck with me up to this point. We now have a Windows system that is all configured and set up with a storage system. Now we need to add programs that will start filling up that storage. The first thing we are going to set up is the Newsgroup access.

What are newsgroups?

Newsgroups (also known as Usenet) use an old protocol for discussion over the internet. Support for binary files was added along the way and it then became a source for sharing files. This not a peer-to-peer system, meaning you do not connect directly to other people sharing their files. Instead, you connect to a Usenet service provider, browse groups (think of them as bulletin boards), and download what you want. How long these files stay available depends on your Usenet service provider. Most providers offer over 2000 days.

Because boards can have files from several years and anyone is free to post whatever they want, newsgroups are understandably hard to browse. A solution to this surfaced in the form of NZB files. Similar to torrent files, these files don't contain any content by themselves, and instead tell some client software where to look and what to download. There are NZB indexing sites which host these files and make them easily searchable.

How to get access

Let me start by saying that access to Newsgroups is not free. If you are looking for a completely free setup, then skip this section and move along to when we set up our torrent software.

If you are still reading, then you are okay with paying for access. The first step is picking a Usenet provider. A bing search for "usenet providers" or "newsgroups" will show you that there is no shortage of selection. A good resource for comparing them is the usenet subreddit and it's corresponding wiki of providers. The wiki page is a little out of date, but it's still generally accurate.

Personally, I use newshosting.com who offer an unlimited download package for $10/mo or $99/year. What's nice is that the yearly package also includes VPN, which will come in handy if you will also be using torrents. The only place cheaper I've found is newsdemon, who are notorious for shady tactics. I encourage you to check out the subreddit linked above and investigate who will work best for you.

If you've read through other guides, you may have seen people recommend that you pick two newsgroup providers: one monthly plan, and a backup that provides block plans. The reasoning for this is that content creators can send the providers DMCA take down notices and make them pull down the content. This results in your downloads failing. The block providers that are often recommended are usually overseas and it is much harder for American companies to get them to take down content. My opinion is that a second provider is not needed. There are two reasons for this:

  1. We are setting up an automated system that will start downloading content within minutes of it being posted. Some companies are quick with their DMCA notices, but none seem to be THAT quick. Failed downloads are usually only seen on backfills (downloading content that has already been posted for a while).
  2. We will be setting up a torrent failover system. If the content can't be grabbed off of newsgroups, our system will search for torrents, which can't get taken down because they're peer-to-peer.

So unless you want to avoid torrents because you don't trust peer-to-peer, you can probably skip the secondary provider.

When you do chose a provider and sign up with them, they will likely provide you with customized software to install. You can if you want, but we will not be using it for our purposes so feel free to skip it and go on to the next step...

NZB indexing sites

As I mentioned earlier, NZBs are small files that tell a download client where to find content. There are many sites out there dedicated to indexing these NZB files to make it easy to search for and download content. The media managers make use of these sites to function, so you will need to sign up for a few or else you'll have a server that can't find anything to download. Unfortunately, most have gone invite-only to avoid too much exposure, but there are still a few that anyone is welcome to sign up for.

The three that we will be making use of in this guide are nzbzombie.com, usenet-crawler.com, and OZnzb.com. Take a minute and sign up for all three and validate your email address if required. 

We won't be using these sites directly, instead we will be interfacing with their API. Once you are logged into each site, navigate to their Profile page (usually by clicking your username). Somewhere on this profile page, you will find something labelled API Key, or RSS Key. It will be a long string of characters, consisting of hexadecimal digits. We will need these keys often, so copy it and paste it in a notepad text file for easy access. Make sure you mark which site each key belongs to.

Now that we have access to the NZB sites, let's install the program that will use the files.

Installing SABnzbd

SABnzbd is the software that will do the actual downloading of files off of newsgroups. The media managers we will be installing later will interface with the software and tell it what to download. SABnzbd will then grab the files and pass it back to the managers so they can sort it.

Begin by downloading the latest version from their website. Run the installer and make sure you check Run at startup when you reach the Choose Components stage.

After it is installed, launch it from the desktop or start menu. You will get a Windows Firewall prompt asking you if it's okay to allow this program to have network access, click Allow access. The interface is entirely web based, so don't be confused when your browser opens. Click Start Wizard.

On the first screen, you need to set up the connection information that you received from your Usenet provider. One tip is that you don't need to use the maximum amount of Connections that your provider allows. Unless you are on a gigabit fiber, 10 connections is likely more than enough to saturate your download speed. Once this is all filled out, click Test Server to make sure you entered it all correctly then click Next.

The next screen will have you set up access to SABnzbd. Since we want to be able to remotely manage our server, change this to I want SABnzbd to be viewable by any pc on my network. We are going to make this accessible from the internet as well, so make sure you password protect it and that you use a decent password. Since this program will be starting with the PC, you will probably want to uncheck Launch my internet browser with the SABnzbd page when the program start. Click Next.

The third step is to enable OZnzb integration. This is optional, but we are already registered so we might as well. Paste in the API key that we got in the last step. Click Next and then wait for the program to relaunch. Click the Go to SABnzbd button when you are able to.

You are now presented with the SABnzbd interface.

We need to make a few more configuration tweaks, so click Config in the upper left corner.

In the General section, change the SABnzbd Port to 38080. Click Save Changes and then Restart SABnzbd. Since we changed the port, it will fail to reconnect on it's own. Go to your address bar and enter http://localhost:38080/sabnzbd/config/general/ to get back to where we were.

Go to the Folders section and make the following changes:

  • Change Temporary Download Folder to ..\Downloads\Incomplete
  • Set Minimum Free Space for Temporary Download Folder to 1G
  • Change Completed Download Folder to ..\Downloads\Complete
  • Click Save Changes

You may be wondering why we aren't using the folders we set up on our Storage Space. Remember that it is our media managers that organize the files and move them to our storage drives. SABnzbd, despite having some sorting functionality built-in, won't need to worry about any of that. 

Go to the Switches section and make the following changes:

  • Check Abort jobs that cannot be completed
  • Change Action when encrypted RAR is downloaded to Abort
  • (Optional) Change Action when unwanted extension detected to Abort
  • (Optional) Set Unwanted extensions to exe, com
  • Change Ignore Samples to Do not download
  • If your ISP has data caps, you can add values to the Quota section.
  • Click Save Changes

Go to the Categories section.

  • In the blank row, type in sonarr for the Category. Type in ..\Incomplete\sonarr for the Folder/Path. Click Add.
  • In the new blank row, type in couchpotato for the Category. Type in ..\Incomplete\couchpotato for Folder/Path. Click Add.
  • Once more, in the blank row type in headphones for Category. Type in ..\Incomplete\headphones for the Folder/Path. Click Add.

This will make more sense later, but in a nutshell, this is how our managers know where to find completed downloads.

We are now all set with SABnzbd, but we will need to come back to it later, so leave it open in a tab. Move on to the next part and we will get our torrent downloader installed and configured!