Tag: servers

SSL Security: Forward Secrecy
Forward secrecy is actually what it sounds like. It forwards on secrecy. This means that the secret keys you have won’t be compromised even if the server’s key gets hacked. The way it works is by constantly changing things. Basically it’s rotating its encryption forever.

“You’re constantly generating new keys for new messages,” says Nadim Kobeissi, the creator of Cryptocat, one instant messaging app that was early adopter of perfect forward secrecy. That means if a user’s device is stolen or hacked and eavesdroppers steal a decryption key, it doesn’t matter. “The latest message gets compromised, but any message prior to that message or after it can’t be decrypted,” Kobeissi says.

This is a good thing to have on your own server, as it means if a secure transaction (like an ebook sale) is compromised, only that one instance is impacted. No one else gets hacked. No other part of the sale is affected. You minimize damage considerably.

Let’s do this thing!

On Monday we killed off RC4 by going to WHM Home » Service Configuration » Apache Configuration » Global Configuration and changing the cipher suite to this: ALL:!ADH:!RC4:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP:!kEDH

In order to properly handle perfect forward security, we have to reorder things. Instead of using ALL, which calls everything, we need to put the ciphers in a preference and omit the ones we don’t want. That gives us this:
```
ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4
```
Awesome. Once we set that, and restart Apache, we go to Service Configuration » Apache Configuration » Include Editor » “Pre Main Include” and pick ‘All Versions’ from the drop down. I’ve edited this section before, when adding HSTS support. But then I edited pre_virtualhost_global.conf (“Pre VirtualHost Include”). This is just a different section.

In the Pre Main Include section, put this:
```
SSLProtocol all -SSLv3 -SSLv2
SSLHonorCipherOrder On
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4
```
This forces the SSL protocol to v3 and then v2, but then it demands the order of the ciphers be honored.

Restart Apache and you’re done.

Testing it

If you want to test, go to ssllabs.com/ssltest and you should see something that says A or A+. If you don’t have it right, you’ll max out at an A-.

This site? It’s currently an A+, however that target is always moving. Keep safe out there.
1 February, 2017
SSL Security: Killing RC4

In the middle of debugging what turned out to be an unrelated problem, my friend James pointed out that my server was throwing a warning about the RC4 cipher.

Ciphers?

When you want to transmit secure data, you use a cipher to encrypt the data. RC4 is a (hah) cipher. It generates a stream of random bits and bytes (they’re not really random) and combine them with plain text using a pattern to make an encrypted message. The secure connection unravels the bits because it knows the cipher too.

Ciphers only work if no one knows how to crack them. That isn’t the case with RC4 anymore and it was hacked. Sad news. Time to disable it!

Disable it!

Why was it enabled at all? As it happens, it was enabled on my servers for browsers that can only use it. Old browsers. The funny part is that those old browsers can’t support SNI (which is what lets me have multiple certificates and multiple https sites on one IP address), so the domains were broken anyway.

To disable, I went to WHM Home » Service Configuration » Apache Configuration » Global Configuration and changed my cipher suite to ALL:!ADH:!RC4:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP:!kEDH

The !RC4 is the part that says “Not this one!”

Restart Apache and you’re safe to go!

30 January, 2017
Let’s Encrypt cPanel
On August 10th, cPanel announced provisional support for Let’s Encrypt via AutoSSL.

For hosts like DreamHost to be able to implement Let’s Encrypt is a lot easier than a behemoth like cPanel. See, DreamHost only had to make sure it worked on their own servers. They have a homegrown panel that they have 100% full control over. Adding in how to install the code on all servers and how to integrate it requires less testing than cPanel, who has to make sure everyone who uses cPanel can use this.

As of version 58 of cPanel, everyone can. And it works.

Installation

Log in via SSH and run this as root:
```
$ /scripts/install_lets_encrypt_autossl_provider
```
That installs everything you need. Keep in mind, this only adds LE to the AutoSSL feature. It’s AutoSSL that whips up SSL certificates for cPanel accounts. Doing this install does not install certs on your domains. We have to configure it for that.

Configuration

Once you’ve installed the code, go to WHM: Home » SSL/TLS » Manage AutoSSL and set it to Let’s Encrypt:

If it’s your first time, yes, check Create a new registration with the provider. because you’re new. You only need to mess with that if you’re new or have to reset registration for some reason.

By default, AutoSSL is set to run based on your “Feature List Setting” (under Home » Packages » Feature Manager » Edit Feature List). Mine has it checked, which means it will automatically run.

Adding Existing Domains

This worked great except I had a bunch of domains using StartSSL. First off, I adore StartSSL, and the recent changes to their website make it so much easier to use. But I was using it for external free certificates where I didn’t want to pay for them, on domains that never see money. Some of them (most of them) I wanted to convert to LE.

For that, I deleted the StartSSL certificates in WHM and cPanel for the domains/account in question. Then I went to AutoSSL, clicked on the tab “Manage Users,” and clicked “Check USERNAME.” I did not pick check all users (which is at the top of the page) because I don’t want to check all users.

Adding New Domains

I love this part.

Do nothing.

No, really. Add the new domain, wait twelve hours, boom. New certificate. If you have to have it right now, go into WHM and click check for that user. But it’s automatic. Hence ‘Auto’ SSL you see.

Caveats!

This is something only controlled by the server admin. Per-site cPanel doesn’t get an option, however if they delete the LE cert and add their own, that will override it.

There’s a limit to how many times you can make certificates and how many you can make. As the warning says:

Certificates that Let’s Encrypt provides through AutoSSL can secure a maximum of 100 domains per virtual host.
Let’s Encrypt will issue a maximum of 20 certificates per week that contain a domain or its subdomains. If you include subdomains of a domain on more than 20 certificates, Let’s Encrypt will issue those during the next window, up to the limit for that week.

If you’re using a wildcard subdomain (*.ipstenu.org for example) in order to make things easier with Multisite, this won’t work. You’ll see a ton of errors in your logs. Not to mention it won’t make SSL certs for all the virtual subdomains. That’s because they’re too virtual. You’ll have to make an actual add-on subdomain or use a domain alias for LE to pick that up.

You can’t revoke a certificate either, which can be a problem should there be a security issue along the lines of Heartbleed. When that happened, we all had to reset our SSL certs as well as patch our servers. Lots of fun. Should that happen again, cPanel users will have a big problem.

It’s because of that I don’t want to use Let’s Encrypt on everything. I’ll use it on this domain, and my other normal ones, but my WMH domain and my stores use a Comodo Certificate.
19 August, 2016
Greylist, RBLs, and Spam

Recently I noticed I had 13 spam emails all from the same ‘company.’ The content was incredibly similar, though subtly different. The from email was always different, but you could tell by looking at it that it was the same. And even more damming, it all had ‘junk’ content and 100+ recipients. But for some reason, SpamAssassin wasn’t catching it!

After 5 emails came in back to back, I decided to do something about it.

At first I was trying to find a way to tell Spamassassin or Exim how to auto-turf the emails with 100+ people listed in the ‘To’ field. This proved to be a little more difficult and complicated than I wanted, and I was sure that these spammers would catch on to that sooner or later.

What I really wanted was for Spamcop to pick up on this, but I’ve been sending them in to no avail for a while. That got me looking into how cPanel handles Spamcop in the first place.

Real-Time Blackhole Lists

cPanel uses RBLs, Real-time Blackhole Lists, to determine if an email sent to you is spam or not. By default, it comes with SpamCop and Spamhaus. That means it will reject mail at SMTP time if the sender host is in the bl.spamcop.net or zen.spamhaus.org RBL. Well that was well and good, but could I add more to that list?

Of course. I pulled up cPanel’s documentation on RBLs and determined I could add as many as I wanted. On the top of the Basic EXIM Editor is a link to Manage Custom RBLs which is what I wanted. All I had to do was figure out what to add.

After reading through WikiPedia’s comparison of DNS blacklists, I picked a few and tested the latest emails that had come through, looking for ones that caught them. Then I tested known good emails and made sure they weren’t caught. I ended up adding Barracudacentral and IPRange.

Greylisting

The next thing I did was introduce Greylisting to my email. They way Greylisting works is that if it doesn’t recognize the email, it will temporarily reject it and tell it to resend. If the email is real, the server tries to send it again after a little while. There are some downsides to this, as it’s possible for a legit email to be trapped for a few hours (or days) if someone’s set up their server poorly. On the other hand, within half an hour, I blocked 11 emails.

I mean. I’m pretty sure monica@getoffherpes.com is spam. You know what I mean?

This was super easy to do, too. I turned on Greylisting, I restarted Exim, I walked away.

Okay no, I didn’t. I sat and watched it to see if anyone legit got caught (one did, it passed itself through properly).

Result?

A little less spam. I don’t expect this to work for everything, but it had an immediate impact on many of the spam emails that were annoying me.

10 August, 2016
SSD: An Anecdotal Journey

After I upgraded to Ogra, I noticed that my server’s load stats were a little janky. They were pretty high, actually, bouncing between .5 and 2 all the time. Now this isn’t really all that bad, but as I looked at my server, I realized I was paying a little more than I needed, since I’d cut down on space by offloading backups better.

This matters because I needed only 50G of diskspace (I only use about 30 right now for all the sites), and while I did want 2G of RAM, I could easily switch to an SSD on my hosting plan. It was cheaper, and it was as easy as pressing a button. I’ve never really gotten to play with SSD for this site, so I thought “what the hell” and pressed the button.

A weekend later…

The five minute load average shows a massive drop. It’s incredible. The red and blank spots are where I was running the actual migration process and disabled monitoring for a bit. Literally all I did was reboot the server and the move it to SSD. That’s it. It’s insane when you think about it.

By the way, I saved disk space by offloading the backups to Amazon S3. Amazon costs me about $3 a month, and while I detest their interface, the integration is built into WHM. I have an open feature requests to allow arbitrary CEPH destinations as backup so please vote for that if you like the idea.

SSDs had the bonus of changing my backup from taking 2 or 3 hours to taking 15 minutes.

Anecdotally SSD is an incredible improvement of the old bog hard drives.

3 June, 2016
Blocking Referrer Spam Server Wide Sucks
A while back I talked about Referrer Spam in Google Adsense and I mentioned how you could block referrer spam with some .htaccess calls. That’s cool, but when you have 12 sites on a server, making this one more thing to manage per site is a pain in the ass. Well okay, what can we do constructively? And sadly the answer is “Not much.”

First of all, forget the idea of using a robots.txt file. If they were real seo crawlers, they would honor this. They don’t and that’s how I know they’re evil.

Secondly, this will only work if you have server wide access. That should be obvious, but server wide settings need server wide access, and that’s just how it is. I say that it sucks because it can be a little complicated and messy to understand where you put things.

If you have your own server, like I do, then you can make a custom VirtualHost template

Since I’m using Apache 2.4, I made local templates:
```
$ cd /var/cpanel/templates/apache2_4/
$ cp ssl_vhost.default ssl_vhost.local
$ cp vhost.default vhost.local
```
If you’re using 2.2 then the files are in /var/cpanel/templates/apache2_2/ instead. In each file, I added this to the top of the VirtualHost settings.
```
  RewriteEngine On
  RewriteOptions Inherit
```
What that does is it tells Apache that it should inherit Rewrite rules from the main server. That means each virtual host (i.e. each website) will abide by any rules in the local settings.

Where you put this in that file can be weird. I ended up looking for this section, and putting it right below:
```
[% IF !vhost.hascgi -%]
  Options -ExecCGI -Includes
  RemoveHandler cgi-script .cgi .pl .plx .ppl .perl
[% END -%]
```
Put that in both files. Because you use HTTPS, right? Then you need to bounce httpdconf:

/scripts/rebuildhttpdconf

Since I’m using WHM, the next step is to go in to the Apache Configuration section and open the Include Editor. Then you want to add your blocking directive in ‘Pre-Virtual Host Include’ for All versions. If you don’t use it, you’ll want to edit /usr/local/apache/conf/includes/pre_virtualhost_global.conf and bounce Apache after.

As you can see, I have some content in there already.

I added this below:
```
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{HTTP_REFERER} spammerseocompany\.com [NC,OR]
  RewriteCond %{HTTP_REFERER} keywords-monitoring-your-success\.com [NC]
  RewriteRule .* - [F,L]
</IfModule>
```
Does it work? Yes. It blocks ‘spammerseocompany’ from all my domains on my server. I put in the other URL since that’s the one they have that’s currently spamming the heck out of my stuff. There are a lot of other options with Apache 2.4, like sending them a 403 and so on. You should read up on using mod_rewrite to control access and pick the method you find most sustainable. For example, you could single like it:
```
  RewriteCond %{HTTP_REFERER} (spammerseocompany|keywords-monitoring-your-success)\.com [NC]
```
I find that a bit clunky.

If you’re using nginx, you’ll want this I believe:
```
if ($http_referer ~* "keywords-monitoring-your-success\.com|spammerseocompany\.com") {
    return 403;
}
```
A big note of caution here. If your list gets too long, you’ll end up slowing your server down. A lot. So keep it as simple as you can. I find that CSF does a dandy job of blocking the most of my trouble makers, and I only need this for the unnamed spammerseocompany because they don’t abide by the common rules of robots.

If, one day, they do, I will stop blocking them and allow their robots. As it stands, they’re idiots and need to go away.
1 June, 2016