W(e)blinks | Notes

Cross Site Request Forgery (CSRF/XSRF)

xml-feeds@wblinks.com (Rich Adams) — Fri, 04 Jun 2010 13:58:00 -0700

If you're building a site that allows users to update any sort of information (so most websites), then you should probably think about protecting against Cross Site Request Forgery (referred to as CSRF or XSRF). Being susceptible to this type of attack can be annoying in some cases, but extremely dangerous in others. Unfortunately, it's not the type of attack that's easy to understand at first, and it's not immediately obvious how to prevent such an attack. Because of this, protecting against XSRF is often overlooked, even on some big name websites.

What is XSRF?

An XSRF vulnerability is one which allows a malicious user or website to make an unsuspecting user follow a URL or post a form which will perform an action on your site which the user didn't want to happen.

As a basic example, imagine you allow users to post images in your comments. If a user puts "http://example.com/logout.php" as the image's URL, where example.com is your domain, then any time a logged in user views that comment, they will be logged out.

Or maybe it could be that you allow a user account to be deleted without confirming the action or protecting it from XSRF in any way, so any user visiting the page would then get deleted instantly!


I thought I'd make a comment on your site. Check out this cool image!
<img src='http://example.com/delete_my_account.php" />

A different example would be if a malicious website contains a form, which an unsuspecting user then submits, it can actually submit information to another site which the user is logged in to, and that site will think the request came from the logged in user and will just perform the update as if the user had done it intentionally.

It doesn't even have to be a link/button the user clicks on, as shown in the first example it could happen even if just viewing a site. Obviously, this could become quite annoying for your users.

Annoying is one thing, but it can also be pretty dangerous. For example, let's say I'm logged into an account on a simple shopping cart site. I then go and browse to another unrelated website. The other website has a button which just says "Click here to register". Seems simple enough, this other website wants me to register an account. However, this is a malicious site and when I click the link, it's actually submitting a request to the original shopping cart website as if I'd clicked the "Order me 1000 of some expensive item in one-click" button. If the shopping cart website is susceptible to XSRF then it will think the request to order 1000 items was genuinely submitted by me, and I'll get a nice surprise in the mail and on my credit card statements.

It's a particularly difficult type of attack to get your head around as it's very subtle, but once you understand how it works it's not that difficult to protect against it. It's very easy to go down the bad route though.

Protecting against XSRF - The bad way

So what can you do as a web developer to prevent such attacks? Effectively you're just wanting to make sure a request came from your site and was actually intended to be run by the user, so as a first step you can just check the referer header to make sure the request came from your own site.


<?php
$url = parse_url($_SERVER['HTTP_REFERER']);
if ($url['host'] != "wblinks.com")
{
    die("You're not coming from my site. Possible XSRF attack");
    exit();
}
?>

While this may work in some cases, it's going to be about as effective as an underwater hair dryer as far as stopping an XSRF attack. Altering the referer header is pretty trivial and it will also have the downside of making the site unusable for lots of people, since many browser or proxies can strip the referer header when in "private" mode.

This also wouldn't protect against the first example of XSRF, where someone just uses the logout URL as an image URL in a comment. As the request would come from the correct referer in that case.

So not only will you not prevent XSRF attacks, but you'll also annoy some of your users. Not a good solution.

Protecting against XSRF - The good way

What we really need is a one-time key (or token) which allows us to validate that the request came from a form we presented to the user intentionally. Here's how I accomplish that on my sites (although you'll probably notice not on this site yet... that's because this is just a blog, and doesn't have registered users anyway. Also I haven't updated this site to use the latest version of my framework just yet as it requires some refactoring).

I have an Auth class which handles everything to do with user authentication on my sites as well as authenticating that a request was valid, etc. I have 3 very simple functions in this class, which are as follows,


// This is a key used to stop CSRF/XSRF attacks. It's stored in the user session. 
public static function getNewRKey() 
{ 
    $rkey    = substr(sha1(uniqid(rand(), true)), 0, 20); 
    $rkeys   = Session::getInstance()->getVar("rkeys"); 
    $rkeys[] = $rkey; 
    Session::getInstance()->setVar("rkeys", $rkeys); 
    return $rkey; 
} 
    
// This will determine if an rkey is valid 
public static function validateRKey($rkey) 
{ 
    $rkeys = Session::getInstance()->getVar("rkeys"); 
    if ($rkeys == null) { return false; } 
    return in_array($rkey, $rkeys); 
}   

// Creates the form input to use 
public static function formRkey() 
{ 
    return "<input type=\"hidden\" name=\"rkey\" value=\"".Auth::getNewRKey()."\" />"; 
}

So what's going on here? Anytime I have a form on the site which POSTs data, I will output the formRKey() function to add a hidden field to this form. This hidden field contains the value returned from getNewRKey(), this method generates a random 20 character hash value, and stores it into the user session (I'll get to why that's an array in a moment) and then returns this hash so it can be put into the hidden input.


<form action=”do_something.php” method=”POST”>
    <fieldset>
        <?=Auth::formRKey();?>
        <input ... />
    </fieldset>
</form>

So, the user session will now contain an array called “rkeys”, which contains a random 20 character hash value.


$_SESSION['rkeys'] (1) => 
{
    [0] => '3748ab53cf129d536eca'
}

This has also added a hidden field to the HTML form which will look like this,


<input type="hidden" name="rkey" value="3748ab53cf129d536eca" />";

So when the user submits this form, the idea is that we take the rkey value and check it against the ones we've stored in the user session. If it's there, then the request is valid and came from a form which the site generated. If not, then it's a possible XSRF attack and the output should be stopped and logged. The validation is done in the validateRKey() method.

Now the final step, is to run something like the following code on any page that takes input. I actually have it in my bootstrap file which is run on every page request to the site.


if (count($_POST) > 0 && class_exists("Auth")) 
{ 
    if (!Auth::validateRKey(Sanitizer::sanitize($_POST['rkey']))) 
    { 
        Log::create("XSRF", "Possible XSRF attack", LOG::NOTIFY, LOG::NOTIFY_ADMIN);
        die("Detected a possible cross site request forgery attack. " 
           ."Perhaps you tried to refresh a POST request?");
        exit();
    } 
} 
// Clear any current rkeys 
Session::getInstance()->clearVar("rkeys");

So if anything has been POSTed to the page load, and the Auth class exists (this is because it's part of the framework I use on a lot of sites.. not all of which require Auth stuff). I validate that the POSTed rkey is in the list of expected ones. If not, I log the error and then kill the page load and present an error to the user. You could perhaps redirect there instead of just calling die(), and present something more friendly to the user, it's up to you. For my purposes, this suffices.

Importantly, you also need to clear the session variable of acceptable rkeys, since a request has now come in and been validated, so all keys are now expired. If you don't do this, then it's pointless, since someone can just screenscrape the rkey and it would always be valid in the user session.

Why an array of rkeys?

Originally it was just one rkey instead of an array, but that presented a problem. If I had more than one form on the page, only the final form would be valid, any other form would present the XSRF error, since a new rkey was generated each time a form was created and then only that new rkey was valid. An easy mistake to make, but something I noticed quite quickly. Of course, if you only ever have one form on a page, then feel free just to use one value, but I would still recommend using an array since it allows you to easily expand in future if you do end up using more than one form on a page.

Why only POST?

You may be wondering why I'm only checking POST variables to prevent XSRF rather than both POST and GET. The answer is because GET should never be susceptible to such an attack if you're using it correctly.

So when should you use GET and when should you use POST? Well, it's all in the name. GET variables ideally should be used when the contents of the page are read-only, so nothing on the page gets changed by the request. So a GET request should be idempotent (I should be able to trigger the same GET request as many times as I want and it shouldn't affect the result I get. Basically, you want to make sure it doesn't have any side-effects). POST should be used whenever it causes a destructive action (I don't just mean deletes... I mean destructive as in something changes). So login, logout, updates, creation, deletion, comments, voting, etc.

This is why browsers will generally prompt you to confirm when resending a POST request, whereas they won't with a GET request. This is because a POST request will generally change the outcome each time it is run so you want to make sure you're not going to accidentally run it again and change something you didn't want to. A GET request shouldn't change any data that way, and so it doesn't need to be confirmed.

If you do want to use a GET for a destructive action, say you want it to be an anchor tag rather than a button otherwise your style will look strange, then you must make sure that you at least redirect to a confirmation page to confirm the YES/NO, ideally this confirmation page should use POST.

If not, then say you were to have a link in your admin area which deleted an item using GET, without any form of confirmation. Some browsers do what's called pre-fetching, where they examine the links on a page, and pre-fetch the websites you'd get by cliking these links, under the assumption that you will go to them. Then when you do click them, the page can be displayed very quickly. If you just delete with a GET, then simple visiting the admin page could cause the browser to follow all those links in the background and delete everything. Obviously not something you want. Yes, this happened to me in the past, so please learn from my mistakes.

POST Refresh

Another thing to keep in mind, is that you should redirect users to a GET based URL once the POST request has been handled. This means users will be able to hit F5 and refresh the result page without being prompted if they want to resubmit the POST request (since that will now cause an XSRF error to be displayed). This is common practice on most sites and something users now expect to be able to do. So they'll probably not be too happy if they have to re-send requests and then get errors saying they're XSRF attacking the site.

Do we Really Need to Keep Typing www?

xml-feeds@wblinks.com (Rich Adams) — Tue, 09 Feb 2010 13:23:00 -0800

I'll admit it, I'm the kind of programmer who'll spend 10 hours writing some code to do a job that would only have taken 2 hours to do anyway. This isn't because I'm stupid (well... maybe a little), but more because I just like writing programs and it'll usually teach me something new. I also have a notion that if I'm going to do something once, I'm probably going to have to do it again at some point, so the next time it'll only take a few seconds because I have a program to do it.

Even though I will happily spend hours writing software that I didn't really need to do, I don't like to waste my time on something that to me seems pointless. Even small insignificant things which in the long run probably have no real impact on my time anyway. I'm kinda strange like that.

One of these small insignificant things is typing out the "www." before a web address. Saving myself the milliseconds it would take to type that out is a big deal for me, because I don't see what the point of me typing it out is. It's redundant information and can simply be implied. Yet all the time I'm coming across sites which will not work if you miss off those four characters at the beginning. This ends up costing my time since I won't realise until the request times out, and then I need to type out the "www." anyway in order to get it work. There's no real excuse for a site to behave this way, it's just rude.

A story of woe

Just before the holidays I wanted to book a train ticket back home, so I opened up my browser and typed "thetrainline.co.uk", hit enter and expected the site to pop right up. But it didn't, instead it timed out. So figuring the site was down I typed "virgintrains.co.uk" instead, hit enter and the site popped right up. I then went ahead and bought my train tickets home.

Later on however, I wanted to check some train times, so I went back to try the trainline, and discovered that the site wasn't down at all, it's just it will only work if you type "www.thetrainline.co.uk".

I've no idea on the trainline's business model, but I would imagine they make at least some of their money from selling the actual train tickets. Because their site doesn't work without the "www." they lost a sale from me. Maybe I'm the only person in the entire world who types URLs this way, and so it would only be my sale that was lost, but one sale is still one sale. However, if a lot of people type their URLs the same way as me, then it could end up being quite a bit of cash that's being lost.

There's no reason for only one of the URLs to work. It's not like they're using the one without the www for something else. It just timed out. It's a tiny piece of site configuration, but if you leave it out then it can cost you visitors and sales.

Why do we use "www." anyway?

I did some searching around, but couldn't find a conclusive answer to this. The only remotely sensible reason I came across is that back in the early days of the world wide web, it was used to distinguish the web site of a domain from the FTP server or mail server (which would be ftp.example.com and mail.example.com, etc). So it seems to be nothing more than a convention that caught on.

One of my personal thoughts on the matter is that it was a way to tell people that what they were looking at was an address for the World Wide Web. In the early days of the internet if you'd seen something like "jurassicpark.com" on the bottom of a movie poster, it wouldn't have been obvious what it was. Is it the name of the production company?, something that got printed by accident? Adding the "www." to the beginning made it immediately obvious to everyone that this was something to do with the internet or World Wide Web.

Despite this mainstream use of "www." in domains, there is no technical reason why it's there. There's nothing in the HTTP specification that requires the use of "www.", it's simply a standard that caught on.You can host an FTP/mail/SSH/web/news server all on the same domain since these services all use different ports. If you need them to connect to different machines, then yes, maybe you would perhaps need a separate domain to use in the DNS records, but there's nothing stopping you just using example.com for a normal website.

To "www." or not to "www."?

So should you use "http://www.example.com" or "http://example.com" for you site? Well, the answer is both, at least from the users' point of view. Make one the actual home page for your site, and just make sure the other one redirects (using a 301 redirect) to the correct one. That way, no matter what the user types into their browser, they'll get to the site.

But then which one is going to be the home page and which one is going to redirect? Some people see this as a non-question saying that it doesn't matter. Well, that's not always the case. The choice you make for this can have big impacts depending on how your site is set up.

You could make the same page appear for both rather than redirecting, but this can have negative impacts on your search rankings. Search engines may think www.example.com and example.com are actually separate sites, so it will appear twice. Google Webmaster Tools will show you if that's the case and generally they'll just pick one, but it could cause problems with other search engines.

There are advantages and disadvantages of both ways and it all depends on what your site is really used for, or your personal preference. For me, I prefer to not use the "www.", but that's because I don't use cookies on this domain. At the end of the day, if you don't have any major reasons for using one over the other, then it doesn't really matter so long as you stay consistent.

What if the site uses cookies?

Without the "www." cookies will be served on all subdomains. This may be the behaviour you want, or it may not. So you would need to analyse this for your own site and decide the best approach. By including the "www." it means subdomains can have different cookies from the main site, whereas without it, all cookies set on the main domain will propagate to all the subdomains.

Why does this matter? Well, if you intend to serve static data then you'll need a cookieless subdomain. If you've gone for the option without the "www." then you won't be able to do this, and instead you'll need to use a completely separate domain name. So pick wisely.

Microreasons

Microblogging is big. Yes, I'm talking about Twitter. Everyone and their dog seems to use twitter, and people will often want to link to places. Having four less characters to type can mean the difference between some free publicity for your domain on twitter, or someone using a URL shortener because your URL is too long. Since most URLs are generally too long anyway, if you don't use the "www." you could even implement your own URL shortener. A little free publicity never hurt anyone.

Summary

It all boils down to this; a list of some top tips for web developers with regards to "www.",

Make your site work both including and excluding the "www.".
Since including the "www." is social convention, you should make sure that users are sent to the same place if they include it or not, they shouldn't be sent to a different site if they miss off the "www." as that will just confuse matters.
Pick one URL to use as the main destination, and redirect the other URL(s) to it. This way, all of the URLs to your site should stay consistent.
Use a 301 redirect as this tells search engines that it's a permanent redirect and that both URLs are actually for the same thing. This should preserve your rankings and keep things consistent.
If you want to use a cookieless subdomain for static data, then you need to use the "www." for your main site
If possible with your domain structure (some websites use subdomains for other things, and that's fine) also try to make "w.domain.com" and "ww.domain.com" point to the main site. This will catch anyone who makes a typo. It takes very little time to setup and can save someone even a few seconds of grief when they realise their typing mistake.

It's the little touches like this which could make the difference between a visitor, or someone who goes elsewhere. It's probably not going to lose you many customers, but every little helps in making people's lives less irritating.

If you follow the tips, then no matter whether someone bothers to type those extra four characters or not, they'll still be able to get to your site or order their train tickets home without hassle!

Creating a 'Database is Down' Page

xml-feeds@wblinks.com (Rich Adams) — Thu, 14 Jan 2010 22:47:00 -0800

Earlier today the database for this site was unavailable for around 30 minutes, I imagine something was being rebooted somewhere. This doesn't really concern me too much, since this is just a personal site. So during this time anyone visiting the site was sent to a holding page explaining that the database was down, and providing some links of places for people to visit instead. I make the choice to hide the error details from the user, rather than displaying a page with cryptic error messages on it, or even worse an error message which prints out some critical information. After all, the user probably doesn't really care why my database is down, they just care that they can't get to the information they want. So instead, I log the error details internally and just give a nice page to the user.

I received an email from someone who obviously tried to visit my site during this time, asking how this was done. Since I haven't updated in a while, I thought it's be good to give a breif overview of how it's done.

It's actually not very difficult. If there's an error when attempting to the connect to the database then I log it and email myself the error, and then redirect the user to the holding page.

In the DBAccess class for wblog (the name I gave to the backend code that runs this site), the connect() method looks like this,


function connect()
{
    global $DB_CONFIG;
    
    if ($this->getHandle() != null) return; // Already connected

    $pdo_string = 'mysql:dbname='.$DB_CONFIG['database'].';host='.$DB_CONFIG['hostname'];
    
    try
    {
        Log::create("DB","Connecting to ".$pdo_string." as ".$DB_CONFIG['username'], Log::INFO);
        
        $this->_handle = new PDO($pdo_string, $DB_CONFIG['username'], $DB_CONFIG['password']);
        $this->getHandle()->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    }
    catch (PDOException $e)
    {
        Log::create("DB","Exception caught while connecting. ".$e->getMessage(), Log::ERROR, Log::NOTIFY_ADMIN);
        header("Location: /dbdown/");
    }
    Log::create("DB","Connected to database", Log::INFO);
}

The relevant bit of the code is this,


        $this->getHandle()->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    }
    catch (PDOException $e)
    {
        Log::create("DB","Exception caught while connecting. ".$e->getMessage(), Log::ERROR, Log::NOTIFY_ADMIN);
        header("Location: /dbdown/");
    }

I set the ATTR_ERRMODE attribute so that it will throw exceptions (See the PHP manual for details on setting attributes for PDO). I then catch any PDOException that's thrown. The "Log" class for wblog takes care of recording the error into a log file on the server, and the extra argument Log::NOTIFY_ADMIN tells it to drop me an email with a copy of the log (so that I get alerted to something going on). It's smart enough not to just email me the same thing everytime a user visits though, and has an increasing delay before it emails me the same thing again. (So first 5 minutes, then 10, 15, 20.. etc). You'll find people keep refreshing the site when they get an error page, so you'll just fill your inbox with emails if you're not careful.

Since you don't know what's in my Log class, you can just ignore those bits of the code. I'll probably make another post one day with the full code for this site anyway.

Once the error has been logged, I use the Location header to redirect the user to my holding page. Then I just make sure the holding page is static HTML file and doesn't touch the DB. It could also be PHP, but for my purposes a static HTML file is fine. As long as it doesn't touch the database, otherwise you're going to get into an infinite loop.

So there you have it, that's how I deal with issues when my database is down. If you have any better ways, or see something wrong with how I'm doing it, then let me know in the comments.

Taking the Security out of Security Questions

xml-feeds@wblinks.com (Rich Adams) — Sat, 12 Dec 2009 12:43:00 -0800

A security system is only as strong as the weakest point and this weakest point tends to be the bit where a human is involved (putting your password on a sticky note attached to your screen, for example). As a programmer, there are some things you just can't protect against, and human failure is one of them (Velociraptors are another). However, developers could at least try to make it a bit harder for people to break into other users' accounts.

I came across the following scenario a few weeks ago when attempting to sign up to a new system. Imagine the scene, you've come up with a really secure password. You're happy that it would take someone a very long time to break such a password, and you haven't noted it down anywhere. Now the website you've signed up to is asking you to enter answers to some security questions in case you forget your password. Questions like "What was the name of your first pet?". You only have 4 different questions to choose from, and you have to pick at least two. It is mandatory to provide an answer.

This scenario is not as far fetched as you might first imagine, behold the following image!

I won't name and shame where this screenshot comes from, but it's from a product that's live and out there in the wild (and I should point out, not associated to anything I've worked on).

If you have any sort of sense at all, you'll see why this is a very bad idea.

All The Hard Work, For Nothing

I really don't know what goes through a developer's head when they come up with things like this. You've built yourself a great web application, you're using salted hashes for the password storage, and you're happy that your users have picked secure passwords.You've not forced some silly password restrictions on people, that actually make it easier to guess passwords. You're feeling pretty happy about it all.

But then you wonder about a certain use case. What happens when someone forgets their password? You've used salted hashes so you have no way to retrieve the original password, which means you'll need users to verify who they are somehow first.

Here is where you have some options,

If a user requests a password reset, simply generate a new password and send it to the email address on file.
Get the user to verify who they are by validating some information you have stored under their account. First line of address, day and month of birth, etc.
Security questions. Get them to fill in some security questions when creating their account, then we can just use that!

Security questions seems to be the popular choice, and the one that I come across on lots of websites. If done right, then it can be a life saver when you've forgotten your pasword, but if done wrong, you might as well give out your password.

The Wrong Way

Simply, it's forcing a set of pre-defined questions on users.For example,

What is your mother's maiden name?
What is your mother's middle name?
What was the name of your first pet?
What was the name of your first school?

Now, think about this for a moment. You're going to allow anyone to "recover" their account and change their password if they answer these simple questions.

Not only have you reduced an attack on the site to a simple dictionary attack (since all those questions require dictionary word/sentence responses) but even worse, if you actually know someone, then you'll probably be able to answer those questions pretty easily. I'm pretty certain I could answer at least 3 of those for most of my friends. You'd also be surprised just how many people grew up with a dog called "Rex" or "Rover", etc.

Also, what if the user has never had a pet? Or their mother doesn't have a middle name? By forcing the questions users can pick from is not only going to irritate people but make it much easier for attackers to just randomly try and break the security questions rather than the password.

The Better Way

Once again, I'm going to stop myself from saying "The Right Way", since there's never going to be a foolproof way to recover someone's account if the password is lost. Every method of account recover has it's drawbacks.

If you're going to go down the security question route, then a much better way is to let the user decide the question themselves. It allows the user to come up with something really cryptic, which only that person would know, yet no one else will. For example,

Q. 2X4B?
A. 523P

Unless you're a fan of Red Dwarf, you wouldn't get the reference that 2X4B is the middle part of Kryten's name, with the end being 523P. Ok, a simple Google search could probably find that out, so probably a bad example, but you get the idea.

Obviously this is still vulnerable to the "human element". A lot of people might choose very simple questions, not realising the implication that they're making it easier for others to get at their account. I've actually seen the following in a live environment before as a custom security question.

Q. My password is "iamawesome"?

Seriously... the mind boggles at what that particular user was thinking. So it may be worth adding a quick notice to remind users that silly questions like that will make their account insecure.

Don't Force Your Users

From a security point of view, allowing someone to recover their account if the password is lost is not a very good idea. The password is there for a reason, to secure the account. If it's lost, then no one should be able to get at the account. Account recovery procedures such as security questions were created purely because of customer service.

Users are only human, and humans forget passwords. It's that simple. Not wanting to lose a customer, you will want to provide a way for a user to recover their account and prove they are who they say they are. If security questions are implemented correctly, then the benefits to customer service far outweight the potential security implications of having two access points to an account, especially if both access points are just as secure as the other.

However, part of customer satisfaction is not to force something on your users. I cannot stand it when I have to sign up for a website and fill in all sorts of information as mandatory, before I'm allowed to continue. By all means provide the user with the ability to create a custom security question, but don't make it mandatory. If a user doesn't want to provide another means to access their account, don't force it on them.

Forcing also has the side effect that most users will just think "Eugh, I just want to continue, I don't care about this" and they'll enter something stupid which is easily breakable. Putting "I don't care" as the answer for example (Yes, I've done this myself).

Adding Salt

It's worth noting that answers to security questions should be treated in exactly the same way as the password. After all, they are effectively the same thing, they both provide the means to access the account in some way. I've seen systems which will store passwords as salted hashes, but then store the security answers in plaintext, completely negating the point of salting the password in the first place. Don't forget to store the security question answers as slated hashes!

Combination Lock

The most effective method of account recovery I've come across is a combination approach. Ask the user to answer their custom security question, and then email a new password to their registered email address. This means unless an attacker knows both the security question answer and has access to the email account, they won't be able to attack the account in this manner. At least not very easily.

Special Case

As with everything, there's always going to be at least one person who manages to lock themselves out of their account, with no access to an old email address and they can't remember their security question response. In these cases it's best not to attempt to provide the user with an automated method for account recovery. Another automated method is yet another point of entry for an account, and another place you need to secure.

In cases like this, it's often better to have a customer service email or phone number which the user can contact, and have a person resolve the issue once they're satisfied the user is who they say they are (by verifying other details on the account).

Get To The Point!

Basically my point is that if you're going to go down the security questions route, don't shoot yourself in the foot. I'm sick of signing up to sites that force me to choose from some security questions before I can proceed. It's the illusion of security, because it actually does nothing except annoy me, and make it easier for an attacker to gain access to my account.

Security questions are great for the user when the password is lost, and they're definitely a nice option. But be sure to take note of the following,

Don't force a certain set of questions on the user. Let them chose their own question.
Don't force a user to fill in the questions as mandatory. If the user doesn't want to, they shouldn't have to. Of course, feel free to ridicule the user if they then forget their password, but it was at least the user's choice.
Hash the answer, and salt the hash before storing.
Don't rely purely on a security question response. Generate a new password and send it to the account email, just so you know it gets to the right person.
If the user doesn't have access to the email, or can't remember the question answer, don't use an automated process. Get another human involved to verify who the person is.

So there you have it. Bought on by my recent voyage into websites that force things on to me, let me know if you've had similar experiences, what methods you use for account recovery (if any) and whether you think anything I've said is wrong. If you have some novel method of account recovery, I'd love to hear about it!

Update

January, 2010 Someone sent me the following image, which are the security questions for a banking website. Another good example of how not to do it.

Only allowing four different questions is a very bad idea. For starters, what if you didn't have a high school mascot? Also most of those questions could be answered by anyone who was a close friend (or enemy). This is not the way to implement security questions, especially not for a bank.

Password Rules Don't Always Help

xml-feeds@wblinks.com (Rich Adams) — Tue, 03 Nov 2009 11:46:00 -0800

A while ago, I wrote about how users can't be trusted to come up with good passwords, and that it's up to us as programmers and web developers to hash the password (and salt it) so that it means bad password choices aren't immediately obvious to someone who gets hold of your stored data.

Of course, if people still use dictionary words, a simple brute force attack will work. So while some password tips such as "Don't use a dictionary word" are quite legitimate, there are plenty of rules and restrictions that do nothing but infuriate users and make passwords less secure.

There is absolutely no need to enforce certain password rules which seems to be forced on people throughout the corporate world. So here are a few which I find the most annoying, which supposedly make people use passwords which are more “secure”, but in reality do just the opposite. I wonder how many of you have come across these before.

The Rotating Password

"You must change your password every x days"

Usually it's every 30 days, so that each month you have to have a new password. There are then crazy rules, like you can't use a password you've used the last 12 times. Sure, if someone discovers a user's password, and waits a month to use it, it will no longer work.. but that's pretty much the only case this protects against. But it does a lot more harm.

If you force a user to change their password every 30 days, there's very little chance they are going to come up with a truly unique password each time. Instead, they'll tend to keep the same basic password and just add something extra in. You'd be shocked at just how many people use the name or number of the month as that extra thing.

Since we now know a likely part of the password, it can make it a bit easier to crack with brute force attacks. It could even mean if you do discover a user's password, and it has a number or name of the month.. you'll always know that users password for any month, making this measure completely pointless.

To add further annoyance to this, a lot of system adminitrators feel it's necessary to remind users that they'll soon need to change their password. So 14 days before your password expires in Windows for example, you'll get a popup saying "Your password expires in 14 days, do you want to change it now?" This has to be the most useless dialog I've ever come across. If my password expires in 14 days, why would I want to change it sooner? It makes no sense. Just tell me when it's expired and I'll change it. Making me select Yes or No every single time I login just irritates me.

The Ridiculous Restriction

Your password must be at least 6-8 characters long, contain uppercase and lowercase letters, and have at least two digits, but not for the first two characters. Oh, and it shouldn't contain the name of a species of dog.

How many websites or applications have you come across with ridiculous password restrictions like this? I see it all the time. Well.. maybe not the dog one. Yet it doesn't actually do anything to make the password more secure! Forcing users to use a certain pattern, means that an attacker also knows the expected pattern, reducing their search space from "anything in the world" to "well, now I know there's at least two digits and will be mixed case".

I've also seen things like "You must choose a password of the form XxxxxxN" where X is an uppercase letter, x is a lowercase letter and N is a digit. Now, that has to be one of the worst. You've given an attacker everything they need to crack the passwords in no time at all using a simple brute force approach.

Not only does it make attacks easier, but it's more frustrating for users. As I said in the previous post, most users will use the same password for everything. It's not their fault, it's just easier to remember one password than one hundred. If you suddenly force a policy that means users can't use that password, they'll have to come up with a new one, and they'll inevitably forget it, frustrating the user even more the next time they login.

So you're 0 for 2. You've made it easier for attackers and pissed off your users, well done!

The Giveaway

"You cannot use that password because another user already has it"

Yes, I have actually seen this in a live environment before. If you have someone on your team who insists on using this method, take them into a room and slap them. This is a pretty big mistake since you've given away another users password. Usernames are much easier to find out that passwords, especially if it's a public website.

I'm really trying hard to imagine what was going through the designer/programmers head when they thought of using this technique. There is no need for a system to enforce unique passwords between users.. none at all! If your database model uses a password as a primary key, you have failed as a developer.

The Normalizer

"Your password cannot contain special characters, only normal letters and numbers"

Why do you care what I use in my password? As long as it's a character understood by the computer, and as long as it will be hashed the same way each time, you shouldn't care what people use. Restricting the use of special characters is on par with dictating the exact format. It's usually done because of some limitation to the system, that you can't store those characters in the database for example, or that it will mess up their admin area. Which means either you're not sanitizing your inputs properly, aren't using prepared statemens and probably don't hash your passwords. Alarm bells should be ringing.

If I want to use a percentage sign or a Japanese character in my password, I should be able to do so. It will mean my password is harder for anyone else to guess.

The Length Restriction

Your password cannot be longer than 16 chars.

Why not? The most common excuse for something like this, is that the database can only store 16 chars in the field, or that a designer only wants the input box to be a certain size.

Well, the designers argument is nonsense, since they can size the box however they want without restricting the amount of information that can be input. The database excuse is also nonsense, since you should be hashing passwords, so that they'll always be the same length after hashing anyway. So the password itself can be any length you want, yet you only have to store the 40 character hash output (or however many characters you force the hash to be).

If I want to type a novel as my password, then that should be my choice. Restricting password length just restricts the search space an attacker needs to use.

So what are the lessons?

I should make it clear that I'm not against all password restrictions. It can depend on the environment. If you're storing the password for a system which has personal data, then enforcing a certain amount of complexity in the password is probably a good idea. I guess the point I'm trying to make though, is that it's easy to go too far. There's a point when it just becomes a hinderance to users, and ends up creating patterns which make the system less secure.

Try not to restrict user password choice. By resitrcting what a user can enter as a password you're simply annoying your users and making it easier for an attacker to guess or brute force the password. By enforcing too many password restrictions you are making the system more predictable, and when it comes to passwords (and cryptography in general), predictability is a very bad thing.

It basically comes down to some very simple points.

Don't resitrct what can go into a password, whether it's a max length or strange characters.
Don't force a users password to follow a certain pattern.
Don't force a user to change their password every x days.
Hash passwords and salt your hashes.

If you follow these simple rules, your users will be happier, your database will always store the same length of data no matter what the user enters and you can be safe in the knowledge that you're not giving away any information to make it easier for an attacker.

text-transform - Content or Presentation?

xml-feeds@wblinks.com (Rich Adams) — Sat, 15 Aug 2009 07:04:00 -0700

HTML and CSS are all about separating the content of a site, from the presentation. As with most things though, there are grey areas, and for a lot of people text-transform is one of them. Some people regard changing the case of text as being a content issue, others see it as a presentation issue.

Personally, I prefer to think of it as a presentation issue for one very good reason; to cover as many scenarios as possible. Suppose you have some mark-up, where the text needs to be in uppercase, because that's how you want a menu to look. You could write the HTML like this,


<ul>
	<li>SOME OPTION</li>
	<li>ANOTHER OPTION</li>
</ul>

This is all well and good, and is perfectly reasonable. Now suppose though that the design needs to change, and these need to be in lowercase instead. Not a problem, since you can use text-transform to convert them to lowercase.


ul li {
	text-transform: lowercase;
}

But now what if it needs to be capitalized instead? You can just use text-transform: capitalize; right? Unfortunately not. A little known part of "text-transform: capitalize" is that is only changes the first letter of each word to be uppercase. It doesn't touch any other letters. This isn't a bug, it's specifically designed this way. But it could give you problems if you haven't thought ahead, since most people expect it to also transform the other letters to lowercase.

In the above example, you would never be able to capitalize the words. Yes, you could start with lowercase in the HTML rather than uppercase. Then you can use text-transform to make it uppercase, and capitalized. Except it would still be all in lowercase if someone were to view the site without a stylesheet.

My suggestion then, is to capitalize things like this in the HTML. So taking my above example again, I would write it like this in the HTML,


<ul>
	<li>Some Option</li>
	<li>Another Option</li>
</ul>

This way you can have full control over the case by using CSS. You can either leave it as it is, make it uppercase, or lowercase. It also means it will be properly capitalized if someone were to view the site without a stylesheet. This way you're not dictating the style in the content.

As with anything though, there are of course exceptions. If a product or trademark name is in all uppercase, or uses different cases within a word, then this would actually be considered the content rather than the presentation. An example of this would be something like "JavaScript", where the J and S should always be uppercase. In those kinds of scenarios, I wouldn't imagine that the case is ever going to change when the design changes, since it's a trademark, brand name etc.

If you need another option, like every other character uppercase or something, then you'll need to do something server-side to change it. CSS2.1 is good, but not that good. You could come up with a JavaScript solution, but JavaScript isn't always the answer.

It's all a matter of analysing the situation and deciding the best way to do it. Could the design change in future, and if so, would it be possible to have the control needed using just CSS? A good rule of thumb is to think the following, "If I view the page without a stylesheet, does the text still accurately reflect what I need it to?". But remember not to be caught out by how text-transform: capitalize; works.

JavaScript is Good, But Should Not be Relied Upon

xml-feeds@wblinks.com (Rich Adams) — Sun, 19 Jul 2009 09:09:00 -0700

There was a time, years ago, when the only reason to use JavaScript on a website was to produce cliché effects; flashing, scrolling, fading and popups to name but a few. It was slow, clunky and not a very nice language to write code in. Browsers required different code to do the same thing as another browser, the whole thing was a mess. People would overuse sites like dynamic drive to achieve all sorts of pointless effects, falling snow, page transition effects, and who can forget, the disabling of right mouse clicking by making a popup appear, which was about as effective as putting up a sign which says "Please don't push this button".

Recently however, JavaScript has lost it's status as as annoyance and has become common place on lots of main stream websites. It's picked up a certain bit of elegance and if used correctly, can add a lot to the experience of a site. Browsers can now parse JavaScript at speeds which make it viable to use for visual effects, and it can be used to trigger events and to change parts of the page dynamically without having to refresh the entire page (AJAX for example).

With libraries like jQuery and Prototype it has become even easier, as they add a new layer between the browser and the programmer, meaning you don't need to know about all the little inconsistencies between the browser implementations of JavaScript. The library will hide these from you, so you can concentrate on writing the code and let the library deal with getting it to behave the same in every browser.

A combination of browser support, libraries, speed and ease of use means JavaScript is now much more attractive than it was 5 or 10 years ago. It no longer has the stigma associated with 1990s sites and has become a much more civilised solution to web development. All the big sites use it, Google has auto-completion when you type, Digg uses it to show extra comments, etc.

There's no doubt that JavaScript is incredibly useful and a great way to make the interactive experience of a website seamless. But you should never rely on JavaScript for a part of your website to function.

Everyone Uses JavaScript

Everyone has JavaScript enabled right? Well I'm afraid not. If you've made any part of your website only accessible to those with JavaScript enabled, then you've cut out 6% of the people who use the internet (June 2009 figures from thecounter). That's a lot of people, no matter how you look at it. Yes, you can just put a disclaimer on the site saying "This only works with JavaScript". But that isn't going to make it any less annoying for those 6% of users.

You may be asking yourself, "Why would anyone have JavaScript disabled in this day and age?". Well, for any number of reasons. Companies sometimes disable JavaScript for security reasons, people may disable it because it makes their browser experience faster if they leave it off, people could be using a text browser, or a mobile phone browser that doesn't support JavaScript.

If a visitor comes to your site and doesn't have JavaScript enabled, yet your site heavily relies on JavaScript to function, then they'll leave immediately. Yet it is so simple to make things work without JavaScript.

Progressive Enhancement vs Graceful Degredation

There are two main programming philosophies when dealing with a new technology. Progressive Enhancement and Graceful Degredation. The latter means that you build the site using the new technology first. Make sure it works in all the latest browsers and that all of your fancy code runs well. Then you go back and add bits to make sure that if someone views it in an older browser, it will still work fine. Progressive Enhancement is the other way around. You make the main functionality of the site work without the new technology, then add it in afterwards so that it works for everyone.

Think of it like this. If you're building a skyscraper, how would you do it? You have two options.

Build the foundations, making sure it's a solid base. Then add each floor one by one.
Build all of the floors right to the top, then go back and make sure the base is strong enough for people to walk around on the bottom floor.

It would seem number 1 is the best choice, but for me, it depends on the scenario. For CSS I generally use Graceful Degredation, not from any pre-thought out plan, it's just how I happen to do it. When I write a new style, I will be developing on just one browser (generally Firefox/Linux), once I am happy with the style, I will validate it with the W3C validator service. I will have generally made a few typos here and there. Once the CSS is valid, I'll check it out in other browsers, usually the latest versions of Opera, Chrome, IE and Firefox. It will generally look the same in all of those, with just perhaps a few minor tweaks here and there. Then I will try it out in the older versions, IE7, Firefox 2, etc. Based on how many inconsistencies there are, I may just use a quick and nasty CSS hack to target just those browsers, or give a new CSS file entirely (in the case of IE6/7). This is graceful degredation at work. I know it works in the later stuff, now I'm making it work in the older stuff without breaking it in the newer stuff.

The problem with this method, is that sometimes you'll edit something to work in an older browser, but then find you've broken it in the newer browsers, so you start going back and forth with yourself trying to get it to work in both.

Progressive Enhancement is the other way around. You first make it work without any "extra stuff", then you add on the extra bits afterwards, for those who have the technology to view the extra bits. It means current people can still use the site, but those who are ahead in technology make use of the latest stuff. This is my preferred method, because it means you won't leave anything out, and will slowly build your site up from a base. It also means you generally aren't going to break something you've already coded, which saves time in the long run.

This is how you should add JavaScript to a site. First make the site work without JavaScript at all. Disable it in your browser and make the site function. Then go ahead and add all of the JavaScript goodness you want, safe in the knowledge that those 6% of people will still be able to use the site. Yeah, so it won't have lovely fade effects, or quick AJAX, but it will still work.

Progressive Enhancement also has an added bonus. It's actually generally a quicker and easier method than Graceful Degredation. It should be a no brainer!

You're a Hypocrite!

I recently (well, end of last year) made the decision that I will no longer build website for IE6. It is my strong opinion that IE6 is holding back web development. It takes nearly double the amount of time to make a site work in IE6 as well as all the other browsers out there. To quote a bit from Falling Down, this isn't economically viable! Yes OK, a lot of people still use IE6 (around 9%), but it won't go away unless we make a stand.

Unfortunately, it isn't going to go anywhere soon, because it was around for so long without any sort of competition, which means lots and lots of companies used it to build internal tools and sites, which now will only work in IE6. They would need to spend lots of money to upgrade their internal things to work in more modern browsers, so they will simply stick with what they have. Why spend money, when it works as it is. There is also the fact that "Internet Explorer" was very cleverly named to have the word "Internet" in it. You'll find the majority of non technical people will not understand what a browser is, and simply think that "Internet Explorer" is in fact "The Internet".

You could say I'm being quite hypocritical. Here I am trying to argue that we should make the time to make our sites work for 6% of people who don't have JavaScript enabled, yet I'm advocating cutting off the 9% of IE6 users out there for exactly the same reasons.

Well, it's not that simple. There are some fundamental differences,

To make a site work without JavaScript takes very little effort. Compared to the effort required to make a site work in IE6, which normally takes double the time as you have to build two sites. One to work in IE6, and one to work in everything else.
There is no excuse for someone to be using IE6 exclusively. It's heavily outdated, and I'm betting most people's kettle's are actually newer than IE6. JavaScript on the other hand has legitimate reasons to be turned off. Security, speed, text browser, mobile browser, accessibility.
Making your site work in IE6 only benefits people who use IE6. Making your site work without JavaScript has several other advantages. If you accidentally introduce a JavaScript bug or the JavaScript fails to load correctly, your site is resilient and won't be crippled, as it will still work without JavaScript. Search Engines generally can't follow JavaScript links, so making them work without JavaScript means search engines will be able to index your site better, etc
As new browsers come out, they may introduce inconsistencies in the way JavaScript works. Being able to fallback to a non-JavaScript solution means you don't have to immediately spend time and effort getting it to work in new browsers. Once you've made the site work in IE6 and put the effort in, that's it, it's never going to be useful for anything else.

Maybe you still think I'm a hypocrite, and that's fine. Let me know your thoughts in the comments. I'm curious to know where people stand on this subject.

Stop Waffling and Get to The Good Stuff

OK, so now to get to the code. Let's imagine you have a shopping website that relies entirely on JavaScript. Your links might look something like this,


<a href="javascript:addToBasket()">Buy Item</a>

That's great, it works for you and adds the item to your shopping cart. Now turn JavaScript off, and you're not going anywhere. There's a link on the page which doesn't work. A user is furiously clicking it, but it won't add anything to their cart. They go to another site instead, and you just lost a sale. Good luck if anyone using a text based browser wants to buy anything, as it won't work for them either.

(This has actually happened to me in the past, not because I've had JavaScript disabled, but because the code had a typo in it and wasn't loading correctly, so I couldn't purchase the item and went somewhere else instead).

If you build a website, you should make it work for as many people as you possibly can. (Yes, even IE6 if you have the time and the money). Imagine buying a car, only to find you can only use it on 94% of the roads. Great, until you hit one of those 6% of roads.

Here's a better way to make the link. This follows the Graceful Degredation process I talked about earlier. We had our original link, which works in JavaScript, and now we're editing it to work without JavaScript too.


<a href="buyitem.php?item=awesomestuff" onlick="addToBasket()">Buy Item</a>

(We'll ignore for the moment it's a bad idea to use a GET parameter for something like this, that's a PHP issue and not in the scope of this post. I'll write about it sometime in the future).

If you make the "addToBasket" function return false at the end, then if JavaScript is enabled, the link will not be followed and only the JavaScript will execute. But if JavaScript is turned off, then the link is followed, and the user can still add things to their basket, (with a script you've written, but for this example, I'm assuming the JavaScript would have called the same PHP script with AJAX anyway, it just will now redirect back to the same page). Everyone is happy and everyone can buy things.

But there's a better way.

The Better Way

Now there's another issue. If I don't have JavaScript enabled, I'm still downloading the "onclick='addToBasket()'" but of the code. It's not much, but over a whole page it could be a lot. It also makes the markup look untidy. Not a big issue, but I'm OCD about that sort of stuff. Also as we saw earlier, there's a better way to do things. Lay the foundations of your skyscraper first.

If you've never heard the term "unobtrusive JavaScript" before, I suggest you do a quick Google search to see what it's all about. Basically it's called unobtrusive because the markup stays the same and has no JavaScript calls in it at all. It is pure HTML, as it was meant to be. HTML for markup, CSS for presentation and JavaScript for behaviour. So rather than littering your HTML with "onclick=''", etc. It is nice and clean.

Since JavaScript only needs to work when JavaScript is enabled, you can use the JavaScript itself to modify the DOM once the page has loaded, in order to add the "onclick" event handlers for you. But this also means there's no chance of someone clicking a JavaScript link before the page has fully loaded, which can sometimes cause problems, since the JavaScript version of the link won't be active until the page has fully loaded. jQuery makes this sort of stuff extremely easy.

On page load, you modify your links to add the JavaScript handlers. If a person has JavaScript disabled, they don't even need to waste the bandwidth of downloading the JavaScript, and the HTML will be nice and clean making the page load faster for them. But if they do have JavaScript enabled, the site will be modified to work as a JavaScript version.

Suppose we now create the link like this,


<a href="buyitem.php?item=awesomestuff" id="buyitem-awesomestuff" class="buyitem">Buy This</a>

This is nice and clean, has an ID to unique identify it, has a CSS class so we can style it appropriately, and is free of JavaScript. This will work for everyone. Now we can add a JavaScript file to the page and "AJAXify" this link without ever needing to touch the HTML again. So an example using jQuery would be something like this,


$(document).ready(function()
{
    $("a.buyitem").click(
        function()
        {
            var url = "buyitem.php?item=" + $(this).attr("id").substring(8);
            $.get(url,
                  "",
                  function(val) { alert("Item added to cart"); },
                  "text"
            );    
            
            return false;  // So we don't follow the normal link.  
        }
    );
});

When the page loads, this will go over all links with the "buyitem" class, pick out the item string from the id, and add an "onclick" event handler to submit the AJAX request. We haven't touched the HTML at all, but by just adding JavaScript we've made it work with AJAX for those with JavaScript, and we're safe in the knowledge it will still work for those without.

Note : The example is just to demonstrate unobtrusive JavaScript. You will want to have a much better callback than the one above. The callback should check that the item was actually added, rather than just show an alert either way as the example above does. You probably want to parse the id to get everything after the "-" character, rather than just substring(8), since the id may change in future, etc.

Final Thoughts

Every so often I come across forums, or answers on the internet, which give JavaScript solutions to common problems (which is perfectly fine), but then they say something like "This will only work with JS enabled, but everyone has it on nowadays, so screw those who don't, they probably won't want to look at your site anyway". This is just completely the wrong attitude. With a little bit of extra work you can make it work both ways and have a site that works for 100% of users, not just 94%. People argue that because it's 2009, everyone should have JavaScript enabled. While perhaps they should it doesn't necessarily mean that they have to. There are legitimate reasons to have it disabled, and those people shouldn't be excluded.

Another common issue people tend to solve with JavaScript is input validation. If your input validation is just a JavaScript file then you have failed as a web developer. All I need to do to bypass your checks is to disable JavaScript. It's the equivalent of using JavaScript for login/password (something I have actually seen on a production site before). By all means use JavaScript for input validation, it means the user can see something is wrong without having to submit the form, a great interface response. But don't rely on it. Always always always make sure you also have server-side validation for the same things, otherwise you're just asking for a world of trouble. It may feel like you're just repeating code, but it's worth it in the long run.

So I guess my point is this. Never rely on JavaScript to do a job. Form validation, prevent dual submits, links, etc. Always make sure they also work without JavaScript, otherwise you've either opened your site up for abuse, or made it unusable to 6% of the people who use the internet. Either way it's not very good.

Storing Passwords - The Wrong, Better and Even Better Way

xml-feeds@wblinks.com (Rich Adams) — Sun, 21 Jun 2009 10:19:00 -0700

If you've ever had to sign up to use a website, you'll no doubt have been prompted to provide a username and password, so that when you next visit the site you can login without having to fill in all of your details again. Your password has to be stored somewhere, otherwise you won't be able to login the next time you visit. Right? Unfortunately, a few sites I've come across do just this. They store your password, which means if the information is stolen, someone has got your password.

In a perfect world, everyone would use a different password for every account they sign up for, and that password would be a combination of numbers, letters (uppercase and lowercase) and special characters, and would be at least about 20 characters long. But let's be honest here, we're not all memory machines and remembering cryptic combinations like that isn't something everyone can do. So people are tempted to choose just one password and use it on everything they sign up for, including their email account. Which means there's the potential for someone to have the email address and the password. Not a very good thing.

There's no way around this, the weakest point in any security system is the human element. People are always going to chose easy passwords, or use the same passwords for multiple sites. So it's up to us as web developers to help to keep the passwords secure so that these people never have to go through the problems associated with someone getting into their other accounts.

This all comes down to how you store the password. Do you do it the “Wrong Way”, the “Better Way” or the “Even Better Way”? (I'm not going to say the “Right Way”, because I don't think there is such a thing when it comes to password security).

The Wrong Way

I was shocked to recently discover a website which stored their passwords in plain text. I couldn't quite grasp how in modern web development anyone could store their passwords like that, but it still happens. Storing passwords as plain text is a very bad thing. It means that if I sign up for a website using the password of “iamawesome”, then the value that's stored in their database is also “iamawesome”. This means any employees with access to the database (and there will always be at least one), can see your password. They can also probably see your email address associated to the account. So if you're one of those people that uses the same password, there's nothing to stop that person going and getting into your accounts. I like to think the majority of people out there in this position are honest and would never use that information for such nefarious means, but by the same token, I'm willing to bet there are people out there who aren't so honest.

So how can you tell if a website stores your passwords securely? The simple answer is that you can't. But there are a few tell tale signs that they're not doing all they can. Many websites have a “Forgotten password” function. If you go through this procedure and you get an email with your original password in, then it is highly likely they're storing you password in plain text. (Although this isn't always the case, they could be using reversible encryption, but again.. the developers will now how to reverse the encryption, so while it may stop anyone who steals the database, it wouldn't stop dishonest employees).

Personally, I would stop using such a service straight away (and in most cases I will email the relevant department to warn them that they're storing things insecurely). There is no need for a website to store that information in plain text. None at all. It puts your information at risk. There's a much better way to store passwords.

The Better Way

In order to login to a website, there's no need for the site to actually know what that user's password is, they just need to be able to tell if you've entered the same password as when you registered.

You can use something called a “hash” to do this. A hash is a one-way mathematical function, which given an input A, will always produce the same output B. But ideally it's very difficult to get from B back to A (but not impossible, you can use a rainbow table to lookup A based on the hash B).

Note : It is possible for more than one value to result in the same hash. This is called hash collision. It's pretty unlikely to get hash collision unless you're using very large datasets, but it is a possibility. I've never come across it myself though. But it does happen.

Different algorithms will hash things in different ways, and some are better than others. Some, such as MD5, have almost complete rainbow tables, so it's possible to find all the values of A which could map to the hash B, etc. MD5 is now considered an insecure hashing algorithm and shouldn't really be used to secure passwords on it's own. Others, such as the SHA-2 family of algorithms, remain secure at the moment and have no complete rainbow table lookups. I use SHA-256 myself. (SHA-1 also has some weaknesses such as hash collision, but it's better than MD5).

PHP 5 has built in functions for hashing with SHA-256


$password_hash = hash("sha256", "iamawesome");
// 4aa4029d0d0265c566c934de4f5e0a36496c59c54b6df8a72d9c52bdf0c1a0e8

The idea behind using a hash, is that you store this hash in your database, rather than the plain text password. In order to verify a user has entered the correct password when logging in, you use the same hash function on whatever text they enter, then compare the result to what you have stored in the database. If they match, then the correct password was entered.


$user_entered = hash("sha256", $_POST['password']);
return ($user_entered == $password_from_db);

This way, only the user ever knows the real password. If someone were to look at the database of your stored hashes (whether it's a dishonest employee, or because it was stolen) they'll only ever be able to see the hash, and won't be able to go around getting into people's email accounts.

So you might be thinking that it's problem solved, let's just do this hashing stuff and we'll be secure. Well no. There are still some problems with this method. Suppose two people have the same password, this means they will have the same hash in the database. Now suppose you manage to trick one of these people into giving you their real password (via however, email scam, etc). This means you would now have the password for anyone with the same hash.

Also, one of the most common password attacks is called a dictionary attack. It involves trying every dictionary word in order to login as someone, as it's quite common for people to just use a dictionary word as their password. A dictionary attack is still possible when using hashes. You can generate a list of hashes from common dictionary words and compare it to the hashes in the database, you'll probably find a few matches, and then you have the password for those people.

So really all we've done is obfuscated the password from people viewing it directly, but certain attacks are still possible. There is still a better way.

The Even Better Way

The way around this problem is to use something called a “salted hash”. The definition of a salt is “random bits that are used as an input to a key derivation function”. Normally to create your hash you provide one thing as input (the original password) and you get the hash as an output. A salt is a random string of characters you use as another input into the hash function in order to get the output.

So now when storing a users password, you can take the hash as you had before, but also generate a random hash, concatenate this with the password hash, and then hash that result again. So in PHP it would look something like this,


$salted_hash = hash(hash($password) . $random_salt);

You can make your function however you want, as long as the random salt and password are both used in order to construct the hash you want to store. The method you use doesn't change the effectiveness of your password storage, one is not really any more secure than the other. Relying on the design of how you hash the password and salt together to provide security is called security through obscurity and should be avoided, since in reality the method you use doesn't affect the security.


$salted_hash = hash($password . $random_salt);
$salted_hash = hash(hash($password) . hash($random_salt));
// ...etc

In the database, it's also important to store the salt for each user as well as the completed salted hash, otherwise you won't be able to tell if the user has entered the correct password. Each user should have their own random salt, you shouldn't use the same one for the entire database or for multiple users, otherwise you've completely negated the point of using a salt in the first place.

You're table should look something like this,


username         password_hash          password_salt
rich             2bae773debd80de        e99aa878c15879a554f
bob              d82ff2c12d5065f        d59cdc5abc7f7f3a207

So now, this means the hash you store will be different for every user in your database, even if they have the same password. So if anyone wants to do a dictionary attack, they have to do it for each user individually, rather than the entire database at once, making it much more difficult. This method also makes using a rainbow table to attack the password quite difficult and often infeasible, since an attacker would need to precompute rainbow tables for each possible salt value.

(Note: You can never make it impossible to crack someone's password, there will always be a way. You can just make it very very difficult)

But now you have another problem, how do you implement the “I've forgotten my password” functionality if you can't tell the user their password?. Rather then just retrieving the user's original password (which you can't do when using salted hashes), you instead want to either generate a new one for them and email it to the address you have for their account, or to verify the user by other means (such as security questions) and ask them to reset their password once you are sure they are who they say they are. Most mainstream websites nowadays will do this (and if they don't then I'd be very worried).

If you have dishonest employees, they won't be able to get the passwords. If someone were to steal your database, they'd have to perform a dictionary attack on every single user in the database. Making it very difficult for them.

Anyway, that's my two cents. Anyone have any better ways to store passwords?

I was prompted to write about this after posting my tip on Tipster and then coming across a project which still stored passwords as plain text.

Always Include a Print Stylesheet

xml-feeds@wblinks.com (Rich Adams) — Wed, 29 Apr 2009 10:17:00 -0700

Earlier today I added a new tip to Tipster entitled Use a Print Stylesheet. I thought I'd follow it up here in a bit more detail.

A print stylesheet is a stylesheet that's only applied to your website when the user goes to print it. A lot of sites will provide a "print version" of their page, which is a page which has the same content as the original page, but has different markup and layout to make it look better when printed. But creating a separate "print version" of your page is not necessary, as you can simply use a print stylesheet to do all of the work.

Most websites have a lot of content which is very useful when browsing the site, but makes no sense being printed out on paper. Navigation menus for example are great for getting around a site, but are a waste of ink and paper if it gets printed every time a user prints a page from your site. You also don't really want background colours to be printed, as if your site has a dark background it's going to waste a lot of ink.

The internet and paper are two completely different types of media, and your site should adjust so that it's best suited for the media it is being presented on. While small fonts may look fine on a screen, they may be very difficult to read when printed. Banner images may make your website stand out from the rest, but it's just going to waste paper if it gets printed. Now I could go down the green route here and say that having a print stylesheet helps to save the environment. Less ink equals less paper, and not wasting paper is good for the environment. For most people this would be more than enough to encourage them to use a print stylesheet. But for the more business minded, print stylesheets take time to develop, and is it really worth the extra cost? Well the answer is yes.

How many times have you gone to print a website, whether it's a flight itinerary or just a product description page, and you've had pointless images printed, the text goes over the edge of the page, and 5 pages or printed material comes out of the printer when you only really need the first page since the other 4 are just banner images? I've lost count of the number of times this has happened to me, and it's extremely frustrating. So much so in fact that I generally resort to just cut and pasting the text I want into a new document and printing it myself. This is a mild annoyance for me, but not the end of the world. But it means the company behind the site doesn't have their logo on my printed copy, and that's a missed opportunity. Plus a less frustrated customer is always an advantage right?

A print stylesheet takes minutes to build, but can make a world of difference for your users. For me, it's not so much the reason that I want my users to have a better experience (although I admit that should be driving force), it's that I want my designs to look as good on paper as it does on a screen. I want my sites to look great, no matter how a user chooses to view it.

So now that you're convinced that adding a print stylesheet is a good thing (or maybe you're not convinced, I'm still going to assume you are though, since otherwise it makes the next bit of this rather pointless) you may be wondering what a typical print stylesheet looks like, and how to include on on your page. Well, that's what this next part is all about.

Including a print stylesheet on your site is pretty straight forward. You add the CSS file like you would any other, but you make sure to select the media type of "print". That way the styles are only applied when the user goes to print the site, rather than all the time.


<link rel="stylesheet" type="text/css" media="print" href="print.css"/>

Generally you will want to hide some elements that don't need to be printed such as sidebars, headers, menus, etc. This can be done by specifying "display: none". I've included the "!important" declaration just to be sure it gets applied and isn't overridden by anything else.


#header,
#menu,
#additional-content   {
    display: none !important;
}

Another useful thing you can do with print stylesheets is to show the URL of any link after the link text. When viewing a website this isn't really necessary, since you can click the link. But when printed you have no way of knowing the URL for a link. I'm normally not a fan of using CSS to include content, but in this case it's very useful, so I like it.


a[href]:after {
    content: " (" attr(href) ") ";
}

You can use the print stylesheet to add page breaks too. If you have a list of blog posts for example, and you want them to print one to a page, then you can use the following bit of code,


.blog-post {
    page-break-before: always;
}

Does anyone else have any tips and tricks for things to include in print stylesheets? Does anyone prefer to create a "print version" of their page rather than using a print stylesheet, and if so, why? Comments on a postcard please, or you could just comment to this post.. your choice :)

Reorganisation

xml-feeds@wblinks.com (Rich Adams) — Sun, 26 Apr 2009 18:51:00 -0700

I've finally started to re-build this site. I can't even remember the last time I had chance to just sit down and play with the code for it. After taking one look at the PHP, I decided it really wasn't worth the hassle to try and fix it. I originally wrote the code for this site back when I first learnt PHP and it was not a pretty sight. I couldn't believe some of the code, I want to believe I could never write such horrible code, but well.. I was learning, so you can't really blame me.

So I have re-built the entire back-end of this site using a lovely object oriented approach and I've restructured the entire database so that it's a little quicker on the lookups. I've also taken the opportunity to re-write some of the markup for the front-end, although I'm still in the process of sorting that out. I also updated the Flux style (I've called this new version "Flux Refresh", since it's pretty much the same style, just with a slightly more polished look). I intend to create an entirely new style for the site sometime soon, but that's a job for another day.

My next task is to organise the content of the site. I have many more projects to add, and perhaps some new sections to add. I'm also in the process of re-building my photos section, which will be back up and available sometime soon. As part of this process I'm also going to try and actually write some more posts for the site. I already have a few drafted out which I'll put up soon, but it would be nice to actually get some content added every now and then rather than me just being too lazy to update as has been in the past.

In a bold move, I've also dropped IE6 support. I'm sick and tired of having to deal with such a completely useless browser. So you now get a non-styled version of the site if you use IE6. If you don't like it, tough! We don't like your kind around these parts!

digg2del.icio.us

xml-feeds@wblinks.com (Rich Adams) — Tue, 24 Feb 2009 11:38:00 -0800

I was trying to find a tool the other day, which would take all of the stories I've "dugg" over on digg.com, and add them as bookmarks to my del.icio.us account. Unfortunately, my search didn't turn up anything, so I decided to make one myself.

It's just one PHP file which you can use to export the digg stories for your username (or any username on Digg for that matter), and import them into your del.icio.us account.

digg2delicious-1.0.tar.gz - 5.0KB

Alternatively, you can just try it out without downloading the file yourself.

Chrome Beta Released

xml-feeds@wblinks.com (Rich Adams) — Tue, 02 Sep 2008 13:30:00 -0700

Google's new browser, Chrome, is now available for download on Windows XP/Vista only.

First impressions are that it's a very fast install, and imported all of my Firefox bookmarks with no issues. Loads much quicker than any of my other browsers, and looks quite nice. The address bar is a bit thick for my liking, but that's just me being picky. None of the usual "File", "Edit" menus, etc as the entire interface is pretty simplistic. Exactly what we've come to expect from Google products. It's got the usual Google feel about it, a nice clean interface, no status bar showing useless information, etc. When the status bar does show, you can move your mouse towards it, and it'll drop below the window so it's not in the way of the page. A nice touch.

Some things are noticeably missing though, such as a "home" button, which is a feature common to most browsers. It is available, but not enabled by default. It's an option on the "Basics" part of the options page. I guess it depends on how much you use it whether that will be an annoyance or a blessing.

Sites are loading pretty fast, as I would expect from something based on WebKit. One feature I have noticed, is that when you're on a page with a download link, it gives you a list of the available downloads at the bottom of the page. I've already found this useful.

They've taken a page from IE8, and fade out the rest of the URL that's not the main domain. I'm not really sure why this is becoming common, and I don't really see the point. Regarding the address bar though, it's also a search bar. You can either enter a URL, or a search term and it will search Google. This combines the two bars you get in Firefox quite nicely. Of course, in FF you can choose the dropdown for various sites (for example I have a Wikipedia one), and I haven't found a way to do that easily in Chrome yet.

Next up are the screenshots.

The home page is pretty nice, shows you some recently visited pages along with your bookmarks. You can edit the things which are shown on it from the options screen.

The download bar which shows when you go to download a file.

Not really many options to go at, but it's simple enough for the average user.

"Incognito" mode is the privacy option. Does exactly what it says on the tin.

Google Chrome passes the ACID2 test with flying colours.

Only gets 61/100 on the ACID3 test, but that's better than a lot of other browsers. It takes a very long time to run on my machine though.

The task manager is quite nifty, and gives you a breakdown of the memory usage of each tag. It'll even tell you how much Flash is taking up if the page has Flash on it.

A new feature is the "about:memory" page, which gives you a breakdown of the memory usage of the browser, along with any other browsers you happen to have open too. I can imagine this being useful for testing purposes.

Overall, I'm impressed. It's clean, quick and will be great for most users. I won't be using it as my main browser though, as I really can't do without some of the FF extensions I have, they're just too useful. If they include an Add-on, or extensions feature, they're onto a winner! I think the hardest part for Google is going to be getting people to actually download Chrome, and then switch to it as their main browser. People don't like change. As long as they're moving away from IE though, I've got no complaints.

A great start, and much better than the IE8 Beta. I see amazing potential in this if it gets released on the various platforms, and has the ability to customize it to your needs.

For reference. Here's what happened the first time I loaded IE8b2 after a fresh install. I'm not making this up just because I hate IE.. this is actually what happened. Look how cluttered it is compared to FF, Chrome or any other browser!! Plus the fact that it crashed. (I should point out, it only crashed the first time. It seems to work fine on second load).

Internet Explorer 8 the first time I ran it.

Google Chrome

xml-feeds@wblinks.com (Rich Adams) — Mon, 01 Sep 2008 13:32:00 -0700

Came across this today.

Google Chrome is Google's open source browser project. As rumored before under the name of "Google Browser", this will be based on the existing rendering engine Webkit. Furthermore, it will include Google's Gears project.

Apparently, it also includes its own JavaScript VM to help speed up scripts. Sounds interesting. I wonder how it'll render CSS ;) Seen as it's based on Webkit, I'm guessing it'll be very very good.

In other news, I also came across a FF extension called Pixel Perfect (well, actually it's a Firebug extension). It allows you to overlay an image onto a website, so you can see how close the coded site is to the original designs. Quite handy if you do that sort of thing.

Update There's a load of screenshots now available.

You can read the comic if you want. It describes some of the features.

Due to be released tomorrow, so I'll be giving it a go then.

IE8 Beta 2 Released

xml-feeds@wblinks.com (Rich Adams) — Thu, 28 Aug 2008 04:32:00 -0700

The new beta of Internet Explorer 8 has been released.

Complete with the new InPrivate option that's been all over the news the past week or so. They've also got Compatibility View, which quickly switches to render the page as IE7 would render it (so.. incorrectly). They had IE7 rendering in the old beta, but it was crap, as you had to reload the browser. Apparently this one does it instantly.

It fares well against current CSS standards though, a lot better than any current iteration of IE. There's an updated compliance table over at quirksmode.org.

I shall be trying it out myself later on to see how it is.

Number 10 goes Web 2.0

xml-feeds@wblinks.com (Rich Adams) — Wed, 13 Aug 2008 04:02:00 -0700

Another in the long list of sites to redesign this year is Number10, the website of the prime minister's office has taken on the whole web 2.0 look.

It's got Flickr, YouTube and Twitter all in the sidebar, plus links to submit stories to Facebook, Digg, etc and of course, RSS feeds.. just in case you want to keep up to date on everything Downing Street it twittering.

Seems like it's built on top of WordPress (based on the code), despite the fact there's no mention of it anywhere. They've already got some rather interesting code cock-ups. Nested anonymous <div>'s for absolutely no reason, links pointing to local IP addresses (e.g. the "Home" link, which points to http://10.10.0.215/), not to mention the fact they're not clearing floats on the "Communicate" page, so the images are pushed off to the left.

There's a grand total of 72 errors if you try and validate the XHTML. Although to be fair, most of those are because of using & instead of &. At least the CSS is valid, although they didn't write it, it's a pre-made WordPress theme called NetWorker.

Also, no print stylesheet. C'mon people!!! All websites should have a print stylesheet. Trying to print their website is painful, and is going to waste a hell of a lot of paper, along with lots of ink. Is that really the environmental message they want to give out?

It's even in "Beta", a true web 2.0 site! Which would explain the above issues, can't expect everything to be perfect first time. It does look very good though, and is definitely an improvement on the old site. Nice to see they're actually making an effort!