W(e)blinks | Notes

Secure Session Management Tips

xml-feeds@wblinks.com (Rich Adams) — Sun, 06 Feb 2011 22:39:00 -0800

Most (if not all) modern websites use sessions to control the experience for individual users, and to maintain state between requests (since HTTP is a stateless protocol after all). Sessions are fantastic and incredibly useful, but if managed incorrectly they can expose your website to security vulnerabilities and potentially allow a malicious attacker to gain unauthorised access to user accounts.

Of course, the biggest tip is that you should really just use a pre-built framework which has tried and tested session management code where security experts have tested and verified it, and the bugs have been identified and fixed. But I never listen to myself...I've been building a new site in my spare time recently and got to the point of writing the session management code. This seemed like a good subject to try and get myself into the habit of updating my notes more regularly.

So while certainly not an exhaustive list, here are 11 of my tips on managing sessions and avoiding some common security vulnerabilities (yes, this post goes all the way to 11). I'm using PHP in the code examples, but the principles apply to any other language. In fact, PHP does a very good job of automatically protecting against most of the attacks the tips discuss, but this isn't necessarily the case for other languages, so the principles are still important to understand.

1. Always regenerate a session ID (SID) when elevating privileges or changing between HTTP and HTTPS.

By elevating privileges, I don't mean when you modify a user's record to give them more permissions, I mean any time an action is performed which makes the current user of the website have more privileges than they had moments before. Things such as logging in, where the user now has more privileges than they did a moment ago yet are still the same user of the website (and will be on the same session). In this case you should immediately generate a new session ID for them and destroy their previous session.

Regenerating a SID is extremely important to protect against session fixation. To understand what session fixation is, consider the following example of horizontal privilege escalation. User M (malicious) is a malicious user who is trying to access the account of User V (victim) on a website.

User M: Visits the website, and has a session ID assigned to them. They either look at the GET parameters (?sid=xxxxx) or at the headers (Set-Cookie: sid=xxxxx) to determine their session ID. Once they have the ID, they craft either a direct link using the GET parameters (http://example.com/?sid=xxxxx) or they construct a non-direct link which will add the relevant headers. This link is then sent to User V.

User V: Gets an email from User M which says "Hey, check out your new look banking account! http://example.com/?sid=xxxxx" Since the link looks legitimate (example.com is the real URL) then the user clicks the link confident it's not a scam. They then login with their account.

User M: Since this user already had the same session or already knows the SID, they will now be logged in to the site as if they are user V. If you regenerate the SID, then this wouldn't be possible since user M would no longer know the correct SID.

There are other ways session fixation can be performed, such as setting a cross-site cookie (Site A sets a cookie for Site B with the session ID, etc), or if you allow a sid to be set from a GET/POST variable then the malicious user can just pick one they want and don't even have to visit in the first place.

Remember it's not just for login, any privilege escalation should get a new ID. You could just do it on every request since the user won't care that the session ID changes and it makes any session ID immediately invalid after the next request, but that might be overkill. Regenerating the session ID will make the attack useless, since by the time User V has done anything, the session ID they were sent was invalid, so User M cannot access their account.

You should also regenerate the SID when switching between HTTP and HTTPS, since you want to be sure not to use an insecure value over a new secure connection and vice versa.

In PHP it's easy to regenerate a session id using session_regenerate_id(), just remember to take extra care to always use true as the optional parameter so that the old session file gets deleted.

session_regenerate_id(true);

2. Check for suspicious activity and immediately destroy any suspect session.

You want to be sure the user who started the session is the same as the user who is actively using the session. There is no 100% accurate way to do this, the best you can do is to look for suspicious signs and get the user to re-authenticate if you suspect at any point they're not the same user.

This is to help prevent session hijacking. Suppose a user visits your site and comes from IP address 999.999.999.999 and is using the Chrome browser. Then all of a sudden on the next request they visit from 888.888.888.888 and are using the Internet Explorer browser. This would be pretty unlikely to happen if it were the genuine user, and so can be considered suspicious. You should actively monitor for these types of events and immediately destroy the user session and/or get them to re-authenticate.

So it's simple then, just check if the IP changes and we can be safe knowing we're protected? WRONG. Many people can be using the same IP address and this alone doesn't prevent a malicious user from hijacking a session.

Ok, so we just need to check the User Agent and if it changes then we're protected? WRONG. Changing the user agent header is trivial and should not be relied upon to protect against this attack.

So what's the answer? There isn't one (that I'm aware of). There's no foolproof way to positively determine that the user is the same user who started the session. We can only be suspicious that a particular user isn't the original user. That's what this tip is about, not knowing the positive, but potentially knowing a negative and acting on it pre-emptively.

So if the user agent changes and the IP changes, you should implement some sort of policy to re-generate the session and get the user to re-authenticate. Be careful about just implementing one of these though, as it can have some bad side effects. Users behind a proxy will find their IP changes on every request, and the user agent string is easy to change manually. It depends on how cautious you want to be and what the site is used for. There's no right answer here.

Some guides and tutorials will suggest also checking the referer to make sure it came from your own site. This would seem to be completely useless to me though. Firstly, a lot of browsers don't send a referer when in private mode. Secondly, referers aren't sent over HTTPS, so if you're doing things securely, this won't work. Finally, just like the user agent, it can easily be changed. Referer will give you more false negatives than is necessary and will just degrade the experience for your users.

If ($_SESSION['_USER_IP'] != $_SERVER['REMOTE_ADDR']
    || $_SESSION['_USER_AGENT'] != $_SERVER['HTTP_USER_AGENT'])
{
    session_unset(); // Same as $_SESSION = array();
    session_destroy();
    session_start();
    session_regenerate_id(true);
    Log::create("Possible session hijacking attempt.", Log::NOTIFY_ADMIN)
    Auth::getCurrentUser()->reAuthenticate(Auth::SESSION_SUSPICIOUS);
}

$_SESSION['_USER_IP']    = $_SERVER['REMOTE_ADDR'];
$_SESSION['_USER_AGENT'] = $_SERVER['HTTP_USER_AGENT'];

A loose IP check is a good option if you don't want to screw over proxy users. Just check the first 2 blocks of the IP address. It will catch anyone quickly changing countries for example. You can add even more information into the mix too, such as if the "Accept" headers change, since these will generally stay the same if it's the same user.

If ($_SESSION['_USER_LOOSE_IP'] != long2ip(ip2long($_SERVER['REMOTE_ADDR']) 
                                           & ip2long("255.255.0.0"))
    || $_SESSION['_USER_AGENT'] != $_SERVER['HTTP_USER_AGENT']
    || $_SESSION['_USER_ACCEPT'] != $_SERVER['HTTP_ACCEPT']
    || $_SESSION['_USER_ACCEPT_ENCODING'] != $_SERVER['HTTP_ACCEPT_ENCODING']
    || $_SESSION['_USER_ACCEPT_LANG'] != $_SERVER['HTTP_ACCEPT_LANGUAGE']
    || $_SESSION['_USER_ACCEPT_CHARSET'] != $_SERVER['HTTP_ACCEPT_CHARSET'])
{
    // Destroy and start a new session
    session_unset(); // Same as $_SESSION = array();
    session_destroy(); // Destroy session on disk
    session_start();
    session_regenerate_id(true);

    // Log for attention of admin
    Log::create("Possible session hijacking attempt.", Log::NOTIFY_ADMIN)

    // Flag that the user needs to re-authenticate before continuing.
    Auth::getCurrentUser()->reAuthenticate(Auth::SESSION_SUSPICIOUS);
}

// Store these values into the session so I can check on subsequent requests.
$_SESSION['_USER_AGENT']           = $_SERVER['HTTP_USER_AGENT'];
$_SESSION['_USER_ACCEPT']          = $_SERVER['HTTP_ACCEPT'];
$_SESSION['_USER_ACCEPT_ENCODING'] = $_SERVER['HTTP_ACCEPT_ENCODING'];
$_SESSION['_USER_ACCEPT_LANG']     = $_SERVER['HTTP_ACCEPT_LANGUAGE'];
$_SESSION['_USER_ACCEPT_CHARSET']  = $_SERVER['HTTP_ACCEPT_CHARSET'];

// Only use the first two blocks of the IP (loose IP check). Use a
// netmask of 255.255.0.0 to get the first two blocks only.
$_SESSION['_USER_LOOSE_IP'] = long2ip(ip2long($_SERVER['REMOTE_ADDR']) 
                                      & ip2long("255.255.0.0"));

3. Store all session information server-side, never store anything except the SID in the client-side cookie.

A friend of mine (OK... me) built a site a very long time ago and thought that storing the username and password in a cookie on the client-side was the correct way to do things. Those details were then authenticated again on each request. As it turns out, this is a very bad idea. For starters, cookies are generally stored in plaintext on the client-side, so anyone with access to the computer can see them. Secondly, there are many attacks which can steal cookies and then use the information to impersonate another user. If all an attacker gets is a session id which has probably been regenerated since, it's pretty useless.

I... I mean my friend then decided that it might be better to hash the password. But again they were wrong. Even if you hash the password, it shouldn't be stored client-side, EVER. It allows someone to brute force the password and have all the time in the world to do it. Basically, never trust the client, you can't rely on client-side information being accurate, you should always keep things server side where you know that they are accurate.

The worst example of this was something I saw a few years ago (not me this time, thankfully). A site would start a session and let you login. The cookie would have a flag which said if the user was logged in or not, and then another variable with the username. All state was stored in the cookie rather than in the session. All you had to do to login as a different user was to change the username in the cookie, no password needed. A cookie can always be manipulated by the user.

Store all information server-side and only store the session ID on the client-side. The cookie should just be a pointer to the information server-side. You should treat cookies in the same way as any other user input (validate it and sanitize it).

When setting your cookies remember to always specify the domain, an expiry, and the HttpOnly option. HttpOnly prevents JavaScript from accessing the cookie, only the server can access it (assuming the user's browser implements it correctly of course). A common method of stealing cookies (and hence the session ID) is to inject some JavaScript onto a site using XSS, and then this JavaScript will steal the user cookie and post it to a malicious domain where the information is collected. Adding HttpOnly helps to prevent this.

As of PHP5.2 you can specify HttpOnly in the setcookie() method as the last parameter, or you can just set it directly into your PHP configuration to have session_start() make use of it.

// Manually set the cookie
setcookie("sid",                // Name
          session_id(),         // Value
          strtotime("+1 hour"), // Expiry
          "/",                  // Path
          ".wblinks.com",       // Domain 
          false,                // HTTPS Only
          true);                // HTTP Only

// Or, in php.ini
session.cookie_lifetime = "3600";      // Expiry
session.cookie_httponly = "1";         // HTTP Only
session.cookie_secure = "0";           // HTTPS Only
session.cookie_domain = ".wblinks.com" // Domain

// Then session_start will use the above config.
session_start();

4. Confirm SIDs aren't from an external source, and verify the session was generated by your server.

Never just blindly accept a session ID and assume it's valid. If you grab a session ID from a cookie, confirm that the cookie was set by the domain of your website (and not an invalid sub-domain for example), and make sure the session exists already and was generated by your server (so don't allow users to set their own session ID). We tend to assume that browsers correctly handle cookies so that cross-site cookies aren't possible. This might not be the case for all the browsers that your users use though, which can allow cross-site cooking. If the domain is invalid or the session wasn't created by your server, then destroy the session immediately and regenerate a fresh one.

Checking the session was generated by your server is as simple as adding a value into the session variable and checking for it's existence.

If (!isset($_SERVER['MY_SERVER_GENERATED_THIS_SESSION']))
{
    session_unset(); 
    session_destroy();
    session_start();
    session_regenerate_id(true);
}

$_SESSION['MY_SERVERT_GENERATED_THIS_SESSION'] = true;

5. Don't append the SID to URLs as a GET parameter.

Not really an issue in PHP 5.3.0 and later, since the default configuration of session.use_only_cookies will protect against this, but it could be important for another language or earlier PHP version.

Doing this will leak the SID to various people, and SID leakage can lead to a session fixation attack if you haven't protected against it (see Tip 1). It will be stored in the user's browser history, it will be stored in a bookmark if they bookmark the page. If they copy/paste the link then it will be copied too. Use a cookie instead. This isn't a huge deal if you regenerate the id on every request, but still something to avoid.

Cookies are very rarely disabled nowadays and I've yet to see anyone (in person, or in logs) who visits without them enabled. I'm all for having a fallback, but you need to decide if it's worth it based on site traffic.

6. Expire sessions on the server side, don't rely on cookie expiration to end a user session.

You should expire a session after both an overall lifetime and an inactivity time. Make sure you mark when a session was last used on every request. Just add a session variable with the current time,

$_SESSION['_USER_LAST_ACTIVITY'] = time();

When starting the session you should also store the time and then check a longer delay to make sure the session cannot last too long.

$_SESSION['SESSION_START_TIME'] = time();

Before doing any of this though, check if the last activity or start time value is older than some pre-defined time limit. If so, then destroy the session immediately. While adding a cookie time limit is important too, it should not be relied upon. You can't just clear the cookie on expiry and think it's over. The session will still be active on the server side and the session ID can still be used. You must clear it server-side too.

When the session expires you should make the user login again, and regenerate the SID as per Tip 1.

if ($_SESSION['SESSION_START_TIME'] < (strtotime("-1 hour"))
    || $_SESSION['_USER_LAST_ACTIVITY'] < (strtotime("-20 mins")))
{
    session_unset();
    session_destroy();
    Auth::getCurrentUser()->reAuthenticate(Auth::SESSION_EXPIRED);
}

7. Use long and unpredictable session IDs.

Quite basic this one, but never use sequential session ID's! If you rely on using a session ID which increments every time you need a new one, stop immediately and re-think your strategy. Even if you regenerate your session IDs to prevent fixation, even if you don't allow the session ID to be given as a GET parameter, even if you don't leak it in a GET parameter, none of that matters if you have predictable session IDs as an attacker can just know your current session ID no matter what. Session prediction is very bad.

In PHP this is all taken care of for you, and is not something you need to be concerned about, but in other languages that might not be the case.

session_start();
session_regenerate_id(true);

You can configure PHP a bit deeper than the defaults and decide on the entropy source (session.entropy_file) used to create the session IDs (/dev/random, etc), the hash function used (session.hash_function), how many bytes are used (session.entropy_length), etc. Change these if you like, but the defaults usually suffice.

Don't try to get clever and generate a session ID based on a hash of the IP or user-agent or anything like that. That's what I mean by predictable data, if you can find the pattern then an attacker can generate a session ID for anyone, opening you up to session hijacking.

Use something as random as possible, but also make sure it's actually a good pseudo random generator. Don't make the mistake of assuming the PHP rand() method on Windows is good for randomness for example, because you'd be surprised.

8. Properly sanitize user input before setting headers with them.

PHP will automatically protect against this as it will only allow one header to be set in the header() function, but for other languages it may not be the case.

This might seem a strange tip and appear unrelated to sessions, but all will become clear after an example. Suppose you have the following code, where the sanitize function strips things like HTML/JavaScript/SQL to prevent cross site scripting, but doesn't strip CRLFs (carriage return and line feed, or 0x0D and 0x0A in ASCII).

$nextPage = Sanitizer::sanitize($_GET['next_page']);
header("Location: $nextPage");

Even though it seems like you're protected from XSS attacks, session fixation is still possible using this method, despite the fact that you've technically not allowed the user to set sessions via the URL. This would appear to have nothing to do with sessions, but suppose I give the following link to a user,

http://example.com/?next_page=login%0d%0aSet-Cookie:%20sessionID%3d12345678

If you get rid of the HTML encoded characters you get,

?next_page=login\r\nSet-Cookie: sessionID=12345678

A sanitize method which doesn't protect against this by removing CRLFs will allow the above string through, in which case the following header has just been sent,

Location: login
Set-Cookie: sessionID=12345678

So even though you never allowed users to explicitly set the session ID, and you're sure you're safe against session fixation, a seemingly unrelated bit of code has allowed a sessionID to be fixed. An attacker can also modify the headers to make sure the cookie never expires for example, making it more likely the attack will succeed. This type of attack is called HTTP response splitting and is not something PHP users need to be too concerned with, as header() only allows one header to be set at a time, purposely to prevent this type of attack.

If you're not using PHP, you should sanitize any input which sets headers by removing CRLFs to prevent response splitting.

9. When a user logs out, destroy their session explicitly on the server.

Don't rely on garbage collection to destroy the session information on disk for you after a user logs out. Garbage collection may never run on a slow traffic site, even if you've set session.gc_probability, session.gc_divisor and session.gc_maxlifetime up properly. You can never absolutely guarantee garbage collection will run when calling session_start(). Always manually use session_destroy() to end the session and delete the data from disk. Don't rely on a cookie expiry to do it for you either, since if you don't manually destroy the session it will still be available on the server.

session_unset(); 
session_destroy();
session_start();
session_regenerate_id(true);

It's not generally possibly to delete a cookie explicitly, instead you need to re-set a cookie with the same name, but set it's expiry time to the past. I've seen tutorials which suggest using some JavaScript to clear the cookies, don't do this. Firstly relying on JavaScript is a bad idea, but also if you're storing your cookies correctly in the first place with HttpOnly (see Tip 3) then it shouldn't be possible to access your cookies via JavaScript anyway.

So first, here are some ways you shouldn't clear cookies.

unset($_COOKIE);
// This will only remove it from the superglobal and will do nothing to
// the actual client-side cookie.

setcookie("sid", "", 0);
// 0 sets the expiry time to when the browser is closed and doesn't 
// immediately expire it. Don't use 0!

setcookie("sid", "", strtotime("-1 hour"));
// Sets the expiry to one hour in the past right?
// In server time yes, but cookies are stored on the client in their
// local timezone, so depending on where that is, it may not expire for a
// few more hours!

The correct way to clear a cookie is to just pass in 1 as the expiry time. This is one second after the unix epoch and will always be in the past. (If you really want you can set it to some time over 24 hours in the past, but "1" is always going to be less verbose).

setcookie("sid", "", 1);

Don't forget to destroy the session on the server-side too!

10. Check your session configuration.

Check your session configuration carefully to ensure you're not sharing things you shouldn't. For example, by default PHP stores sessions in the "/tmp" directory. All well and good if you have a dedicated server, but if you're on shared hosting then it could allow anyone else on that server to see the session data and hijack them. Of course, you can still use /tmp, just make sure to set the file system permissions properly so only you can read the session data.

It's recommended to read through all of the configuration options PHP or the language of your choice provides and to make sure they're all set up correctly for your needs. A slight misconfiguration can open you up to all sorts of strange attacks. In general the default configuration is pretty good, but there are still some things you should consider changing (like the session.save_path mentioned above for example). The parts of the configuration you need to change will always depend on the specific needs of your application, so make sure to understand all of the options available to you.

11. Force users to re-authenticate on any destructive or critical actions.

A quick tip to end on. Any time a user wants to perform something destructive or critical (delete account, change password, etc) then you should force them to re-authenticate with their password. This will prevent anyone from performing the critical actions if they've stolen a valid session ID since they don't know the password.

Auth::getCurrentUser()->reAuthenticate(Auth::ACTION_SENSITIVE_CRITICAL);

You don't have to worry about the malicious user knowing the password, since if they knew that then it's a moot point and game over anyway, attempting the hijack the session would be pointless.

Summary

Always regenerate a session ID (SID) when elevating privileges or changing between HTTP and HTTPS.
Check for suspicious activity and immediately destroy any suspect session.
Store all session information server-side, never store anything except the SID in the client-side cookie.
Confirm SIDs aren't from an external source, and verify the session was generated by your server.
Don't append the SID to URLs as a GET parameter.
Expire sessions on the server side, don't rely on cookie expiration to end a user session.
Use long and unpredictable session IDs.
Properly sanitize user input before setting headers with them.
When a user logs out, destroy their session explicitly on the server.
Check your session configuration.
Force users to re-authenticate on any destructive or critical actions.

None of this is cutting edge and there are no new session based attacks out there that have prompted this post, all of these tips are things that have been known for years. But that doesn't mean people aren't always started to learn about web development and these things need to be known. Even if you rely on pre-built frameworks, knowing this stuff is useful for other areas.

As I said at the start, the list is certainly not exhaustive and there are plenty of excellent tutorials and articles on the subject just a Google search away. Secure session management is a complicated subject, so it's well advised to read around before trying to implement your own system.

As I have said many times in past notes, I am not a security expert. Before trying to write any session management code yourself, seriously consider using something pre-built and open source. Many web frameworks have session management abilities as part of them which have been tried and tested by many users and security experts, people who are much smarter than me.

If you want to share any other tips on session management though, please post them in the comments.

Cross Site Request Forgery (CSRF/XSRF)

xml-feeds@wblinks.com (Rich Adams) — Fri, 04 Jun 2010 13:58:00 -0700

If you're building a site that allows users to update any sort of information (so most websites), then you should probably think about protecting against Cross Site Request Forgery (referred to as CSRF or XSRF). Being susceptible to this type of attack can be annoying in some cases, but extremely dangerous in others. Unfortunately, it's not the type of attack that's easy to understand at first, and it's not immediately obvious how to prevent such an attack. Because of this, protecting against XSRF is often overlooked, even on some big name websites.

What is XSRF?

An XSRF vulnerability is one which allows a malicious user (or website) to make an unsuspecting user perform an action on your site which they didn't want to happen.

As a basic example, imagine you allow users to post images in your comments. If a malicious user puts "http://example.com/logout.php" as the image's URL, where example.com is your domain, then any time a logged in user views that comment they will be logged out if you don't protect against XSRF. It's not a valid URL for the image, but that doesn't matter as the unsuspecting user's browser will still make the request and your site will perform the action thinking the user wanted it.

A more dangerous example could be that you allow a user account to be deleted without confirming the action or protecting it from XSRF in any way, so any user visiting the page would then get their account deleted instantly!

I thought I'd make a comment on your site. Check out this cool image!
<img src='http://example.com/delete_my_account.php" />

Even if you have no user submitted content on your site, you can still be vulnerable. If a malicious website contains a form which the unsuspecting user submits, it can submit the information to a different site which the user is logged in to, and that site will think the request came from the logged in user and will just perform the update as if the user had done it intentionally.

It doesn't even have to be a link/button the user clicks on, as shown in the first example it could happen even if just viewing a site. Obviously, this could become quite annoying for your users.

While annoying is one thing, it can also be dangerous. For example, let's say I'm logged into an account on a simple shopping cart site. I then go and browse to another unrelated website. The other website has a button which just says "Click here to register". Seems simple enough, this other website wants me to register an account. However, this is a malicious site and when I click the link, it's actually submitting a request to the original shopping cart website as if I'd clicked the "Order me 1000 of some expensive item in one-click" button. If the shopping cart website is susceptible to XSRF then it will think the request to order 1000 items was genuinely submitted by me, and I'll get a nice surprise in the mail and on my credit card statements.

It's a particularly difficult type of attack to get your head around as it's very subtle, but once you understand how it works it's not that difficult to protect against it. It's very easy to go down the bad route though and think you're safe when in reality you're still wide open to attack.

Protecting against XSRF - The bad way

So what can you do as a web developer to prevent such attacks? Effectively you're just wanting to make sure a request came from your site and was actually intended to be run by the user, so as a first step you can just check the referer header to make sure the request came from your own site.

<?php
$url = parse_url($_SERVER['HTTP_REFERER']);
if ($url['host'] != "wblinks.com")
{
    die("You're not coming from my site. Possible XSRF attack");
    exit();
}
?>

While this may work in some cases, it's going to be about as effective as an underwater hair dryer as far as stopping an XSRF attack. Altering the referer header is pretty trivial and it will also have the downside of making the site unusable for lots of people, since many browser or proxies can strip the referer header when in "private" mode.

This also wouldn't protect against the first example of XSRF, where someone just uses the logout URL as an image URL in a comment. As the request would come from the correct referer in that case.

So not only will you not prevent XSRF attacks, but you'll also annoy some of your users. Not a good solution.

Protecting against XSRF - The good way

What we really need is a nonce (one-time key/token) which allows us to validate that the request came from a form we presented to the user intentionally. The following code samples show the method I use to achieve this on my site.

For the purposes of the following code examples, you can assume the Session class is just a wrapper for the $_SESSION superglobal. It actually does some other stuff, but that doesn't matter for this.

I have an Auth class which handles everything to do with user authentication on my sites as well as authenticating that a request was valid, etc. In this class I have 3 very simple static functions,

// This is a nonce used to stop CSRF/XSRF attacks. It's stored in the user session. 
public static function getNewRKey() 
{ 
    $rkey    = substr(sha1(uniqid(rand(), true)), 0, 20); 
    $rkeys   = Session::getInstance()->getVar("rkeys"); // $keys = $_SESSION['rkeys'];
    $rkeys[] = $rkey;
    Session::getInstance()->setVar("rkeys", $rkeys); // $_SESSION['rkeys'] = $rkeys;
    return $rkey; 
} 
    
// This will determine if an rkey is valid 
public static function validateRKey($rkey) 
{ 
    $rkeys = Session::getInstance()->getVar("rkeys"); // $rkeys = $_SESSION['rkeys']
    if ($rkeys == null) { return false; } 
    return in_array($rkey, $rkeys); 
}   

// Creates the form input to use 
public static function formRkey() 
{ 
    return "<input type=\"hidden\" name=\"rkey\" value=\"".Auth::getNewRKey()."\" />"; 
}

So what's going on here? Anytime I have a form on the site which POSTs data, I will output the formRKey() function to add a hidden field to this form. This hidden field contains the value returned from getNewRKey(), this method generates a random 20 character hash value, and stores it into the user session (I'll get to why that's an array in a moment) and then returns this hash so it can be put into the hidden input.

<form action=”do_something.php” method=”POST”>
    <fieldset>
        <?=Auth::formRKey();?>
        <input ... />
    </fieldset>
</form>

So, the user session will now contain an array called “rkeys”, which contains a random 20 character hash value.

$_SESSION['rkeys'] array(1) => 
{
    [0] => '3748ab53cf129d536eca'
}

This has also added a hidden field to the HTML form which will look like this,

<input type="hidden" name="rkey" value="3748ab53cf129d536eca" />";

So when the user submits this form, the idea is that we take the rkey value and check it against the ones we've stored in the user session. If it's there, then the request is valid and came from a form which the site generated. If not, then it's a possible XSRF attack and the output should be stopped and logged. This validation is done in the validateRKey() method.

Now the final step, is to run something like the following code on any page that takes input. I actually have it in my bootstrap file which is run on every page request to the site.

if (count($_POST) > 0 && class_exists("Auth")) 
{ 
    if (!Auth::validateRKey(Sanitizer::sanitize($_POST['rkey']))) 
    { 
        Log::create("XSRF", "Possible XSRF attack", LOG::NOTIFY, LOG::NOTIFY_ADMIN);
        die("Detected a possible cross site request forgery attack. " 
           ."Perhaps you tried to refresh a POST request?");
        exit();
    } 
} 
// Clear any current rkeys 
Session::getInstance()->clearVar("rkeys"); // unset($_SESSION['rkeys']);

So if anything has been POSTed to the page load, and the Auth class exists (this is because it's part of the framework I use on a lot of sites.. not all of which require Auth stuff). I validate that the POSTed rkey is in the list of expected ones. If not then I log the error, kill the page load and present an error to the user. You could perhaps redirect there instead of just calling die(), and present something more friendly to the user, it's up to you. For my purposes, this suffices.

Importantly, you also need to clear the session of acceptable rkeys, since a request has now come in and been validated, so all keys are now expired. If you don't do this then it's pointless, since someone can just screenscrape the rkey and it would always be valid in the user session. Though you may now be thinking, what if the user doesn't make another request? The rkey will remain valid until they do, allowing someone to use the value. Well, that's because we can improve this method further by including an expiry time.

Making it a little bit better!

I lied a little bit when saying the above code is how I do it. It's how I used to do it before having the thought above regarding if the user doesn't make another request. So the above method can improved slightly by adding an expiry time to the rkey when we first generate and store it in the session. When the rkey is validated it can then be checked to see if it has expired or not. This gets around the issue of never expiring an rkey because the user didn't revisit the site and allow us to clear the keys manually. Only the following two functions need to change,

// This is the nonce used to stop CSRF/XSRF attacks. It's stored in the user session. 
public static function getNewRKey() 
{ 
    $rkey         = substr(sha1(uniqid(rand(), true)), 0, 20); 
    $rkeys        = Session::getInstance()->getVar("rkeys"); // $keys = $_SESSION['rkeys'];
    $rkeys[$rkey] = strtotime("+15 mins"); 
    Session::getInstance()->setVar("rkeys", $rkeys); // $_SESSION['rkeys'] = $rkeys;
    return $rkey; 
} 
    
// This will determine if an rkey is valid 
public static function validateRKey($rkey) 
{ 
    $rkeys = Session::getInstance()->getVar("rkeys"); // $rkeys = $_SESSION['rkeys']
    if ($rkeys == null) { return false; }
    
    // Check that the rkey exists, and time has not expired
    foreach ($rkeys as $key => $expires)
    {
        if ($key == $rkey
            && time() <= $expires)
        {
            return true;
        }
    } 
    return false; 
}

The "rkeys" array in the user session will now look like this instead, where the hash is the array key and the expiry time of that hash is the array value.

$_SESSION['rkeys'] array(1) => 
{
    ["3748ab53cf129d536eca"] => int(1275675864)
}

When a request comes in, we now not only validate that the rkey exists, but that it also hasn't expired. Restricting the validity time of the rkey reduces the likelyhood of an attack succeeding since it would have to be mounted quickly. Note that you should still clear all of the rkeys on every request once you've dealt with it though rather than relying entirely on the expiry time. If you know the key is done with then invalidate it instantly rather than waiting for a timeout.

Why an array of tokens?

Originally I coded this as just one rkey instead of an array, but that presented a problem. If I had more than one form on the page, only the final form would be valid, any other form would present the XSRF error, since a new rkey was generated each time a form was created and then only that new rkey was valid. An easy mistake to make, but something I noticed quite quickly when using lots of forms. Of course, you don't have to generate a fresh token for each form, you can just use one per user and apply that to every form on the page. But this is the way I've implemented it.

Why only POST?

You may be wondering why I'm only checking POST variables to prevent XSRF rather than both POST and GET. The answer is because GET should never be susceptible to such an attack if you're using it correctly.

So when should you use GET and when should you use POST? Well, it's all in the name. GET variables ideally should be used when the contents of the page are read-only, so nothing on the page gets changed by the request. So a GET request should be idempotent (I should be able to trigger the same GET request as many times as I want and it shouldn't affect the result I get. Basically, you want to make sure it doesn't have any side-effects). POST should be used whenever it causes a destructive action (I don't just mean deletes... I mean destructive as in something changes). So login, logout, updates, creation, deletion, comments, voting, etc.

This is why browsers will generally prompt you to confirm when resending a POST request, whereas they won't with a GET request. This is because a POST request will generally change the outcome each time it is run so you want to make sure you're not going to accidentally run it again and change something you didn't want to. A GET request shouldn't change any data that way, and so it doesn't need to be confirmed.

If you do want to use a GET for a destructive action, say you want it to be an anchor tag rather than a button otherwise your style will look strange, then you must make sure that you at least redirect to a confirmation page to confirm the YES/NO, ideally this confirmation page should use POST.

If not, then say you were to have a link in your admin area which deleted an item using GET, without any form of confirmation. Some browsers do what's called pre-fetching, where they examine the links on a page, and pre-fetch the websites you'd get by cliking these links, under the assumption that you will go to them. Then when you do click them, the page can be displayed very quickly. If you just delete with a GET, then simple visiting the admin page could cause the browser to follow all those links in the background and delete everything. Obviously not something you want. Yes, this happened to me in the past, so please learn from my mistakes.

POST Refresh

Another thing to keep in mind, is that you should redirect users to a GET based URL once the POST request has been handled. This means users will be able to hit F5 and refresh the result page without being prompted if they want to resubmit the POST request (since that will now cause an XSRF error to be displayed). This is common practice on most sites and something users now expect to be able to do. So they'll probably not be too happy if they have to re-send requests and then get errors saying they're XSRF attacking the site.

Do we Really Need to Keep Typing www?

xml-feeds@wblinks.com (Rich Adams) — Tue, 09 Feb 2010 13:23:00 -0800

I'll admit it, I'm the kind of programmer who'll spend 10 hours writing some code to do a job that would only have taken 2 hours to do anyway. This isn't because I'm stupid (well... maybe a little), but more because I just like writing programs and it'll usually teach me something new. I also have a notion that if I'm going to do something once, I'm probably going to have to do it again at some point, so the next time it'll only take a few seconds because I have a program to do it.

Even though I will happily spend hours writing software that I didn't really need to do, I don't like to waste my time on something that to me seems pointless. Even small insignificant things which in the long run probably have no real impact on my time anyway. I'm kinda strange like that.

One of these small insignificant things is typing out the "www." before a web address. Saving myself the milliseconds it would take to type that out is a big deal for me, because I don't see what the point of me typing it out is. It's redundant information and can simply be implied. Yet all the time I'm coming across sites which will not work if you miss off those four characters at the beginning. This ends up costing my time since I won't realise until the request times out, and then I need to type out the "www." anyway in order to get it work. There's no real excuse for a site to behave this way, it's just rude.

A story of woe

Just before the holidays I wanted to book a train ticket back home, so I opened up my browser and typed "thetrainline.co.uk", hit enter and expected the site to pop right up. But it didn't, instead it timed out. So figuring the site was down I typed "virgintrains.co.uk" instead, hit enter and the site popped right up. I then went ahead and bought my train tickets home.

Later on however, I wanted to check some train times, so I went back to try the trainline, and discovered that the site wasn't down at all, it's just it will only work if you type "www.thetrainline.co.uk".

I've no idea on the trainline's business model, but I would imagine they make at least some of their money from selling the actual train tickets. Because their site doesn't work without the "www." they lost a sale from me. Maybe I'm the only person in the entire world who types URLs this way, and so it would only be my sale that was lost, but one sale is still one sale. However, if a lot of people type their URLs the same way as me, then it could end up being quite a bit of cash that's being lost.

There's no reason for only one of the URLs to work. It's not like they're using the one without the www for something else. It just timed out. It's a tiny piece of site configuration, but if you leave it out then it can cost you visitors and sales.

Why do we use "www." anyway?

I did some searching around, but couldn't find a conclusive answer to this. The only remotely sensible reason I came across is that back in the early days of the world wide web, it was used to distinguish the web site of a domain from the FTP server or mail server (which would be ftp.example.com and mail.example.com, etc). So it seems to be nothing more than a convention that caught on.

One of my personal thoughts on the matter is that it was a way to tell people that what they were looking at was an address for the World Wide Web. In the early days of the internet if you'd seen something like "jurassicpark.com" on the bottom of a movie poster, it wouldn't have been obvious what it was. Is it the name of the production company?, something that got printed by accident? Adding the "www." to the beginning made it immediately obvious to everyone that this was something to do with the internet or World Wide Web.

Despite this mainstream use of "www." in domains, there is no technical reason why it's there. There's nothing in the HTTP specification that requires the use of "www.", it's simply a standard that caught on.You can host an FTP/mail/SSH/web/news server all on the same domain since these services all use different ports. If you need them to connect to different machines, then yes, maybe you would perhaps need a separate domain to use in the DNS records, but there's nothing stopping you just using example.com for a normal website.

To "www." or not to "www."?

So should you use "http://www.example.com" or "http://example.com" for you site? Well, the answer is both, at least from the users' point of view. Make one the actual home page for your site, and just make sure the other one redirects (using a 301 redirect) to the correct one. That way, no matter what the user types into their browser, they'll get to the site.

But then which one is going to be the home page and which one is going to redirect? Some people see this as a non-question saying that it doesn't matter. Well, that's not always the case. The choice you make for this can have big impacts depending on how your site is set up.

You could make the same page appear for both rather than redirecting, but this can have negative impacts on your search rankings. Search engines may think www.example.com and example.com are actually separate sites, so it will appear twice. Google Webmaster Tools will show you if that's the case and generally they'll just pick one, but it could cause problems with other search engines.

There are advantages and disadvantages of both ways and it all depends on what your site is really used for, or your personal preference. For me, I prefer to not use the "www.", but that's because I don't use cookies on this domain. At the end of the day, if you don't have any major reasons for using one over the other, then it doesn't really matter so long as you stay consistent.

What if the site uses cookies?

Without the "www." cookies will be served on all subdomains. This may be the behaviour you want, or it may not. So you would need to analyse this for your own site and decide the best approach. By including the "www." it means subdomains can have different cookies from the main site, whereas without it, all cookies set on the main domain will propagate to all the subdomains.

Why does this matter? Well, if you intend to serve static data then you'll need a cookieless subdomain. If you've gone for the option without the "www." then you won't be able to do this, and instead you'll need to use a completely separate domain name. So pick wisely.

Microreasons

Microblogging is big. Yes, I'm talking about Twitter. Everyone and their dog seems to use twitter, and people will often want to link to places. Having four less characters to type can mean the difference between some free publicity for your domain on twitter, or someone using a URL shortener because your URL is too long. Since most URLs are generally too long anyway, if you don't use the "www." you could even implement your own URL shortener. A little free publicity never hurt anyone.

Summary

It all boils down to this; a list of some top tips for web developers with regards to "www.",

Make your site work both including and excluding the "www.".
Since including the "www." is social convention, you should make sure that users are sent to the same place if they include it or not, they shouldn't be sent to a different site if they miss off the "www." as that will just confuse matters.
Pick one URL to use as the main destination, and redirect the other URL(s) to it. This way, all of the URLs to your site should stay consistent.
Use a 301 redirect as this tells search engines that it's a permanent redirect and that both URLs are actually for the same thing. This should preserve your rankings and keep things consistent.
If you want to use a cookieless subdomain for static data, then you need to use the "www." for your main site
If possible with your domain structure (some websites use subdomains for other things, and that's fine) also try to make "w.domain.com" and "ww.domain.com" point to the main site. This will catch anyone who makes a typo. It takes very little time to setup and can save someone even a few seconds of grief when they realise their typing mistake.

It's the little touches like this which could make the difference between a visitor, or someone who goes elsewhere. It's probably not going to lose you many customers, but every little helps in making people's lives less irritating.

If you follow the tips, then no matter whether someone bothers to type those extra four characters or not, they'll still be able to get to your site or order their train tickets home without hassle!

Creating a 'Database is Down' Page

xml-feeds@wblinks.com (Rich Adams) — Thu, 14 Jan 2010 22:47:00 -0800

Earlier today the database for this site was unavailable for around 30 minutes, I imagine something was being rebooted somewhere. This doesn't really concern me too much, since this is just a personal site. So during this time anyone visiting the site was sent to a holding page explaining that the database was down, and providing some links of places for people to visit instead. I make the choice to hide the error details from the user, rather than displaying a page with cryptic error messages on it, or even worse an error message which prints out some critical information. After all, the user probably doesn't really care why my database is down, they just care that they can't get to the information they want. So instead, I log the error details internally and just give a nice page to the user.

I received an email from someone who obviously tried to visit my site during this time, asking how this was done. Since I haven't updated in a while, I thought it's be good to give a breif overview of how it's done.

It's not very difficult and just involves a simple catch statement. If there's an error when attempting to the connect to the database then I log it and email myself the error, and then redirect the user to the holding page.

In the DBAccess class for wblog (the name I gave to the backend code that runs this site), the connect() method looks like this,

function connect()
{
    global $DB_CONFIG;
    
    if ($this->getHandle() != null) return; // Already connected

    $pdo_string = 'mysql:dbname='.$DB_CONFIG['database'].';host='.$DB_CONFIG['hostname'];
    
    try
    {
        Log::create("DB","Connecting to ".$pdo_string." as ".$DB_CONFIG['username'], Log::INFO);
        
        $this->_handle = new PDO($pdo_string, $DB_CONFIG['username'], $DB_CONFIG['password']);
        $this->getHandle()->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    }
    catch (PDOException $e)
    {
        Log::create("DB","Exception caught while connecting. ".$e->getMessage(), Log::ERROR, Log::NOTIFY_ADMIN);
        header("Location: /dbdown/");
    }
    Log::create("DB","Connected to database", Log::INFO);
}

The relevant bit of the code is this,

        $this->getHandle()->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    }
    catch (PDOException $e)
    {
        Log::create("DB","Exception caught while connecting. ".$e->getMessage(), Log::ERROR, Log::NOTIFY_ADMIN);
        header("Location: /dbdown/");
    }

I set the ATTR_ERRMODE attribute so that it will throw exceptions (See the PHP manual for details on setting attributes for PDO). I then catch any PDOException that's thrown. The "Log" class for wblog takes care of recording the error into a log file on the server, and the extra argument Log::NOTIFY_ADMIN tells it to drop me an email with a copy of the log (so that I get alerted to something going on). It's smart enough not to just email me the same thing everytime a user visits though, and has an increasing delay before it emails me the same thing again. (So first 5 minutes, then 10, 15, 20.. etc). You'll find people keep refreshing the site when they get an error page, so you'll just fill your inbox with emails if you're not careful.

Since you don't know what's in my Log class, you can just ignore those bits of the code. I'll probably make another post one day with the full code for this site anyway.

Once the error has been logged, I use the Location header to redirect the user to my holding page. Then I just make sure the holding page is static HTML file and doesn't touch the DB. It could also be PHP, but for my purposes a static HTML file is fine. As long as it doesn't touch the database, otherwise you're going to get into an infinite loop.

So there you have it, that's how I deal with issues when my database is down. If you have any better ways, or see something wrong with how I'm doing it, then let me know in the comments.

Taking the Security out of Security Questions

xml-feeds@wblinks.com (Rich Adams) — Sat, 12 Dec 2009 12:43:00 -0800

A security system is only as strong as the weakest point and this weakest point tends to be the bit where a human is involved (putting your password on a sticky note attached to your screen, for example). As a programmer, there are some things you just can't protect against, and human failure is one of them (Velociraptors are another). However, developers could at least try to make it a bit harder for people to break into other users' accounts.

I came across the following scenario a few weeks ago when attempting to sign up to a new system. Imagine the scene if you will, you've come up with a really secure password, you're happy that it would take someone a very long time to break such a password, and you haven't noted it down anywhere. Now the website you've signed up to is asking you to enter answers to some security questions in case you forget your password. Questions like "What was the name of your first pet?". You only have 4 different questions to choose from, and you have to pick at least two. It is mandatory to provide an answer.

This scenario is not as far fetched as you might first imagine, behold the following image!

I won't name and shame where this screenshot comes from, but it's from a product that's live, out there in the wild and worryingly also deals with money (and I should point out, not associated to anything I've worked on).

If you have any sort of sense at all, you'll see why this is a very bad idea.

All The Hard Work, For Nothing

I really don't know what goes through a developer's head when they come up with things like this. You've built yourself a great web application, you're using salted hashes for the password storage, and you're happy that your users have picked secure passwords.You've not forced some silly password restrictions on people, that actually make it easier to guess passwords. You're feeling pretty happy about it all.

But then you wonder about a certain use case. What happens when someone forgets their password? You've used salted hashes so you have no way to retrieve the original password, which means you'll need users to verify who they are somehow first.

Here is where you have some options,

If a user requests a password reset, generate a new password and send it to the email address on file.
Get the user to verify who they are by validating some information you have stored under their account. First line of address, day and month of birth, etc. Email them with a confirmation link so that you can verify they have access to the email account on file. That confirmation link then sends them to a page where the user can set a new password.
Security questions. Get them to fill in some security questions when creating their account, then we can just use that!

Don't bother with the first option, you should never send a password in plaintext over email. The second option is my preferred method, yet security questions seems to be the popular choice, and the one that I come across on lots of websites. If done right, then security questions can be a life saver when you've forgotten your pasword, but if done wrong, you might as well give out your password.

The Wrong Way

Simply, it's forcing a set of pre-defined questions on users. For example,

What is your mother's maiden name?
What is your mother's middle name?
What was the name of your first pet?
What was the name of your first school?

Now, think about this for a moment. You're going to allow anyone to "recover" their account and change their password if they answer these simple questions.

Not only have you reduced an attack on the site to a simple dictionary attack (since all those questions require dictionary word/sentence responses) but even worse, if you actually know someone, then you'll probably be able to answer those questions pretty easily. I'm pretty certain I could answer at least 3 of those for most of my friends. You'd also be surprised just how many people grew up with a dog called "Rex" or "Rover", etc.

Also, what if the user has never had a pet? Or their mother doesn't have a middle name? By forcing the questions users can pick from is not only going to irritate people but make it much easier for attackers to just randomly try and break the security questions rather than the password.

The Better Way

Once again, I'm going to stop myself from saying "The Right Way", since there's never going to be a foolproof way to recover someone's account if the password is lost. Every method of account recover has it's drawbacks. The very idea of having another method to access an account inherantly makes it less secure.

If you're going to go down the security question route, then a much better way is to let the user decide the question themselves. It allows the user to come up with something really cryptic, which only that person would know, yet no one else will. For example,

Q. 2X4B?
A. 523P

Unless you're a fan of Red Dwarf, you wouldn't get the reference that 2X4B is the middle part of Kryten's name, with the end being 523P. Ok, a simple Google search could probably find that out, so probably a bad example, but you get the idea.

Obviously this is still vulnerable to the "human element". A lot of people might choose very simple questions, not realising the implication that they're making it easier for others to get at their account. I've actually seen the following in a live environment before as a custom security question.

Q. My password is "iamawesome"?

Seriously... the mind boggles at what that particular user was thinking. So it may be worth adding a quick notice to remind users that silly questions like that will make their account insecure.

Don't Force Your Users To Create Security Questions

From a security point of view, allowing someone to recover their account if the password is lost is not a very good idea. The password is there for a reason, to secure the account. If it's lost, then no one should be able to get at the account. Account recovery procedures such as security questions were created purely because of customer service, not for security.

Users are only human, and humans forget passwords. It's that simple. Not wanting to lose a customer, you will want to provide a way for a user to recover their account and prove they are who they say they are. If security questions are implemented correctly, then the benefits to customer service can outweight the potential security implications of having two access points to an account.

However, part of customer satisfaction is not to force something on your users. I cannot stand it when I have to sign up for a website and fill in all sorts of information as mandatory, before I'm allowed to continue. By all means provide the user with the ability to create a custom security question, but don't make it mandatory. If a user doesn't want to provide another means to access their account, don't force it on them.

Forcing also has the side effect that most users will just think "Eugh, I just want to continue, I don't care about this" and they'll enter something stupid which is easily breakable. Putting "I don't care" as the answer for example (Yes, I've done this myself).

Adding Salt - Security Question Answers Are Passwords!

It's worth noting that answers to security questions are password equivalent and should be treated in exactly the same way as the main account password. I've seen systems which will store passwords as salted hashes, but then store the security answers in plaintext, completely negating the point of salting the password in the first place. Don't forget to store the security question answers as salted, strengthened hashes!

Combination Lock

The most effective method of account recovery I've come across is a combination approach. Ask the user to answer their custom security question, and then get them to verify their email address by sending a confirmation link they need to click. This means unless an attacker knows both the security question answer and has access to the email account, they won't be able to attack the account in this manner. At least not very easily. Once the user has clicked the confirmation link, they should be taken to a screen where they can just set a new password for their account.

Special Case

As with everything, there's always going to be at least one person who manages to lock themselves out of their account, with no access to an old email address and they can't remember their security question response. In these cases it's best not to attempt to provide the user with an automated method for account recovery. Another automated method is yet another point of entry for an account, and another place you need to secure.

In cases like this, it's often better to have a customer service email or phone number which the user can contact, and have a person resolve the issue once they're satisfied the user is who they say they are (by verifying other details on the account). In some cases that's just not going to be possible though, and their account will just have to remain inaccessible forever.

Get To The Point!

Basically my point is that if you're going to go down the security questions route, don't shoot yourself in the foot. I'm sick of signing up to sites that force me to choose from some security questions before I can proceed. It's the illusion of security, because it actually does nothing except annoy me, and make it easier for an attacker to gain access to my account since there are now two access points when I only want one.

If at all possible, don't bother implementing security questions at all. A forgotten password page should get the user to verify some other information in their account and then get them to confirm their email before allowing them to just chose a new password. This will always be better than security questions. But if you decide to implement them anyway, at the very least be sure to take note of the following,

Don't force a certain set of questions on the user. Let them chose their own question.
Don't force a user to fill in the questions as mandatory. If the user doesn't want to, they shouldn't have to. Of course, feel free to ridicule the user if they then forget their password, but it was at least the user's choice.
Hash the answer, salt the hash, and strengthen it before storing.
Don't rely purely on a security question response. Confirm the users email somehow and then send them straight to a page where they can set a new password. Avoid generating and sending a new password via email.
If the user doesn't have access to the email, or can't remember the question answer, don't use an automated process. Get another human involved to verify who the person is.

So there you have it. Bought on by my recent voyage into websites that force things on to me, let me know if you've had similar experiences, what methods you use for account recovery (if any) and whether you think anything I've said is wrong. If you have some novel method of account recovery, I'd love to hear about it!

Update (January, 2010): Someone sent me the following image, which are the security questions for a banking website. Another good example of how not to do it.

Only allowing four different questions is a very bad idea. For starters, what if you didn't have a high school mascot? Also most of those questions could be answered by anyone who was a close friend (or enemy). This is not the way to implement security questions, especially not for a bank.

Password Rules Don't Always Help

xml-feeds@wblinks.com (Rich Adams) — Tue, 03 Nov 2009 11:46:00 -0800

A while ago, I wrote about how users can't be trusted to come up with good passwords, and that it's up to us as programmers and web developers to hash the password (and salt it) so that it means bad password choices aren't immediately obvious to someone who gets hold of your stored data.

Of course, if people still use dictionary words, a simple brute force attack will work. So while some password tips such as "Don't use a dictionary word" are quite legitimate, there are plenty of rules and restrictions that do nothing but infuriate users and make passwords less secure.

There is absolutely no need to enforce certain password rules which seems to be forced on people throughout the corporate world and on many websites. Here are a few which I find the most annoying, supposedly implemented to make people use passwords which are more “secure”, but in reality do just the opposite. Passwords become more predictable and expert users who create complex passwords get infuriated when they're force to make them less complex in order to fit with the restrictions. I wonder how many of you have come across these before.

The Rotating Password

"You must change your password every x days"

Usually it's every 30 days, so that each month you have to have a new password. There are then crazy rules, like you can't use a password you've used the last 12 times. Sure, if someone discovers a user's password, and waits a month to use it, it will no longer work.. but that's pretty much the only case this protects against. But it does a lot more harm.

If you force a user to change their password every 30 days, there's very little chance they are going to come up with a truly unique password each time. Instead, they'll tend to keep the same basic password and just add something extra in. You'd be shocked at just how many people use the name or number of the month as that extra thing.

Since we now know a likely part of the password, it can make it a bit easier to crack with brute force attacks. It could even mean if you do discover a user's password, and it has a number or name of the month.. you'll always know that users password for any month, making this measure completely pointless.

To add further annoyance to this, a lot of system adminitrators feel it's necessary to remind users that they'll soon need to change their password. So 14 days before your password expires in Windows for example, you'll get a popup saying "Your password expires in 14 days, do you want to change it now?" This has to be the most useless dialog I've ever come across. If my password expires in 14 days, why would I want to change it sooner? It makes no sense. Just tell me when it's expired and I'll change it. Making me select Yes or No every single time I login just irritates me.

The Ridiculous Restriction

Your password must be at least 6-8 characters long, contain uppercase and lowercase letters, and have at least two digits, but not for the first two characters. Oh, and it shouldn't contain the name of a species of dog.

How many websites or applications have you come across with ridiculous password restrictions like this? I see it all the time. Well.. maybe not the dog one. Yet it doesn't actually do anything to make the password more secure! Forcing users to use a certain pattern, means that an attacker also knows the expected pattern, reducing their search space from "anything in the world" to "well, now I know there's at least two digits and will be mixed case".

I've also seen things like "You must choose a password of the form XxxxxxN" where X is an uppercase letter, x is a lowercase letter and N is a digit. Now, that has to be one of the worst. You've given an attacker everything they need to crack the passwords in no time at all using a simple brute force approach.

Not only does it make attacks easier, but it's more frustrating for users. As I said in the previous post, most users will use the same password for everything. It's not their fault, it's just easier to remember one password than one hundred. If you suddenly force a policy that means users can't use that password, they'll have to come up with a new one, and they'll inevitably forget it, frustrating the user even more the next time they login. Some would argue this is a good thing, but I think if a user has a sufficiently complex password, they should be able to use it everywhere if they really want to (although if they really value security, they'll use a different one everywhere, that's a choice the user should make).

So you're 0 for 2. You've made it easier for attackers and pissed off your users, well done!

The Giveaway

"You cannot use that password because another user already has it"

Yes, I have actually seen this in a live environment before. If you have someone on your team who insists on using this method, take them into a room and slap them. This is a pretty big mistake since you've given away another users password. Usernames are much easier to find out that passwords, especially if it's a public website.

I'm really trying hard to imagine what was going through the designer/programmers head when they thought of using this technique. There is no need for a system to enforce unique passwords between users.. none at all! If your database model uses a password as a primary key, you have failed as a developer.

The Normalizer

"Your password cannot contain special characters, only normal letters and numbers"

Why do you care what I use in my password? As long as it's a character understood by the computer, and as long as it will be hashed the same way each time, you shouldn't care what people use. Restricting the use of special characters is on par with dictating the exact format. It's usually done because of some limitation to the system, that you can't store those characters in the database, it will mess up their admin area, or the website doesn't support multibyte characters, etc. Which means either you're not sanitizing your inputs properly, aren't using prepared statemens and probably don't hash your passwords. Alarm bells should be ringing.

If I want to use a percentage sign or a Japanese character in my password, I should be able to do so. It will mean my password is harder for anyone else to guess. I cannot stand it when I come up with a really complex password like "l-.(mR{n<Pg1$Yh=u'C?^Wp_$ANNn'4gE+?@VE" and am told that my password can only contain letters and numbers. You're forcing me to choose a password that is much less secure than one I've chosen myself. That is unacceptable.

The Length Restriction

Your password cannot be longer than 16 chars.

Why not? The most common excuse for something like this, is that the database can only store 16 chars in the field, or that a designer only wants the input box to be a certain size.

Well, the designers argument is nonsense, since they can size the box however they want without restricting the amount of information that can be input. The database excuse is also nonsense, since you should be hashing passwords, so that they'll always be the same length after hashing anyway. So the password itself can be any length you want, yet you only have to store the 40 character hash output (or however many characters you force the hash to be).

A limit can be enforced, but it should be at least 100 characters at the minimum.

If I want to type a novel as my password, then that should be my choice. Restricting password length just restricts the search space an attacker needs to use and infuriates users who actually pick long complicated passwords by making them pick something less secure.

So what are the lessons?

Enforce some complexity if you have to, but don't restrict complexity. I should make it clear that I'm not against all password restrictions. It can depend on the environment. If you're storing the password for a system which has personal data, then enforcing a certain amount of complexity in the password is probably a good idea. I guess the point I'm trying to make though, is that it's easy to go too far. There's a point when it just becomes a hinderance to users, and ends up creating patterns which make the system less secure.

Try not to restrict user password choice. By heavily restricting what a user can enter as a password you're simply annoying your users and making it easier for an attacker to guess or brute force the password. By enforcing too many password restrictions you are making the system more predictable, and when it comes to passwords (and cryptography in general), predictability is a very bad thing.

It comes down to some very simple points.

Don't resitrct what can go into a password, whether it's a max length or strange characters.
Don't force a users password to follow a certain pattern.
Don't force a user to change their password every x days unless you really have to.
Store your passwords securely (strengthened, salted hashes).

If you follow these simple rules, your users will be happier, your database will always store the same length of data no matter what the user enters and you can be safe in the knowledge that you're not giving away any information to make it easier for an attacker.

Update (December, 2009): The best method I've seen recently is simply a blacklist of common passwords and dictionary words (this is the method Twitter use apparently). Users can pick a password of any length, with any characters they want and don't have to change it at any interval. They just can't use a common password or dictionary word found in the blacklist. Simple, effective and only reduces the search space by the common dictionary words which would've been easy to crack using brute force anyway.

text-transform - Content or Presentation?

xml-feeds@wblinks.com (Rich Adams) — Sat, 15 Aug 2009 07:04:00 -0700

HTML and CSS are all about separating the content of a site, from the presentation. As with most things though, there are grey areas, and for a lot of people text-transform is one of them. Some people regard changing the case of text as being a content issue, others see it as a presentation issue.

Personally, I prefer to think of it as a presentation issue for one very good reason; to cover as many scenarios as possible. Suppose you have some mark-up, where the text needs to be in uppercase, because that's how you want a menu to look. You could write the HTML like this,

<ul>
	<li>SOME OPTION</li>
	<li>ANOTHER OPTION</li>
</ul>

This is all well and good, and is perfectly reasonable. Now suppose though that the design needs to change, and these need to be in lowercase instead. Not a problem, since you can use text-transform to convert them to lowercase.

ul li {
	text-transform: lowercase;
}

But now what if it needs to be capitalized instead? You can just use text-transform: capitalize; right? Unfortunately not. A little known part of "text-transform: capitalize" is that is only changes the first letter of each word to be uppercase. It doesn't touch any other letters. This isn't a bug, it's specifically designed this way. But it could give you problems if you haven't thought ahead, since most people expect it to also transform the other letters to lowercase.

In the above example, you would never be able to capitalize the words. Yes, you could start with lowercase in the HTML rather than uppercase. Then you can use text-transform to make it uppercase, and capitalized. Except it would still be all in lowercase if someone were to view the site without a stylesheet.

My suggestion then, is to capitalize things like this in the HTML. So taking my above example again, I would write it like this in the HTML,

<ul>
	<li>Some Option</li>
	<li>Another Option</li>
</ul>

This way you can have full control over the case by using CSS. You can either leave it as it is, make it uppercase, or lowercase. It also means it will be properly capitalized if someone were to view the site without a stylesheet. This way you're not dictating the style in the content.

As with anything though, there are of course exceptions. If a product or trademark name is in all uppercase, or uses different cases within a word, then this would actually be considered the content rather than the presentation. An example of this would be something like "JavaScript", where the J and S should always be uppercase. In those kinds of scenarios, I wouldn't imagine that the case is ever going to change when the design changes, since it's a trademark, brand name etc.

If you need another option, like every other character uppercase or something, then you'll need to do something server-side to change it. CSS2.1 is good, but not that good. You could come up with a JavaScript solution, but JavaScript isn't always the answer.

It's all a matter of analysing the situation and deciding the best way to do it. Could the design change in future, and if so, would it be possible to have the control needed using just CSS? A good rule of thumb is to think the following, "If I view the page without a stylesheet, does the text still accurately reflect what I need it to?". But remember not to be caught out by how text-transform: capitalize; works.

JavaScript is Good, But Should Not be Relied Upon

xml-feeds@wblinks.com (Rich Adams) — Sun, 19 Jul 2009 09:09:00 -0700

There was a time, years ago, when the only reason to use JavaScript on a website was to produce cliché effects; flashing, scrolling, fading and popups to name but a few. It was slow, clunky and not a very nice language to write code in. Browsers required different code to do the same thing as another browser, the whole thing was a mess. People would overuse sites like dynamic drive to achieve all sorts of pointless effects, falling snow, page transition effects, and who can forget, the disabling of right mouse clicking by making a popup appear, which was about as effective as putting up a sign which says "Please don't push this button".

Recently however, JavaScript has lost it's status as as annoyance and has become common place on lots of main stream websites. It's picked up a certain bit of elegance and if used correctly, can add a lot to the experience of a site. Browsers can now parse JavaScript at speeds which make it viable to use for visual effects, and it can be used to trigger events and to change parts of the page dynamically without having to refresh the entire page (AJAX for example).

With libraries like jQuery and Prototype it has become even easier, as they add a new layer between the browser and the programmer, meaning you don't need to know about all the little inconsistencies between the browser implementations of JavaScript. The library will hide these from you, so you can concentrate on writing the code and let the library deal with getting it to behave the same in every browser.

A combination of browser support, libraries, speed and ease of use means JavaScript is now much more attractive than it was 5 or 10 years ago. It no longer has the stigma associated with 1990s sites and has become a much more civilised solution to web development. All the big sites use it, Google has auto-completion when you type, Digg uses it to show extra comments, etc.

There's no doubt that JavaScript is incredibly useful and a great way to make the interactive experience of a website seamless. But you should never rely on JavaScript for a part of your website to function.

Everyone Uses JavaScript

Everyone has JavaScript enabled right? Well I'm afraid not. If you've made any part of your website only accessible to those with JavaScript enabled, then you've cut out 6% of the people who use the internet (June 2009 figures from thecounter). That's a lot of people, no matter how you look at it. Yes, you can just put a disclaimer on the site saying "This only works with JavaScript". But that isn't going to make it any less annoying for those 6% of users.

You may be asking yourself, "Why would anyone have JavaScript disabled in this day and age?". Well, for any number of reasons. Companies sometimes disable JavaScript for security reasons, people may disable it because it makes their browser experience faster if they leave it off, people could be using a text browser, or a mobile phone browser that doesn't support JavaScript.

If a visitor comes to your site and doesn't have JavaScript enabled, yet your site heavily relies on JavaScript to function, then they'll leave immediately. Yet it is so simple to make things work without JavaScript.

Progressive Enhancement vs Graceful Degredation

There are two main programming philosophies when dealing with a new technology. Progressive Enhancement and Graceful Degredation. The latter means that you build the site using the new technology first. Make sure it works in all the latest browsers and that all of your fancy code runs well. Then you go back and add bits to make sure that if someone views it in an older browser, it will still work fine. Progressive Enhancement is the other way around. You make the main functionality of the site work without the new technology, then add it in afterwards so that it works for everyone.

Think of it like this. If you're building a skyscraper, how would you do it? You have two options.

Build the foundations, making sure it's a solid base. Then add each floor one by one.
Build all of the floors right to the top, then go back and make sure the base is strong enough for people to walk around on the bottom floor.

It would seem number 1 is the best choice, but for me, it depends on the scenario. For CSS I generally use Graceful Degredation, not from any pre-thought out plan, it's just how I happen to do it. When I write a new style, I will be developing on just one browser (generally Firefox/Linux), once I am happy with the style, I will validate it with the W3C validator service. I will have generally made a few typos here and there. Once the CSS is valid, I'll check it out in other browsers, usually the latest versions of Opera, Chrome, IE and Firefox. It will generally look the same in all of those, with just perhaps a few minor tweaks here and there. Then I will try it out in the older versions, IE7, Firefox 2, etc. Based on how many inconsistencies there are, I may just use a quick and nasty CSS hack to target just those browsers, or give a new CSS file entirely (in the case of IE6/7). This is graceful degredation at work. I know it works in the later stuff, now I'm making it work in the older stuff without breaking it in the newer stuff.

The problem with this method, is that sometimes you'll edit something to work in an older browser, but then find you've broken it in the newer browsers, so you start going back and forth with yourself trying to get it to work in both.

Progressive Enhancement is the other way around. You first make it work without any "extra stuff", then you add on the extra bits afterwards, for those who have the technology to view the extra bits. It means current people can still use the site, but those who are ahead in technology make use of the latest stuff. This is my preferred method, because it means you won't leave anything out, and will slowly build your site up from a base. It also means you generally aren't going to break something you've already coded, which saves time in the long run.

This is how you should add JavaScript to a site. First make the site work without JavaScript at all. Disable it in your browser and make the site function. Then go ahead and add all of the JavaScript goodness you want, safe in the knowledge that those 6% of people will still be able to use the site. Yeah, so it won't have lovely fade effects, or quick AJAX, but it will still work.

Progressive Enhancement also has an added bonus. It's actually generally a quicker and easier method than Graceful Degredation. It should be a no brainer!

You're a Hypocrite!

I recently (well, end of last year) made the decision that I will no longer build website for IE6. It is my strong opinion that IE6 is holding back web development. It takes nearly double the amount of time to make a site work in IE6 as well as all the other browsers out there. To quote a bit from Falling Down, this isn't economically viable! Yes OK, a lot of people still use IE6 (around 9%), but it won't go away unless we make a stand.

Unfortunately, it isn't going to go anywhere soon, because it was around for so long without any sort of competition, which means lots and lots of companies used it to build internal tools and sites, which now will only work in IE6. They would need to spend lots of money to upgrade their internal things to work in more modern browsers, so they will simply stick with what they have. Why spend money, when it works as it is. There is also the fact that "Internet Explorer" was very cleverly named to have the word "Internet" in it. You'll find the majority of non technical people will not understand what a browser is, and simply think that "Internet Explorer" is in fact "The Internet".

You could say I'm being quite hypocritical. Here I am trying to argue that we should make the time to make our sites work for 6% of people who don't have JavaScript enabled, yet I'm advocating cutting off the 9% of IE6 users out there for exactly the same reasons.

Well, it's not that simple. There are some fundamental differences,

To make a site work without JavaScript takes very little effort. Compared to the effort required to make a site work in IE6, which normally takes double the time as you have to build two sites. One to work in IE6, and one to work in everything else.
There is no excuse for someone to be using IE6 exclusively. It's heavily outdated, and I'm betting most people's kettle's are actually newer than IE6. JavaScript on the other hand has legitimate reasons to be turned off. Security, speed, text browser, mobile browser, accessibility.
Making your site work in IE6 only benefits people who use IE6. Making your site work without JavaScript has several other advantages. If you accidentally introduce a JavaScript bug or the JavaScript fails to load correctly, your site is resilient and won't be crippled, as it will still work without JavaScript. Search Engines generally can't follow JavaScript links, so making them work without JavaScript means search engines will be able to index your site better, etc
As new browsers come out, they may introduce inconsistencies in the way JavaScript works. Being able to fallback to a non-JavaScript solution means you don't have to immediately spend time and effort getting it to work in new browsers. Once you've made the site work in IE6 and put the effort in, that's it, it's never going to be useful for anything else.

Maybe you still think I'm a hypocrite, and that's fine. Let me know your thoughts in the comments. I'm curious to know where people stand on this subject.

Stop Waffling and Get to The Good Stuff

OK, so now to get to the code. Let's imagine you have a shopping website that relies entirely on JavaScript. Your links might look something like this,

<a href="javascript:addToBasket()">Buy Item</a>

That's great, it works for you and adds the item to your shopping cart. Now turn JavaScript off, and you're not going anywhere. There's a link on the page which doesn't work. A user is furiously clicking it, but it won't add anything to their cart. They go to another site instead, and you just lost a sale. Good luck if anyone using a text based browser wants to buy anything, as it won't work for them either.

(This has actually happened to me in the past, not because I've had JavaScript disabled, but because the code had a typo in it and wasn't loading correctly, so I couldn't purchase the item and went somewhere else instead).

If you build a website, you should make it work for as many people as you possibly can. (Yes, even IE6 if you have the time and the money). Imagine buying a car, only to find you can only use it on 94% of the roads. Great, until you hit one of those 6% of roads.

Here's a better way to make the link. This follows the Graceful Degredation process I talked about earlier. We had our original link, which works in JavaScript, and now we're editing it to work without JavaScript too.

<a href="buyitem.php?item=awesomestuff" onlick="addToBasket()">Buy Item</a>

(We'll ignore for the moment it's a bad idea to use a GET parameter for something like this, that's a PHP issue and not in the scope of this post. I'll write about it sometime in the future).

If you make the "addToBasket" function return false at the end, then if JavaScript is enabled, the link will not be followed and only the JavaScript will execute. But if JavaScript is turned off, then the link is followed, and the user can still add things to their basket, (with a script you've written, but for this example, I'm assuming the JavaScript would have called the same PHP script with AJAX anyway, it just will now redirect back to the same page). Everyone is happy and everyone can buy things.

But there's a better way.

The Better Way

Now there's another issue. If I don't have JavaScript enabled, I'm still downloading the "onclick='addToBasket()'" but of the code. It's not much, but over a whole page it could be a lot. It also makes the markup look untidy. Not a big issue, but I'm OCD about that sort of stuff. Also as we saw earlier, there's a better way to do things. Lay the foundations of your skyscraper first.

If you've never heard the term "unobtrusive JavaScript" before, I suggest you do a quick Google search to see what it's all about. Basically it's called unobtrusive because the markup stays the same and has no JavaScript calls in it at all. It is pure HTML, as it was meant to be. HTML for markup, CSS for presentation and JavaScript for behaviour. So rather than littering your HTML with "onclick=''", etc. It is nice and clean.

Since JavaScript only needs to work when JavaScript is enabled, you can use the JavaScript itself to modify the DOM once the page has loaded, in order to add the "onclick" event handlers for you. But this also means there's no chance of someone clicking a JavaScript link before the page has fully loaded, which can sometimes cause problems, since the JavaScript version of the link won't be active until the page has fully loaded. jQuery makes this sort of stuff extremely easy.

On page load, you modify your links to add the JavaScript handlers. If a person has JavaScript disabled, they don't even need to waste the bandwidth of downloading the JavaScript, and the HTML will be nice and clean making the page load faster for them. But if they do have JavaScript enabled, the site will be modified to work as a JavaScript version.

Suppose we now create the link like this,

<a href="buyitem.php?item=awesomestuff" id="buyitem-awesomestuff" class="buyitem">Buy This</a>

This is nice and clean, has an ID to unique identify it, has a CSS class so we can style it appropriately, and is free of JavaScript. This will work for everyone. Now we can add a JavaScript file to the page and "AJAXify" this link without ever needing to touch the HTML again. So an example using jQuery would be something like this,

$(document).ready(function()
{
    $("a.buyitem").click(
        function()
        {
            var url = "buyitem.php?item=" + $(this).attr("id").substring(8);
            $.get(url,
                  "",
                  function(val) { alert("Item added to cart"); },
                  "text"
            );    
            
            return false;  // So we don't follow the normal link.  
        }
    );
});

When the page loads, this will go over all links with the "buyitem" class, pick out the item string from the id, and add an "onclick" event handler to submit the AJAX request. We haven't touched the HTML at all, but by just adding JavaScript we've made it work with AJAX for those with JavaScript, and we're safe in the knowledge it will still work for those without.

Note : The example is just to demonstrate unobtrusive JavaScript. You will want to have a much better callback than the one above. The callback should check that the item was actually added, rather than just show an alert either way as the example above does. You probably want to parse the id to get everything after the "-" character, rather than just substring(8), since the id may change in future, etc.

Final Thoughts

Every so often I come across forums, or answers on the internet, which give JavaScript solutions to common problems (which is perfectly fine), but then they say something like "This will only work with JS enabled, but everyone has it on nowadays, so screw those who don't, they probably won't want to look at your site anyway". This is just completely the wrong attitude. With a little bit of extra work you can make it work both ways and have a site that works for 100% of users, not just 94%. People argue that because it's 2009, everyone should have JavaScript enabled. While perhaps they should it doesn't necessarily mean that they have to. There are legitimate reasons to have it disabled, and those people shouldn't be excluded.

Another common issue people tend to solve with JavaScript is input validation. If your input validation is just a JavaScript file then you have failed as a web developer. All I need to do to bypass your checks is to disable JavaScript. It's the equivalent of using JavaScript for login/password (something I have actually seen on a production site before). By all means use JavaScript for input validation, it means the user can see something is wrong without having to submit the form, a great interface response. But don't rely on it. Always always always make sure you also have server-side validation for the same things, otherwise you're just asking for a world of trouble. It may feel like you're just repeating code, but it's worth it in the long run.

So I guess my point is this. Never rely on JavaScript to do a job. Form validation, prevent dual submits, links, etc. Always make sure they also work without JavaScript, otherwise you've either opened your site up for abuse, or made it unusable to 6% of the people who use the internet. Either way it's not very good.

Storing Passwords - The Wrong, Better and Even Better Way

xml-feeds@wblinks.com (Rich Adams) — Sun, 21 Jun 2009 10:19:00 -0700

If you've ever had to sign up to use a website, you'll no doubt have been prompted to provide a username and password, so that when you next visit the site you can login without having to fill in all of your details again. Your password has to be stored somewhere, otherwise you won't be able to login the next time you visit. Right? Unfortunately, a few sites I've come across do just this. They store your password, which means if the information is stolen, someone has got your password.

In a perfect world, everyone would use a different password for every account they sign up for, and that password would be a combination of numbers, letters (uppercase and lowercase) and special characters, and would be at least about 20 characters long. But let's be honest here, we're not all memory machines and remembering cryptic combinations like that isn't something everyone can do. So people are tempted to choose just one password and use it on everything they sign up for, including their email account. Which means there's the potential for someone to have the email address and the password. Not a very good thing.

There's no way around this, the weakest point in any security system is the human element. People are always going to chose easy passwords, or use the same passwords for multiple sites. No amount of security can make up for users picking a bad password to begin with, but we can still protect those use the same password for everything. So it's up to us as web developers to help to keep their passwords secure so that these people never have to go through the problems associated with someone getting into their other accounts.

This all comes down to how you store the password. Do you do it the “Wrong Way”, the “Better Way” or the “Even Better Way”? (I'm not going to say the “Right Way”, because I don't think there is such a thing when it comes to password security).

The Wrong Way

I was shocked to recently discover a website which stored their passwords in plain text. I couldn't quite grasp how in modern web development anyone could store their passwords like that, but it still happens. Storing passwords as plain text is a very bad thing. It means that if I sign up for a website using the password of “iamawesome”, then the value that's stored in their database is also “iamawesome”. This means any employees with access to the database (and there will always be at least one), can see your password. They can also probably see your email address associated to the account. So if you're one of those people that uses the same password, there's nothing to stop that person going and getting into your accounts. I like to think the majority of people out there in this position are honest and would never use that information for such nefarious means, but by the same token, I'm willing to bet there are people out there who aren't so honest.

So how can you tell if a website stores your passwords securely? The simple answer is that you can't. But there are a few tell tale signs that they're not doing all they can. Many websites have a “Forgotten password” function. If you go through this procedure and you get an email with your original password in, then it is highly likely they're storing you password in plain text. (Although this isn't always the case, they could be using reversible encryption, but again.. the developers will now how to reverse the encryption, so while it may stop anyone who steals the database, it wouldn't stop dishonest employees).

Personally, I would stop using such a service straight away (and in most cases I will email the relevant department to warn them that they're storing things insecurely). There is no need for a website to store that information in plain text. None at all. It puts your information at risk. There's a much better way to store passwords.

The Better Way

In order to login to a website, there's no need for the site to actually know what that user's password is, they just need to be able to tell if you've entered the same password as when you registered.

You can use something called a “hash” to do this. A hash is a one-way mathematical function, which given an input A, will always produce the same output B. But ideally it's very difficult to get from B back to A (but not impossible, you could use a rainbow table to lookup A based on the hash B).

Note : It is possible for more than one input value to result in the same hash. This is called hash collision. It's pretty unlikely to get hash collision unless you're using very large datasets, but it is a possibility. I've never come across it myself, although it does happen. The danger with hash collision is that someone doesn't need to know the original password, if they can find something which hashes to the same result, then they can just use that.

Different algorithms will hash things in different ways, and some are better than others. MD5 is a common one used in lots of tutorials but it has serious issues with collisions and widely considered to be cryptographically broken. SHA1 also has similiar issues with hash collision. The bottom line is you shouldn't use MD5 or SHA1. Others, such as the SHA-2 family of algorithms, currently have no known collision issues. I'm going to use SHA-256 in these examples, but you could use any other number of hashing algorithms. There is still a problem with using these algorithms for passwords (they're to fast), but I'll get to that later.

PHP 5 has the built in hash function, which I will use to hash with SHA-256,

$password_hash = hash("sha256", "iamawesome");
// 4aa4029d0d0265c566c934de4f5e0a36496c59c54b6df8a72d9c52bdf0c1a0e8

The idea behind using a hash, is that you store this hash in your database, rather than the plain text password. In order to verify a user has entered the correct password when logging in, you use the same hash function on whatever text they enter, then compare the result to what you have stored in the database. If they match, then the correct password was entered.

$user_entered = hash("sha256", $_POST['password']);
return ($user_entered == $password_from_db);

This way, only the user ever knows the real password. If someone were to look at the database of your stored hashes (whether it's a dishonest employee, or because it was stolen) they'll only ever be able to see the hash, and won't be able to go around getting into people's email accounts.

So you might be thinking that it's problem solved, let's just do this hashing stuff and we'll be secure. Well no. There are still some problems with this method. Suppose two people have the same password, this means they will have the same hash in the database. Now suppose you manage to trick one of these people into giving you their real password (via however, email scam, etc). This means you would now have the password for anyone with the same hash.

One of the most basic password attacks is called a dictionary attack. At the most basic level this would involve trying every word in the English dictionary as the password to login as someone, but more commonly the dictionary (a list of strings, not a language dictionary) contains a list of every combination of characters up to a certain length. This will generally work since it's quite common for people to just use a dictionary word or a common sequence of characters as their password. A dictionary attack is still possible when using hashes. You can generate a list of hashes from a dictionary and compare it to the hashes in the database, you'll probably find a few matches, and then you have the password for those people. The list of hashes is called a rainbow table, basically a massive lookup table of hashes and the corresponding input.

So really all we've done is obfuscated the password from people viewing it directly, but certain attacks are still possible. There is still a better way.

The Even Better Way

The way around these problems is to use something called a “salted hash”. The definition of a salt is “random bits that are used as an input to a key derivation function”, basically just another word for a nonce. Normally to create your hash you provide one thing as input (the original password) and you get the hash as an output. A salt/nonce is a random string of characters you use as another input into the hash function in order to get the output.

So now when storing a users password, instead of just hashing the password, you concatenate the password and the salt/nonce and hash that instead. So in PHP it would look something like this,

$salted_hash = hash($password . $random_salt);

The salt should be a string of random characters, ideally it should be long (more than 20 characters) and not just alphanumeric, it should have special characters too.

You can make your function however you want, as long as the random salt and password are both used in order to construct the hash you want to store. The method you use doesn't change the effectiveness of your password storage, one is not really any more secure than the other. Relying on the design of how you hash the password and salt together to provide security is called security through obscurity and should be avoided, since in reality the method you use doesn't affect the security.

$salted_hash = hash(hash($password) . $random_salt);
$salted_hash = hash(hash($password) . hash($random_salt));
// ...etc

In the database, it's also important to store the salt for each user as well as the completed salted hash, otherwise you won't be able to tell if the user has entered the correct password. Each user should have their own random salt, you shouldn't use the same one for the entire database or for multiple users, otherwise you've completely negated the point of using a salt in the first place. You can however use a site-wide salt in addition to a per-user salt if you want (this is sometimes called a pepper), the idea being if your database is compromised then an attacker will only have the user salt, not the application-wide one.

You're database table should look something like this,

username         password_hash          password_salt
rich             2bae773debd80de...     ?hb-:4a-loDC90^n#=R...
bob              d82ff2c12d5065f...     2g}iT'JG><,?wP6{#VG...

So now, this means the hash you store will be different for every user in your database, even if they have the same password. So if anyone wants to do a dictionary attack by precomputing a rainbow table, they have to precompute it for each user individually, rather than the entire database at once, making it much more difficult and often infeasible.

(Note: You can never make it impossible to crack someone's password, there will always be a way. You can just make it very very difficult)

But now you have another problem, how do you implement the "I've forgotten my password" functionality if you can't tell the user their password? Rather then just retrieving the user's original password (which you can't do when using salted hashes), you instead want to verify the user by getting them to confirm some other information on their account, then send them a email with a confirmation URL/number in it. When they click this URL (or enter the number) they should then be taken to a page where they can set a new password for their account. You can now feel safe that the person setting the new password knows information about the account, and has access to the email address. Sometimes a site will just email you a newly generated password, but I don't like this method, since the first thing I'll do once someone sends me a password is to change it, so letting me change it first just makes things quicker for me. Most mainstream websites nowadays will do this, and this is usually a good sign that they're storing your password properly.

Slow It Down

But wait, there's still more to do! While in most algorithms you aim to make things as fast as possible, in password hashing algorithms you want the opposite. The traditional hashing algorithms like MD5, SHA-256, SHA-512, etc. all have a serious problem when it comes to password storage, they're too fast. Now that you have salted passwords stored, what if someone were to try and crack those passwords?

While a rainbow table is a precomputed table of all the hashes corresponding to a set of plaintext passwords, another method of cracking involves feeding every possible combination of characters into your hashing algorithm rather than precomputing the results. These are called incremental password crackers. Basically rather than using space to attack the passwords (a massive rainbow table lookup), they use time (try a dictionary on your hashing function until a result is found). With services like Amazon EC2, you can have massive amounts of computing power for very little cost able do all the hard work for you.

If your password hashing function is very fast, then the incremental method will work faster and the password can be cracked quickly. If it takes 0.00001 seconds for your hash function to return, someone can try 100,000 passwords a second until they find the password. If it takes 1 second for your hash function to spit out the result, it's not a big deal as far as someone logging into your application is concerned, but for cracking the password it's a very big deal since each attempt will now take 1 second to get a result, meaning it would take 100,000 times as long to find the password as it would using your original hash function.

So how do you slow it down? Either use a hashing algorithm specifically designed to be slow (like bcrypt), or use a standard hash function lots of times. This is called key strengthening (or sometimes key stretching), and is just the idea of running the hash function through thousands of iterations.

So now your password hashing method becomes this,

$iterations = 100000;
$salted_hash = hash($password . $random_salt);
for ($i = 0; $i < $iterations; $i++)
{
    $salted_hash = hash($salted_hash);
    // Or $salted_hash = hash($salted_hash . $random_salt); .. etc
}

You should also store the number of iterations in your database somewhere, since you will want to update the number of iterations in future if you get faster hardware. In that case, not storing it would mean older users would be unable to login. Your database table should now look like this,

username      password_hash          password_salt           hash_iterations
rich          2bae773debd80de...     ?hb-:4a-loDC90^n#=R...  100000
bob           d82ff2c12d5065f...     2g}iT'JG><,?wP6{#VG...  150000

Some people like to store these all in one field, with some pre-defined separator. That's fine too, it doesn't really make much difference other than how you write the code to extract those values. Just make sure the separator you use can't appear in the hash or salt, or you'll run into issues when extracting the values.

username      password
rich          $100000$2bae773debd80de...$?hb-:4a-loDC90^n#=R...$
bob           $150000$d82ff2c12d5065f...$2g}iT'JG><,?wP6{#VG...$

You can also take this a step further to future-proof it by including an identifier for the algorithm you're using. If you're using sha256 to hash passwords right now, in a few years you may want to use something better (there could be a sha1024 for example), if you include something that tells you with algorithm was used to hash, then you'll be able to use newer or more appropriate algorithms without having to change your code other than to add the new algorithm (you'll still be able to check passwords that were hashed using the older method, etc). Supposing sha256 had an identifier of 5 (which it does in crypt), then your database would now look like this,

username      password
rich          $5$100000$2bae773debd80de...$?hb-:4a-loDC90^n#=R...$
bob           $5$150000$d82ff2c12d5065f...$2g}iT'JG><,?wP6{#VG...$

Hashed, salted, strengthened and future-proofed. If a better hashing algorithm becomes available, you can implement it without affecting current users. Likewise, if you get new hardware, you can increase the number of iterations without affecting current users. If you have dishonest employees, they won't be able to get the passwords since they're hashed. If someone were to steal your database, they'd either need to compute a rainbow table for every single user in the database, taking a very long time, or take 1 second per password to try a dictionary in an incremental password cracker, also taking a very long time. Either way it's going to be difficult to crack. But not impossible, it never is!

The Even Better Than The Even Better Way

Security is hard, and I'm not a security expert. Everything I've written about above is just how I understand it right now, there could be important things I've missed, or even worse there could be things I've misunderstood and presented incorrectly (and please do let me know if that's the case). Unless you understand the intricacies involved in cryptographic algorithms and cryptography in general you shouldn't try to implement a security system yourself. There are plenty of great libraries out there built by real security experts, which have been tried and tested in the real world. A simple mistake somewhere in a custom built system can go unnoticed until it's too late.

Nothing I've written about here is new, in fact most of it dates back to 1976. If any of this page is news to you, then it just proves my point that you shouldn't custom build a security system.

I'll say this once more, because it's very important, use a well established library that has been tried and tested in the field, and written by real security experts. There are plenty of options out there, bcrypt and scrypt being popular choices. PHP has the built-in crypt function, and there are many frameworks such as phpass which handle everything for you. All of these incorporate everything I've talked about above.

But if you like to ignore good advice and decide to implement a password storage system yourself (seriously, don't do it!), at the very least remember to store your passwords the even better way by using salted, strengthened hashes!

Always Include a Print Stylesheet

xml-feeds@wblinks.com (Rich Adams) — Wed, 29 Apr 2009 10:17:00 -0700

Earlier today I added a new tip to Tipster entitled Use a Print Stylesheet. I thought I'd follow it up here in a bit more detail.

A print stylesheet is a stylesheet that's only applied to your website when the user goes to print it. A lot of sites will provide a "print version" of their page, which is a page which has the same content as the original page, but has different markup and layout to make it look better when printed. But creating a separate "print version" of your page is not necessary, as you can simply use a print stylesheet to do all of the work.

Most websites have a lot of content which is very useful when browsing the site, but makes no sense being printed out on paper. Navigation menus for example are great for getting around a site, but are a waste of ink and paper if it gets printed every time a user prints a page from your site. You also don't really want background colours to be printed, as if your site has a dark background it's going to waste a lot of ink.

The internet and paper are two completely different types of media, and your site should adjust so that it's best suited for the media it is being presented on. While small fonts may look fine on a screen, they may be very difficult to read when printed. Banner images may make your website stand out from the rest, but it's just going to waste paper if it gets printed. Now I could go down the green route here and say that having a print stylesheet helps to save the environment. Less ink equals less paper, and not wasting paper is good for the environment. For most people this would be more than enough to encourage them to use a print stylesheet. But for the more business minded, print stylesheets take time to develop, and is it really worth the extra cost? Well the answer is yes.

How many times have you gone to print a website, whether it's a flight itinerary or just a product description page, and you've had pointless images printed, the text goes over the edge of the page, and 5 pages or printed material comes out of the printer when you only really need the first page since the other 4 are just banner images? I've lost count of the number of times this has happened to me, and it's extremely frustrating. So much so in fact that I generally resort to just cut and pasting the text I want into a new document and printing it myself. This is a mild annoyance for me, but not the end of the world. But it means the company behind the site doesn't have their logo on my printed copy, and that's a missed opportunity. Plus a less frustrated customer is always an advantage right?

A print stylesheet takes minutes to build, but can make a world of difference for your users. For me, it's not so much the reason that I want my users to have a better experience (although I admit that should be driving force), it's that I want my designs to look as good on paper as it does on a screen. I want my sites to look great, no matter how a user chooses to view it.

So now that you're convinced that adding a print stylesheet is a good thing (or maybe you're not convinced, I'm still going to assume you are though, since otherwise it makes the next bit of this rather pointless) you may be wondering what a typical print stylesheet looks like, and how to include on on your page. Well, that's what this next part is all about.

Including a print stylesheet on your site is pretty straight forward. You add the CSS file like you would any other, but you make sure to select the media type of "print". That way the styles are only applied when the user goes to print the site, rather than all the time.

<link rel="stylesheet" type="text/css" media="print" href="print.css"/>

Generally you will want to hide some elements that don't need to be printed such as sidebars, headers, menus, etc. This can be done by specifying "display: none". I've included the "!important" declaration just to be sure it gets applied and isn't overridden by anything else.

#header,
#menu,
#additional-content   {
    display: none !important;
}

Another useful thing you can do with print stylesheets is to show the URL of any link after the link text. When viewing a website this isn't really necessary, since you can click the link. But when printed you have no way of knowing the URL for a link. I'm normally not a fan of using CSS to include content, but in this case it's very useful, so I like it.

a[href]:after {
    content: " (" attr(href) ") ";
}

You can use the print stylesheet to add page breaks too. If you have a list of blog posts for example, and you want them to print one to a page, then you can use the following bit of code,

.blog-post {
    page-break-before: always;
}

Does anyone else have any tips and tricks for things to include in print stylesheets? Does anyone prefer to create a "print version" of their page rather than using a print stylesheet, and if so, why? Comments on a postcard please, or you could just comment to this post.. your choice :)

Reorganisation

xml-feeds@wblinks.com (Rich Adams) — Sun, 26 Apr 2009 18:51:00 -0700

I can't remember the last time I had a chance to just sit down and play with the code for this site. After taking one look at the current PHP code, I decided it just wasn't worth the hassle trying to fix it. I originall wrote the backend of this site when I first learnt PHP and it was not a pretty sight. I want to believe I could never write such horrible code, but I was learning so it's predictable that the code would suck.

I've now rebuilt the entire backend of the site using a much nicer object oriented approach, restructured the database so that it actually makes sense and re-written all of the database queries so that they're fast(er). I've also taken the opportunity to fix some of the front-end bugs and updated the main style of the site (which was called "Flux", but is now called "Flux-Refresh", since it's pretty much the same style but with a more polished look). I would like to create a new frontend style for the site at some point, but it's likely to take a while since I have little free time.

My next task is to organise the content of the site. The projects section in particular is horribly outdated and I have some new projects to add. I've started to rebuild my photo album so that it looks better and is easier to maintain. Also, as part of the reorganisation I'm going to start actually writing some decent notes for the site. In the past I've generally just posted a link to something which I found interesting, or talked about new versions of browsers and how they handle standard compliance, etc. From now on, anything like that I'll just post on my Twitter account and the notes on this site will be more like full articles or tutorials on a particular idea. I already have a few ideas drafted out which I'll post up soon. It would be nice to actually get some content added every now and then rather than me just being too lazy to update, as it has been in the past.

In a bold move, I've also completely dropped IE6 support. This is my personal site, and less than 1% of visitors use IE6. While I always like to make things available to as wide an audience as possible, it's just not worth my time to do twice the work to get this working in IE6. I'm sick and tired of having to deal with such a completely useless browser. So you now get a non-styled version of the site if you use IE6. If you don't like it, tough! We don't like your kind around these parts!

digg2del.icio.us

xml-feeds@wblinks.com (Rich Adams) — Tue, 24 Feb 2009 11:38:00 -0800

I was trying to find a tool the other day, which would take all of the stories I've "dugg" over on digg.com, and add them as bookmarks to my del.icio.us account. Unfortunately, my search didn't turn up anything, so I decided to make one myself.

It's just one PHP file which you can use to export the digg stories for your username (or any username on Digg for that matter), and import them into your del.icio.us account.

digg2delicious-1.0.tar.gz - 5.0KB

Alternatively, you can just try it out without downloading the file yourself.

Chrome Beta Released

xml-feeds@wblinks.com (Rich Adams) — Tue, 02 Sep 2008 13:30:00 -0700

Google's new browser, Chrome, is now available for download on Windows XP/Vista only.

Google Chrome is Google's open source browser project. As rumored before under the name of "Google Browser", this will be based on the existing rendering engine Webkit. Furthermore, it will include Google's Gears project.

First impressions are that it's a very fast install, and imported all of my Firefox bookmarks with no issues. Loads much quicker than any of my other browsers, and looks quite nice. The address bar is a bit thick for my liking, but that's just me being picky. None of the usual "File", "Edit" menus, etc as the entire interface is pretty simplistic. Exactly what we've come to expect from Google products. It's got the usual Google feel about it, a nice clean interface, no status bar showing useless information, etc. When the status bar does show, you can move your mouse towards it, and it'll drop below the window so it's not in the way of the page. A nice touch.

Some things are noticeably missing though, such as a "home" button, which is a feature common to most browsers. It is available, but not enabled by default. It's an option on the "Basics" part of the options page. I guess it depends on how much you use it whether that will be an annoyance or a blessing.

Sites are loading pretty fast, as I would expect from something based on WebKit. One feature I have noticed, is that when you're on a page with a download link, it gives you a list of the available downloads at the bottom of the page. I've already found this useful.

They've taken a page from IE8, and fade out the rest of the URL that's not the main domain. I'm not really sure why this is becoming common, and I don't really see the point. Regarding the address bar though, it's also a search bar. You can either enter a URL, or a search term and it will search Google. This combines the two bars you get in Firefox quite nicely. Of course, in FF you can choose the dropdown for various sites (for example I have a Wikipedia one), and I haven't found a way to do that easily in Chrome yet.

Next up are the screenshots.

The home page is pretty nice, shows you some recently visited pages along with your bookmarks. You can edit the things which are shown on it from the options screen.

The download bar which shows when you go to download a file.

Not really many options to go at, but it's simple enough for the average user.

"Incognito" mode is the privacy option. Does exactly what it says on the tin.

Google Chrome passes the ACID2 test with flying colours.

Only gets 61/100 on the ACID3 test, but that's better than a lot of other browsers. It takes a very long time to run on my machine though.

The task manager is quite nifty, and gives you a breakdown of the memory usage of each tag. It'll even tell you how much Flash is taking up if the page has Flash on it.

A new feature is the "about:memory" page, which gives you a breakdown of the memory usage of the browser, along with any other browsers you happen to have open too. I can imagine this being useful for testing purposes.

Overall, I'm impressed. It's clean, quick and will be great for most users. I won't be using it as my main browser though, as I really can't do without some of the FF extensions I have, they're just too useful. If they include an Add-on, or extensions feature, they're onto a winner! I think the hardest part for Google is going to be getting people to actually download Chrome, and then switch to it as their main browser. People don't like change. As long as they're moving away from IE though, I've got no complaints.

A great start, and much better than the IE8 Beta. I see amazing potential in this if it gets released on the various platforms, and has the ability to customize it to your needs.

For reference. Here's what happened the first time I loaded IE8b2 after a fresh install. I'm not making this up just because I hate IE.. this is actually what happened. Look how cluttered it is compared to FF, Chrome or any other browser!! Plus the fact that it crashed. (I should point out, it only crashed the first time. It seems to work fine on second load).

Internet Explorer 8 the first time I ran it.

IE8 Beta 2 Released

xml-feeds@wblinks.com (Rich Adams) — Thu, 28 Aug 2008 04:32:00 -0700

The new beta of Internet Explorer 8 has been released.

Complete with the new InPrivate option that's been all over the news the past week or so. They've also got Compatibility View, which quickly switches to render the page as IE7 would render it (so.. incorrectly). They had IE7 rendering in the old beta, but it was crap, as you had to reload the browser. Apparently this one does it instantly.

It fares well against current CSS standards though, a lot better than any current iteration of IE. There's an updated compliance table over at quirksmode.org.

Number 10 goes Web 2.0

xml-feeds@wblinks.com (Rich Adams) — Wed, 13 Aug 2008 04:02:00 -0700

Another in the long list of sites to redesign this year is Number10, the website of the prime minister's office has taken on the whole web 2.0 look.

It's got Flickr, YouTube and Twitter all in the sidebar, plus links to submit stories to Facebook, Digg, etc and of course, RSS feeds.. just in case you want to keep up to date on everything Downing Street it tweeting.

Seems like it's built on top of WordPress (based on the code), despite the fact there's no mention of it anywhere. They've already got some rather interesting code cock-ups. Nested anonymous <div>'s for absolutely no reason, links pointing to local IP addresses (e.g. the "Home" link, which points to http://10.10.0.215/), not to mention the fact they're not clearing floats on the "Communicate" page, so the images are pushed off to the left.

There's a grand total of 72 errors if you try and validate the XHTML. Although to be fair, most of those are because of using & instead of &. At least the CSS is valid, although they didn't write it, it's a pre-made WordPress theme called NetWorker.

Also, no print stylesheet. C'mon people!!! All websites should have a print stylesheet. Trying to print their website is painful, and is going to waste a hell of a lot of paper, along with lots of ink. Is that really the environmental message they want to give out?

It's even in "Beta", a true web 2.0 site! Which would explain the above issues, can't expect everything to be perfect first time. It does look very good though, and is definitely an improvement on the old site. Nice to see they're actually making an effort!