Buca Bay - Always nice

Dua tiko noqu toa loaloa, na yacana ko… laga mai…

Secure password hashing and storage in PHP

November7

Everyone knows you should never store passwords as plain text, right?

Recently I’ve come across a lot of bad advice on how to store passwords securely. I thought I’d share a bit of my research into the subject. Like most web developers, I’m not a cryptography expert. However, this should not be the excuse we use when we make a database of thousands of passwords available to hackers. It is quite simple to save passwords in a reasonably secure manner.

Main points: secure password hashing

  1. Force your users to use secure passwords
  2. Do not use 2 way encryption, save passwords as hashes
  3. Salt each password with a unique salt
  4. Hash the password and salt together multiple times (1000 or more)
  5. Use an adaptive hashing technique

Cryptographic Hashes

Passwords should never be stored in their original form, instead a cryptographic hash must be stored. A hash function is a function that performs a one way operation on an input producing an output of fixed length called a digest.

A popular hash function is MD5. An example in PHP:

$hash = md5('test'); // 098f6bcd4621d373cade4e832627b4f6

From the hash, the original text cannot be computed, however, knowing the original text you can easily compute the hash. This is the basis of saving passwords securely.

To verify the password, we see if it hashes to the same hash as the one we have stored.

if (md5($password) == $hash) { /* password is valid */ }

Notice how we never store the actual password, only the hash. We also don’t need to be able to decrypt the hash, we only have to verify it given the password supplied by the user.

Note I’m only using MD5 for example, since it is well known. You probably want to use a more secure hash such as SHA256 or Whirlpool as MD5 and SHA1 have a few problems (don’t ask me what they are, I don’t know).

Key Based Encryption

I’ve come across developers advising the use of a key based encryption. However, this is not a secure way of storing passwords. You’re only deferring the task of having to hide the password, to having to hide the secret key.

Another argument is that you need to be able to retrieve the password if the user forgets it. If we backtrack to why we use passwords, we realize that allowing the user to gain access to their account is the concern, not retrieving their password. So instead, send the user a unique key (token) through email or some other authenticated resource, so the user can generate a new password.

Precomputation Attacks - Rainbow Tables and salts

Now, the problem with just hashing a password, is that users normally use very easy to remember passwords, which are thus very easy to guess. Due to this, there are many databases available that map all the possible passwords made up of up of alpha-numeric characters in upper and lower case, as well as a lot of special characters. There are usually in the form of rainbow tables. Most do not have matches for passwords longer then 14 characters however due to the amount of computation required to generate these tables.

The way to beat rainbow tables or any other precomputation attack is to use a salt. A salt is a randomly generated input that you add to the password before hashing. This is done so that the input becomes long enough that generating a table that includes it is computationally infeasible.

So our hash function becomes:

$salt = 'f39ae656bc79a9b2398890bb4';
$hash = md5('test'.$salt); // 1db36e73cf2e68df3640fb1052e801da 

Now even if the user supplies a very easy to guess password, such as “love”, it still cannot be looked up in a rainbow table because the table would have to hold the whole input, “lovef39ae656bc79a9b2398890bb4″. No public rainbow tables have that many characters in them.

Use a unique salt for each password

If you use the same salt for each password (a common practice in PHP applications) is a very bad idea. It allows the attacker to crack all your passwords in one go using brute force. Imagine a database with a million passwords. Each guess becomes one million times more likely to match one of those passwords. Using a unique salt for each password prevents this.

Does salt size matter?

Another piece of advice I’ve come across is that the size of the salt does not matter. Using common sense this is not correct. A salt such as “52e801da” will still generate the input “love52e801da” which will be in rainbow tables.

Early Unix user passwords had a vulnerability that was partly due to the size of the salt.

Brute Force Attacks on passwords

Now even though we have mitigated precomputation attacks such as rainbow tables, we still have one issue. We have no way to hide the salt. Since we cannot hide the salt securely, we must assume it is visible. If the salt is visible, the attacker still cannot use a rainbow table, however, they can just guess the password by simply guessing passwords and then hashing them with the visible salt, then comparing the result to the stored hash. This is called a brute force attack.

A brute force attack in it’s simplest form, is to try every combination of passwords in sequence. However, sophisticated forms will use precomputated data such as the word combinations in a dictionary (dictionary attack) combined with statistics to find the password “love” very quickly.

Better Passwords - Key Stretching and Adaptive Hashing

Now the first way to combat this is to force your users to have a password at minimum 6 characters and consist of numbers as well as alphabetic characters in both upper and lower case. This is probably the best thing you could do as you have made the range of possible passwords large enough that a brute force will take a while.

Now you need to slow down the brute force. One way to do this is by using key strengthening. This is the process of hashing multiple times, in order to make the hashing process longer. Hash functions such as MD5, SHA1 are very fast. This is not a good thing for you, since the attacker can do numerous hashes very quickly. However, if you hash 1000 times, then the attacker has to also hash each possible password the same number of times, making their brute force attack 1000 times slower.

Now it is up to you to figure out how many times you want to hash the password. Computational power is increasing every day, so you may also want to increase the number of times you hash the password as time progresses. There are hash functions that take this into account such as MD5 crypt and BCrypt. A lot of why this is needed is explained at this Matasano security article.

PHP implementation of secure password hashing

Now I know this has been a rather long article, so here is what I have come up with that incorporates these methods of mitigating these password cracking techniques.

/**
 * Generate cryptographic Hashes for passwords
 *
 * Features:
 * 	Harderned against precomputation attacks like rainbow tables (using unique salts)
 * 	Harderned against brute force and dictionary attacks (using key stretching and optional secret key)
 *
 *  http://en.wikipedia.org/wiki/Password_cracking
 *
 *  Note: for PHP4 and lower, just remove the "public static" before function declaration
 *
 * @author gabe@fijiwebdesign.com
 * @link http://www.fijiwebdesign.com/
 * @version $Id$
 */
class Password_Hash
{

	/**
	 * Generate the Hash
	 * @return String
	 * @param $password String
	 * @param $salt String[optional]
	 * @param $iterations Int[optional]
	 * @param $secret String[optional]
	 */
	public static function generate($password, $salt = null, $iterations = 10000, $hash_function = 'sha1', $secret = '')
	{
		$salt or $salt = self::generateToken();
		$hashes = array();
		$hash = $password;
		// hash a sequence of hashes, each hash depends on the last one, so any implementation must hash each one individually
		$i = $iterations;
		while(--$i)
		{
			$hash = $hash_function($hash.$salt.$secret);
		}
		return implode(':', array($hash, $iterations, $hash_function, $salt));
	}

	/**
	 * Verify a password meets a hash
	 * @return Bool
	 * @param $password String
	 * @param $hash String
	 * @param $secret String[optional]
	 */
	public static function verify($password, $hash, $secret = '')
	{
		list($_hash, $iterations, $hash_function, $salt) = explode(':', $hash);
		return ($hash == self::generate($password, $salt, $iterations, $hash_function, $secret));
	}

	/**
	 * Generate a random hex based token
	 * @return String
	 * @param $length Int[optional]
	 */
	public static function generateToken($length = 40)
	{
		$token = array();
		for( $i = 0; $i < $length; ++$i )
		{
			$token[] =	dechex( mt_rand(0, 15) );
		}
		return implode('', $token);
	}

}

Example using SHA1 with default of 10000 iterations.

// generating the hash
$password = 'test';
$hash = Password_Hash::generate($password);

// verifying a password
$result = Password_Hash::verify($password, $hash);

// dump results
var_dump($hash, $result);

Example using whirlpool as the hash function, a 128 length salt as well as a secret.

// define our custom hash function
function whirlpool($str) {
	return hash('whirlpool', $str);
}

$password = 'test';
$salt = password_Hash::generateToken(128);
$secret = password_Hash::generateToken(128);
$iterations = 10000;

// generate hash
$hash = Password_Hash::generate($password, $salt, $iterations, 'whirlpool', $secret);

// verify
$result = Password_Hash::verify($password, $hash, $secret);

// dump results
var_dump($result);

Base conversion in JavaScript

September4

I just realized recently that you can convert between number bases in JavaScript using the built in method Object.prototype.toString() and parseInt().

Math.base = function(n, to, from) {
     return parseInt(n, from || 10).toString(to);
}

For example, to convert from decimal to hex, or hex to decimal:

// convert the decimal 10 to hex
Math.base(10, 16); // 'a'

// convert the hex 'a' to decimal
Math.base('a', 10, 16); // 10
// or
Math.base(0xA, 10); // 10

Or from hex or dec to binary:

// convert the decimal 10 to binary
Math.base(10, 2); // '1010'

// convert the hex 'a' to decimal
Math.base('a', 2, 16); // 1010

This should work for bases between 2 and 36. ie: the number of characters from 0-9a-z.

Edit: changed base() to Math.base() for better namespace

I’ve also added base conversion in PHP up to a radix of 255.

Fun with JavaScript bookmarks

July15

Here are some random JavaScript sinippets I wrote for bookmarks.

How they work is when you click the bookmark, the JavaScript is executed. So you can actually do a bit of programming inside a browser bookmark.

How to save JavaScript bookmarks:

  1. Copy the Bookmarks JavaScript code
  2. Right Click on the bookmark toolbar
  3. Choose “new bookmark” or the equivalent on your browser
  4. Fill in a name of choice
  5. In the location/url field paste the javascript code
Here is the list:
Translator:

javascript:window.location='http://www.google.com/translate_c?u='+window.location
Calculator:
javascript:void(alert(eval(prompt('Calculate:'))))
Binary Convertor:
javascript:str='';i=0;c=prompt('Encode in Binary:');while(n=c.charCodeAt(i)){ b = '';while(n>0) { b = (n&1 ? '1' : '0')+b; n >>= 1; } i++;str+=b+' ' } void(alert(str))

Real-time Concurrency Control and Version Control

July11

Almost every software developer has used some sort of source code version control system. You’re probably familiar with the acronyms GIT, SVN or the now aging CVS. Here is a list of version control systems.

Version control is a form of concurrency control with the benefit of revisions. Todays major versioning systems work by creating a copy of a resource, and allowing a user to work on that copy. The user then checks their modified copy back to the central source code repository. If the copy at the repository has changed, then the user must merge the two changed copies together.

Since each user is working on a remote copy, there is no real time collaboration involved at all. Each user must edit their copy individually without input from the other users. The collaboration is somewhat chunked, instead of a smooth integration of two or more users inputs.

I’ve been waiting for the day when real time concurrency control is integrated with source code version control. Just lately Google launched Google Wave, which essentially implements real time concurrency control in document editing. A while before that, Appjet Launched their Etherpad product, which implemented real time concurrency control in document and JavaScript source code editing.

Neither of these currently implement source code version control. However, they are proofs that real time concurrency control can be achievable over the web (HTTP) using Operantional Transformation (OT).

Now the missing link is to tie together a real time concurrency control system, with a version control system. Imagine checking out a document with a number of colleges, editing it concurrently in real time, and checking it back into version control.

The model of source code versioning will change dramatically with integrated real-time concurrency control. Checking out a copy may be a thing of the past. You can just join a a group that is editing a live copy, and revisions can be saved automatically for individuals and/or the group as a whole.

Add your own thoughts here…

PHP Object Cache

June25

The last two days I’ve been writing an Object cache in PHP as part of a larger project. I released it today as open source so it will help those doing something similar and get some that helpful feedback open source offers.

PHP Object Cache is a Memory Object cache, implemented with PHP Sockets. It runs on PHP4 or PHP5+ and requires your PHP build to have sockets enabled.

The project is a web (browser) based chat system called Joomla Ajax Chat. The initial development of the chat was 3 and a half years ago, so many of what worked then (HTTP Polling and AJAX) is becoming old school now and not as efficient as what is possible with browsers today.

One of the new features is implementing comet like HTTP, which essentially means keeping the HTTP connection open for as long as you can. XMPP defined a specification for this called BOSH for their Instant Messaging Protocol to work over HTTP. With a good implementation, Comet can be very efficient.

Our problem is that we have to implement Comet on regular ol’ Apache, Lighty, Nginx, IIS servers running different versions and builds of PHP. The code will have to run on the average shared hosting, dedicated servers to cloud based solutions. The other problem is it has to run on top of Joomla, and other CMSs. A call to a minimal Joomla page alone, is around 5-6 database reads and one or two writes and about 5-6Mb of ram. It wouldn’t take much to crash a shared hosting account trying to implement comet on top of that.

So the solution would have to be an object cache of some sort, even if it means a file based cache (last resort). Thus, PHP Object Cache, which hopefully will work for the percentage of shared server that allow sockets. The idea it to take the most intensive IO processes, such as session management, chat events and put them into the Object cache. When everyone online has viewed what they need from the cache, flush that bit to the database or some other persistent storage if needed.

Since the browser is able to do more as time goes on, hopefully php on old shared servers can keep up.

Backup and export Appjet Applications

June10

Appjet recently anounced that they are closing down. They however offer an open source download that allows you to run the appjet engine on your own server.

JGate has gone to the intiative of cloning the Appjet service. This allows you to transfer your apps from Appjet to JGate, including your storage and libraries.

Backup your Appjet Applications

For those needing to backup your applications, I’ve written a backup application that will take all your published applications and create a single compressed download in Zip or Gzip format.

http://export.appjet.net/

This app is actually cloned from two other apps. It has to contact a PHP server on the background in order to create the compressed files. I couldn’t find any JavaScript implementations of ZIP and Gzip readily available.

Using the Application is simple, just input your username and click on “backup my apps” and it will prompt a download after a few seconds.

Export your Appjet Storage

To export your storage you need to import an Appjet Library.

http://lib-export.appjet.net/
http://apps.jgate.de/lib-export

This library exists both on Appjet and JGate, and if you are hosting your own Appjet server then you need to include that library.

Here are the steps to export your appjet storage:

  • Import lib-export into both apps ie: import(’lib-export’);
  • Show the Admin Panel. On appjet.net, you can do this via the preview window. In JGate y ou need to include showExportAdminPanel(); after your import command and view the app in regular mode.
  • First set a password on both apps.
  • Then in the app you want to export from, turn off the admin panel. ie: remove the line showExportAdminPanel();
  • In the app you want to import to, click on the “import” link in the admin panel.
  • Fill in the details, make sure the URL does not include a trailing slash. eg: http://example-app.appjet.net
  • Submit the form.
You can also view the JSON representation of your root storage object (storage) by going to the URL: http://appname.appjet.net/export?password=mypass where appname and mypass are substituted for your own.

Other uses of lib-export:

If you’re interested in how it works, check out the source for lib-export.
Since appjet uses object storage instead of a traditional relational database, you can serialize the storage Objects into regular objects, and then into JSON, for export.
You can also do the opposite and unserialize JSON into native JavaScript Objects, and native JavaScript objects into Appjet storage Objects. (StorageObject and StorableCollection instances).
This can be useful if you want to storage random JavaScript Objects without knowing their type. It should even store your custom JavaScript instances, though I haven’t tried it.
eg: storing native and custom Objects
import('storage');
import('lib-export');
// native object
storage.date = storablelizeObjects(new Date());
function MyCustomObject() {
   this.name = 'my custom object';
}
// custom object
storage.customObj = storablelizeObjects(new MyCustomObject());
eg: storing Arrays
import('storage');
import('lib-export');
storage.myArray = storablelizeObjects(['hi', 'bye']);
eg: Store anything
import('storage');
import('lib-export');
storage.myArray = storablelizeObjects({
   'myarr': ['hi', 'bye', {"another object": "value"}],
   'myObj': new MyCustomObject(),
   'date': new Date(),
   'url': wget('http://example.com')
});
Convert Storage Object to native Objects
import('storage');
import('lib-export');
var myObj = objectizeStorables(storage.myObj);
Export your Storage Objects as JSON
import('storage');
import('lib-export');
var myObj_str = serializeStorables(storage.myObj);
Note that properties resolved from Object.prototype is not stored. Obviously, using StorableCollection() and StorableObject() will be faster then casting native Objects to Storables using lib-export, though the latter is more convenient for storing objects of unknown types.

Notes

The library lib-export is provided without any guarantees. Please do not use it if you do not take full responsibility for any outcome. ie: It works for me, but I cannot guarantee that it will work for your app.
Please note that the export lib is still in development, so there may be some changes to it which I will document here.

Appjet service ends

June2

Appjet, a web based javascript application development platform, sent out an email to members today that they are closing down the service.

According to the email, they are closing their Appjet service because of the success of Etherpad - which is a real time document collaboration service that they built on Appjet. I think what they really mean is, appjet isn’t making any money and has become a liability. In that case, they just didn’t find a good way to capitalize on what they have, because it is quite awesome.

Appjet has been the only server side JavaScript development platform that offered a web based editor so you could create applications with just a connection to the internet, nothing else. There are many server side JavaScript alternatives, and many JavaScript Application development Platforms, none had the simplicity Appjet had. Nor is there a platform that encourages the sharing of libraries amongst developers like Appjet.

I have quite a few applications running on Appjet. One the fetches updates from freelance sites and posts it to a twitter account, freelance_jobs. Another pings websites and keeps a log of their uptime.Yet another acts as a proxy allowing RSS feeds to be fetched cross domain as a JSON callback. Theres even one that allows to to cheat at facebook’s wordtwist game by guessing all the possible combinations of words available for that round. There’s more then 10 of these apps I wrote and they were all hosted freely by Appjet. These were all developed in my spare time and at first I thought of them only as fun apps, but a few of them have become quite useful.

Now comes the porting of these apps to a new location. The great thing is that the appjet engine is available for download.

I’m considering installing this on my own server, or just rewriting everything in PHP.

Incoming mail with PHP Mime Mail Parser

May13

I’ve just added the ability to parse mime mail from standard input into PHP Mime Mail Parser. This allows you to receive and parse email in PHP efficiently and effortlessly. 

To pipe email to PHP, follow one of these articles:

Evolt - Incoming Mail and PHP
DevPapers - Incoming Mail and PHP
DevArticles - Incoming Mail and PHP

Then for your PHP code, get the PHP Mime Mail Parser, and use:

<?php
// include the mime-mail-parser class
require_once('MimeMailParser.class.php');

// instantiate
$Parser = new MimeMailParser();
// read the email from stdin
$Parser->setStream(STDIN);

// get the email parts
$to = $Parser->getHeader('to');
$delivered_to = $Parser->getHeader('delivered-to');
$from = $Parser->getHeader('from');
$subject = $Parser->getHeader('subject');
$text = $Parser->getMessageBody('text');
$html = $Parser->getMessageBody('html');
$attachments = $Parser->getAttachments();

?>

Monster - 3d in JavaScript

April24

I was reading the Qt blog’s on Webkit when I came aross this post on the chrome experiment Monster.

The experiment is written by Dean McNamee and is a JavaScript 3d rendering engine. There is a great article on it at the google code blog.

It renders a cube that turns into a sphere then a monster. All with just JavaScript, no Flash or other plugins.

I tried launching the demo (click to open in a new window) in Chrome, Safari and Firefox. Chrome was the only browser that handled the animation well. It didn’t even use more then a few MB of memory to do it. Safari climbed to about 200MB of memory and didn’t render the more intensive parts of the animation. Firefox was slow in rendering and also took up to 200MB of memory. This just shows how vastly superior Chrome’s Javascript engine (V8) is. You’d also have to take into account the implementation of the Canvas element, but the Javascript execution is what really matters.

The code for Monster isn’t released yet. Would be cool to play around with it when it is.

A PHP Mime Mail parser using MailParse Extension

April21

Ever tried parsing Mime Mail? Not for the faint hearted, I assure you. The great thing is that PHP has an extension for parsing Mime Messages called MailParse. The bad news is that using this extension is probably just as hard as writing your own Mime Parser. 

Fortunately with a bit of help I’ve put together a Mime Mail Parser Class that wraps the MailParse extension functions making it simple, efficient and fast to parse mime mail in PHP. 

Why another Mime Mail parser? Well for two main reasons. 

1) Pure PHP implementations are slow and inefficient compared to MailParse.

2) MailParse is too hard to use, and is not OO. 

Therefore welcome to MimeMailParser.

Here is a an example that shows how easy it is to parse raw mime mail using MimeMailParser:

<?php

require_once('MimeMailParser.class.php');

$path = 'path/to/mail.txt';
$Parser = new MimeMailParser();
$Parser->setPath($path);

$to = $Parser->getHeader('to');
$from = $Parser->getHeader('from');
$subject = $Parser->getHeader('subject');
$text = $Parser->getMessageBody('text');
$html = $Parser->getMessageBody('html');
//$attachments = $Parser->getAttachments();
$attachments = $Parser->getAttachmentsAsStreams();

?>

You can find the source code here for your enjoyment.

« Older Entries
Tag Cloud