Moce toka Viti. Nomu lasa e rui divi.

Week two in Florida. Florida is a lot like Fiji, tropical weather and multi cultural. Though it would be winter, the mornings and evenings are cool, and the days almost hot. The beaches are extremely wide and smoothened by bulldozers each morning before the city wakes. Though the manicured beaches, tall buildings and bright lights keep you on a permanent high, it also reminds me of the insustainability of our destruction of this planet.

My brother flew down the day I got here. I hadn’t seen him in a few years. It has been great seeing him again but we haven’t had much time to enjoy Florida due to tight work schedule. Every Sunday we go down to the beach and play some volleyball and watch the waves. At times it almost feels like Fiji.

I can’t wait for Vara and Joana to get here. I know they will love Florida.

Dear God,

I’m Sorry.

Be nice.

To Mommy and Daddy.

Amen.

Everyone knows you should never store passwords as plain text, right?

Recently I’ve come across a lot of bad advice on how to store passwords securely. I thought I’d share a bit of my research into the subject. Like most web developers, I’m not a cryptography expert. However, this should not be the excuse we use when we make a database of thousands of passwords available to hackers. It is quite simple to save passwords in a reasonably secure manner.

Main points: secure password hashing

1. Force your users to use secure passwords
2. Do not use 2 way encryption, save passwords as hashes
3. Salt each password with a unique salt
4. Hash the password and salt together multiple times (1000 or more)
5. Use an adaptive hashing technique

Cryptographic Hashes

Passwords should never be stored in their original form, instead a cryptographic hash must be stored. A hash function is a function that performs a one way operation on an input producing an output of fixed length called a digest.

A popular hash function is MD5. An example in PHP:

$hash = md5('test'); // 098f6bcd4621d373cade4e832627b4f6

From the hash, the original text cannot be computed, however, knowing the original text you can easily compute the hash. This is the basis of saving passwords securely.

To verify the password, we see if it hashes to the same hash as the one we have stored.

if (md5($password) == $hash) { /* password is valid */ }

Notice how we never store the actual password, only the hash. We also don’t need to be able to decrypt the hash, we only have to verify it given the password supplied by the user.

Note I’m only using MD5 for example, since it is well known. You probably want to use a more secure hash such as SHA256 or Whirlpool as MD5 and SHA1 have a few problems (don’t ask me what they are, I don’t know).

Key Based Encryption

I’ve come across developers advising the use of a key based encryption. However, this is not a secure way of storing passwords. You’re only deferring the task of having to hide the password, to having to hide the secret key.

Another argument is that you need to be able to retrieve the password if the user forgets it. If we backtrack to why we use passwords, we realize that allowing the user to gain access to their account is the concern, not retrieving their password. So instead, send the user a unique key (token) through email or some other authenticated resource, so the user can generate a new password.

Precomputation Attacks - Rainbow Tables and salts

Now, the problem with just hashing a password, is that users normally use very easy to remember passwords, which are thus very easy to guess. Due to this, there are many databases available that map all the possible passwords made up of up of alpha-numeric characters in upper and lower case, as well as a lot of special characters. There are usually in the form of rainbow tables. Most do not have matches for passwords longer then 14 characters however due to the amount of computation required to generate these tables.

The way to beat rainbow tables or any other precomputation attack is to use a salt. A salt is a randomly generated input that you add to the password before hashing. This is done so that the input becomes long enough that generating a table that includes it is computationally infeasible.

So our hash function becomes:

$salt = 'f39ae656bc79a9b2398890bb4';
$hash = md5('test'.$salt); // 1db36e73cf2e68df3640fb1052e801da 

Now even if the user supplies a very easy to guess password, such as “love”, it still cannot be looked up in a rainbow table because the table would have to hold the whole input, “lovef39ae656bc79a9b2398890bb4″. No public rainbow tables have that many characters in them.

Use a unique salt for each password

If you use the same salt for each password (a common practice in PHP applications) is a very bad idea. It allows the attacker to crack all your passwords in one go using brute force. Imagine a database with a million passwords. Each guess becomes one million times more likely to match one of those passwords. Using a unique salt for each password prevents this.

Does salt size matter?

Another piece of advice I’ve come across is that the size of the salt does not matter. Using common sense this is not correct. A salt such as “52e801da” will still generate the input “love52e801da” which will be in rainbow tables.

Early Unix user passwords had a vulnerability that was partly due to the size of the salt.

Brute Force Attacks on passwords

Now even though we have mitigated precomputation attacks such as rainbow tables, we still have one issue. We have no way to hide the salt. Since we cannot hide the salt securely, we must assume it is visible. If the salt is visible, the attacker still cannot use a rainbow table, however, they can just guess the password by simply guessing passwords and then hashing them with the visible salt, then comparing the result to the stored hash. This is called a brute force attack.

A brute force attack in it’s simplest form, is to try every combination of passwords in sequence. However, sophisticated forms will use precomputated data such as the word combinations in a dictionary (dictionary attack) combined with statistics to find the password “love” very quickly.

Better Passwords - Key Stretching and Adaptive Hashing

Now the first way to combat this is to force your users to have a password at minimum 6 characters and consist of numbers as well as alphabetic characters in both upper and lower case. This is probably the best thing you could do as you have made the range of possible passwords large enough that a brute force will take a while.

Now you need to slow down the brute force. One way to do this is by using key strengthening. This is the process of hashing multiple times, in order to make the hashing process longer. Hash functions such as MD5, SHA1 are very fast. This is not a good thing for you, since the attacker can do numerous hashes very quickly. However, if you hash 1000 times, then the attacker has to also hash each possible password the same number of times, making their brute force attack 1000 times slower.

Now it is up to you to figure out how many times you want to hash the password. Computational power is increasing every day, so you may also want to increase the number of times you hash the password as time progresses. There are hash functions that take this into account such as MD5 crypt and BCrypt. A lot of why this is needed is explained at this Matasano security article.

PHP implementation of secure password hashing

Now I know this has been a rather long article, so here is what I have come up with that incorporates these methods of mitigating these password cracking techniques.

/**
 * Generate cryptographic Hashes for passwords
 *
 * Features:
 * 	Harderned against precomputation attacks like rainbow tables (using unique salts)
 * 	Harderned against brute force and dictionary attacks (using key stretching and optional secret key)
 *
 *  http://en.wikipedia.org/wiki/Password_cracking
 *
 *  Note: for PHP4 and lower, just remove the "public static" before function declaration
 *
 * @author gabe@fijiwebdesign.com
 * @link http://www.fijiwebdesign.com/
 * @version $Id$
 */
class Password_Hash
{

	/**
	 * Generate the Hash
	 * @return String
	 * @param $password String
	 * @param $salt String[optional]
	 * @param $iterations Int[optional]
	 * @param $secret String[optional]
	 */
	public static function generate($password, $salt = null, $iterations = 10000, $hash_function = 'sha1', $secret = '')
	{
		$salt or $salt = self::generateToken();
		$hashes = array();
		$hash = $password;
		// hash a sequence of hashes, each hash depends on the last one, so any implementation must hash each one individually
		$i = $iterations;
		while(--$i)
		{
			$hash = $hash_function($hash.$salt.$secret);
		}
		return implode(':', array($hash, $iterations, $hash_function, $salt));
	}

	/**
	 * Verify a password meets a hash
	 * @return Bool
	 * @param $password String
	 * @param $hash String
	 * @param $secret String[optional]
	 */
	public static function verify($password, $hash, $secret = '')
	{
		list($_hash, $iterations, $hash_function, $salt) = explode(':', $hash);
		return ($hash == self::generate($password, $salt, $iterations, $hash_function, $secret));
	}

	/**
	 * Generate a random hex based token
	 * @return String
	 * @param $length Int[optional]
	 */
	public static function generateToken($length = 40)
	{
		$token = array();
		for( $i = 0; $i < $length; ++$i )
		{
			$token[] =	dechex( mt_rand(0, 15) );
		}
		return implode('', $token);
	}

}

Example using SHA1 with default of 10000 iterations.

// generating the hash
$password = 'test';
$hash = Password_Hash::generate($password);

// verifying a password
$result = Password_Hash::verify($password, $hash);

// dump results
var_dump($hash, $result);

Example using whirlpool as the hash function, a 128 length salt as well as a secret.

// define our custom hash function
function whirlpool($str) {
	return hash('whirlpool', $str);
}

$password = 'test';
$salt = password_Hash::generateToken(128);
$secret = password_Hash::generateToken(128);
$iterations = 10000;

// generate hash
$hash = Password_Hash::generate($password, $salt, $iterations, 'whirlpool', $secret);

// verify
$result = Password_Hash::verify($password, $hash, $secret);

// dump results
var_dump($result);

ReadWrite web posted an article titled “Google Should Stop Playing Around With Wave and Focus on Spreadsheet” a few days ago.

As the title suggests, the author believes Google is wasting time working on Google Wave and should put their efforts into Google Docs SpreadSheets.

I believe there are a few misconceptions. Google wave is not the Google Wave Client, that is in private beta. That is like saying GMail is Email. Google wave is an open protocol (like SMTP for email, or XMPP for IM) for realtime concurrency control.

On the other hand, ordinary folk create real apps on spreadsheets all day long.

More people use Email, IM and Social Networking. This is the areas Google Wave will be used in.

Real-Time Spreadsheets? That Is So 1980s, Dude!…
Online spreadsheets that anyone can edit concurrently solved the version control problem. Problem solved! Done, finished. Can we move on now?

No, not just yet. Version control is not concurrency control. However, they are often used synonymously. Simplistically, version control, is the saving of changes made to a document, while concurrency control, is the merging of these changes.

Here is an overview of version control systems in use today. Most of today’s version control systems such as used in Subversion, Mercurial and GIT implement a copy-modify-merge for their concurrency control. I’ve already blogged about how concurrency control in today’s version control systems is different from realtime concurrency control.

In the case of Google Docs, including Google Spreadsheets, there is no concurrency control. If you simultaneously edit a document with another user, any save overwrites the previous save. So if your update had not reached the user yet, it would be lost.

Google Wave implements true realtime concurrency control. It does this using Operational Transformation (OT), which is not new, and used by other software. What is new is that Google went ahead and defined a standard protocol for editing documents concurrently.

Most importantly Google built Google Wave on top of XMPP Federation (Open Source Instant messaging), so that anyone can host a Google Wave server by hosting an XMPP server and/or XMPP component, just like hosting an SMTP (Email) server. They also went ahead an made an open source implementation of the Google Wave Server. (I’ll be adding a post on how to build the Google Wave Server on CentOS 5). Already, there are other implementations of Google Wave Servers such as pygowave and yet to be release EJabberd extension (XMPP for client-server also).

This is a very important factor. If Google had not made Google Wave an Open Standard that can be implemented by anyone, they there would be very little hype about Google Wave from developers. I wouldn’t be writing this article. Instead I’d probably be joined by others criticizing Google for creating a monopoly on a new technology. However, Google made the “right choice” and made Google Wave and open standard like the rest of the open internet standards.

Now that we have a better understanding of what Google Wave is, we know that it:

1. Is not a software like spreadsheets, it is a protocol
2. It offers a basis for real time concurrency control that spreadsheets do not have
3. Spreadsheets and similar applications will be built on Google wave in the future
4. Better communication and collaboration is a much greater need then better spreadsheets

So you cannot say stop working on Google Wave and work on the Spreadsheets since they are not mutually exclusive. Working on Google wave will bring much needed true realtime collaboration features to spreadsheets. In fact it will bring realtime collaboration to the whole web, allowing the creation of applications that surpasses the uses of the spreadsheet.

By the time we got into Lautoka town today I was starving. I hadn’t had anything to eat all day.

We stopped at Pa’s kitchen, a very good restaurant  with Fijian food, but it was closed. So we head down the road to a Chinese restaurant  we hadn’t been to before, “Nina’s”. The restaurant was only a quarter full so we order immediately at the counter. We decide to order takeaway, and and have it in the restaurant since we were sure Joana wouldn’t finish her food, and thus need a takeaway box anyway.

We all leisurely sit down, open each box and we’re just about to eat when we notice that we don’t have forks. Tiare, Vara’s sister walks over to the counter and asks for forks. The restaurant owner starts yelling at her and then Vara and I. “No, forks for takeaway”. Vara asks, “Do we have to pay for the forks?”. The owner yells again, “No, why you want to buy takeaway. If you want to eat here, don’t buy takeaway”. At first we’re a bit puzzled. The owner seems really hot an annoyed and then one of the waitresses starts chiming in. Vara is still puzzled and asks, “Do we have to pay extra to eat here?”. “No, we no want your money”, the restraunt owner is now fully upset.

The guy sitting next us blurts out, “C’mon Nina, just let them eat their food”. The two of them start quarreling. I look over to Vara, “I think we should just go, I don’t feel like eating here”. As we start leaving the owner yells at us, “You buy takeaway, you take it away!”.

Vara and Tiare were pretty shocked, but I’m was just about to die from laughter. That has got to be the funniest thing I’ve heard a restaurant owner say. ”You buy takeaway, you TAKE IT AWAY!”

Today were were in Lautoka town, doing some shopping - mostly toys for Joana. On our way back I decided to stop at a shop and buy a rugby ball. Noticing the Adidas shop I asked our regular taxi driver, Darman, to stop there. Vara and I got out and walked up to the shop, which looked closed. There were two female employees with the same outfit standing just inside the door talking so I made hand gestures to them asking if it was closed. They affirmed my suspicious with gestures from behind the glass door.

We walked across the street and were just about to get into our taxi when a male employee bursts out of the shop, runs half way across the street out and yells out, “Boss boss, come come we open”. I walk back across the street and he motions for me to go inside so I do. I’m in the store looking around and there’s just him, a security guard and the too girls.

He asks “what you looking for boss?”. “Rugby ball”, I reply. He motions for me to follow him and there in between a bunch of rugby boots was a single rugby ball. I ask, “How much?”, pointing at it. He takes it of the shelf and hands it to me, “for you, good price, only 99 dollars”.

“What!”, I exclaim, taken by surprise. “You got anything else?”. He studies my expression for a while then asks, “How much you want?”. “I was looking more for something around 40 dollars, or 60 if its a match ball. This is just a size 5 training ball.”. The guy walks over to another shelf and brings out a stuffed ball that you can fit in your fist and hands it to me. “This one 30 dollars”.

I don’t know whether to laugh or swear at him. “Are you serious?”. All serious he exclaims, “This good ball, good souvenir.”. I look around, everyone in the shop is staring. In my best Fijian accent, “Boss, I’m from Lautoka, whats regular price for this ball”. A bit startled he looks a me, “Sorry Boss, all out of stock”.

I have a large collection of videos on my Windows PC. Before today, in order to watch the videos on my computer somewhere else in the house, we’d have to copy the movie to a USB stick and transfer it to the other computer.

Since this is an almost daily occurrence, and copying videos  can take a while, I decided to look into streaming the videos  over the Local network.

My first thought was to create a virtual host on my local Apache server, and have the videos  streamed by Apache over HTTP. This worked reasonably well for a while, but after about 10 minutes or so  the video started to lag on the other side. This also happened when I tried streaming the file via FTP, using Filezilla FTP server.

I also tried using Nginx which has an FLV streaming module. The videos on my PC are mostly .avi or .divx, so I’d have to transcode to FLV before hand in order to stream. I didn’t want to have to transcode each file to FLV, since it was already taking up a huge amount of space on my hard disk.

This whole time I was using VLC to play the videos. I had totally overlooked the fact that VLC was also a video streaming server. It turns out, VLC can transcode and stream at the same time (using FFMpeg internally), so you don’t need to transcode files before hand, and keep around multiple copies of the same file in different encodings.

Here is the article on streaming video with VLC that I followed. Pretty soon I had VLC on my PC streaming movies to multiple PCs and laptops on the local network. The videos playback can also be controlled from the streaming server, so I could start and play movies for my 2 and a half year old daughter Joana while doing my work.

The only problems with streaming with VLC is that you’ll have to mix and match video encodings and streaming protocols to figure out which works. For me, streaming H264 streamed over RTP seemed to work best. Edit: I somehow missed their documentation on supported streaming protocols.

Step by step to setting up streaming a video with VLC over your LAN:

1. Open VLC on your PC with the video
2. Click on: Media -> Streaming…
3. Choose the video file you want to stream (double click) A window opens with the streaming options.
4. Choose RTP for the streaming protocol. For the address put in the IP* of the computer to stream to. You can leave the port number as is.
5. Choose H264 from the “profiile” dropdown. This selects “MPEG-TS” for encapsulation, H264 for Video and MPEG-4 for audio.
6. Click Stream.
7. Then on the computer you want to watch from, Open VLC.
8. Click on Media -> Open Network
9. Choose RTP and the computers IP or localhost. Leave the port number as is.
10. Click “Play”

You can also use the command line.

To open the command line on XP: Start -> Run -> type in “cmd” and click enter
On Vista: Start -> type “cmd” into the search box and click enter
In the command prompt, type: net view
For the list of IPs: arp -a
To get the IP of a computer given its name: ping <name>

To get the name of the current computer: hostname

Before trying out VLC, I was thinking of setting up Red5 to stream videos, since I have worked with it before to stream video and audio for XMPP (Instant Messaging), but since VLC does the job really well and is so easy to set up, I’m sticking to that for now.

I was woken up this morning by Vara, yelling into my ear. “There is a Tsunami warning for Fiji, wake up. Do you think it’s gonna come this far”?

We’re about a mile from the beach, but also at an elevation of about 50m or so. I replied, “No, not gonna come this far”. I was up all night on a project so this was not at the slightest bit interesting to me right now.

In my half alseep state however, each passing car started to sound like an approaching wave, crashing through the coconut trees and quickly tearing it’s way up the hill towards us. I decided to was time to wake up.

I started making my tea, listening to the radio. It was going on about the Tsunami warning and evacuations etc. There was a report about that the sea was retreating in the Yasawa’s. OK, sounds like this is a real disaster.

The first thing that came to mind was to see if I could get more up to date information online. I typed in http://twitter.com/ and did a search for “tsunami fiji“.

There were 2-3 updates every minute, and most of them stating the Tsunami warnings were already withdrawn. However, the radio was still going on with the warning. I didn’t really want to trust twitter solely, since most of it was just word of mouth.

I tried google, which would not have been useful in this situation a week ago. However, early this month they had implemented “search options” which allowed you to filter search results by date, showing the most recent results first.

Google proved to be up to date, with trusted information. Twitter had been just as, or maybe a few minutes ahead, but it took weeding through a few posts to finally get a trusted source.

It didn’t take long for Fiji Times to post an update on the cancelled Tsunami warning for Fiji. Which was immediately picked up by google, as well as twitter.

A few months ago I had the need to search for some very up to date information. Twitter provided the best source, in which google was quite useless. Now google seems to have noticed that they needed to provide realtime results. Twitter however still has the edge, with human interaction in near real-time and a wider range of resources. For example, if the Tsunami had actually hit, you could have watched it from around the world via Fiji Webcam link posted on twitter.

Now looking at this, I’m amazed at how close up to date information, especially those of large human interest such as disasters, is on the web now, mostly attributed to Twitter. Forget the radio, I’m doing a twitter search.

Allows you to convert to any base between 2 and 255, effectively using all the ASCII characters.

In order to convert very large numbers with arbitrary precision you’ll need the BCMath lib. Without BCMath the large numbers will not be converted correctly due to PHP not being able to do the arithmetic.

If you need to convert between bases 2-36, you can use the base_convert() function. However, converting to higher bases such as 255 has some benefits, such as “compressing” the characters.

You can successfully compress SHA1 from a 40 byte hex to a 20 byte string.

echo base255(base_convert(sha1('test'), 16, 10)));

Since there is no loss of data, it can be used as a lossless compression. Normal compression such as zlib won’t work on a SHA1 since there are no repeating patterns.