Friday, September 12, 2014

Node.js, MongoDB, JWT, BCrypt and authentication

If you write software for end-users today you need a really wide-ranging knowledge. One thing you need to deal with regularly is the question about where to save your data. It is very common to save data on a computer in the WAN or a LAN to make sure the data is available everywhere. You will need to distinguish different users. Your users will have to authenticate with a username and password combination. This is something we need so often in our daily work so that I want to describe a handy way to do so.

We are going to write an application to save favorite books for different users.
During this post we concentrate on writing an  API.
We use Node.js to provide an API in order to interact with our service. Node.js will be the glue between the network and a MongoDB to save users and book lists in JSON formated documents.

I assume that you have Node.js and MongoDB installed already.
If not then a quick google search should help you to do that quickly.

Create a directory books.
We start with a file named package.json that should look like this:

{
  "name": "books",
  "dependencies": {
    "express": "~4.8.7",
    "body-parser": "~1.7.0",
    "mongoose": "~3.8.15",
    "jwt-simple": "~0.2.0",
    "bcrypt": "~0.8.0"
  }
} 

We will write a powerfull API in Javascript using Node.js and express. Express will use body-parser as middle-ware to parse the body of posted data and put it into json formated objects. Mongoose provides a straight-forward, schema-based solution to modeling our application data. JWT provides the ability to encode and decode tokens and bcrypt will be used to hash our passwords.

Use /usr/bin/npm install to install all the dependencies.

We want to store our favorite books. Favorite books without a user who owns them are useless.
Let's express that in code.

Because we want to store our data in  MongoDB we need a way to connect to it.
Create a file named db.js with the following content:

var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/books', function () {
  console.log('mongodb connected to books');
});
module.exports = mongoose;

We use Mongoose to connect to our MongoDB on localhost to the database books.
The first time we write to the database it will be created if it does not exist.

Now that we can connect to the database we need to care about the data we want to save.
We want to store users. A user has a username and a password. Furthermore an array of favorite books is linked to a user. A user who authenticated successfully with a correct username and password combination will have read and write access to the array of favorite books linked to his user object stored in MongoDB. Subsequent requests to the service will send a valid token to differentiate authenticated from unauthenticated users.

Let's describe our idea of data representation in a file named datamodel.js.

var db = require('./db');
var Schema = db.Schema;
var BookSchema = new Schema({
        title: { type: String, required: true },
        category: { type: String, required: true },
        link: { type: String, required: true, default: "http://www.amazon.com" },
        date: { type: Date, required: true, default: Date.now }
});

var UserSchema = new Schema({
  username: { type: String, required: true },
  password: { type: String, required: true, select: false },
  booklist: { type: Array(BookSchema), required: false, default: [] }
});

var DataModel = {
        Book: db.model('Book' , BookSchema),
        User: db.model('User', UserSchema)
};

module.exports = DataModel;

This is worth to talk about it for a moment.
We defined our idea of a user and a book in a kind of blueprint called schema.
The ability to express it like that is offered by Mongoose.
We export two models, one for a book and one for a user combined in a DataModel.
Except two things the contents of the datamodel.js file are self explanatory.

The first thing is select: false in the definition of the user's password property. As mentioned before we will hand around a token as soon as a user is authenticated.
It will be simple to extract the username from the token and send it back to the user. With select: false as an option for the password property we make sure that the password won't be extractable. Although we store the password encrypted it is good practice to avoid to send the password back and forth more often than necessary. The findOne method of Mongoose /MongoDB will respect our wishes and return a user object without the password if a user exist for our token.

The second thing is the type definition for the booklist property in the UserSchema.
UserSchema has a sub-schema called BookSchema. Means that the booklist array can only include books that fulfill the BookSchema or nothing. Required is false because we want to be able to register a new user even if the user has no favorite book yet.

I really suggest to try out what we have so far. The models have a function save. If the data you want to store don't fulfill your defined model then the save-function will throw an error telling you what went wrong. You got a database backend and data validation with just a few lines of code. Here an example to try out before you go ahead.

peer@cdevel:~/web-stuff/books> /usr/bin/node
> var DataModel = require('./datamodel');
undefined
> mongodb connected to books
undefined
> var user = new DataModel.User();
undefined
> user.username = 'Peer';
'Peer'
> user.password = 'topsecret';
'topsecret'
> var book = new DataModel.Book();
undefined
> book.title = 'Junglebuch';
'Junglebuch'
> book.category = 'children';
'children'
> user.booklist.push(book);
1
> user.save();

With the data still in memory try to add something to the booklist that violates our rules
and save the user again. This will throw an exception:

> user.booklist.push({ foo: 'bar' , bar: 'baz'});
2
> user.save();
...
ValidationError: Path `category` is required., Path `title` is required.

So far so good. Now we need to define how to allow access to the data - our API .
  • /api/register - http method post to create a new user with username and password
  • /api/login - http method post to authenticate a user with username and password
    • send a token in case the user is valid
    • send an error in case the user is invalid
  • /api/user - for valid, authenticated users
    • method get to send a user record back
    • method post to save a user's record book list
We write Javascript code and use Node.js to execute it.
Create a new file called server.js.
First we write and discuss the API and test it with CURL.
At the end it is up to you to write a UI.
Put the following into you server.js file.

var express = require('express');
var bodyParser = require('body-parser');
var DataModel = require('./datamodel');
var bcrypt = require('bcrypt');
var jwt = require('jwt-simple');
var mysecret = require('./secret');

var app = express();
app.use(bodyParser.json());

app.post('/api/register', function(req, res, next) {
        var user = new DataModel.User();
        user.username = req.body.username;
        user.booklist = [] ;
        bcrypt.hash(req.body.password, 10, function (err, hash) {
                user.password = hash;
                user.save(function(err,user) {
                        if(err) { return(next(err)); }
                        res.status(200).send();
                });
        });
});

app.listen(8080, function () {
  console.log('Server listening on', 8080)
});

Let's concentrate on /api/register. If the method is POST and the URI is /api/register then we create a new empty user object offered by our data model. The body-parser makes the posted data available as Javascript object with properties inside req.body. We assign the username we got to our user object, assign an empty array to the booklist property and the encrypted password hash to the password property of our user object. Then we execute the "save" function of the user object. If you sent valid and well-formed data then you get a status 200 otherwise the response will be pretty detailed because we lack our own error handler function, something you want to avoid in production.

Here is the command to register a user using curl:

curl -X POST -d '{"username": "peer", "password": "passwd"}' \
 -H "Content-Type: application/json" localhost:8080/api/register

Use the MongoDB client to check the result:

/usr/bin/mongo
> use books
switched to db books
> db.users.find();
{ 
   "password" : "$2a$10$PpF6Ysdx2zRcjDay8hubmOu4IdpizjPNUELHk5yzcKYAdTCxrvpIK", 
   "username" : "peer", 
   "_id" : ObjectId("540cd784d1c95c703fcdd378"), 
   "booklist" : [ ], 
   "__v" : 0 
}

Next we care about how to login.
Create a file secret.js with the following content:

module.exports = {
    secret: 'Keep it private'
}

We will use it with jwt-simple in order to encrypt a token with the username.
Your server.js file should look like this now:

var express = require('express');
var bodyParser = require('body-parser');
var DataModel = require('./datamodel');
var bcrypt = require('bcrypt');
var jwt = require('jwt-simple');
var mysecret = require('./secret');

var app = express();
app.use(bodyParser.json());

app.post('/api/register', function(req, res, next) {
 var user = new DataModel.User();
 user.username = req.body.username;
 user.booklist = [] ;
 bcrypt.hash(req.body.password, 10, function (err, hash) {
  user.password = hash;
  user.save(function(err,user) {
   if(err) { return(next(err)); }
   res.status(200).send();
  });
 });
});

app.post('/api/login', function(req, res, next) {
 DataModel.User.findOne({username: req.body.username})
 .select('password').select('username')
 .exec(function (err, user) {
  if (err) { return next(err); }
  if (!user) { return res.status(401).send(); }
  bcrypt.compare(req.body.password, 
                               user.password, function (err, valid) {
   if (err) { return next(err); }
   if (!valid) { return res.status(401).send(); }
   var token = jwt.encode({
                                                   username: user.username
                                               }, mysecret.secret);
   res.status(200).send(token);
  });
 });
});

app.listen(8080, function () {
  console.log('Server listening on', 8080)
});

We expect a POST request for URI /api/login. We need a username and password to authenticate the user. In production you want to make sure that the transport of the clear text password happens via https. We use our data model of a user to look for the username in our MongoDB.
With .select('password').select('username') we ensure that the username and the encrypted password will be available if the user exists. You probably remember that we set select: false in our model.
If we found the user we compare the encrypted clear text password with the stored encrypted password. If this is a match then we create a token with jwt-simple that contains an encoded version of a JSON formated Javascript object with a username property set to the username. We send the token back with a status of 200. In case something fails we send a status 401 in order to signal a failure.

Try it with curl:

Get a valid token
curl -v -X POST -d '{"username": "peer", "password": "passwd"}' \
-H "Content-Type: application/json" localhost:8080/api/login

Get an error
curl -v -X POST -d '{"username": "peer", "password": "passw"}' \
-H "Content-Type: application/json" localhost:8080/api/login

Next we care about what to do with the token.
Create a file authtransform.js with the following content:

var jwt = require('jwt-simple');
var mysecret = require('./secret');

module.exports = function (req, res, next) {
    if (req.headers['x-auth']) {
        req.auth = jwt.decode(req.headers['x-auth'], mysecret.secret);
  
    }   
    next();
}

The reason for that is that we want to deal with the clear text username of authenticated users on server side. The way we use it in the server.js file will make sure that this piece of code is executed for each request. If we find a token then we decode it and hand it around in req.auth.

Next we want to be able to send a user object (from the perspective of Node.js to the user) if the user is authenticatet and valid.
Your server.js file should look like this now:

var express = require('express');
var bodyParser = require('body-parser');
var DataModel = require('./datamodel');
var bcrypt = require('bcrypt');
var jwt = require('jwt-simple');
var mysecret = require('./secret');

var app = express();
app.use(bodyParser.json());
app.use(require('./authtransform'));

app.post('/api/register', function(req, res, next) {
 var user = new DataModel.User();
 user.username = req.body.username;
 user.booklist = [] ;
 bcrypt.hash(req.body.password, 10, function (err, hash) {
  user.password = hash;
  user.save(function(err,user) {
   if(err) { return(next(err)); }
   res.status(200).send();
  });
 });
});

app.post('/api/login', function(req, res, next) {
 DataModel.User.findOne({username: req.body.username} })
 .select('password').select('username')
 .exec(function (err, user) {
     if (err) { return next(err); }
     if (!user) { return res.status(401).send(); }
     bcrypt.compare(req.body.password, user.password, function (err, valid) {
  if (err) { return next(err); }
  if (!valid) { return res.status(401).send(); }
         var token = jwt.encode({username: user.username}, mysecret.secret);
  res.status(200).send(token);
            });
 });
});

app.get('/api/user', function(req, res, next) {
 if (!req.auth) {
  return res.status(401).send();
 }

 DataModel.User.findOne({username: { $in: [req.auth.username] } }, function (err, user) {
  if (err) { return next(err); }
  res.status(200).json(user);
 });
});

app.listen(8080, function () {
  console.log('Server listening on', 8080)
});

This time we concentrate on the URI /api/user. If a valid, authenticated user send a GET request to that URI then req.auth is set to { username: user}. We try to find the data record for our user and send it back in json format or an error if there is no such user. Test it with curl like this and use the received token as TOKEN when you logged in:

curl -v -H 'x-auth: TOKEN' localhost:8080/api/user

As you can see we send our token in a header field called x-auth. The reason is that we don't want to rely on cookies. The API should be useful for a wide range of applications. If the test worked as expected then you received something like below:

{"username":"peer","_id":"540e2becf7c7648f444823eb","__v":0,"booklist":[]}

Now we perform the final step to finish writing the API. If our client application has a user object and want to store the book list of the user object then it sends a post request for URI /api/user.
Make sure your server.js file looks like below:

var express = require('express');
var bodyParser = require('body-parser');
var DataModel = require('./datamodel');
var bcrypt = require('bcrypt');
var jwt = require('jwt-simple');
var mysecret = require('./secret');

var app = express();
app.use(bodyParser.json());
app.use(require('./authtransform'));

app.post('/api/register', function(req, res, next) {
 var user = new DataModel.User();
 user.username = req.body.username;
 user.booklist = [] ;
 bcrypt.hash(req.body.password, 10, function (err, hash) {
  user.password = hash;
  user.save(function(err,user) {
   if(err) { return(next(err)); }
   res.status(200).send();
  });
 });
});

app.post('/api/login', function(req, res, next) {
 DataModel.User.findOne({username: req.body.username})
 .select('password').select('username')
 .exec(function (err, user) {
     if (err) { return next(err); }
     if (!user) { return res.status(401).send(); }
     bcrypt.compare(req.body.password, user.password, function (err, valid) {
  if (err) { return next(err); }
  if (!valid) { return res.status(401).send(); }
         var token = jwt.encode({username: user.username}, mysecret.secret);
  res.status(200).send(token);
            });
 });
});

app.get('/api/user', function(req, res, next) {
 if (!req.auth) {
  return res.status(401).send();
 }

 DataModel.User.findOne({username: { $in: [req.auth.username] } }, function (err, user) {
  if (err) { return next(err); }
  res.status(200).json(user);
 });
});

app.post('/api/user', function(req, res, next) {
        if (!req.auth) {
                return res.status(401).send();
        }

        DataModel.User.findOne({username: { $in: [req.auth.username] } }, function (err, user) {
                if (err) { return next(err); }
                user.booklist = req.body.booklist;
                user.save(function(err,user) {
                        if(err) { return(next(err)); }
                        res.status(200).json(user);
                });
        });
});

app.listen(8080, function () {
  console.log('Server listening on', 8080)
});

First we check if the authenticated user is still valid. If so, then we will find the user in the database. The client sent a user object and we replace the book list of the user we found in the database with the book list the client sent. Then we execute the save method of ther user object. If the user and the book list are valid according to our defined data model then the new record will be saved in the database and we send back the new user object to the client.

You can try it with curl like that:

curl -v -X POST -d '{"username":"peer","_id":"540e2becf7c7648f444823eb",\
"__v":0,"booklist":[{"title": "C Programming" , "category": "development"}]}' \
-H 'x-auth: TOKEN' -H "Content-Type: application/json" localhost:8080/api/user

If you sent a valid token then you got the new user object back, otherwise an error.
Check your MongoDB and figure out if your data arrived there as expected.

I hope this will give you a starting point for your own thought.