NodeJs in Visual Studio Code querying CouchDB, running on a Vagrant Linux Mint box

03.png

Introduction

Last post, I created a pretty nice Linux Mint 18 virtual image using Vagrant to run the Hyperledger-Fabric Docker demos. The host OS running that virtual image, my good old Windows 7 desktop, has 16 GB of RAM but it is still slow to boot (i.e. “time to go get a coffee”). On the other hand, I found that assigning 4GB out of that 16 GB of RAM to my Linux Mint image makes it so responsive, it actually feels faster than the host it’s running on.

In this post, I’m going to do some NodeJs server-side JavaScript scripts communicating with the non-relational database Apache CouchDB, which is used in Hyperledger-Fabric. CouchDB cannot be queried using SQL, it has Map/Reduce built in.  I’m going to create a few scripts and run them on Linux Mint using the free IDE Microsoft Visual Studio Code.

A free version of Visual Studio on Linux? Well, yes. Microsoft now provides a Linux .deb installer download, too. Nothing could be easier! Er, almost. A step by step install will be described below.

Assumptions

If you intent to follow along, the assumptions are:

  • You have followed all the steps to install Vagrant and Cygwin as described in my first post, right up to and including the section “Test Go in Cygwin”;
  • You installed Linux Mint 18 as described in my last post as described in the first section “Getting a Vagrant Linux Mint virtual image”;
  • You are comfortable using a Linux command prompt;
  • You don’t make typos — especially in the Map/Reduce queries because the update view scripts will save fine to CouchDB, but the queries will fail in the most baffling fashion.

Steps we will follow

  • Install the Visual Studio Code pre-requisites on Linux Mint
  • Install Visual Studio Code
  • Install NodeJs
  • Install Apache CouchDB
  • Set up the Visual Studio Code project
  • Create a first database, add a document to the database
  • Delete the document from the database, delete the database
  • Create a second database, add documents, update a document
  • Create a design view,  update the design view with a map
  • Query with only the map
  • Update the design view adding a reduce aggregator
  • Query the map/reduce view with a grouping
  • Update the design view to add a second, more complex map/reduce function
  • Query the map/reduce view with second level grouping

Install the Visual Studio Code pre-requisites on Linux Mint

It turns out the Visual Studio Code .deb installer has a hidden dependency on… Google Chrome.

Log in to Mint, open a terminal. Run commands:

sudo add-apt-repository “deb http://dl.google.com/linux/chrome/deb/ stable main”

wget -q -O – https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add  –

01.png

sudo apt-get update

sudo aptitude install google-chrome-stable

Accept warning with:

yes

02.png

OK, done.

Install Visual Studio Code

This Linux Mint image comes with Firefox pre-installed. Usually on a new Windows box I have to open Internet Explorer once to download and install Firefox.

Open Firefox, go to URL:

https://code.visualstudio.com/docs/?dv=linux64_deb

Select to Save to disk:

03.png

Click the arrow > right-click the .deb file > Open Containing Folder

04.png

In the Downloads dialog, double-click the .deb file. The Package Installer dialog opens. Click the “Install Package” button:

05.png

I find the conclusion of the Package Installer ambiguous, so I always check the “Automatically close after the changes have been successfully applied” checkbox:

06.png

Once closed, I assume it’s safe to close the Package Installer, and Firefox.

Install NodeJs

In a terminal, do the commands:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash –

sudo apt-get install nodejs

Next, we create a folder to work in under the /vagrant folder synchronized by Vagrant between the Windows host and the Linux Mint virtual image.

cd /vagrant

mkdir nodecode

cd nodecode

The next command will create our package.json for us:

npm init --yes

Actually, I got issues later on with npm not liking my generated my package.json file. In the end I ended up with the following:

19.png

I tried a few different approaches to connect to CouchDB from NodeJs and the one that had the simplest code and worked for me was with the “request” add-on. You install it this way:

cd /vagrant/nodecode

sudo npm install –no-bin-links request –save

20.png

Install Apache CouchDB

The database is easy to install on Linux Mint, with the command:

sudo apt-get install couchdb

Accept with:

Y

CouchDB is up and running after that, this can be tested by opening the CouchDB admin page (named Futon) in Firefox at URL:

http://localhost:5984/_utils/ 

08.png

Set up the Visual Studio Code project

In Linux Mint, open Visual Studio Code:

09.png

menu File > Open Folder… > open /vagrant/nodecode

10.png

11.png

The editor is ready:

12.png

We open the integrated terminal:

menu View > Integrated Terminal

13.png

And now we are ready to do some NodeJs server-side JavaScript code:

14.png

Create a first database, add a document to the database

Hover the mouse in Visual Studio Code over the folder name and click on the “create new file” icon:

15.png

We’re going to call the file add_first_db.js

16.png

and hit ENTER. Ready to type!

Here is our first NodeJs script to create our first CouchDB database named firstbd:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'firstdb/';

request.put(myurl + mydb, function (err, resp, body)
{
    if(err != null)
    {
        console.log('ERROR in add_first_db:', err);
    }

    console.log('statusCode:', resp && resp.statusCode);
    console.log('body:', body);
});

As you type, you have intellisense, which was a bit of a revelation for me after working with JavaScript in a text editor for a couple of decades:

17.png

Another nice feature is the upper left indicator to tell me the number of open files I need to save:

18.png

menu File > Save or CTRL-S which works too.

To run, type in the Integrated terminal:

node add_first_db

Success, our database “firstdb” was created:

21.png

“King of the world!” But let’s double-check anyway to be sure, open Firefox and go to Futon:

22.png

OK.

CouchDB does not have a schema, or tables, or primary keys, or foreign keys. You insert JSON documents. Let’s add one now.

Create a new file named “add_doc0.js” with:

15.png

Here is the code:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'firstdb/';
var myid = 'doc0';

request.put(
    {
        url: myurl + mydb + myid,
        body: { "firstname": "Popeye", "lastname": "Sailorman" },
        json: true
    },
    function (err, resp, body)
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc0: ', err);
        }

        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body);
    }
);

Run this in the Integrated Console with command:

node add_doc0

Success:

23.png

And in Futon, we click on the “firstdb” hyperlink, the document is there:

24.png

If we click on the red “doc0” hyperlink in Futon, we can view the document JSON in the Source tab:

25.png

Delete the document from the database, delete the database

To update or delete a document in CouchDB, you need to retrieve its “_rev” revision number. Therefore, our next script will first get the document, extract the revision number through JSON object notation, then will do another call to delete. Script will be called get_and_delete_doc0.js and here it is:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'firstdb/';
var myid = 'doc0';
var myrevision = '?rev=';

request(myurl + mydb + myid, function ()
{
    request(myurl + mydb + myid, function (error, response, body) 
    {
        if(error != null)
        {
            console.log('error:', error);
            console.log('statusCode:', response && response.statusCode); 
            console.log('body:', body);
        }
        else
        {
            // extract the revision number
            var obj = JSON.parse(body);
            console.log("Revision of doc was " + obj._rev);
            myrevision += obj._rev;

            // delete doc0
            request.delete(
            {
                url: myurl + mydb + myid + myrevision
            },
            function (ERROR, response, bod)
            {
                if(error != null)
                {
                    console.log('ERROR occurred in delete_doc0: ', error);
                }

                console.log('statusCode:', response && response.statusCode);
                console.log('body:', bod);
            }); // end function
        } // end else
    });
});

Don’t forget to save.

Run this in the Integrated Terminal with:

node delete_doc0

Success:

26.png

In Futon, our database “firstdb” no longer has a document “doc0“:

27.png

Finally we create a new script called delete_db_firstdb.js

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'firstdb';

request.delete(
{
    url: myurl + mydb
    },
    function (err, resp, body)
    {
        if(err != null)
        {
            console.log('ERROR occurred in delete_db_firstdb: ', err);
        }

        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body);
    }
);

Result is successful after we save and run the command:

node delete_db_firstdb

28.png

Nothing left of “firstdb” in Futon:

30.png

Create a second database, add documents, update a document

Here’a s new script called “add_db_dbtwo.js” to create a second database named “dbtwo“:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';

request.put(myurl + mydb, function (err, resp, body)
{
    if(err != null)
    {
        console.log('ERROR in add_db_dbtwo:', err);
    }

    console.log('statusCode:', resp && resp.statusCode);
    console.log('body:', body);
});

When we run it:

32.png

33.png

Next, I create six new scripts to populate database dbtwo. Here they are:

Script “add_doc1.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc1'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: 
        { 
            "firstname": "Steve", 
            "lastname": "Canyon", 
            "street": "123 Skyways Park",
            "city": "Portland",
            "state": "Oregon",
            "sex": "male"
        },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc1: ', err); 
        }
        
        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body); 
    }
);

Script “add_doc2.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc2'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: 
        { 
            "firstname": "Pat", 
            "lastname": "Ryan", 
            "street":"23 Mao Way", 
            "city": "Shanghai",
            "sex": "male"
         },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc2: ', err); 
        }
        
        console.log('statusCode:', resp && resp.statusCode); 
        console.log('body:', body);
    }
);

Script “add_doc3.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc3'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: { "firstname": "Dick", 
                "lastname": "Tracy", 
                "street": "435 North Michigan Avenue", 
                "city": "Chicago", 
                "state": "Illinois",
                "sex": "male"
        },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc3: ', err);
        }
        
        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body);
    }
); 

Script “add_doc4.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc4'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: 
        { 
            "firstname": "Dragon", 
            "lastname": "Lady", 
            "street":"24 Mao Way", 
            "city": "Shanghai",
            "sex": "female"
         },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc4: ', err); 
        }
        
        console.log('statusCode:', resp && resp.statusCode); 
        console.log('body:', body);
    }
);

Script “add_doc5.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc5'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: { "firstname": "Olive", 
                "lastname": "Oyl", 
                "street": "1 Sweapea Lane", 
                "city": "Portland", 
                "state": "Oregon",
                "sex": "female"
        },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc5: ', err);
        }
        
        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body);
    }
);

Script “add_doc6.js“:

 
var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc6'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: { "firstname": "Brutus", 
                "lastname": "Manhandler", 
                "street": "5 Sweapea Lane", 
                "city": "Portland", 
                "state": "Oregon",
                "sex": "male"
        },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_doc6: ', err);
        }
        
        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body);
    }
);

When we run them:

34.png

Let’s now update doc3 and give Dick Tracy the occupation of detective with script “update_doc3.js“:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = 'doc3'
var myrevision = '?rev=';

// get doc3
request(myurl + mydb + myid, function () 
{
    request(myurl + mydb + myid, function (error, response, body) 
    {
        if(error != null)
        {
            console.log('ERROR occurred in get_and_update_doc3 (get part):', error);
            console.log('statusCode:', response && response.statusCode);
            console.log('body:', body);
        }
        else
        {
            // extract the revision number of doc3
            var obj = JSON.parse(body);
            console.log("Revision of doc3 was " + obj._rev);
            myrevision += obj._rev;

            // update doc3
            request.put(
            { 
                url: myurl + mydb + myid + myrevision, 
                body: { 
					"firstname": "Dick", 
					"lastname": "Tracy", 
					"street": "435 North Michigan Avenue", 
					"city": "Chicago", 
					"state": "Illinois",
					"sex": "male",
					"occupation": "Detective"
                },
                json: true
            },  
            function (err, resp, body) 
            {
                if(err != null)
                {
                  console.log('ERROR in get_and_update_doc3 (update part): ', err); 
                }
                
                console.log('statusCode:', resp && resp.statusCode);
                console.log('body:', body);
            }); // end put
        } // end else
    });
});

Result is successful:

35.png

In Futon:

36.png

Looking good!

Create a design view,  update the design view with a map

First, we create an design view called ‘_design/query‘ with script “add_design_view.js“:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query'

request.put(
    { 
        url: myurl + mydb + myid, 
        body: 
        { 
            "_id": myid
        },
        json: true
    },  
    function (err, resp, body) 
    {
        if(err != null)
        {
            console.log('ERROR occurred in add_design_view: ', err); 
        }
        
        console.log('statusCode:', resp && resp.statusCode);
        console.log('body:', body); 
    }
);

Result:

37.png

In Futon we have:

38.png

Next we update the design view by adding a map function with script “get_and_update_design_view.js“:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query'
var myrevision = '?rev=';

request(myurl + mydb + myid, function () 
{
    request(myurl + mydb + myid, function (error, response, body) 
    {
        if(error != null)
        {
            console.log('ERROR in get_and_update_design_view (get):', error);
            console.log('statusCode:', response && response.statusCode);
            console.log('body:', body);
        }
        else
        {
            // extract the revision number
            var obj = JSON.parse(body);
            console.log("Revision was " + obj._rev);
            myrevision += obj._rev;

            // update 
            request.put(
            { 
                url: myurl + mydb + myid + myrevision, 
                body: 
                {
                    "_id": myid,
                    "views": 
                    {
                        "city":
                        {
                            "map": "function(doc) {if(doc.city) emit(doc.city, 1)}"
                        }
                    } 
                },
                json: true
            },  
            function (err, resp, body) 
            {
                if(err != null)
                {
                 console.log('ERROR in get_and_update_design_view (update): ', err); 
                }
                
                console.log('statusCode:', resp && resp.statusCode);
                console.log('body:', body);
            }); // end put
        } // end else
    });
});

If we run it we get:

39.png

Query with only the map

Let us now query our database to list documents that have the “city” element, with script “run_design_view.js” with no grouping (notice the code in blue):

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query/_view/';
var mysearchtype = 'city';

request(myurl + mydb + myid + mysearchtype, function () 
{
    request(myurl + mydb + myid + mysearchtype, function (error, response, body) {
        if(error != null)
        {
            console.log('ERROR in run_design_view');
        }
        
        console.log('statusCode:', response && response.statusCode);
        console.log('body:', body);
    });
});

Result:

40.png

Update the design view adding a reduce aggregator

Let’s now add a reduce function to our map with script “get_and_update_design_view2.js“:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query'
var myrevision = '?rev=';

request(myurl + mydb + myid, function () 
{
    request(myurl + mydb + myid, function (error, response, body) 
    {
        if(error != null)
        {
            console.log('ERROR in get_and_update_design_view2 (get part):', error);
            console.log('statusCode:', response && response.statusCode);
            console.log('body:', body);
        }
        else
        {
            // extract the revision number 
            var obj = JSON.parse(body);
            console.log("Revision of doc4 was " + obj._rev);
            myrevision += obj._rev;

            // update
            request.put(
            { 
                url: myurl + mydb + myid + myrevision, 
                body: 
                {
                    "_id": myid,
                    "views": 
                    {
                        "city":
                        {
                            "map": "function(doc) {if(doc.city) emit(doc.city, 1)}",
                            "reduce": "function(keys,values){ return sum(values); }"
                        }
                    } 
                },
                json: true
            },  
            function (err, resp, body) 
            {
                if(err != null)
                {
                 console.log('ERROR in get_and_update_design_view2 (update): ', err); 
                }
                
                console.log('statusCode:', resp && resp.statusCode);
                console.log('body:', body);
            }); // end put
        } // end else
    });
});

When we run this:

43.png

In Futon:

44.png

Query the map/reduce view with a grouping

Let us now query our database for documents with a city element aggregated this time with script “run_design_view2.js” with grouping (notice the code in purple):

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query/_view/';
var mysearchtype = 'city?group=true';

request(myurl + mydb + myid + mysearchtype, function () 
{
    request(myurl + mydb + myid + mysearchtype, function (error, response, body) {
        if(error != null)
        {
            console.log('ERROR in run_design_view2');
        }
        
        console.log('statusCode:', response && response.statusCode);
        console.log('body:', body);
    });
});

We get result:

45.png

So, same results as before, but grouped. We have one document with the city “Chicago”, three with the value “Portland”, and Pat Ryan and the Dragon Lady live a couple of doors down from each other in Shanghai.

Update the design view to add a second, more complex map/reduce function

Next, we add a second view called “demographic”. We will group our returned documents by city and sex. The reduce function (in purple) will have the same effect as the other view’s but will shorter syntax. Here is the “get_and_update_design_view3.js” script:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query'
var myrevision = '?rev=';

request(myurl + mydb + myid, function () 
{
    request(myurl + mydb + myid, function (error, response, body) 
    {
        if(error != null)
        {
            console.log('ERROR in get_and_update_design_view3 (get part):', error);
            console.log('statusCode:', response && response.statusCode);
            console.log('body:', body);
        }
        else
        {
            // extract the revision number 
            var obj = JSON.parse(body);
            console.log("Revision of doc4 was " + obj._rev);
            myrevision += obj._rev;

            // update
            request.put(
            { 
                url: myurl + mydb + myid + myrevision, 
                body: 
                {
                    "_id": myid,
                    "views": 
                    {
                        "city":
                        {
                            "map": "function(doc) {if(doc.city) emit(doc.city, 1)}",
                            "reduce": "function(keys,values){ return sum(values); }"
                        },
                        "demographic":
                        {
               "map": "function(doc) {if(doc.city) emit([doc.city, doc.sex], 1)}",
               "reduce": "_count"
                        }
                    } 
                },
                json: true
            },  
            function (err, resp, body) 
            {
                if(err != null)
                {
                 console.log('ERROR in get_and_update_design_view3 (update): ', err); 
                }
                
                console.log('statusCode:', resp && resp.statusCode);
                console.log('body:', body);
            }); // end put
        } // end else
    });
});

When we run the script:

01.png

Query the map/reduce view with second level grouping

Next we create a new query to list the “demographic” documents:

var request = require('request')

var myurl = 'http://127.0.0.1:5984/';
var mydb = 'dbtwo/';
var myid = '_design/query/_view/';
var mysearchtype = 'demographic?group=true;group_level=2';

request(myurl + mydb + myid + mysearchtype, function () 
{
    request(myurl + mydb + myid + mysearchtype, function (error, response, body) {
        if(error != null)
        {
            console.log('ERROR in run_design_view');
        }
        
        console.log('statusCode:', response && response.statusCode);
        console.log('body:', body);
    });
});

When we run this, we can see our grouping now discriminates between the sexes:

02.png

(end of post)

Bertrand Szoghy, June 2017.

Advertisements

One thought on “NodeJs in Visual Studio Code querying CouchDB, running on a Vagrant Linux Mint box

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s