Mindtravel.nl Just a blog about Coding, Gaming and sports

Offline npm packages

local-npm is a Node server that acts as a local npm registry. It serves modules, caches them, and updates them whenever they change.

Last weekend I was on a plane flying to Lisbon and sometimes its handy to have a local cache of npm packages. This could be being on a flight or during a workshop. Getting this working is pretty simple.

local-npm

Using local-npm is like using a local npm mirror without a complete replication. It's not as big as sinopia where you can also have private repo's.

Your npm installs are fetched from npmjs and then modules and their deps are stored in a local PouchDB. It also takes care of keeping modules updated when they change.

To get local-npm installed, run:

# npm install -g local-npm
# local-npm

local-npm replicates the skimdb part of the database and start replicating. You don't have to wait for it to be at 100%, you can already start using the registry. It will fallback to the online version for any library that isn't in it's proxy yet.

To complete setup, you need to set npm to point to the local server local-npm:

# npm set registry http://127.0.0.1:5080

The local-npm's server setup also has a simple UI for browsing cached modules and searching for them. You can access this at http://localhost:5080/_browse.

npmrc

Switch between different .npmrc files with ease and grace

It's handy to have a local proxy, but sometimes its easier to just use a live one or use the private repo from work.
If that's the case you know it's annoying to do the switching between a bunch of different .npmrc files and manually managing symlinks.

npmrc is a handy tool to save the day. It will switch your .npmrc with a specific named version.

Installation:

# npm install -g npmrc

First time you run it, it will create the links:

# npmrc
Initialising npmrc...  
Creating /Users/leon/.npmrcs  
Making /Users/leon/.npmrc the default npmrc file  
Activating .npmrc "default"  

Lets create the local-proxy profile:

# npmrc -c local-proxy
Removing old .npmrc (default)  
Activating .npmrc "proxy"  

A blank profile will be created. To point your profile to the local registry and start using it:

# npm set registry http://127.0.0.1:5080

and switching back to the default is as simple as just using the alias:

# npmrc default

It also has some nice generic npmjs repos as an alias:

# npmrc -r eu
Using http://registry.npmjs.eu/ registry.  

For more options check the help.

Running softether vpn on smartos

An Open-Source Free ​Cross-platform Multi-protocol VPN Program.

Getting SoftEther running on Smartos in a branded zone isn't that difficult, it's just a lot of work :)

Downloading and compiling

Import debian 8:

# imgadm import 445d04f4-cad6-11e5-a1a0-9f6c0ce02707 

And create a nice lx branded zone:

{
  "brand": "lx",
  "image_uuid": "445d04f4-cad6-11e5-a1a0-9f6c0ce02707",
  "kernel_version": "3.13.0",
  "autoboot": true,
  "alias": "lxvpn",
  "hostname": "lxvpn",
  "dns_domain": "mindtravel.nl",
 "nics": [
  {
    "nic_tag": "admin",
    "ip": "192.168.1.123",
    "netmask": "255.255.255.0",
    "gateway": "192.168.1.1"
  }
 ],
  "resolvers": [
    "192.168.1.5"
  ],
 "max_physical_memory": 512,
 "quota": 10
}

Make sure to update the system:

# apt-get update

and install the basic stuff to be able to compile stuff:

# apt-get install build-essential -y

You can not find softether as a .deb so I downloaded the install package directly from the website using lynx

# apt-get install lynx

Now browse to the download website and download the file softether-vpnserver-v4.18-9570-rtm-2015.07.26-linux-x64-64bit.tar.gz by selecting the file and press D to save it.

# lynx http://www.softether-download.com/files/softether/v4.18-9570-rtm-2015.07.26-tree/Linux/SoftEther_VPN_Server/64bit_-_Intel_x64_or_AMD64/

Unpack the file:

# tar zxfv softether-vpnserver-v4.18-9570-rtm-2015.07.26-linux-x64-64bit.tar.gz

Time to build the server code and press a couple of times 1 for accepting some stuff and move it to a nice location:

# cd vpnserver
# make
# cd ..
# mv vpnserver /usr/local
# cd /usr/local/vpnserver/
# chmod 600 *
# chmod 700 vpnserver
# chmod 700 vpncmd

Create a nice service file:

# vi /etc/init.d/vpnserver

with the content:

#!/bin/sh
# chkconfig: 2345 99 01
# description: SoftEther VPN Server
DAEMON=/usr/local/vpnserver/vpnserver  
LOCK=/var/lock/subsys/vpnserver  
test -x $DAEMON || exit 0  
case "$1" in  
start)  
$DAEMON start
touch $LOCK  
;;
stop)  
$DAEMON stop
rm $LOCK  
;;
restart)  
$DAEMON stop
sleep 3  
$DAEMON start
;;
*)
echo "Usage: $0 {start|stop|restart}"  
exit 1  
esac  
exit 0  

Start the server and make it start on boot:

# chmod 755 /etc/init.d/vpnserver && /etc/init.d/vpnserver start
# update-rc.d vpnserver defaults

Configuring the server

Start the cli client and configure the server password:

# ./vpncmd
// and select option 1

And type:

ServerPasswordSet  

To use Softether we need to setup a virtual hub. I named it VPN.
Lets create a VirtualHub and select it:

HubCreate VPN  
Hub VPN  
SecureNatEnable  
bridgecreate VPN /DEVICE:soft /TAP:yes  

The last command sets the way it can connect to the internal network.
Create a user test and set a password:

UserCreate test  
UserPasswordSet test  

I use an OSX laptop to connect. For this case you need to enable L2TP/IPsec with the following command:

IPsecEnable  

The system will ask some questions, just press YES.
Also you will need to create a pre-shared key, used to connect if you are not using a cert to connect to the server.

Make sure you have the UDP port 500 and 4500 open to the server and your done and ready to connect with the user test to your vpn :D

Setting up your OSX client is very easy, just check this page

Running Stackstorm on smartos

StackStorm is a powerful automation tool that wires together all of your apps, services and workflows.

Running Stackstorm on Smartos in a branded zone was a bit more work than I expected but in the end its working like a charm :)

Basic installation

Import ubuntu 14.04, I couldn't get the debian 8 to work (had problems with postgresql) :(

# imgadm import 5cdc6dde-d6ad-11e5-8b11-8337e6f86725

And create a nice lx branded zone:

{
  "brand": "lx",
  "image_uuid": "5cdc6dde-d6ad-11e5-8b11-8337e6f86725",
  "kernel_version": "3.13.0",
  "autoboot": true,
  "alias": "lxstackstorm",
  "hostname": "lxstackstorm",
  "dns_domain": "mindtravel.nl",
 "nics": [
  {
    "nic_tag": "admin",
    "ip": "192.168.1.84",
    "netmask": "255.255.255.0",
    "gateway": "192.168.1.1"
  }
 ],
  "resolvers": [
    "192.168.1.5"
  ],
 "max_physical_memory": 4096,
 "quota": 20
}

install dependencies (and grab a coffee):

# apt-get update
# apt-get install -y apt-transport-https mongodb-server rabbitmq-server postgresql

setup the repositories:

# wget -qO - https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -
# echo "deb https://dl.bintray.com/stackstorm/trusty_staging stable main" | sudo tee /etc/apt/sources.list.d/st2-staging-stable.list
# apt-get update

time to install the main package (and grab more coffee):

# apt-get install -y st2 st2mistral

config the database:

## Create Mistral DB in PostgreSQL
# cat << EHD | sudo -u postgres psql
CREATE ROLE mistral WITH CREATEDB LOGIN ENCRYPTED PASSWORD 'StackStorm';  
CREATE DATABASE mistral OWNER mistral;  
EHD  

you can ignore the following errors:

shell-init: error retrieving current directory: getcwd: cannot access parent directories: Permission denied  
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: Permission denied  
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: Permission denied  
could not identify current directory: Permission denied  
could not identify current directory: Permission denied  
## Setup Mistral DB tables, etc.
# /opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
## Register mistral actions
# /opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate

Again ignoring the warnings.

Stackstorm uses a user called stanley to do cmd stuff. So we will need to add it to sudo:

# useradd stanley
# mkdir -p /home/stanley/.ssh
# chmod 0700 /home/stanley/.ssh
# ssh-keygen -f /home/stanley/.ssh/stanley_rsa -P ""
# cat /home/stanley/.ssh/stanley_rsa.pub >> /home/stanley/.ssh/authorized_keys
# chmod 0600 /home/stanley/.ssh/authorized_keys
# chown -R stanley:stanley /home/stanley
# echo "stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL" >> /etc/sudoers.d/st2

Time to start the stackstorm service and afterwards restart and register all the packs

# st2ctl start
# st2ctl reload --register-all

Setup a user

Time to create a user with a password

# apt-get install -y apache2-utils
# echo "Ch@ngeMe" | htpasswd -i /etc/st2/htpasswd test

edit the /etc/st2/st2.conf:

[auth]
# ...
enabled = True  

Frontend installation

By default the nginx frontend is not installed.
So we install nginx and remove the default server and replace it with the default that comes with the stackstorm package

# apt-get install -y st2web nginx
# mkdir -p /etc/ssl/st2
# openssl req -x509 -newkey rsa:2048 -keyout /etc/ssl/st2/st2.key -out /etc/ssl/st2/st2.crt \
-days XXX -nodes -subj "/C=US/ST=California/L=Palo Alto/O=StackStorm/OU=Information \
Technology/CN=$(hostname)"

# rm /etc/nginx/sites-enabled/default
# cp /usr/share/doc/st2/conf/nginx/st2.conf /etc/nginx/sites-available/
# ln -s /etc/nginx/sites-available/st2.conf /etc/nginx/sites-enabled/st2.conf
# service nginx restart

Installing and getting the Bot running

The documentation of Stackstorm describes that it runs the hubot in a docker container. Unfortunately the lxbrand doesnt support Docker, but the bot is a nice nodejs project so we will run it native.

Lets make sure that stackstorm can handle hubot:

# st2 run packs.install packs=hubot
# st2ctl reload --register-rules

The nodejs part:

# apt-get install build-essential redis-server
# curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
# apt-get install nodejs
# npm install -g hubot coffee-script yo generator-hubot

Time to create the bot itself:

# mkdir -p /opt/hubot
# chown stanley:stanley /opt/hubot
# sudo -H -u stanley bash -c 'cd /opt/hubot && echo "n" | yo hubot --name=stanley --description="Stanley StackStorm bot" --defaults'
# sudo -H -u stanley bash -c 'cd /opt/hubot && npm install hubot-slack hubot-stackstorm --save'
# sudo -H -u stanley sed -i 's/.*\[.*/&\n  "hubot-stackstorm",/' /opt/hubot/external-scripts.json

Create a service to start the bot like this:

cat /etc/init/hubot.conf

# hubot - HUBOT
#

description "It's a Hubot!"

start on filesystem or runlevel [2345]  
stop on runlevel [!2345]  
respawn  
respawn limit 5 60

script  
# Start Hubot
exec /opt/hubot/start.sh  
end script  

and the start file (dont forget to chmod 755)

cat /opt/hubot/start.sh

#!/bin/bash

cd /opt/hubot  
ST2_AUTH_USERNAME=testu ST2_AUTH_PASSWORD=Ch@ngeMe ST2_AUTH_URL=http://localhost:9100 HUBOT_SLACK_TOKEN="xoxb-16090510417-aslacktoken" PORT=8181 bin/hubot --name "stanley" --adapter slack --alias !  

start with a simple service hubot start
In the future I will explain how to create a pack to use with the bot.

First steps into Kong on Smartos, the easy way :)

Kong allows developers to reduce complexity and deployment times in implementing an API solution on NGINX.

Kong is a Lua application running in Nginx. It can take care of all the stuff we always need to redo. Like throttling, oauth, ssl and more. Hoping to add more blog posts about how to add plugins and use them.

Kong depends heavy on Lua and that's where the problem with Smartos comes in. For now I decided to do the easy way with a simple debian zone.

Running Kong on Smartos in a branded zone.

Import debian 8:

# imgadm import 445d04f4-cad6-11e5-a1a0-9f6c0ce02707

And create a nice lx branded zone:

{
  "brand": "lx",
  "image_uuid": "445d04f4-cad6-11e5-a1a0-9f6c0ce02707",
  "kernel_version": "3.13.0",
  "autoboot": true,
  "alias": "lxkong",
  "hostname": "lxkong",
  "dns_domain": "mindtravel.nl",
 "nics": [
  {
    "nic_tag": "admin",
    "ip": "192.168.1.65",
    "netmask": "255.255.255.0",
    "gateway": "192.168.1.1"
  }
 ],
  "resolvers": [
    "192.168.1.5"
  ],
 "max_physical_memory": 2048,
 "quota": 20
}

Install the needed packages:

# apt-get install netcat openssl libpcre3 dnsmasq procps

Ignore the following, after a reboot all is fine.
Fyi, I use dnsmasq from another zone.

Setting up dnsmasq (2.72-3+deb8u1) ...  
Job for dnsmasq.service failed. See 'systemctl status dnsmasq.service' and 'journalctl -xn' for details.  
invoke-rc.d: initscript dnsmasq, action "start" failed.  

You will need java to run cassandra. Here is how to add java8 to debian:

# echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list
# echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
# apt-get update
# apt-get install oracle-java8-installer

Make sure you run 2.xx cassandra, not 3.xx

# wget http://apache.cs.uu.nl/cassandra/2.2.5/apache-cassandra-2.2.5-bin.tar.gz
# tar zxfv apache-cassandra-2.2.5-bin.tar.gz
# cassandra/bin/cassandra

Install Kong itself, you can find the download link on https://getkong.org/install/debian/

# dpkg -i ./jessie_all.deb

Start Kong:

# kong start
# kong status
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong is running
# Kong is running
# curl 127.0.0.1:8001

Yep, its that simple.

How to install couchbase on smartos

It took me some time to get it working, but this is how i did it :)

[root@00-a0-b0-1d-c3-d4 /opt/zones]# cat couchbase.json
{
  "alias": "couchbase",
  "hostname": "couchbase",
  "brand": "joyent",
  "max_physical_memory": 4096,
  "quota": 25,
  "dataset_uuid": "9eac5c0c-a941-11e2-a7dc-57a6b041988f",
  "dns_domain": "mindtravel.nl",
  "resolvers": [
    "192.168.1.5"
  ],
  "nics": [
    {
      "nic_tag": "external",
      "ip": "192.168.1.52",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.254"
    }
  ]
}

Make sure you have enough memory, at least 2gig as maxphysicalmemory otherwise you will get some weird compile errors.

Login on the vm and install the following packages:

pkgin -y in libtool-base autoconf automake scmgit-base gcc47 gnupg gmake libevent icu py27-expat snappy erlang-15.1.3.1 subversion-base  
wget -O /opt/local/bin/repo http://commondatastorage.googleapis.com/git-repo-downloads/repo  
chmod a+x /opt/local/bin/repo  
mkdir /opt/couchbase  

Install gperftools:

wget --no-check-certificate https://gperftools.googlecode.com/files/gperftools-2.1.tar.gz  
gtar xfz gperftools-2.1.tar.gz  
cd gperftools-2.1  
./configure --enable-minimal --enable-shared --disable-static --prefix=/opt/couchbase
gmake install  
cd ..  

Next is install V8:

wget --no-check-certificate -Ov8.tar.gz https://github.com/v8/v8/archive/3.19.0.tar.gz  
gtar xfz v8.tar.gz  
cd v8-3.19.0  
gmake dependencies  
gmake x64 library=shared -j 4  
cp out/x64.release/lib.target/libv8.so /opt/couchbase/lib  
cp include/* /opt/couchbase/include/  
cd  

Add some extra Makefile options:

mkdir -p ~/.couchbase/build  
cat > ~/.couchbase/build/Makefile.extra  
OPTIONS += CPPFLAGS="-I$(PREFIX)/include -m64"  
OPTIONS += LDFLAGS="-R/opt/local/lib -L$(PREFIX)/lib -R$(PREFIX)/lib -m64"  
OPTIONS += CXX="g++ -m64 -L/opt/local/lib -I/opt/local/include"  
OPTIONS += CC="gcc -m64 -I/opt/local/include -L/opt/local/lib"  
memcached_EXTRA_OPTIONS +=  --enable-tcmalloc-minimal  

Warning, looks like my blog changes the output above:
it should be memcached_EXTRA_OPTIONS

Checkout couchbase version 2.2.0

mkdir compile  
cd compile  
repo init -u git://github.com/membase/manifest.git -m released/2.2.0.xml  
repo sync  

We need to patch one file:

vi ep-engine/src/couch-kvstore/couch-kvstore.cc  

and change line 524 into:

if (access(fname.c_str(), F_OK) == -1) {  

finale steps:

rm /opt/local/lib/libsnappy.la /opt/local/lib/libevent.la  
gmake PREFIX=/opt/couchbase  

Create a user couch and:

chown couch /opt/couchbase  

After that you can just test the build with:

su - couch  
cd /opt/couchbase/bin  
./couchbase-server

Open you browser to http://installedhost:8091/index.html
Party time! :)

Thanx go to http://trondn.blogspot.no/2013/08/running-couchbase-211-on-smartos.html for the inspitation

Older Posts