MongoDB Design and Docker

rfcvg3

MongoDB Design and Docker

I am design my MongoDB schema right now. Actually, I just set up some JSON format model there.
I learn something from articles from google. One document can not be over 16m, so I only put few embed objects in one model, only IDs if there is hundreds, use one to many when there is thousands or more.

1 Recall Mongo Knowledge (The things I will use)
SQL Statement                       Mongo Statement
alter table users add …

select a, b from users              db.users.find({}, {a:1, b:1})

2 Set up Mongo Cluster
Based on the documents from meteor, we may only do Replica Set there at first.

Set Up the Docker Information
#Prepre the OS
FROM            ubuntu:14.04
MAINTAINER      Carl Luo <luohuazju@gmail.com>

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade

#Install MongoDB
RUN             mkdir /tool/
WORKDIR         /tool/

ADD             install/mongodb-linux-x86_64-ubuntu1404-3.2.0.tgz /tool/
RUN             ln -s /tool/mongodb-linux-x86_64-ubuntu1404-3.2.0 /tool/mongodb
ENV   PATH  /tool/mongodb/bin:$PATH

#Define moutable directories
RUN             mkdir -p /data/mongodb
RUN             mkdir /logs/
#VOLUME ["/data/", "/logs/"]

#Expose ports for Mongo replica set
EXPOSE 27017

#Copy configuration files
COPY conf/ /conf/

#Start the Application
RUN     mkdir -p /app/
ADD     start.sh /app/
WORKDIR /app/
CMD [ "./start.sh" ]

Makefile
IMAGE=sillycat/public
TAG=ubuntu-mongo-3.2
NAME=ubuntu-mongo-3.2
#REPOSITORY=sillycat.ddns.net

docker-context:

#tag-local:
# docker tag $(IMAGE):$(TAG) $(REPOSITORY)/$(IMAGE):$(TAG)

#push-local:
# docker push $(REPOSITORY)/$(IMAGE):$(TAG)

prepare:
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1404-3.2.0.tgz -P install/

install:
docker pull sillycat/public:ubuntu-mongo-3.2
mkdir -p data/mongodb
mkdir logs

build: docker-context
# sudo docker build -t $(REPOSITORY)/$(IMAGE):$(TAG) .
# sudo docker build --no-cache -t $(IMAGE):$(TAG) .
sudo docker build -t $(IMAGE):$(TAG) .

run:
docker run -d --name $(NAME) -p 27017:27017 -v $(shell pwd)/data:/data -v $(shell pwd)/logs:/logs $(IMAGE):$(TAG)

debug:
docker run -ti -p 27017:27017 --name $(NAME) -v $(shell pwd)/data:/data -v $(shell pwd)/logs:/logs $(IMAGE):$(TAG) /bin/bash

clean:
docker stop ${NAME}
docker rm ${NAME}

logs:
docker logs ${NAME}

publish:
docker push ${IMAGE}

fetch:
docker pull ${IMAGE}


carl@ubuntu-build:~/work/sillycat-docker/ubuntu-mongo$ cat conf/mongod.conf
# mongod.conf

# Where to store the data.

# Note: if you run mongodb as a non-root user (recommended) you may
# need to create and set permissions for this directory manually,
# e.g., if the parent directory isn't mutable by the mongodb user.
dbpath=/data/mongodb/

#where to log
logpath=/logs/mongodb-db.log

# process id
fork = true
pidfilepath = /var/run/mongodb-db.pid

logappend=true

port=27017
smallfiles=true

# Listen to local interface only. Comment out to listen on all interfaces.
#bind_ip = 127.0.0.1

# Disables write-ahead journaling
# nojournal = true

# Enables periodic logging of CPU utilization and I/O wait
#cpu = true

# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true

# Verbose logging output.
#verbose = true

# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck = true

# Enable db quota management
#quota = true

# Set oplogging level where n is
#   0=off (default)
#   1=W
#   2=R
#   3=both
#   7=W+some reads
#diaglog = 0

# Ignore query hints
#nohints = true

# Enable the HTTP interface (Defaults to port 28017).
#httpinterface = true

# Turns off server-side scripting.  This will result in greatly limited
# functionality
#noscripting = true

# Turns off table scans.  Any query that would do a table scan fails.
#notablescan = true

# Disable data file preallocation.
#noprealloc = true

# Specify .ns file size for new databases.
# nssize = <size>

# Replication Options

# in replicated mongo databases, specify the replica set name here
replSet=sillycatMongoReplica
# maximum size in megabytes for replication operation log
#oplogSize=1024
# path to a key file storing authentication info for connections
# between replica set members
#keyFile=/path/to/keyfile

Login My Docker Registry
> sudo docker login
Username: sillycat
Password:
Email: luohuazju@gmail.com
WARNING: login credentials saved in /home/carl/.docker/config.json
Login Succeeded

Command to Pull Out the Things
> docker pull sillycat/public

Or with tag
> docker pull sillycat/public:ubuntu-mongo-3.2

Running things on top of ReplicaSet

Set Up Time Zone and Time Sync
> sudo dpkg-reconfigure tzdata

> sudo apt-get update

> sudo apt-get install ntp

Then we reboot the system, it will automatically sync the time.

Set Up Replica
> mongo --host ubuntu-master --port 27017
MongoDB shell version: 3.2.0
connecting to: ubuntu-master:27017/test
Server has startup warnings:
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten]
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten]
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten]
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-16T16:55:59.526+0000 I CONTROL  [initandlisten]

I may fix the warning later, but right now. I will not spend much time on that.

> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "0b3c81ad5928:27017",
"ok" : 1
}

It will share us the id of that service.
primary  0b3c81ad5928:27017

secondary 1  ded2fe6b86bc:27017

secondary 2  a1117cd7eb5a:27017

Error Message when I try to add secondary to primary
> rs.add('ubuntu-dev1:27017')
{
"ok" : 0,
"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 0b3c81ad5928:27017; the following nodes did not respond affirmatively: ubuntu-dev1:27017 failed with HostUnreachable",
"code" : 74
}

Solution:
https://docs.mongodb.org/manual/tutorial/enable-internal-authentication/

rs.initiate() should only be run on the primary node.

Use IP instead
> mongo --host 192.168.56.104 --port 27017

rs.add('192.168.56.106:27017')

rs.add('192.168.56.105:27017')

Error Message:
rs.status()
2015-12-16T11:51:13.070-0600 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetGetStatus' on host '192.168.56.104:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:132:1
DB.prototype.adminCommand@src/mongo/shell/db.js:149:12
rs.status@src/mongo/shell/utils.js:1005:34

Solution:
I do not know what happened. I will just use single mongoDB for now.

References:
mongoDB does not support arm system
http://sillycat.iyunv.com/blog/2256088

Mongo Blogs
http://sillycat.iyunv.com/blog/2155801  2.6.5
http://sillycat.iyunv.com/blog/603890   Basic Mongo Installation on Windows and Basic Query in command and java driver
1~6
http://sillycat.iyunv.com/blog/1547291   Compare MySQL and MongoDB
http://sillycat.iyunv.com/blog/1547292   velocity and groovy MVC
http://sillycat.iyunv.com/blog/1547294   springData MongoDB
http://sillycat.iyunv.com/blog/1965857   MAC Setup and Version Update 2.4.7, knowledge recall and Replication, 1 primary 2 secondaries; kill the master, 1 of the secondary will become the PRIMARY, after we restart the old primary, the old primary will join as secondary
http://sillycat.iyunv.com/blog/1965880   Scala Client/Driver - casbah
http://sillycat.iyunv.com/blog/2066225   Update Version and HA Proxy

MongoDB Design
http://www.cnblogs.com/egger/archive/2013/04/27/3047191.html
http://www.jianshu.com/p/bb0caddff60a

https://mongodb.github.io/casbah/

MongoDB Cluster
https://www.digitalocean.com/community/tutorials/how-to-create-a-sharded-cluster-in-mongodb-using-an-ubuntu-12-04-vps
http://www.mongodbspain.com/en/2015/01/26/how-to-set-up-a-mongodb-sharded-cluster/
http://yhv5.com/mongodb-shard_821.html
http://blog.iyunv.com/uid-25135004-id-4170240.html
http://hnr520.blog.iyunv.com/4484939/1698306
https://meteorhacks.com/mongodb-replica-sets-with-meteor

Docker
https://medium.com/@arunoda/production-quality-mongodb-setup-with-docker-65136a4a9d8#.41jhrtt17
https://github.com/inlight-media/docker-mongodb-replica-set

https://docs.docker.com/engine/examples/mongodb/