scribe 研究

kient88 · 发表于 2015-11-28 16:13:41

scribe 研究- 一种分布式日志系统（facebook）

http://onexin.iyunv.com/source/plugin/onexin_bigdata/file:///C:/DOCUME~1/ADMINI~1/LOCALS~1/Temp/msohtml1/01/clip_image001.jpg一张scribe结构图
先转一下，留作后用。
Installing Scribe For Log Collection
　　Scribeis a newly releasedlogcollection tool that dumps log files from various nodes in a cluster toScribe servers, where the logs are stored for further use. Facebook describestheir usage of Scribe by saying, “[Scribe] runs on thousands of machines andreliably delivers tens of billions of messages a day.” It turns out thatScribe is rather difficult to install, so the hope of this post is to helpthose of you attempting to install Scribe. The first step is to get dependenciesinstalled.
　　Dependencies
Scribe has many dependencies that must be installed in order for Scribe to bebuilt properly. They are listed here:

ruby(ruby and ruby-dev)
python(python and python-dev)
libevent(libevent and libevent-dev)
boostv1.36
Thrift
fb303 (included in Thrift incontrib/fb303)

　　The order in which these are installed is important.First, you must install libevent, then libevent-dev, then boost, then Thrift,and finally fb303. I installed libevent and libevent-dev from RPMs, whereasboost, Thrift, and fb303 were installed from source. I was unable to getScribe, Thrift, and fb303 to locate the boost libraries and includescorrectly with the default boost install directories, so I installed boost in/usr/local/boost, /usr/local/boost/bin, /usr/local/boost/lib, and/usr/local/boost/include. Run ‘./configure –help’ when configuring boost tosee how to specify these options.
　　When configuring Thrift and fb303, you must specify yourlocation of boost with the “–with-boost=/path/to/boost/root” option and alsoset your BOOST_ROOT environment variable.
　　Finally, you must make sure your LD_LIBRARY_PATHenvironment variable contains the lib folders that house the Thrift, fb303,boost, and libevent C++ libraries. LD_LIBRARY_PATH follows the same patternas the PATH variable. That is, directories that contain libraries areseparated by colons. If you forget to set your LD_LIBRARY_PATH variable, thenyou’ll get the following error when running scribed:
scribed:error while loading shared libraries: libboost_system-gcc41-mt-1_36.so.1.36.0:cannot open shared object file: No such file or directory
Install Scribe
Once you’ve successfully installed alldependencies, installing Scribe is easy. Scribe ships with a fairlycomprehensive README file, but the instructions involving boost’sconfiguration are slightly incorrect. I needed to pass the“–with-boost=/usr/local/boost” options while configuring. However, the READMEfile says to use “–with-boost /usr/local/boost”. Here is my full configurestatement:
./configure–with-boost=/usr/local/boost –with-boost-system=boost_system-gcc41-mt-1_36–with-boost-filesystem=boost_filesystem-gcc41-mt-1_36
　　We installed Scribe from trunk instead ofreleases/scribe-2.0.
Configure Scribe
Scribe ships with a few good examples in$SCRIBE_ROOT/examples; just take a look at the README in the examplesdirectory, and you should be ready to rock. However, the README doesn’tdocument that the scribe_ctrl and scribe_cat programs are in the examplesdirectory.
　　I hope this tutorial was helpful! Send us anemailif you have any issues. We’ll make a follow-up post latertalking about more in-depth Scribe configurations and benchmarking.
Tuesday,October 28th, 2008 at 9:13 pm by Alex Loddengaard, filed under datacollection.
转自：
http://www.cloudera.com/blog/2008/10/28/installing-scribe-for-log-collection/

centos 5 安装
在开源Thrift，Hive和Cassandra之后，Facebook于2008年10月29日开源了其分布式日志收集框架Scribe，而且一上来就是stable的2.0版本。Scribe是一个用于分布式收集日志的Thrift服务，Facebook描述它”运行于数以千计的节点之上，每天可靠的发送数以百亿的消息“。

Scribe本身的代码量不大，但安装相当烦琐，本文的安装环境是CentOS 5，其他distribution如Ubuntu的安装类似。Scribe安装的依赖项包括：

* ruby (ruby and ruby-dev)
* python (python and python-dev)
* libevent (libevent and libevent-dev)
* boost 1.36
* thrift
* fb303

首先是安装ruby, python和libevent，这些比较容易，
yum install ruby ruby-devel
yum install python python-devel
yum install libevent libevent-devel

然后是boost，Scribe要求1.36版本，先去boost官方(http://www.boost.org/users/download/)上下一个安装包，如果没装g++，

yum install gcc-c++

假设boost安装的目录为/usr/local/boost

tar xzvf boost_1_36_0.tar.gz
cd boost_1_36_0
./configure --prefix=/usr/local/boost
make install

boost编译需要的时间比较长，安装完boost就可以装Thrift了，注意要先thrift依赖automake,libtool等

yum install automake byacc libtool flex bison
wget -O thrift.tgz"http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz"
tar -xzf thrift.tgz
cd thrift
./configure --with-boost=/usr/local/boost
make
make install

然后安装fb303,facebook的支持库,源文件在thrift目录下的contrib/fb303
./configure --with-boost=/usr/local/boost
make
make install

终于可以装Scribe了
设置环境变量BOOST_ROOT为boost安装路径
export BOOST_ROOT=/usr/local/boost

设置环境变量LD_LIBRARY_PATH，指定包含Thrift,fb303, boost, libevent的路径
export LD_LIBRARY_PATH=/usr/local/boost/lib::/usr/lib:/usr/local/lib

安装
./configure --with-boost=/usr/local/boost
make
make install

安装后可以试一下scribe自带的例子，在目录examples下。
转自：
http://www.hadoopchina.com/index.php?option=com_content&task=view&id=67&Itemid=39

账号		自动登录	找回密码
密码			立即注册

大疆运维招人啦，

C++ :try 语句块和异常处理

C++的多态

Red Hat RHCE 8 (EX294) Cert Guide

Java/C++ 区别：看完这一篇，就够用！

别再用过时库了！这 13 个顶级 C++ 库才是

c++ size_t 和 int 的区别

[经验分享] scribe 研究

浏览过的版块

扫码加入运维网微信交流群