互联网项目系统软件集成解决方案.docx

互联网项目系统软件集成解决方案目录前言3Nginx安装4Nignx下载4Nginx安装4JDK安装5Tomcat安装6Memecached安装6第一步安装libevent7第二部安装memcached8第三步启动并检测memcached10配置Nginx+Tomcat集群实现负载均衡10Tomcat集群配置11Nginx配置12Tomcat实现session同步16Tomcat调用memcached配置18优化linux内核19测试Nginx下Tomcat实现session同步19MySQL集群20管理节点（MGM）的安装及配置21存储节点（NDB）的安装及配置22负载均衡23前言本文的主要目的是通过对下面所列出的软件进行合理的配置以及合理的使用来解决互联网项目中如何提高应用系统平台的高可用性及稳定性。本文中主要应用的软件为：Nginx负载均衡Tomcat应用服务器Memcached缓存服务器Mysql数据库相关插件：JDKLibeventPcremsmHeartbeatLdirectord如果说单纯的按照本文档操作配置实现了负载均衡及集群，大大提高了系统的高可用性及系统性能就算完成任务，那么就失去了文档本身的意义。本文档所列内容只是高可用系统的一个基础，只是基于如何去搭建高可用平台，同时也只是运维工作的一个开始。系统的7*24小时高可用并不只是一套高可用软件集成，而是在这个基础上如何去维护它。我们可以通过一套完整的运维管理制度去约束运维团队进行细致的工作，对任何出现的系统问题进行分析及解决。对于运维工程师可以不断的去探索如何去监控并合理利用网络资源；如何去监控并合理使用服务器及存储的硬件资源；如何去深度发掘上述软件的更多使用奥妙及大胆的尝试新的软件及技术；如何去对程序进行监控，与研发团队紧密沟通提高代码的性能；如何去对数据库进行监控并优化，大大提高数据存储及读取的效率等等。旨在不忽略运维工作中的每个细节，虑小患而治大忧，这样才能最大限度的保证系统的稳定性，7*24小时不间断提供服务，为企业通过互联网平台不断获益提供保障。本文档参考了大量的文献，最终整合在一起，希望对从事系统运维工作的朋友带来一些帮助。有愿意进一步进行技术交流的大家可以mail我：50053912。有不足的地方也希望大家多多指出。Nginx安装本文以nginx-0.7.67为例，到官网/en/download.html下载最新的稳定版本，以0.7.67版本为例。Nignx下载rootlocalhost # wget /download/nginx-0.7.67.tar.gz -2010-09-24 14:48:12- /download/nginx-0.7.67.tar.gz Resolving . 37 Connecting to |37|:80. connected. HTTP request sent, awaiting response. 200 OK Length: 608462 (594K) application/octet-stream Saving to: nginx-0.7.67.tar.gz 100% 608,462 44.5K/s in 18s 2010-09-24 14:48:32 (32.8 KB/s) - nginx-0.7.67.tar.gz saved 608462/608462 Nginx安装 解压压缩文件rootlocalhost # tar -zxvf nginx-0.7.67.tar.gz进入安装文件目录rootlocalhost # cd nginx-0.7.67指定安装到rootlocalhost nginx-0.7.67# ./configure prefix=/usr/local/nginx /usr/local/nginx目录下，可用./configure help查看需要哪些编译参数编译rootlocalhost nginx-0.7.67#make 安装 rootlocalhost nginx-0.7.67#make install查看是否安装成功rootlocalhost nginx-0.7.67# ll /usr/local/nginx/ drwxr-xr-x 2 root root 4096 Sep 24 15:12 conf drwxr-xr-x 2 root root 4096 Sep 24 15:12 html drwxr-xr-x 2 root root 4096 Sep 24 15:12 logs drwxr-xr-x 2 root root 4096 Sep 24 15:12 sbin测试Nginx配置文件是否正确rootlocalhost nginx-0.7.67#/usr/local/nginx/sbin/nginx t启动Nginxrootlocalhost nginx-0.7.67#/usr/local/nginx/sbin/nginx 下面验证Nginx是否正常启动，浏览器中输入nginx服务器地址，出现如下图说明nginx正常工作。JDK安装赋权执行权限给JDK*.bin# chmod u+x JDK*.bin运行JDK安装包# ./JDK*.bin 创建指定的jdk文件路径# mkdir /usr/java -pv将jdk移动到指定路径，根据实际情况进行配置# mv jdk* /usr/java声明Java的环境变量修改/etc/profile文件，增加JAVA_HOME，JRE_HOME，PATH，CLASSPATH属性# vi /etc/profileexport JAVA_HOME=/jdk路径 如export JAVA_HOME=/root/jdk1.6.0_18 export JRE_HOME=/jre路径 如export JRE_HOME=/root/jdk1.6.0_18/jre export PATH=/java/bin路径 如export PATH=/root/jdk1.6.0_18/bin export CLASSPATH=/java/./lib路径 如export CLASSPATH=./:/usr/jdk1.6.0_18/lib:/usr/jdk1.6.0_18/jre/lib 保存并退出。运行./etc/profile使环境变量生效。也可以使用echo $PATH看是否包含添加的路径。运行命令：# java -version若出现如下信息则证明java运行成功java version 1.6.0_18Java(TM) SE Runtime Environment (build 1.6.0_18-b07)Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)Tomcat安装安装Tomcat安装包到/usr/local目录下执行以下命令：解压tomcat# tar xf apache-Tomcatt*.tar.gz -C /usr/local建立连接符,根据实际情况配置# ln -sv apache-Tomcat* tomcat# cd tomcat修改/etc/profile文件，导入tomcat目录，声明环境变量加入以下两条命令：export CATALINA_HOME=/usr/local/tomcatecho $CATALINA_HOME赋权执行权限给catalina.sh和startup.sh# chmod u+x catalina.sh# chmod u+x startup.sh手动启动tomcat# bin/catalina.sh start 或# bin/startup.sh在客户端验证toncat是否安装成功,默认端口为8080。访问http:/service_IP:8080Memecached安装下载 Memcache :/files/memcached-1.4.5.tar.gzMemcache用到了libevent这个库用于Socket的处理，所以还需要安装libevent。下载 libevent:http:/ /provos/libevent-1.4.14b-stable.tar.gz第一步安装libevent先安装libevent：解压软件包# tar zxvf libevent-1.4.14b-stable.tar.gz 路径跳转# cd libevent-1.4.14b-stable 配置安装路径# ./configure -prefix=/usr编译# make安装# make install 如果安装过程中，发现提示：configure: error: no acceptable C compiler found in $PATH需要安装C compiler，安装方法：在Linux环境下只要能链接Internet就可以通过本地的yum源来安装，在终端下输入# yum install gcc就可以，自动安装的。安装的时候需保证linux可以上网，因为需要网上下载安装包。然后看看我们的libevent是否安装成功：执行命令：# ls -al /usr/lib | grep libeventlrwxrwxrwx1 root root21 3月2 13:17 libevent-1.4.so.2 - libevent-1.4.so.2.1.2-rwxr-xr-x1 root root280215 3月2 13:17 libevent-1.4.so.2.1.2-rw-r-r-1 root root367094 3月2 13:17 libevent.alrwxrwxrwx1 root root26 3月2 13:17 libevent_core-1.4.so.2 - libevent_core-1.4.so.2.1.2-rwxr-xr-x1 root root107511 3月2 13:17 libevent_core-1.4.so.2.1.2-rw-r-r-1 root root146924 3月2 13:17 libevent_core.a-rwxr-xr-x1 root root 860 3月2 13:17 libevent_core.lalrwxrwxrwx1 root root26 3月2 13:17 libevent_core.so - libevent_core-1.4.so.2.1.2lrwxrwxrwx1 root root27 3月2 13:17 libevent_extra-1.4.so.2 - libevent_extra-1.4.so.2.1.2-rwxr-xr-x1 root root219655 3月2 13:17 libevent_extra-1.4.so.2.1.2-rw-r-r-1 root root281482 3月2 13:17 libevent_extra.a-rwxr-xr-x1 root root 867 3月2 13:17 libevent_extra.lalrwxrwxrwx1 root root27 3月2 13:17 libevent_extra.so - libevent_extra-1.4.so.2.1.2-rwxr-xr-x1 root root 825 3月2 13:17 libevent.lalrwxrwxrwx1 root root21 3月2 13:17 libevent.so - libevent-1.4.so.2.1.2出现以上提示则证明libevent安装完毕。第二部安装memcached解压软件包# tar zxvf memcached-1.4.5.tar.gz跳转到指定路径# cd memcached-1.4.5配置安装路径# ./configure -with-libevent=/usr编译# make安装# make install如果中间出现报错，请仔细检查错误信息，按照错误信息来配置或者增加相应的库或者路径。安装完成后会把memcached放到 /usr/local/bin/memcached 。我们看以下是否安装了：# ls -al /usr/local/bin/mem*-rwxr-xr-x 1 root root 121249 3月 2 13:23 /usr/local/bin/memcached-rwxr-xr-x 1 root root 130179 3月 2 13:23 /usr/local/bin/memcached-debug 安装完成后，现在我们看以下memcache的帮助：# /usr/local/bin/memcached -h出现：error while loading shared libraries: libevent-1.4.so.2: cannot open shared object file: No such file or directory 原因是找不到libevent-1.4.so.2类库。解决办法如下：使用LD_DEBUG=help ./memcached -v来确定加载的类库路径。方法如下:# LD_DEBUG=libs ./memcached -v 2&1 /dev/null | less 则系统会显示：linux:/local/memcached/bin # LD_DEBUG=libs ./memcached -v 2&1 /dev/null | less 20421: find library=libevent-1.4.so.2; searching 20421: search cache=/etc/ld.so.cache 20421: search path=/lib/tls/i686/sse2:/lib/tls/i686:/lib/tls/sse2:/lib/tls:/lib/i686/sse2:/lib/i686:/lib/sse2:/lib:/usr/lib/tls/i686/sse2:/usr/lib/tls/i686:/usr/lib/tls/sse2:/usr/lib/tls:/usr/lib/i686/sse2:/usr/lib/i686:/usr/lib/sse2:/usr/lib (system search path) 20421: trying file=/lib/tls/i686/sse2/libevent-1.4.so.2 20421: trying file=/lib/tls/i686/libevent-1.4.so.2 20421: trying file=/lib/tls/sse2/libevent-1.4.so.2 20421: trying file=/lib/tls/libevent-1.4.so.2 20421: trying file=/lib/i686/sse2/libevent-1.4.so.2 20421: trying file=/lib/i686/libevent-1.4.so.2 20421: trying file=/lib/sse2/libevent-1.4.so.2 20421: trying file=/lib/libevent-1.4.so.2 20421: trying file=/usr/lib/tls/i686/sse2/libevent-1.4.so.2 20421: trying file=/usr/lib/tls/i686/libevent-1.4.so.2 20421: trying file=/usr/lib/tls/sse2/libevent-1.4.so.2 20421: trying file=/usr/lib/tls/libevent-1.4.so.2 20421: trying file=/usr/lib/i686/sse2/libevent-1.4.so.2 20421: trying file=/usr/lib/i686/libevent-1.4.so.2 20421: trying file=/usr/lib/sse2/libevent-1.4.so.2 20421: trying file=/usr/lib/libevent-1.4.so.2 20421: ./memcached: error while loading shared libraries: libevent-1.4.so.2: cannot open shared object file: No such file or directory我们看到，memcached会到很多地方去找，所以根据其它求，我们只需建一个软链接，指定到我们安装的类库上即可方法如下：# ln -s /usr/local/lib/libevent-1.4.so.2 /lib/libevent-1.4.so.2 再测试是否安装成功：# /usr/local/bin/memcached -hmemcached 1.2.6-p TCP port number to listen on (default: 11211)-U UDP port number to listen on (default: 0, off)-s unix socket path to listen on (disables network support)-a access mask for unix socket, in octal (default 0700)-l interface to listen on, default is INDRR_ANY-d run as a daemon-r maximize core file limit-u assume identity of (only when run as root)-m max memory to use for items in megabytes, default is 64 MB-M return error on memory exhausted (rather than removing items)-c max simultaneous connections, default is 1024-k lock down all paged memory. Note that there is a limit on how much memory you may lock. Trying to allocate more than that would fail, so be sure you set the limit correctly for the user you started the daemon with (not for -u user; under sh this is done with ulimit -S -l NUM_KB).-v verbose (print errors/warnings while in event loop)-vv very verbose (also print client commands/reponses)-h print this help and exit-i print memcached and libevent license-b run a managed instanced (mnemonic: buckets)-P save PID in , only used with -d option-f chunk size growth factor, default 1.25-n minimum space allocated for key+value+flags, default 48 出现上面描述，表示安装成功。 第三步启动并检测memcached启动一个Memcache的服务器端：# /usr/local/bin/memcached -d -m 10 -u root -l 73 -p 12000 -c 256 -P /tmp/memcached.pid相关参数-d选项是启动一个守护进程。-m是分配给Memcache使用的内存数量，单位是MB，我这里是10MB。-u是运行Memcache的用户，我这里是root。-l是监听的服务器IP地址，如果有多个地址的话，我这里指定了服务器的IP地址。-p是设置Memcache监听的端口，我这里设置了12000，最好是1024以上的端口。-c选项是最大运行的并发连接数，默认是1024，我这里设置了256，按照你服务器的负载量来设定。-P是设置保存Memcache的pid文件，我这里是保存在/tmp/memcached.pid。如果要结束Memcache进程，执行：# kill -9 cat /tmp/memcached.pid也可以启动多个守护进程，不过端口不能重复。用 netstat -lp|grep memcached 命令可以查看 memcached 是否已经启动:# netstat -lp|grep memcachedtcp 0 0 :12000 *:* LISTEN 9037/memcached udp 0 0 :12000 *:* 9037/memcached 说明 memcached 正在运行。配置Nginx+Tomcat集群实现负载均衡因为我们需要实现tomcat负载均衡，并且在多台用于运行tomcat的服务器以及每台服务器上同时运行多个tomcat服务，故以下我们以一台服务器上运行多个tomcat服务举例进行说明，对于多台服务器操作方法类似，只是需要注意IP地址及端口的使用及合理规划即可。Tomcat集群配置以下为我们以一台服务上同事运行两个tomcat服务为例进行具体说明。安装了两个tomcat，解压tomcat到/usr/local/tomcat1和/usr/local/tomcat2。分别进入/usr/local/tomcat1/bin和/usr/local/tomcat2/bin。目录执行命令：设置执行权限# chmod u+x *.Sh 修改tomcat1下conf中server.xml修改tomcat1下bin中的startup.sh和shutdown.sh添加以下内容export JAVA_HOME=/usr/local/java1.5.0_09export PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=$JAVA_HOME/libexport CATALINA_HOME=$CATALINA_HOME_1export CATALINA_BASE=$CATALINA_BASE_1修改tomcat2下conf中server.xml修改tomcat2下bin中的startup.sh和shutdown.sh添加以下内容export JAVA_HOME=/usr/local/java1.5.0_09export PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=$JAVA_HOME/libexport CATALINA_HOME=$CATALINA_HOME_2export CATALINA_BASE=$CATALINA_BASE_2修改profile配置文件vi /etc/profile 配置系统变量，打开后添加以下内容：CATALINA_BASE_1=/usr/local/tomcat1CATALINA_HOME_1=/usr/local/tomcat1export CATALINA_BASE_1 CATALINA_HOME_1TOMCAT_HOME_1=/usr/local/tomcat1export TOMCAT_HOME_1CATALINA_BASE_2=/usr/local/tomcat2CATALINA_HOME_2=/usr/local/tomcat2export CATALINA_BASE_2 CATALINA_HOME_2TOMCAT_HOME_2=/usr/local/tomcat2export TOMCAT_HOME_2设置tomcat随系统启动vi /etc/rc.d/rc.local 添加以下内容/usr/local/tomcat1/bin/startup.sh/usr/local/tomcat2/bin/startup.sh保存退出，logout 注销一下分别进入/usr/local/tomcat1/bin和/usr/local/tomcat2/bin目录执行./startup.sh访问http:/IP:8011/和http:/IP:8012/看是否可以看到tomcat的欢迎界面Nginx配置安装nginx，在安装nginx之前需要先安装所需的pcre库以nginx-0.7.60-linux及pcre-7.8为例，解压我们下载的pcre-7.8.tar.gz解压pcre# tar zxvf pcre-7.8.tar.gz# cd pcre-7.8配置pcre# ./configure编译# make安装# make install安装完毕。# tar zxvf nginx-0.7.60-linux.tar.gz# ./configure-user=www-group=www-prefix=/usr/local/nginx-with-http_stub_status_module-with-http_ssl_module创建代理文件 vi /usr/local/nginx/conf/proxy.conf 输入以下内容proxy_redirect off;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;client_max_body_size 10m;client_body_buffer_size 128k;proxy_connect_timeout 600;proxy_read_timeout 600;proxy_send_timeout 600;proxy_buffer_size 8k;proxy_buffers 4 32k;proxy_busy_buffers_size 64k;proxy_temp_file_write_size 64k;保存退出。编辑nginx.conf文件 vi /usr/local/nginx/conf/nginx.conf 修改如下#运行用户#user nobody;#启用进程worker_processes 8;#全局错误日志和piderror_log logs/error.log;#error_log logs/error.log notice;#error_log logs/error.log info;pid logs/nginx.pid;#工作模式及连接数上限worker_rlimit_nofile 51200;events worker_connections 51200;#配置http已经反向代理做负载均衡http include mime.types; default_type application/octet-stream; includeproxy.conf; #log_format main $remote_addr - $remote_user $time_local $request # $status $body_bytes_sent $http_referer # $http_user_agent $http_x_forwarded_for; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; #配置负载均衡服务列表,weight参数表示权重值，值越大权重越高 #例子程序 # upstream www.XXX_1.com # server 1:8080; # server 2:8080; # # upstream www.XXX_2.com # server 1:8080; # server 2:8080; # upstream proxyserver #ip_hash; server localhost:8080 weight=6; server localhost:8081 weight=1; server #端口以及域名配置 listen 80; server_name ; #charset utf-8; #access_log logs/host.access.log main; location /NginxStatus/ stub_status on; access_log off; #设定默认首页 location / root /wwwroot/ROOT; index index.html login.jsp; #禁止访问WEB-INF location /(WEB-INF)/ deny all; #设定浏览器访问时nginx直接处理和需缓存的文件后缀以及时间参数 location .(gif|jpg|jpeg|png|bmp|ico|rar|css|zip|txt|flv|swf|mid|doc|ppt|xls|pdf|txt|mp3|wma)$ root /wwwroot/ROOT; expires 24h; location .(htm|html)$ root /wwwroot/ROOT; #设定此后缀的文件利用反向代理转给负载均衡列表中的tomcat处理 location .jsp$ proxy_pass http:/proxyserver; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location .do$ proxy_pass http:/proxyserver; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location .js$ proxy_pass http:/proxyserver; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location .xml$ proxy_pass http:/proxyserver; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location /dwr/ proxy_pass http:/proxyserver; proxy_set_header X-Forwarded-For $proxy_add_x_forward