15-Hadoop-socket测试

Hadoop socket 测试 分类 hadoop 2011 11 01 14 35 89 人阅读 评论 0 收藏 举报 概要 hadoop 上以 streaming 方式运行 map 任务 map 任务为 socke client 端 和在外部系统 运行的 server 交互 代码 server cpp view plaincopy to clipboardprint 1 include 2 include 3 include 4 include 5 include 6 include 7 include 8 include 9 include 10 include 11 include 12 define SERVPORT 9899 服务器监听端口号 13 define BACKLOG 1000 最大同时连接请求数 14 define MAXDATASIZE 100 每次最大数据传输量 15 16 using namespace std 17 18 int main 19 20 int sock fd client fd sock fd 监听 socket client fd 数据传输 socket 21 int sin size 22 char buf MAXDATASIZE 23 struct sockaddr in my addr 本机地址信息 24 struct sockaddr in remote addr 客户端地址信息 25 26 if sock fd socket AF INET SOCK STREAM 0 1 27 28 perror socket 创建出错 29 exit 1 30 31 my addr sin family AF INET 32 my addr sin port htons SERVPORT 33 my addr sin addr s addr INADDR ANY 34 bzero 35 36 if bind sock fd struct sockaddr 39 exit 1 40 41 42 if listen sock fd BACKLOG 1 43 44 perror listen 出错 45 exit 1 46 47 48 while 1 49 50 sin size sizeof struct sockaddr in 51 if client fd accept sock fd struct sockaddr 54 continue 55 56 printf received a connection from s n inet ntoa remote addr sin addr 57 if fork 58 59 子进程代码段 60 if send client fd Hello you are connected n 26 0 1 61 62 perror send 出错 63 64 65 int recvbytes 0 66 if recvbytes recv client fd buf MAXDATASIZE 0 1 67 68 perror recv 出错 69 exit 1 70 71 72 cout buf endl 73 close client fd 74 exit 0 75 76 close client fd 77 78 include include include include include include client cpp view plaincopy to clipboardprint 1 include 2 include 3 include 4 include 5 include 6 include 7 include 8 include 9 include 10 include 11 define SERVPORT 9899 12 define MAXDATASIZE 100 每次最大数据传输量 13 using namespace std 14 15 int main int argc char argv 16 17 int sock fd recvbytes 18 char buf MAXDATASIZE 19 struct hostent host 20 struct sockaddr in serv addr 21 22 读取输入 23 string input 24 while cin input 25 26 cout input endl 27 28 29 if argch addr 49 bzero 50 51 连接 52 if connect sock fd struct sockaddr 55 exit 1 56 57 58 接受数据 59 if recvbytes recv sock fd buf MAXDATASIZE 0 1 60 61 perror recv 出错 62 exit 1 63 64 65 发送数据 66 if send sock fd hello server 13 0 1 67 68 perror send 出错 69 70 buf recvbytes 0 71 printf Received s buf 72 close sock fd 73 return 0 74 include include include include include include 编译程序 g o client client cpp g o server server cpp 测试 测试机上运行 server server hadoop 上运行任务 hadoop streaming input user test txt output user result mapper client domain reducer cat file client jobconf mapred reduce tasks 1 jobconf mapred map tasks 5 jobconf mapre job name socket test input 内容 user test txt 为 1324 jobconf mapred reduce tasks 1 jobconf mapred map tasks 5 mapper 任务 5 个 reduce 任务 1 个 server 输出 received a connection from 10 hello server received a connection from 10 hello server received a connection from 10 hello server received a connection from 10 hello server received a connection from 10 hello server 运行结果 hadoop fs cat user result part 00000 1324 Received