MHuiG

「看庭前花开花落,望天上云卷云舒」

两个 agent 级联

# Flume 两个 agent 级联

## 需求分析

第一个 agent 负责收集文件当中的数据，通过网络发送到第二个 agent 当中去，第二个 agent 负责接收第一个 agent 发送的数据，并将数据保存到 hdfs 上面去

## 第一步：node02 安装 flume

将 node03 机器上面解压后的 flume 文件夹拷贝到 node02 机器上面去

    cd /export/servers
    scp -r apache-flume-1.6.0-cdh5.14.0-bin/ node02:$PWD

## 第二步：node02 配置 flume 配置文件

在 node02 机器配置我们的 flume

    cd /export/servers/apache-flume-1.6.0-cdh5.14.0-bin/conf
    vim tail-avro-avro-logger.conf

tail-avro-avro-logger.conf

    ##################
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /export/servers/taillogs/access_log
    a1.sources.r1.channels = c1
    # Describe the sink
    ##sink avro is a sender
    a1.sinks = k1
    a1.sinks.k1.type = avro
    a1.sinks.k1.channel = c1
    a1.sinks.k1.hostname = 192.168.52.120
    a1.sinks.k1.port = 4141
    a1.sinks.k1.batch-size = 10
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

## 第三步：node02 开发定脚本文件往写入数据

    mkdir -p /export/servers/shells/
    cd /export/servers/shells/
    vim tail-file.sh

tail-file.sh

    #!/bin/bash
    while true
    do
    date >> /export/servers/taillogs/access_log;
     sleep 0.5;
    done

创建文件夹

    mkdir -p /export/servers/taillogs

## 第四步：node03 开发 flume 配置文件

在 node03 机器上开发 flume 的配置文件

    cd /export/servers/apache-flume-1.6.0-cdh5.14.0-bin/conf
    vim avro-hdfs.conf

avro-hdfs.conf

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    # Describe/configure the source
    ##source avro is a receiver
    a1.sources.r1.type = avro
    a1.sources.r1.channels = c1
    a1.sources.r1.bind = 192.168.52.120
    a1.sources.r1.port = 4141
    # Describe the sink
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = hdfs://node01:8020/avro/hdfs/%y-%m-%d/%H%M/
    a1.sinks.k1.hdfs.filePrefix = eventsa1.sinks.k1.hdfs.round = true
    a1.sinks.k1.hdfs.roundValue = 10
    a1.sinks.k1.hdfs.roundUnit = minute
    a1.sinks.k1.hdfs.rollInterval = 3
    a1.sinks.k1.hdfs.rollSize = 20
    a1.sinks.k1.hdfs.rollCount = 5
    a1.sinks.k1.hdfs.batchSize = 1
    a1.sinks.k1.hdfs.useLocalTimeStamp = true
    # Sequencefile,DataStream
    a1.sinks.k1.hdfs.fileType = DataStream
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

## 第五步：顺序启动

node03 机器启动 flume 进程

    cd /export/servers/apache-flume-1.6.0-cdh5.14.0-bin
    bin/flume-ng agent -c conf -f conf/avro-hdfs.conf -n a1 -Dflume.root.logger=INFO,console

node02 机器启动 flume 进程

    cd /export/servers/apache-flume-1.6.0-cdh5.14.0-bin/
    bin/flume-ng agent -c conf -f conf/tail-avro-avro-logger.conf -n a1 -Dflume.root.logger=INFO,console

node02 机器启 shell 脚本生成文件

    cd /export/servers/shells
    sh tail-file.sh

Flume 监控文件变化

监控文件变化 Flume 监控文件变化 需求分析采集需求：比如业务系统使用 log4j 生成的日志，日志内容不断增加，需要把追加到日志文件中的数据实时采集到 hdfs 根据需求，首先定义...

Flume 负载均衡 load balancer

负载均衡 Flume 负载均衡 load balancer 负载均衡是用于解决一台机器 (一个进程) 无法解决所有请求而产生的一种算法。Load balancing Sink Proc...

留言区

Notes
Flume
本文目录
<pjax>
</pjax>
<pjax></pjax>
<pjax> Are you Robot?</pjax>