#Kafka remove duplicate messages code#

If you think Flume That would be a big mistake. Sink You can write logs to HDFS, HBase, Even another one Source wait. such as пјљChannel You can store events in memory temporarily, It can also be persisted to the local hard disk. The combination mode is based on the configuration file set by the user, Very flexible.

Different types of Source,Channel and Sink Can be combined freely. Very straightforward design, Among them, it is worth noting that ,Flume A lot of built-in SourceгЂЃChannel and Sink type. Sink Responsible for persisting logs or pushing events to another Source. You can take Channel Think of it as a buffer, It will save events until Sink Process the event. When Source After the event is captured, it will be formatted specifically, then Source Will push events into ( One or more )Channel in. Event is Flume The basic unit of data, It carries Log data ( Byte array form ) And carry the head message, these Event from Agent External Source, Like the one above Web Server Generate. 2 Flume The core conceptДЇlume Data flow from events (Event) Throughout. Notes : Used in this article Flume Version is flume-1.4.0-cdh4.7.0, No additional installation process is required, Decompression is available.

Use ThriftгЂЃAvro Flume sources It can be downloaded from flume0.9.4 send out events To flume 1.x.

pluggable, Part is for users, Tool or system developers.

No longer need master Nodes and pairs zookeeper Dependence, Configuration file simplification.

No longer distinguish between logic and physics node, All physical nodes are collectively referred to as вЂњagentsвЂќ, Every agents Can run 0 One or more sources and sinks.

This article will not go into details, But here is a brief mention Flume NG пј€1.x.xпј‰ The main changes :ДЎ in-memory channel Non persistent support, Fast Дў JDBC-based channel Persistence support

#Kafka remove duplicate messages code#

But as the FLume The expansion of function ,Flume OG Code engineering is bloated гЂЃ The design of core components is unreasonable гЂЃ The core configuration is not standard and other shortcomings are exposed, Especially in Flume OG The last release of 0.94.0 in, The instability of log transmission is especially serious, To solve these problems ,2011 year 10 month 22 Number ,cloudera It's done Flume-728, Yes Flume A milestone change has been made пјљ Refactor the core components гЂЃ Core configuration and code architecture, The reconstructed version is called Flume NGпј€next generationпј‰ Another reason for the change is to Flume Included in the apache its ,cloudera Flume Renamed as Apache Flume.IBM This article of пјљгЂЉ Flume NGпјљFlume The first revolution in the history of development гЂ‹, From the perspective of basic components and user experience Flume OG To Flume NG Revolutionary changes have taken place.

Flume The initial release is now collectively referred to as Flume OGпј€original generationпј‰, Belong to cloudera. Note that you don't have to store consumed offsets in Kafka, but instead store it in a storage mechanism of your own choosing.Flume As cloudera Development of real-time log collection system, It has been recognized and widely used in the industry. Consumer groups must have unique group ids within the cluster, from a kafka broker perspective.Д¬reating the consumer: const consumer = nsumer( When a consumer fails the load is automatically distributed to other members of the group. Consumer groups allow a group of machines or processes to coordinate access to a list of topics, distributing the load among the consumers.