Apache Flume 1.7.0 发布，日志服务器

FXMAR

　　欢迎加入运维网交流群：263444886  
                            　　Apache Flume 1.7.0 发布了，Flume 是一个分布式、可靠和高可用的服务，用于收集、聚合以及移动大量日志数据，使用一个简单灵活的架构，就流数据模型。这是一个可靠、容错的服务。

　　本次更新如下：
　　新特性

• 　　[FLUME-2498] - Implement Taildir Source
  

　　改进

• 　　[FLUME-1899] - Make SpoolDir work with Sub-Directories
  
• 　　[FLUME-2526] - Build flume by jdk 7 in default
  
• 　　[FLUME-2628] - Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source
  
• 　　[FLUME-2704] - Configurable poll delay for spooling directory source
  
• 　　[FLUME-2718] - HTTP Source to support generic Stream Handler
  
• 　　[FLUME-2729] - Allow pollableSource backoff times to be configurable
  
• 　　[FLUME-2755] - Kafka Source reading multiple topics
  
• 　　[FLUME-2781] - A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
  
• 　　[FLUME-2799] - Kafka Source - Message Offset and Partition add to headers
  
• 　　[FLUME-2801] - Performance improvement on TailDir source
  
• 　　[FLUME-2810] - Add static Schema URL to AvroEventSerializer configuration
  
• 　　[FLUME-2820] - Support New Kafka APIs
  
• 　　[FLUME-2852] - Kafka Source/Sink should optionally read/write Flume records
  
• 　　[FLUME-2868] - Kafka Channel partition topic by key
  
• 　　[FLUME-2872] - Kafka Sink should be able to select which header as the key
  
• 　　[FLUME-2875] - Allow RollingFileSink to specify a file prefix and a file extension.
  
• 　　[FLUME-2909] - Bump Rat version
  
• 　　[FLUME-2910] - AsyncHBaseSink - Failure callbacks should log the exception that caused them
  
• 　　[FLUME-2911] - Add includePattern option in SpoolDirectorySource configuration
  
• 　　[FLUME-2918] - TaildirSource is underperforming with huge parent directories
  
• 
  　　[FLUME-2937] - Integrate checkstyle for non-test>
  
• 
  　　[FLUME-2941] - Integrate checkstyle for test>
  
• 　　[FLUME-2954] - make raw data appearing in log messages explicit
  
• 　　[FLUME-2955] - Add file path to the header in TaildirSource
  
• 　　[FLUME-2959] - Fix issues with flume-checkstyle module
  
• 　　[FLUME-2982] - Add localhost escape sequence to HDFS sink
  
• 　　[FLUME-2999] - Kafka channel and sink should enable statically assigned partition per event via header
  
• 　　[FLUME-2821] - Flume-Kafka Source with new Consumer
  
• 　　[FLUME-2822] - Flume-Kafka-Sink with new Producer
  
• 　　[FLUME-2823] - Flume-Kafka-Channel with new APIs
  

　　Bug 修复

• 　　[FLUME-1668] - Hdfs Sink File Rollover
  
• 　　[FLUME-2132] - Exception while syncing from Flume to HDFS
  
• 　　[FLUME-2143] - Flume build occasionally fails with OutOfMemoryError on Windows.
  
• 　　[FLUME-2215] - ResettableFileInputStream can’t support ucs-4 character
  
• 　　[FLUME-2318] - SpoolingDirectory is unable to handle empty files
  
• 　　[FLUME-2448] - Building flume from trunk failing with dependency error
  
• 　　[FLUME-2484] - NullPointerException in Kafka Sink test
  
• 　　[FLUME-2485] - Thrift Source tests fail on Oracle JDK 8
  
• 　　[FLUME-2514] - Some TestFileChannelRestart tests are extremely slow
  
• 　　[FLUME-2567] - Remove unneeded repository declarations in pom.xml
  
• 　　[FLUME-2573] - flume-ng –conf parameter is not used when starting a flume agent
  
• 　　[FLUME-2593] - ResettableFileInputStream returns negate values from read() method
  
• 　　[FLUME-2619] - Spooldir source does not log channel exceptions
  
• 　　[FLUME-2632] - High CPU on KafkaSink
  
• 　　[FLUME-2652] - Documented transaction handling semantics incorrect
  
• 　　[FLUME-2660] - Add documentation for EventValidator
  
• 　　[FLUME-2672] - NPE in KafkaSourceCounter
  
• 　　[FLUME-2712] - Optional channel errors slows down the Source to Main channel event rate
  
• 　　[FLUME-2725] - HDFS Sink does not use configured timezone for rounding
  
• 　　[FLUME-2732] - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable
  
• 　　[FLUME-2734] - Kafka Channel timeout property is overridden by default value
  
• 　　[FLUME-2738] - Async HBase sink FD leak on client shutdown
  
• 　　[FLUME-2746] - How to include this Flume Patch in Flume 1.5.2 ?
  
• 　　[FLUME-2749] - Kerberos configuration error when using short names in multiple HDFS Sinks
  
• 　　[FLUME-2751] - Upgrade Derby version to 10.11.1.1
  
• 　　[FLUME-2753] - Error when specifying empty replace string in Search and Replace Interceptor
  
• 　　[FLUME-2754] - Hive Sink skipping first transaction in each Batch of Hive Transactions
  
• 　　[FLUME-2761] - Move Hive sink out of preview mode
  
• 　　[FLUME-2763] - flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib
  
• 　　[FLUME-2773] - TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already
  
• 
  　　[FLUME-2797] - SyslogTcpSource uses Deprecated>
  
• 　　[FLUME-2798] - Malformed Syslog messages can lead to OutOfMemoryException
  
• 　　[FLUME-2804] - Hive sink - abort remaining transactions on shutdown
  
• 　　[FLUME-2806] - flume-ng.ps1 Error running script to start an agent on Windows
  
• 　　[FLUME-2835] - Hive Sink tests need to create table with transactional property set
  
• 　　[FLUME-2841] - Upgrade commons-collections to 3.2.2
  
• 　　[FLUME-2844] - ChannelCounter of SpillableMemoryChannel doesn’t register actually.
  
• 　　[FLUME-2881] - Windows Launch Script fails in plugins dir code
  
• 　　[FLUME-2886] - Optional Channels can cause OOMs
  
• 　　[FLUME-2889] - Fixes to DateTime computations
  
• 　　[FLUME-2891] - Revert FLUME-2712 and FLUME-2886
  
• 　　[FLUME-2897] - AsyncHBase sink NPE when Channel.getTransaction() fails
  
• 　　[FLUME-2901] - Document Kerberos setup for Kafka channel
  
• 　　[FLUME-2908] - NetcatSource - SocketChannel not closed when session is broken
  
• 
  　　[FLUME-2913] - Flume>
  
• 　　[FLUME-2915] - The kafka channel using new APIs will be stuck when the sink is avro sink
  
• 　　[FLUME-2920] - Kafka Channel Should Not Commit Offsets When Stopping
  
• 　　[FLUME-2922] - HDFSSequenceFile Should Sync Writer
  
• 　　[FLUME-2923] - Bump AsyncHBase version
  
• 　　[FLUME-2936] - KafkaSource tests arbitrarily fail
  
• 　　[FLUME-2939] - Upgrade recursive SpoolDir to use Java7 features
  
• 　　[FLUME-2948] - Docs: Fixed parameters on Replicating Channel Selector documentation example
  
• 　　[FLUME-2949] - Flume fails to build on Windows
  
• 
  　　[FLUME-2950] ->
  
• 　　[FLUME-2952] - SyslogAgent possible NPE on stop()
  
• 　　[FLUME-2972] - Handle offset migration in the new Kafka Channel
  
• 　　[FLUME-2974] - Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader
  
• 　　[FLUME-2983] - Handle offset migration in the new Kafka Source
  

　　文档

• 　　[FLUME-2575] - FLUME-2548 brings SSLv2Hello back for Avro Sink, but UG says it is one of the protocols to exclude
  
• 　　[FLUME-2713] - Document Fault Tolerant Config parameters in FlumeUserGuide
  
• 　　[FLUME-2737] - Documentation for Pollable Source config parameters introduced in FLUME-2729
  
• 　　[FLUME-2783] - Update Website Team page with new Committer’s
  
• 　　[FLUME-2890] - Typo in Twitter source warning
  
• 　　[FLUME-2934] - Document new cachePatternMatching option for TaildirSource
  
• 　　[FLUME-2963] - FlumeUserGuide - error in Kafka Source properties table
  
• 　　[FLUME-2971] - Document secure Kafka Sink/Source/Channel setup
  
• 　　[FLUME-2975] - Minor mistake in NetCat Source example in documentation
  
• 　　[FLUME-2998] - Add missing configuration parameter to SequenceSource docs
  

　　任务

• 　　[FLUME-2935] - Bump java target version to 1.7
  

　　测试

• 　　[FLUME-3003] - testSourceCounter in TestSyslogUdpSource is flaky
  

　　点此查看发行主页和完整更新日志
　　下载地址