I was using filebeat to listen on port 514 to accept rsyslog messages from AIX servers with the aim of filebeat then having an output to my logstash instance on the same server, to then send this data to Microsoft sentinel I encountered my filebeat log file filling full of the errors shown below.
Filebeat log
2024-01-18T10:51:13.586+1300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 127.0.0.1:56512->127.0.0.1:5146: write: connection reset by peer 2024-01-18T10:56:11.661+1300 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 127.0.0.1:37308->127.0.0.1:5146: write: connection reset by peer 2024-01-18T10:56:13.421+1300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 127.0.0.1:37308->127.0.0.1:5146: write: connection reset by peer
Logstash-plain log
2024-01-11T13:23:09,692][INFO ][org.logstash.beats.BeatsHandler][main][56704f1b81c9f0e05445eff69994b23d064ad2c59c0e35c5c1c7224e7b790895] [local: 0:0:0:0:0:0:0:1:5146, remote: 0:0:0:0:0:0:0:1:53804] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22) [2024-01-11T13:23:09,692][WARN ][io.netty.channel.DefaultChannelPipeline][main][56704f1b81c9f0e05445eff69994b23d064ad2c59c0e35c5c1c7224e7b790895] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.2.6.jar:?] at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.2.6.jar:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] ... 9 more [2024-01-11T13:23:09,693][INFO ][org.logstash.beats.BeatsHandler][main][56704f1b81c9f0e05445eff69994b23d064ad2c59c0e35c5c1c7224e7b790895] [local: 0:0:0:0:0:0:0:1:5146, remote: 0:0:0:0:0:0:0:1:53804] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3) [2024-01-11T13:23:09,693][WARN ][io.netty.channel.DefaultChannelPipeline][main][56704f1b81c9f0e05445eff69994b23d064ad2c59c0e35c5c1c7224e7b790895] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3 at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.2.6.jar:?] at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.2.6.jar:?] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final] ... 11 more
The amount of syslog entries being processed was quite low and sending a single request manually via telnet to the syslog port on my server also resulted in the same error.
Online help messages seemed to suggest firewalls causing issues between filebeat and logstash but as my installs were both on the same server this was not an issue for me. There was also mentions of discrepancies in versions between filebeat and logstash but both of mine were at the same version.
After trying multiple fixes the one which worked was adding the bottom two lines to my filebeat.yml.
output.logstash:
# The Logstash hosts
hosts: ["localhost:5146"]
ttl: 30
pipelining: 0
After restarting my errors disappeared and i could carry on happily. I hope this helps you also!
