Skip to content

Logstash 6.8.4 编译

logstash 6.8.4 编译

  • what: logstash 6.8.4 在arm环境编译
  • where: centos8 aarch64
  • when: 2022.1.17
  • who: JRT
  • why: 记录
  • how:

1.依赖安装

1.1 安装依赖

yum install make cmake gcc git vim wget -y

1.2 安装Java

[root@b156873121b1 logstash]# wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz
[root@b156873121b1 logstash]#tar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz
[root@b156873121b1 logstash]# mkdir -p /opt/
[root@b156873121b1 logstash]# mv jdk8u252-b09 /opt/
[root@b156873121b1 logstash]# vim /etc/profile
export JAVA_HOME=/opt/jdk8u252-b09
export PATH=$JAVA_HOME/bin:$PATH
[root@b156873121b1 logstash]# source /etc/profile

[root@b156873121b1 logstash]# java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
[root@b156873121b1 logstash]#

2.编译

2.1下载源码

[root@b156873121b1 opt]# git clone https://github.com/elastic/logstash.git
Cloning into 'logstash'...
remote: Enumerating objects: 111965, done.
remote: Counting objects: 100% (506/506), done.
remote: Compressing objects: 100% (378/378), done.
remote: Total 111965 (delta 220), reused 285 (delta 109), pack-reused 111459
Receiving objects: 100% (111965/111965), 121.82 MiB | 16.07 MiB/s, done.
Resolving deltas: 100% (70937/70937), done.
[root@b156873121b1 logstash]# git checkout v6.8.4
Note: switching to 'v6.8.4'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 1e4628fc5 syntax fix
[root@b156873121b1 logstash]#

2.2 编译

[root@b156873121b1 logstash]# ./gradlew assembleOssTarDistribution
runcate, logstash-output-email, logstash-output-lumberjack
Installation successful
[artifact:tar] Building tar.gz of default plugins
[artifact:tar] building build/logstash-oss-6.8.4-SNAPSHOT.tar.gz
Complete: build/logstash-oss-6.8.4-SNAPSHOT.tar.gz

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 6m 57s
30 actionable tasks: 30 executed

编译后包地址：

[root@b156873121b1 build]# pwd
/opt/logstash/build
[root@b156873121b1 build]# ls
bootstrap  cache  libs  logstash-oss-6.8.4-SNAPSHOT.tar.gz  tmp

3.启动

[root@b156873121b1 build]# tar -xf logstash-oss-6.8.4-SNAPSHOT.tar.gz 
[root@b156873121b1 build]# ls
bootstrap  cache   libs  logstash-6.8.4-SNAPSHOT  logstash-oss-6.8.4-SNAPSHOT.tar.gz  tmp
[root@b156873121b1 build]# cd /opt/logstash/build/logstash-6.8.4-SNAPSHOT/bin
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# cd bin/
[root@b156873121b1 bin]# ls
benchmark.sh         logstash           logstash.lib.sh  pqrepair
cpdump           logstash.bat       logstash-plugin  ruby
dependencies-report  logstash-keystore      logstash-plugin.bat  setup.bat
ingest-convert.sh    logstash-keystore.bat  pqcheck      system-install
[root@b156873121b1 bin]# 
[root@b156873121b1 bin]# ./logstash -e 'input { stdin { } } output { stdout {} }'
Sending Logstash logs to /opt/logstash/build/logstash-6.8.4-SNAPSHOT/logs which is now configured via log4j2.properties
[2022-01-17T06:48:02,813][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-01-17T06:48:02,827][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.4"}
[2022-01-17T06:48:02,857][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"a8113d05-b502-40f6-8e08-7411352eb3d8", :path=>"/opt/logstash/build/logstash-6.8.4-SNAPSHOT/data/uuid"}
[2022-01-17T06:48:08,329][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2022-01-17T06:48:08,442][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x75a35ac9 sleep>"}
The stdin plugin is now waiting for input:
[2022-01-17T06:48:08,480][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-01-17T06:48:08,701][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
hello
/opt/logstash/build/logstash-6.8.4-SNAPSHOT/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
          "host" => "b156873121b1",
       "message" => "hello",
    "@timestamp" => 2022-01-17T06:48:23.622Z,
      "@version" => "1"
}

4.问题解决

问题1：

问题描述：执行启动命令后报：

[root@b156873121b1 bin]# ./logstash
Sending Logstash logs to /opt/logstash/build/logstash-6.8.4-SNAPSHOT/logs which is now configured via log4j2.properties
[2022-01-17T06:25:37,350][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/opt/logstash/build/logstash-6.8.4-SNAPSHOT/data/queue"}
[2022-01-17T06:25:37,370][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/opt/logstash/build/logstash-6.8.4-SNAPSHOT/data/dead_letter_queue"}
[2022-01-17T06:25:37,706][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) load error: ffi/ffi -- java.lang.NullPointerException: null
[root@b156873121b1 bin]#

问题分析：

lib/ruby/stdlib/ffi/platform/aarch64-linux/需要platform.conf文件，导致代码异常报错。

问题解决：

#进入logstash-6.8.0-SNAPSHOT
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# find ./ -name jruby*.jar
./logstash-core/lib/jars/jruby-complete-9.2.7.0.jar
./vendor/jruby/lib/jruby.jar
./vendor/bundle/jruby/2.5.0/gems/logstash-filter-dissect-1.2.0/vendor/jars/org/logstash/dissect/jruby-dissect-library/1.2.0/jruby-dissect-library-1.2.0.jar
./vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar
./vendor/bundle/jruby/2.5.0/gems/thread_safe-0.3.6-java/lib/thread_safe/jruby_cache_backend.jar
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# 
#beifen
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# cp ./logstash-core/lib/jars/jruby-complete-9.2.7.0.jar ./bak

#解压
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# cd bak
[root@b156873121b1 bak]# unzip jruby-complete-9.2.7.0.jar
[root@b156873121b1 bak]# find ./ -name types.conf
[root@b156873121b1 bak]# cd ./META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/aarch64-linux/
[root@b156873121b1 aarch64-linux]# ls
types.conf
[root@b156873121b1 aarch64-linux]# cp -r types.conf platform.conf
[root@b156873121b1 aarch64-linux]# ls
platform.conf  types.conf
[root@b156873121b1 aarch64-linux]#
#重新打包
[root@b156873121b1 aarch64-linux]# cd /opt/logstash/build/logstash-6.8.4-SNAPSHOT/bak
[root@b156873121b1 bak]# rm -rf jruby-complete-9.2.7.0.jar
[root@b156873121b1 bak]# zip -r jruby-complete-9.2.7.0.jar *
[root@b156873121b1 bak]# ls
com  jni  jnr  jruby  jruby-complete-9.2.7.0.jar  META-INF  module-info.class  org  tables
[root@b156873121b1 bak]# cd ..
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# find ./ -name jruby*.jar
./logstash-core/lib/jars/jruby-complete-9.2.7.0.jar
./bak/jruby-complete-9.2.7.0.jar
./vendor/jruby/lib/jruby.jar
./vendor/bundle/jruby/2.5.0/gems/logstash-filter-dissect-1.2.0/vendor/jars/org/logstash/dissect/jruby-dissect-library/1.2.0/jruby-dissect-library-1.2.0.jar
./vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar
./vendor/bundle/jruby/2.5.0/gems/thread_safe-0.3.6-java/lib/thread_safe/jruby_cache_backend.jar
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# 
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# cp ./bak/jruby-complete-9.2.7.0.jar ./logstash-core/lib/jars/cp: overwrite './logstash-core/lib/jars/jruby-complete-9.2.7.0.jar'? y
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# 
#重新打包
[root@b156873121b1 logstash-6.8.4-SNAPSHOT]# cd ..
[root@b156873121b1 build]# ls
bootstrap  cache   libs  logstash-6.8.4-SNAPSHOT  logstash-oss-6.8.4-SNAPSHOT.tar.gz  tmp
[root@b156873121b1 build]# rm -rf logstash-oss-6.8.4-SNAPSHOT.tar.gz 
[root@b156873121b1 build]# ls
bootstrap  cache  libs  logstash-6.8.4-SNAPSHOT  tmp
[root@b156873121b1 build]# tar -cf logstash-oss-6.8.4-SNAPSHOT.tar.gz logstash-6.8.4-SNAPSHOT
[root@b156873121b1 build]# ls
bootstrap  cache   libs  logstash-6.8.4-SNAPSHOT  logstash-oss-6.8.4-SNAPSHOT.tar.gz  tmp
[root@b156873121b1 build]#