性能测试_BITLOG

测试用例

• 日志行数：1000000

  • 单机器单文件

  • 插入语句数量：5000

  • 每条插入语句插入日志行数：200

硬件环境

~ 211.145.67.150 (云数的服务器)
  CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 40核80线程
  MEM: 128G

应用环境

rule: 20个处理线程
cassandra service： 40个处理线程

测试结果

程序直插模式

1333条/秒 80M/秒

一般缓存模式

• 数据流转过程

  file -> toe(file cache) -> parser(file cache) -> parser rule -> cassandra service
  

• 处理速度结果

无缓存模式

• 数据流转过程

  file -> toe -> parser rule -> cassandra service
  

• 处理速度结果

直接插入模式

• 数据流转过程

  insert data -> cassandra service
  

• 处理速度结果

• 原始命令行结果

[matrix@sjysomdb01 test]$ ./testbucketlog -o -d 1000 -b 1000
  2020-05-30 14:23:46.325453 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:23:46.325557 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:23:49.228911 I | Gse dictionary loaded finished.
  Create log bucket object.
  1/1
  Prepare data size: 1000
  total:1000, success:1000, fail:0, speed:71.43/s 23134540.03B/s


  [matrix@sjysomdb01 test]$ ./testbucketlog -o -d 2000 -b 500 -c
  2020-05-30 14:05:27.772587 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:05:27.772679 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:05:30.557173 I | Gse dictionary loaded finished.
  Create log file testbucketlog.log
  Create log bucket object.
  1/1
  Prepare data size: 2000
  total:2000, success:2000, fail:0, speed:250.00/s 40485614.65B/s


  [matrix@sjysomdb01 test]$ ./testbucketlog -o -d 2500 -b 400
  2020-05-30 14:22:22.643289 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:22:22.643391 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:22:25.419440 I | Gse dictionary loaded finished.
  Create log bucket object.
  1/1
  Prepare data size: 2000
  total:2000, success:2000, fail:0, speed:399.98/s 51815300.29B/s


  [matrix@sjysomdb01 test]$ ./testbucketlog -o -d 5000 -b 200
  2020-05-30 14:17:52.612614 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:17:52.612720 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:17:55.467785 I | Gse dictionary loaded finished.
  Create log bucket object.
  1/1
  Prepare data size: 4000
  total:4000, success:4000, fail:0, speed:1333.29/s 86360140.54B/s


  [matrix@sjysomdb01 test]$ ./testbucketlog -o -d 8000 -b 100
  2020-05-30 14:19:12.528154 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:19:12.528258 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:19:15.306108 I | Gse dictionary loaded finished.
  Create log bucket object.
  1/1
  Prepare data size: 8000
  total:8000, success:8000, fail:0, speed:2666.59/s 86360500.91B/s


  [matrix@sjysomdb01 test]$ ./testbucketlog -o -d 16000 -b 50
  2020-05-30 14:20:24.663286 I | Dict files path:  [/opt/matrix/var/test/dictionary.txt]
  2020-05-30 14:20:24.663394 I | Load the gse dictionary: "/opt/matrix/var/test/dictionary.txt"
  2020-05-30 14:20:27.441471 I | Gse dictionary loaded finished.
  Create log bucket object.
  1/1
  Prepare data size: 16000
  total:16000, success:15999, fail:1, speed:3999.78/s 64764637.39B/s