Skip to content

Dilipr/fluent bit log processing #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
r-dilip merged 9 commits into from
Sep 14, 2018
Merged

Dilipr/fluent bit log processing #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0d4cd8b
Build out_oms.so and include in docker-cimprov package
Sep 6, 2018
abb7282
Adding fluent-bit-config file to base container
Sep 7, 2018
c9163d3
PR Feedback
r-dilip Sep 11, 2018
029a038
Adding out_oms.conf to base_container.data
r-dilip Sep 12, 2018
ccbd668
PR Feedback
r-dilip Sep 13, 2018
75bd22a
Making the critical section as small as possible
r-dilip Sep 13, 2018
ebbf2e1
PR Feedback
r-dilip Sep 13, 2018
13a39b7
merge
r-dilip Sep 14, 2018
12c8bab
Fixing the newline bug for Computer, and changing containerId to Id
r-dilip Sep 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
829 changes: 420 additions & 409 deletions build/Makefile
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions installer/conf/out_oms.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
omsadmin_conf_path=/etc/opt/microsoft/omsagent/conf/omsadmin.conf
cert_file_path=/etc/opt/microsoft/omsagent/certs/oms.crt
key_file_path=/etc/opt/microsoft/omsagent/certs/oms.key
container_host_file_path=/var/opt/microsoft/docker-cimprov/state/containerhostname
container_inventory_refresh_interval=60
kube_system_containers_refresh_interval=300
35 changes: 35 additions & 0 deletions installer/conf/td-agent-bit.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
[SERVICE]
Flush 5
Log_Level info
Parsers_File /etc/td-agent-bit/parsers.conf
Log_File /var/log/fluent-bit.log

[INPUT]
Name tail
Tag oms.container.log.*
Path /var/log/containers/*.log
DB /var/log/fblogs.db
Parser docker
Mem_Buf_Limit 30m
Path_Key filepath

[FILTER]
Name record_modifier
Match oms.container.log.*
Whitelist_key log
Whitelist_key stream
Whitelist_key time
Whitelist_key filepath

[FILTER]
Name modify
Match oms.container.log.*
Rename log LogEntry
Rename stream LogEntrySource
Rename time LogEntryTimeStamp
Rename filepath Filepath
Add_if_not_present SourceSystem Containers

[OUTPUT]
Name oms
Match oms.container.log.*
7 changes: 6 additions & 1 deletion installer/datafiles/base_container.data
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ MAINTAINER: 'Microsoft Corporation'
/opt/microsoft/omsagent/plugin/in_kube_services.rb; source/code/plugin/in_kube_services.rb; 644; root; root
/opt/microsoft/omsagent/plugin/in_kube_nodes.rb; source/code/plugin/in_kube_nodes.rb; 644; root; root


/opt/td-agent-bit/bin/out_oms.so; intermediate/${{BUILD_CONFIGURATION}}/out_oms.so; 755; root; root
Copy link
Member

@vishiy vishiy Sep 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add the 2 log files (& touch chmod as for other log files ex;- kubernetes_client_log)

/etc/opt/microsoft/docker-cimprov/td-agent-bit.conf; installer/conf/td-agent-bit.conf; 644; root; root
/etc/opt/microsoft/docker-cimprov/out_oms.conf; installer/conf/out_oms.conf; 644; root; root

%Links
/opt/omi/lib/libcontainer.${{SHLIB_EXT}}; /opt/microsoft/docker-cimprov/lib/libcontainer.${{SHLIB_EXT}}; 644; root; root
Expand Down Expand Up @@ -76,6 +78,9 @@ MAINTAINER: 'Microsoft Corporation'
/var/opt/microsoft/docker-cimprov/state/ImageInventory; 755; root; root
/var/opt/microsoft/docker-cimprov/log; 755; root; root

/opt/td-agent-bit; 755; root; root;sysdir
/opt/td-agent-bit/bin; 755; root; root;sysdir

%Dependencies

%Postinstall_10
Expand Down
20 changes: 20 additions & 0 deletions source/code/go/src/plugins/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
GITVERSION := 0.1
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S),Linux)
BUILDDATE := $(shell date --rfc-3339=seconds)
endif
ifeq ($(UNAME_S),Darwin)
BUILDDATE := $(shell gdate --rfc-3339=seconds)
endif

fbplugin:
go build -ldflags "-X 'main.revision=$(GITVERSION)' -X 'main.builddate=$(BUILDDATE)'" -buildmode=c-shared -o out_oms.so .

test:
go test -cover -race -coverprofile=coverage.txt -covermode=atomic

glide:
glide install

clean:
rm -rf *.so *.h *~
209 changes: 209 additions & 0 deletions source/code/go/src/plugins/glide.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions source/code/go/src/plugins/glide.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package: plugins
import:
- package: github.com/fluent/fluent-bit-go
subpackages:
- output
- package: github.com/mitchellh/mapstructure
version: ^1.0.0
- package: k8s.io/apimachinery
subpackages:
- pkg/apis/meta/v1
- package: k8s.io/client-go
version: ^8.0.0
subpackages:
- kubernetes
- rest
Loading