diff --git a/.github/workflows/run_unit_tests.yml b/.github/workflows/run_unit_tests.yml new file mode 100644 index 000000000..29f5afc7a --- /dev/null +++ b/.github/workflows/run_unit_tests.yml @@ -0,0 +1,30 @@ +name: Run Unit Tests +on: + pull_request: + types: [opened, synchronize, reopened] + branches: + - ci_dev + - ci_prod +jobs: + Golang-Tests: + runs-on: ubuntu-latest + steps: + - name: Check out repository code + uses: actions/checkout@v2 + - name: Run unit tests + run: | + cd ${{ github.workspace }} + ./test/unit-tests/run_go_tests.sh + Ruby-Tests: + runs-on: ubuntu-latest + steps: + - name: Check out repository code + uses: actions/checkout@v2 + - name: install fluent + run: | + sudo gem install fluentd -v "1.12.2" --no-document + sudo fluentd --setup ./fluent + - name: Run unit tests + run: | + cd ${{ github.workspace }} + ./test/unit-tests/run_ruby_tests.sh diff --git a/.gitignore b/.gitignore index 2e2978e91..b0467519c 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,7 @@ intermediate kubernetes/linux/Linux_ULINUX_1.0_x64_64_Release # ignore generated .h files for go source/plugins/go/src/*.h +*_mock.go +*_log.txt +*.log +*.byebug_history diff --git a/Dev Guide.md b/Dev Guide.md new file mode 100644 index 000000000..7057a4afe --- /dev/null +++ b/Dev Guide.md @@ -0,0 +1,125 @@ +# Dev Guide + +More advanced information needed to develop or build the docker provider will live here + + + +## Testing +Last updated 8/18/2021 + +To run all unit tests run the commands `test/unit-tests/run_go_tests.sh` and `test/unit-tests/run_ruby_tests.sh` + +#### Conventions: +1. Unit tests should go in their own file, but in the same folder as the source code their testing. For example, the tests for `in_kube_nodes.rb` are in `in_kube_nodes_test.rb`. Both files are in the folder `source/plugin/ruby`. + +### Ruby +Sample tests are provided in [in_kube_nodes_test.rb](source/plugin/ruby/in_kube_nodes_test.rb). They are meant to demo the tooling used for unit tests (as opposed to being comprehensive tests). Basic techniques like mocking are demonstrated there. + +#### Conventions: +1. When modifying a fluentd plugin for unit testing, any mocked classes (like KubernetesApiClient, applicationInsightsUtility, env, etc.) should be passed in as optional arguments of initialize. For example: +``` + def initialize + super +``` +would be turned into +``` + def initialize (kubernetesApiClient=nil, applicationInsightsUtility=nil, extensionUtils=nil, env=nil) + super() +``` + +2. Having end-to-end tests of all fluentd plugins is a longshot. We care more about unit testing smaller blocks of functionality (like all the helper functions in KubeNodeInventory.rb). Unit tests for fluentd plugins are not expected. + +### Golang + +Since golang is statically compiled, mocking requires a lot more work than in ruby. Sample tests are provided in [utils_test.go](source/plugin/go/src/utils_test.go) and [extension_test.go](source/plugin/go/src/extension/extension_test.go). Again, they are meant to demo the tooling used for unit tests (as opposed to being comprehensive tests). Basic techniques like mocking are demonstrated there. + +#### Mocking: +Mocks are generated with gomock (mockgen). +* Mock files should be called *_mock.go (socket_writer.go => socket_writer_mock.go) +* Mocks should not be checked in to git. (they have been added to the .gitignore) +* The command to generate mock files should go in a `//go:generate` comment at the top of the mocked file (see [socket_writer.go](source/plugin/go/src/extension/socket_writer.go) for an example). This way mocks can be generated by the unit test script. +* Mocks also go in the same folder as the mocked files. This is unfortunate, but necessary to avoid circular package dependencies (anyone else feel free to figure out how to move mocks to a separate folder) + +Using mocks is also a little tricky. In order to mock functions in a package with gomock, they must be converted to reciever methods of a struct. This way the struct can be swapped out at runtime to change which implementaions of a method are called. See the example below: + +``` +// declare all functions to be mocked in this interface +type registrationPreCheckerInterface interface { + FUT(string) bool +} + +// Create a struct which implements the above interface +type regPreCheck struct{} + +func (r regPreCheck) FUT(email string) bool { + fmt.Println("real FUT() called") + return true +} + +// Create a global variable and assign it to the struct +var regPreCondVar registrationPreCheckerInterface + +func init() { + regPreCondVar = regPreCheck{} +} +``` + +Now any code wishing to call FUT() will call `regPreCondVar.FUT("")` + +A unit test can substitute its own implementaion of FUT() like so + +``` +// This will hold the mock of FUT we want to substitute +var FUTMock func(email string) bool + +// create a new struct which implements the earlier interface +type regPreCheckMock struct{} + +func (u regPreCheckMock) FUT(email string) bool { + return FUTMock(email) +} +``` + +Everything is set up. Now a unit test can substitute in a mock like so: + +``` +func someUnitTest() { + // This will call the actual implementaion of FUT() + regPreCondVar.FUT("") + + // Now the test creates another struct to substitue. After this like all calls to FUT() will be diverted + regPreCondVar = regPreCheckMock{} + + // substute another function to run instead of FUT() + FUTMock = func(email string) bool { + fmt.Println("FUT 1 called") + return false + } + // This will call the function defined right above + regPreCondVar.FUT("") + + // We can substitue another implementation + FUTMock = func(email string) bool { + fmt.Println("FUT 2 called") + return false + } + regPreCondVar.FUT("") + + // put the old behavior back + regPreCondVar = regPreCheck{} + // this will call the actual implementation of FUT() + regPreCondVar.FUT("") + +} +``` + +A concrete example of this can be found in [socket_writer.go](source/plugin/go/src/extension/socket_writer.go) and [extension_test.go](source/plugin/go/src/extension/extension_test.go). Again, if anybody has a better way feel free to update this guide. + + + +A simpler way to test a specific function is to write wrapper functions. Test code calls the inner function (ReadFileContentsImpl) and product code calls the wrapper function (ReadFileContents). The wrapper function provides any outside state which a unit test would want to control (like a function to read a file). This option makes product code more verbose, but probably easier to read too. Either way is acceptable. +``` +func ReadFileContents(fullPathToFileName string) (string, error) { + return ReadFileContentsImpl(fullPathToFileName, ioutil.ReadFile) +} +``` diff --git a/build/linux/installer/datafiles/base_container.data b/build/linux/installer/datafiles/base_container.data index b71cafd49..d104a5084 100644 --- a/build/linux/installer/datafiles/base_container.data +++ b/build/linux/installer/datafiles/base_container.data @@ -148,10 +148,10 @@ MAINTAINER: 'Microsoft Corporation' /etc/fluent/plugin/MdmMetricsGenerator.rb; source/plugins/ruby/MdmMetricsGenerator.rb; 644; root; root /etc/fluent/plugin/MdmAlertTemplates.rb; source/plugins/ruby/MdmAlertTemplates.rb; 644; root; root -/etc/fluent/plugin/omslog.rb; source/plugins/utils/omslog.rb; 644; root; root -/etc/fluent/plugin/oms_common.rb; source/plugins/utils/oms_common.rb; 644; root; root -/etc/fluent/plugin/extension.rb; source/plugins/utils/extension.rb; 644; root; root -/etc/fluent/plugin/extension_utils.rb; source/plugins/utils/extension_utils.rb; 644; root; root +/etc/fluent/plugin/omslog.rb; source/plugins/ruby/omslog.rb; 644; root; root +/etc/fluent/plugin/oms_common.rb; source/plugins/ruby/oms_common.rb; 644; root; root +/etc/fluent/plugin/extension.rb; source/plugins/ruby/extension.rb; 644; root; root +/etc/fluent/plugin/extension_utils.rb; source/plugins/ruby/extension_utils.rb; 644; root; root /etc/fluent/kube.conf; build/linux/installer/conf/kube.conf; 644; root; root diff --git a/build/windows/Makefile.ps1 b/build/windows/Makefile.ps1 index 737abc92a..b9bd1f3e4 100644 --- a/build/windows/Makefile.ps1 +++ b/build/windows/Makefile.ps1 @@ -183,11 +183,7 @@ Write-Host("successfully copied installer files conf and scripts from :" + $inst $rubyplugindir = Join-Path -Path $rootdir -ChildPath "source\plugins\ruby" Write-Host("copying ruby source files from :" + $rubyplugindir + " to :" + $publishdir + " ...") Copy-Item -Path $rubyplugindir -Destination $publishdir -Recurse -Force +Get-ChildItem $Path | Where{$_.Name -Match ".*_test\.rb"} | Remove-Item Write-Host("successfully copied ruby source files from :" + $rubyplugindir + " to :" + $publishdir + " ") -ForegroundColor Green -$utilsplugindir = Join-Path -Path $rootdir -ChildPath "source\plugins\utils" -Write-Host("copying ruby util files from :" + $utilsplugindir + " to :" + $publishdir + " ...") -Copy-Item -Path $utilsplugindir -Destination $publishdir -Recurse -Force -Write-Host("successfully copied ruby util files from :" + $utilsplugindir + " to :" + $publishdir + " ") -ForegroundColor Green - -Set-Location $currentdir \ No newline at end of file +Set-Location $currentdir diff --git a/kubernetes/windows/Dockerfile b/kubernetes/windows/Dockerfile index 0ba64cd75..290deef40 100644 --- a/kubernetes/windows/Dockerfile +++ b/kubernetes/windows/Dockerfile @@ -71,7 +71,6 @@ COPY ./omsagentwindows/installer/scripts/rubyKeepCertificateAlive/*.rb /etc/flue #Copy fluentd ruby plugins COPY ./omsagentwindows/ruby/ /etc/fluent/plugin/ -COPY ./omsagentwindows/utils/*.rb /etc/fluent/plugin/ ENV AGENT_VERSION ${IMAGE_TAG} ENV OS_TYPE "windows" diff --git a/kubernetes/windows/Dockerfile-dev-image b/kubernetes/windows/Dockerfile-dev-image index 6764ef8c4..35aa83bd9 100644 --- a/kubernetes/windows/Dockerfile-dev-image +++ b/kubernetes/windows/Dockerfile-dev-image @@ -33,7 +33,6 @@ COPY ./omsagentwindows/installer/scripts/rubyKeepCertificateAlive/*.rb /etc/flue #Copy fluentd ruby plugins COPY ./omsagentwindows/ruby/ /etc/fluent/plugin/ -COPY ./omsagentwindows/utils/*.rb /etc/fluent/plugin/ ENV AGENT_VERSION ${IMAGE_TAG} ENV OS_TYPE "windows" diff --git a/source/plugins/go/src/extension/extension.go b/source/plugins/go/src/extension/extension.go index c68140ded..4d78380bc 100644 --- a/source/plugins/go/src/extension/extension.go +++ b/source/plugins/go/src/extension/extension.go @@ -1,12 +1,13 @@ package extension -import ( +import ( "encoding/json" "fmt" - "log" + "log" + "strings" "sync" - "strings" - uuid "github.com/google/uuid" + + uuid "github.com/google/uuid" "github.com/ugorji/go/codec" ) @@ -14,31 +15,31 @@ type Extension struct { datatypeStreamIdMap map[string]string } -var singleton *Extension +var singleton *Extension var once sync.Once var extensionconfiglock sync.Mutex var logger *log.Logger -var containerType string +var containerType string -func GetInstance(flbLogger *log.Logger, containerType string) *Extension { - once.Do(func() { - singleton = &Extension{make(map[string]string)} +func GetInstance(flbLogger *log.Logger, containertype string) *Extension { + once.Do(func() { + singleton = &Extension{make(map[string]string)} flbLogger.Println("Extension Instance created") - }) + }) logger = flbLogger - containerType = containerType - return singleton + containerType = containertype + return singleton } func (e *Extension) GetOutputStreamId(datatype string) string { extensionconfiglock.Lock() - defer extensionconfiglock.Unlock() + defer extensionconfiglock.Unlock() if len(e.datatypeStreamIdMap) > 0 && e.datatypeStreamIdMap[datatype] != "" { message := fmt.Sprintf("OutputstreamId: %s for the datatype: %s", e.datatypeStreamIdMap[datatype], datatype) logger.Printf(message) return e.datatypeStreamIdMap[datatype] } - var err error + var err error e.datatypeStreamIdMap, err = getDataTypeToStreamIdMapping() if err != nil { message := fmt.Sprintf("Error getting datatype to streamid mapping: %s", err.Error()) @@ -54,29 +55,30 @@ func getDataTypeToStreamIdMapping() (map[string]string, error) { taggedData := map[string]interface{}{"Request": "AgentTaggedData", "RequestId": guid.String(), "Tag": "ContainerInsights", "Version": "1"} jsonBytes, err := json.Marshal(taggedData) + // TODO: this error is unhandled var data []byte - enc := codec.NewEncoderBytes(&data, new(codec.MsgpackHandle)) + enc := codec.NewEncoderBytes(&data, new(codec.MsgpackHandle)) if err := enc.Encode(string(jsonBytes)); err != nil { return datatypeOutputStreamMap, err } - - fs := &FluentSocketWriter{ } + + fs := &FluentSocket{} fs.sockAddress = "/var/run/mdsd/default_fluent.socket" if containerType != "" && strings.Compare(strings.ToLower(containerType), "prometheussidecar") == 0 { fs.sockAddress = fmt.Sprintf("/var/run/mdsd-%s/default_fluent.socket", containerType) - } - responseBytes, err := fs.WriteAndRead(data) - defer fs.disConnect() + } + responseBytes, err := FluentSocketWriter.writeAndRead(fs, data) + defer FluentSocketWriter.disconnect(fs) logger.Printf("Info::mdsd::Making call to FluentSocket: %s to write and read the config data", fs.sockAddress) if err != nil { return datatypeOutputStreamMap, err } - response := string(responseBytes) + response := string(responseBytes) // TODO: why is this converted to a string then back into a []byte? var responseObjet AgentTaggedDataResponse err = json.Unmarshal([]byte(response), &responseObjet) - if err != nil { + if err != nil { logger.Printf("Error::mdsd::Failed to unmarshal config data. Error message: %s", string(err.Error())) return datatypeOutputStreamMap, err } @@ -84,16 +86,16 @@ func getDataTypeToStreamIdMapping() (map[string]string, error) { var extensionData TaggedData json.Unmarshal([]byte(responseObjet.TaggedData), &extensionData) - extensionConfigs := extensionData.ExtensionConfigs - logger.Printf("Info::mdsd::build the datatype and streamid map -- start") + extensionConfigs := extensionData.ExtensionConfigs + logger.Printf("Info::mdsd::build the datatype and streamid map -- start") for _, extensionConfig := range extensionConfigs { outputStreams := extensionConfig.OutputStreams for dataType, outputStreamID := range outputStreams { logger.Printf("Info::mdsd::datatype: %s, outputstreamId: %s", dataType, outputStreamID) datatypeOutputStreamMap[dataType] = outputStreamID.(string) - } + } } - logger.Printf("Info::mdsd::build the datatype and streamid map -- end") + logger.Printf("Info::mdsd::build the datatype and streamid map -- end") logger.Printf("extensionconfig::getDataTypeToStreamIdMapping:: getting extension config from fluent socket-end") diff --git a/source/plugins/go/src/extension/extension_test.go b/source/plugins/go/src/extension/extension_test.go new file mode 100644 index 000000000..c3b5ef472 --- /dev/null +++ b/source/plugins/go/src/extension/extension_test.go @@ -0,0 +1,74 @@ +package extension + +import ( + "fmt" + "log" + "os" + reflect "reflect" + "testing" + + "github.com/golang/mock/gomock" +) + +type FluentSocketWriterMock struct{} + +func Test_getDataTypeToStreamIdMapping(t *testing.T) { + + type test_struct struct { + testName string + mdsdResponse string + fluentSocket FluentSocket + output map[string]string + err error + } + + // This is a pretty useless unit test, but it demonstrates the concept (putting together a real test + // would require some large json structs). If getDataTypeToStreamIdMapping() is ever updated, that + // would be a good opertunity to add some real test cases. + tests := []test_struct{ + { + "basic test", + "{}", + FluentSocket{}, + map[string]string{}, + nil, + }, + } + + for _, tt := range tests { + t.Run(tt.testName, func(t *testing.T) { + mockCtrl := gomock.NewController(t) + defer mockCtrl.Finish() + mock := NewMockIFluentSocketWriter(mockCtrl) + sock := &FluentSocket{} + sock.sockAddress = "/var/run/mdsd/default_fluent.socket" + mock.EXPECT().writeAndRead(sock, gomock.Any()).Return([]byte(tt.mdsdResponse), nil).Times(1) + mock.EXPECT().disconnect(sock).Return(nil).Times(1) + + // This is where calls to the normal socket writer calls are redirected to the mock. + ActualFluentSocketWriter := FluentSocketWriter // save the old struct so that we can put it back later + FluentSocketWriter = mock + + logfile, err := os.Create("logFile.txt") + if err != nil { + fmt.Println(err.Error()) + } + + // use an actual logger here. Using a real logger then cleaning up the log file later is easier than mocking the logger. + GetInstance(log.New(logfile, "", 0), "ContainerType") + defer os.Remove("logFile.txt") + + got, reterr := getDataTypeToStreamIdMapping() + if reterr != nil { + t.Errorf("got error") + t.Errorf(err.Error()) + } + if !reflect.DeepEqual(got, tt.output) { + t.Errorf("getDataTypeToStreamIdMapping() = %v, want %v", got, tt.output) + } + + // stop redirecting method calls to the mock + FluentSocketWriter = ActualFluentSocketWriter + }) + } +} diff --git a/source/plugins/go/src/extension/socket_writer.go b/source/plugins/go/src/extension/socket_writer.go index 1b16b319c..bfd35f5e6 100644 --- a/source/plugins/go/src/extension/socket_writer.go +++ b/source/plugins/go/src/extension/socket_writer.go @@ -4,20 +4,45 @@ import ( "net" ) +//go:generate mockgen -destination=socket_writer_mock.go -package=extension Docker-Provider/source/plugins/go/src/extension IFluentSocketWriter + //MaxRetries for trying to write data to the socket const MaxRetries = 5 //ReadBufferSize for reading data from sockets //Current CI extension config size is ~5KB and going with 20KB to handle any future scenarios -const ReadBufferSize = 20480 +const ReadBufferSize = 20480 //FluentSocketWriter writes data to AMA's default fluent socket -type FluentSocketWriter struct { - socket net.Conn - sockAddress string +type FluentSocket struct { + socket net.Conn + sockAddress string +} + +// begin mocking boilerplate +type IFluentSocketWriter interface { + connect(fluentSocket *FluentSocket) error + disconnect(fluentSocket *FluentSocket) error + writeWithRetries(fluentSocket *FluentSocket, data []byte) (int, error) + read(fluentSocket *FluentSocket) ([]byte, error) + write(fluentSocket *FluentSocket, payload []byte) (int, error) + writeAndRead(fluentSocket *FluentSocket, payload []byte) ([]byte, error) +} + +type FluentSocketWriterImpl struct{} + +// Methods in this file can by mocked by replacing FluentSocketWriter with a different struct. The methods +// in this file are all tied to the FluentSocketWriterImpl struct, but other structs could implement +// IFluentSocketWriter and be used instead +var FluentSocketWriter IFluentSocketWriter + +func init() { + FluentSocketWriter = FluentSocketWriterImpl{} } -func (fs *FluentSocketWriter) connect() error { +// end mocking boilerplate + +func (FluentSocketWriterImpl) connect(fs *FluentSocket) error { c, err := net.Dial("unix", fs.sockAddress) if err != nil { return err @@ -26,15 +51,15 @@ func (fs *FluentSocketWriter) connect() error { return nil } -func (fs *FluentSocketWriter) disConnect() error { - if (fs.socket != nil) { - fs.socket.Close() +func (FluentSocketWriterImpl) disconnect(fs *FluentSocket) error { + if fs.socket != nil { + fs.socket.Close() fs.socket = nil } return nil } -func (fs *FluentSocketWriter) writeWithRetries(data []byte) (int, error) { +func (FluentSocketWriterImpl) writeWithRetries(fs *FluentSocket, data []byte) (int, error) { var ( err error n int @@ -54,7 +79,7 @@ func (fs *FluentSocketWriter) writeWithRetries(data []byte) (int, error) { return 0, err } -func (fs *FluentSocketWriter) read() ([]byte, error) { +func (FluentSocketWriterImpl) read(fs *FluentSocket) ([]byte, error) { buf := make([]byte, ReadBufferSize) n, err := fs.socket.Read(buf) if err != nil { @@ -64,22 +89,22 @@ func (fs *FluentSocketWriter) read() ([]byte, error) { } -func (fs *FluentSocketWriter) Write(payload []byte) (int, error) { +func (FluentSocketWriterImpl) write(fs *FluentSocket, payload []byte) (int, error) { if fs.socket == nil { // previous write failed with permanent error and socket was closed. - if err := fs.connect(); err != nil { + if err := FluentSocketWriter.connect(fs); err != nil { return 0, err } } - return fs.writeWithRetries(payload) + return FluentSocketWriter.writeWithRetries(fs, payload) } -//WriteAndRead writes data to the socket and sends the response back -func (fs *FluentSocketWriter) WriteAndRead(payload []byte) ([]byte, error) { - _, err := fs.Write(payload) +//writeAndRead writes data to the socket and sends the response back +func (FluentSocketWriterImpl) writeAndRead(fs *FluentSocket, payload []byte) ([]byte, error) { + _, err := FluentSocketWriter.write(fs, payload) if err != nil { return nil, err } - return fs.read() + return FluentSocketWriter.read(fs) } diff --git a/source/plugins/go/src/go.mod b/source/plugins/go/src/go.mod index 4ead145ac..58e668597 100644 --- a/source/plugins/go/src/go.mod +++ b/source/plugins/go/src/go.mod @@ -8,6 +8,7 @@ require ( github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect github.com/dnaeon/go-vcr v1.2.0 // indirect github.com/fluent/fluent-bit-go v0.0.0-20171103221316-c4a158a6e3a7 + github.com/golang/mock v1.4.1 github.com/google/uuid v1.1.2 github.com/microsoft/ApplicationInsights-Go v0.4.3 github.com/philhofer/fwd v1.1.1 // indirect diff --git a/source/plugins/go/src/go.sum b/source/plugins/go/src/go.sum index 7f93bb260..ad9e40089 100644 --- a/source/plugins/go/src/go.sum +++ b/source/plugins/go/src/go.sum @@ -130,6 +130,7 @@ github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfb github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= +github.com/golang/mock v1.4.1 h1:ocYkMQY5RrXTYgXl7ICpV0IXwlEQGwKIsery4gyXa1U= github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= diff --git a/source/plugins/go/src/utils.go b/source/plugins/go/src/utils.go index 02d30607e..6b3036f85 100644 --- a/source/plugins/go/src/utils.go +++ b/source/plugins/go/src/utils.go @@ -12,8 +12,8 @@ import ( "net/url" "os" "strings" - "time" - + "time" + "github.com/Azure/azure-kusto-go/kusto" "github.com/Azure/azure-kusto-go/kusto/ingest" "github.com/Azure/go-autorest/autorest/azure/auth" @@ -87,7 +87,7 @@ func CreateHTTPClient() { } tlsConfig.BuildNameToCertificate() - transport = &http.Transport{TLSClientConfig: tlsConfig} + transport = &http.Transport{TLSClientConfig: tlsConfig} } // set the proxy if the proxy configured if ProxyEndpoint != "" { @@ -105,7 +105,7 @@ func CreateHTTPClient() { HTTPClient = http.Client{ Transport: transport, Timeout: 30 * time.Second, - } + } Log("Successfully created HTTP Client") } @@ -123,57 +123,57 @@ func ToString(s interface{}) string { //mdsdSocketClient to write msgp messages func CreateMDSDClient(dataType DataType, containerType string) { - mdsdfluentSocket := "/var/run/mdsd/default_fluent.socket" + mdsdfluentSocket := "/var/run/mdsd/default_fluent.socket" if containerType != "" && strings.Compare(strings.ToLower(containerType), "prometheussidecar") == 0 { - mdsdfluentSocket = fmt.Sprintf("/var/run/mdsd-%s/default_fluent.socket", containerType) - } + mdsdfluentSocket = fmt.Sprintf("/var/run/mdsd-%s/default_fluent.socket", containerType) + } switch dataType { - case ContainerLogV2: - if MdsdMsgpUnixSocketClient != nil { - MdsdMsgpUnixSocketClient.Close() - MdsdMsgpUnixSocketClient = nil - } - /*conn, err := fluent.New(fluent.Config{FluentNetwork:"unix", - FluentSocketPath:"/var/run/mdsd/default_fluent.socket", - WriteTimeout: 5 * time.Second, - RequestAck: true}) */ - conn, err := net.DialTimeout("unix", - mdsdfluentSocket, 10*time.Second) - if err != nil { - Log("Error::mdsd::Unable to open MDSD msgp socket connection for ContainerLogV2 %s", err.Error()) - //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) - } else { - Log("Successfully created MDSD msgp socket connection for ContainerLogV2: %s", mdsdfluentSocket) - MdsdMsgpUnixSocketClient = conn - } - case KubeMonAgentEvents: - if MdsdKubeMonMsgpUnixSocketClient != nil { - MdsdKubeMonMsgpUnixSocketClient.Close() - MdsdKubeMonMsgpUnixSocketClient = nil - } - conn, err := net.DialTimeout("unix", - mdsdfluentSocket, 10*time.Second) - if err != nil { - Log("Error::mdsd::Unable to open MDSD msgp socket connection for KubeMon events %s", err.Error()) - //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) - } else { - Log("Successfully created MDSD msgp socket connection for KubeMon events:%s", mdsdfluentSocket) - MdsdKubeMonMsgpUnixSocketClient = conn - } - case InsightsMetrics: - if MdsdInsightsMetricsMsgpUnixSocketClient != nil { - MdsdInsightsMetricsMsgpUnixSocketClient.Close() - MdsdInsightsMetricsMsgpUnixSocketClient = nil - } - conn, err := net.DialTimeout("unix", - mdsdfluentSocket, 10*time.Second) - if err != nil { - Log("Error::mdsd::Unable to open MDSD msgp socket connection for insights metrics %s", err.Error()) - //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) - } else { - Log("Successfully created MDSD msgp socket connection for Insights metrics %s", mdsdfluentSocket) - MdsdInsightsMetricsMsgpUnixSocketClient = conn - } + case ContainerLogV2: + if MdsdMsgpUnixSocketClient != nil { + MdsdMsgpUnixSocketClient.Close() + MdsdMsgpUnixSocketClient = nil + } + /*conn, err := fluent.New(fluent.Config{FluentNetwork:"unix", + FluentSocketPath:"/var/run/mdsd/default_fluent.socket", + WriteTimeout: 5 * time.Second, + RequestAck: true}) */ + conn, err := net.DialTimeout("unix", + mdsdfluentSocket, 10*time.Second) + if err != nil { + Log("Error::mdsd::Unable to open MDSD msgp socket connection for ContainerLogV2 %s", err.Error()) + //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) + } else { + Log("Successfully created MDSD msgp socket connection for ContainerLogV2: %s", mdsdfluentSocket) + MdsdMsgpUnixSocketClient = conn + } + case KubeMonAgentEvents: + if MdsdKubeMonMsgpUnixSocketClient != nil { + MdsdKubeMonMsgpUnixSocketClient.Close() + MdsdKubeMonMsgpUnixSocketClient = nil + } + conn, err := net.DialTimeout("unix", + mdsdfluentSocket, 10*time.Second) + if err != nil { + Log("Error::mdsd::Unable to open MDSD msgp socket connection for KubeMon events %s", err.Error()) + //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) + } else { + Log("Successfully created MDSD msgp socket connection for KubeMon events:%s", mdsdfluentSocket) + MdsdKubeMonMsgpUnixSocketClient = conn + } + case InsightsMetrics: + if MdsdInsightsMetricsMsgpUnixSocketClient != nil { + MdsdInsightsMetricsMsgpUnixSocketClient.Close() + MdsdInsightsMetricsMsgpUnixSocketClient = nil + } + conn, err := net.DialTimeout("unix", + mdsdfluentSocket, 10*time.Second) + if err != nil { + Log("Error::mdsd::Unable to open MDSD msgp socket connection for insights metrics %s", err.Error()) + //log.Fatalf("Unable to open MDSD msgp socket connection %s", err.Error()) + } else { + Log("Successfully created MDSD msgp socket connection for Insights metrics %s", mdsdfluentSocket) + MdsdInsightsMetricsMsgpUnixSocketClient = conn + } } } @@ -202,11 +202,15 @@ func CreateADXClient() { } func ReadFileContents(fullPathToFileName string) (string, error) { + return ReadFileContentsImpl(fullPathToFileName, ioutil.ReadFile) +} + +func ReadFileContentsImpl(fullPathToFileName string, readfilefunc func(string) ([]byte, error)) (string, error) { fullPathToFileName = strings.TrimSpace(fullPathToFileName) if len(fullPathToFileName) == 0 { return "", errors.New("ReadFileContents::filename is empty") } - content, err := ioutil.ReadFile(fullPathToFileName) //no need to close + content, err := readfilefunc(fullPathToFileName) //no need to close if err != nil { return "", errors.New("ReadFileContents::Unable to open file " + fullPathToFileName) } else { @@ -228,7 +232,7 @@ func isValidUrl(uri string) bool { func convertMsgPackEntriesToMsgpBytes(fluentForwardTag string, msgPackEntries []MsgPackEntry) []byte { var msgpBytes []byte - + fluentForward := MsgPackForward{ Tag: fluentForwardTag, Entries: msgPackEntries, @@ -239,7 +243,7 @@ func convertMsgPackEntriesToMsgpBytes(fluentForwardTag string, msgPackEntries [] msgpSize += 1 + msgp.Int64Size + msgp.GuessSize(fluentForward.Entries[i].Record) } - //allocate buffer for msgp message + //allocate buffer for msgp message msgpBytes = msgp.Require(nil, msgpSize) //construct the stream @@ -252,6 +256,6 @@ func convertMsgPackEntriesToMsgpBytes(fluentForwardTag string, msgPackEntries [] msgpBytes = msgp.AppendInt64(msgpBytes, batchTime) msgpBytes = msgp.AppendMapStrStr(msgpBytes, fluentForward.Entries[entry].Record) } - - return msgpBytes + + return msgpBytes } diff --git a/source/plugins/go/src/utils_test.go b/source/plugins/go/src/utils_test.go new file mode 100644 index 000000000..ab61ce751 --- /dev/null +++ b/source/plugins/go/src/utils_test.go @@ -0,0 +1,79 @@ +package main + +import ( + "errors" + "testing" +) + +func Test_isValidUrl(t *testing.T) { + type test_struct struct { + isValid bool + url string + } + + tests := []test_struct{ + {true, "https://www.microsoft.com"}, + {true, "http://abc.xyz"}, + {true, "https://www.microsoft.com/tests"}, + {false, "()"}, + {false, "https//www.microsoft.com"}, + {false, "https:/www.microsoft.com"}, + {false, "https:/www.microsoft.com*"}, + {false, ""}, + } + + for _, tt := range tests { + t.Run(tt.url, func(t *testing.T) { + got := isValidUrl(tt.url) + if got != tt.isValid { + t.Errorf("isValidUrl(%s) = %t, want %t", tt.url, got, tt.isValid) + } + }) + } +} + +func Test_ReadFileContents(t *testing.T) { + type mock_struct struct { + expectedFilePath string + fileContents []byte + err error + } + type test_struct struct { + testname string + calledFilePath string + subcall_spec mock_struct + output string + err bool + } + + tests := []test_struct{ + {"normal", "foobar.txt", mock_struct{"foobar.txt", []byte("asdf"), nil}, "asdf", false}, + {"extra whitespace", "foobar.txt ", mock_struct{"foobar.txt", []byte("asdf \t"), nil}, "asdf", false}, + {"empty filename", "", mock_struct{"", []byte(""), nil}, "", true}, + {"file doesn't exist", "asdf.txt", mock_struct{"asdf", []byte(""), errors.New("this error doesn't matter much")}, "", true}, + } + + for _, tt := range tests { + t.Run(string(tt.testname), func(t *testing.T) { + + readfileFunc := func(filename string) ([]byte, error) { + if filename == tt.subcall_spec.expectedFilePath { + return tt.subcall_spec.fileContents, nil + } + return []byte(""), errors.New("file not found") + } + + got, err := ReadFileContentsImpl(tt.calledFilePath, readfileFunc) + + if got != tt.output || !(tt.err == (err != nil)) { + t.Errorf("ReadFileContents(%v) = (%v, %v), want (%v, %v)", tt.calledFilePath, got, err, tt.output, tt.err) + if got != tt.output { + t.Errorf("output strings are not equal") + } + if tt.err == (err != nil) { + t.Errorf("errors are not equal") + } + } + }) + } +} diff --git a/source/plugins/ruby/CAdvisorMetricsAPIClient.rb b/source/plugins/ruby/CAdvisorMetricsAPIClient.rb index da6e94f5f..017bfb08d 100644 --- a/source/plugins/ruby/CAdvisorMetricsAPIClient.rb +++ b/source/plugins/ruby/CAdvisorMetricsAPIClient.rb @@ -40,9 +40,9 @@ class CAdvisorMetricsAPIClient @os_type = ENV["OS_TYPE"] if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0 - @LogPath = "/etc/omsagentwindows/kubernetes_perf_log.txt" + @LogPath = Constants::WINDOWS_LOG_PATH + "kubernetes_perf_log.txt" else - @LogPath = "/var/opt/microsoft/docker-cimprov/log/kubernetes_perf_log.txt" + @LogPath = Constants::LINUX_LOG_PATH + "kubernetes_perf_log.txt" end @Log = Logger.new(@LogPath, 2, 10 * 1048576) #keep last 2 files, max log file size = 10M # @@rxBytesLast = nil diff --git a/source/plugins/ruby/KubernetesApiClient.rb b/source/plugins/ruby/KubernetesApiClient.rb index 4b50e20d8..8925248d7 100644 --- a/source/plugins/ruby/KubernetesApiClient.rb +++ b/source/plugins/ruby/KubernetesApiClient.rb @@ -25,11 +25,12 @@ class KubernetesApiClient #@@IsValidRunningNode = nil #@@IsLinuxCluster = nil @@KubeSystemNamespace = "kube-system" + @os_type = ENV["OS_TYPE"] if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0 - @LogPath = "/etc/omsagentwindows/kubernetes_client_log.txt" + @LogPath = Constants::WINDOWS_LOG_PATH + "kubernetes_client_log.txt" else - @LogPath = "/var/opt/microsoft/docker-cimprov/log/kubernetes_client_log.txt" + @LogPath = Constants::LINUX_LOG_PATH + "kubernetes_client_log.txt" end @Log = Logger.new(@LogPath, 2, 10 * 1048576) #keep last 2 files, max log file size = 10M @@TokenFileName = "/var/run/secrets/kubernetes.io/serviceaccount/token" @@ -87,42 +88,42 @@ def getTokenStr end end - def getClusterRegion - if ENV["AKS_REGION"] - return ENV["AKS_REGION"] + def getClusterRegion(env=ENV) + if env["AKS_REGION"] + return env["AKS_REGION"] else @Log.warn ("Kubernetes environment variable not set AKS_REGION. Unable to get cluster region.") return nil end end - def getResourceUri(resource, api_group) + def getResourceUri(resource, api_group, env=ENV) begin - if ENV["KUBERNETES_SERVICE_HOST"] && ENV["KUBERNETES_PORT_443_TCP_PORT"] + if env["KUBERNETES_SERVICE_HOST"] && env["KUBERNETES_PORT_443_TCP_PORT"] if api_group.nil? - return "https://#{ENV["KUBERNETES_SERVICE_HOST"]}:#{ENV["KUBERNETES_PORT_443_TCP_PORT"]}/api/" + @@ApiVersion + "/" + resource + return "https://#{env["KUBERNETES_SERVICE_HOST"]}:#{env["KUBERNETES_PORT_443_TCP_PORT"]}/api/" + @@ApiVersion + "/" + resource elsif api_group == @@ApiGroupApps - return "https://#{ENV["KUBERNETES_SERVICE_HOST"]}:#{ENV["KUBERNETES_PORT_443_TCP_PORT"]}/apis/apps/" + @@ApiVersionApps + "/" + resource + return "https://#{env["KUBERNETES_SERVICE_HOST"]}:#{env["KUBERNETES_PORT_443_TCP_PORT"]}/apis/apps/" + @@ApiVersionApps + "/" + resource elsif api_group == @@ApiGroupHPA - return "https://#{ENV["KUBERNETES_SERVICE_HOST"]}:#{ENV["KUBERNETES_PORT_443_TCP_PORT"]}/apis/" + @@ApiGroupHPA + "/" + @@ApiVersionHPA + "/" + resource + return "https://#{env["KUBERNETES_SERVICE_HOST"]}:#{env["KUBERNETES_PORT_443_TCP_PORT"]}/apis/" + @@ApiGroupHPA + "/" + @@ApiVersionHPA + "/" + resource end else - @Log.warn ("Kubernetes environment variable not set KUBERNETES_SERVICE_HOST: #{ENV["KUBERNETES_SERVICE_HOST"]} KUBERNETES_PORT_443_TCP_PORT: #{ENV["KUBERNETES_PORT_443_TCP_PORT"]}. Unable to form resourceUri") + @Log.warn ("Kubernetes environment variable not set KUBERNETES_SERVICE_HOST: #{env["KUBERNETES_SERVICE_HOST"]} KUBERNETES_PORT_443_TCP_PORT: #{env["KUBERNETES_PORT_443_TCP_PORT"]}. Unable to form resourceUri") return nil end end end - def getClusterName + def getClusterName(env=ENV) return @@ClusterName if !@@ClusterName.nil? @@ClusterName = "None" begin #try getting resource ID for aks - cluster = ENV["AKS_RESOURCE_ID"] + cluster = env["AKS_RESOURCE_ID"] if cluster && !cluster.nil? && !cluster.empty? @@ClusterName = cluster.split("/").last else - cluster = ENV["ACS_RESOURCE_NAME"] + cluster = env["ACS_RESOURCE_NAME"] if cluster && !cluster.nil? && !cluster.empty? @@ClusterName = cluster else @@ -147,7 +148,7 @@ def getClusterName return @@ClusterName end - def getClusterId + def getClusterId(env=ENV) return @@ClusterId if !@@ClusterId.nil? #By default initialize ClusterId to ClusterName. # In ACS/On-prem, we need to figure out how we can generate ClusterId @@ -155,7 +156,7 @@ def getClusterId # e.g. md5 digest is 128 bits = 32 character in hex. Get first 16 and get a guid, and the next 16 to get resource id @@ClusterId = getClusterName begin - cluster = ENV["AKS_RESOURCE_ID"] + cluster = env["AKS_RESOURCE_ID"] if cluster && !cluster.nil? && !cluster.empty? @@ClusterId = cluster end @@ -777,13 +778,13 @@ def getResourcesAndContinuationToken(uri, api_group: nil) return continuationToken, resourceInventory end #getResourcesAndContinuationToken - def getKubeAPIServerUrl + def getKubeAPIServerUrl(env=ENV) apiServerUrl = nil begin - if ENV["KUBERNETES_SERVICE_HOST"] && ENV["KUBERNETES_PORT_443_TCP_PORT"] - apiServerUrl = "https://#{ENV["KUBERNETES_SERVICE_HOST"]}:#{ENV["KUBERNETES_PORT_443_TCP_PORT"]}" + if env["KUBERNETES_SERVICE_HOST"] && env["KUBERNETES_PORT_443_TCP_PORT"] + apiServerUrl = "https://#{env["KUBERNETES_SERVICE_HOST"]}:#{env["KUBERNETES_PORT_443_TCP_PORT"]}" else - @Log.warn "Kubernetes environment variable not set KUBERNETES_SERVICE_HOST: #{ENV["KUBERNETES_SERVICE_HOST"]} KUBERNETES_PORT_443_TCP_PORT: #{ENV["KUBERNETES_PORT_443_TCP_PORT"]}. Unable to form resourceUri" + @Log.warn "Kubernetes environment variable not set KUBERNETES_SERVICE_HOST: #{env["KUBERNETES_SERVICE_HOST"]} KUBERNETES_PORT_443_TCP_PORT: #{env["KUBERNETES_PORT_443_TCP_PORT"]}. Unable to form resourceUri" end rescue => errorStr @Log.warn "KubernetesApiClient::getKubeAPIServerUrl:Failed #{errorStr}" diff --git a/source/plugins/ruby/constants.rb b/source/plugins/ruby/constants.rb index 40fa80c14..69da56488 100644 --- a/source/plugins/ruby/constants.rb +++ b/source/plugins/ruby/constants.rb @@ -129,5 +129,7 @@ class Constants ONEAGENT_FLUENT_SOCKET_NAME = "/var/run/mdsd/default_fluent.socket" #Tag prefix for output stream EXTENSION_OUTPUT_STREAM_ID_TAG_PREFIX = "dcr-" - + + LINUX_LOG_PATH = $in_unit_test.nil? ? "/var/opt/microsoft/docker-cimprov/log/" : "./" + WINDOWS_LOG_PATH = $in_unit_test.nil? ? "/etc/omsagentwindows/" : "./" end diff --git a/source/plugins/utils/extension.rb b/source/plugins/ruby/extension.rb similarity index 100% rename from source/plugins/utils/extension.rb rename to source/plugins/ruby/extension.rb diff --git a/source/plugins/utils/extension_utils.rb b/source/plugins/ruby/extension_utils.rb similarity index 100% rename from source/plugins/utils/extension_utils.rb rename to source/plugins/ruby/extension_utils.rb diff --git a/source/plugins/ruby/in_kube_nodes.rb b/source/plugins/ruby/in_kube_nodes.rb index bc62756a1..a32a32769 100644 --- a/source/plugins/ruby/in_kube_nodes.rb +++ b/source/plugins/ruby/in_kube_nodes.rb @@ -7,26 +7,13 @@ module Fluent::Plugin class Kube_nodeInventory_Input < Input Fluent::Plugin.register_input("kube_nodes", self) - @@configMapMountPath = "/etc/config/settings/log-data-collection-settings" - @@promConfigMountPath = "/etc/config/settings/prometheus-data-collection-settings" - @@osmConfigMountPath = "/etc/config/osm-settings/osm-metric-collection-configuration" - @@AzStackCloudFileName = "/etc/kubernetes/host/azurestackcloud.json" - - - @@rsPromInterval = ENV["TELEMETRY_RS_PROM_INTERVAL"] - @@rsPromFieldPassCount = ENV["TELEMETRY_RS_PROM_FIELDPASS_LENGTH"] - @@rsPromFieldDropCount = ENV["TELEMETRY_RS_PROM_FIELDDROP_LENGTH"] - @@rsPromK8sServiceCount = ENV["TELEMETRY_RS_PROM_K8S_SERVICES_LENGTH"] - @@rsPromUrlCount = ENV["TELEMETRY_RS_PROM_URLS_LENGTH"] - @@rsPromMonitorPods = ENV["TELEMETRY_RS_PROM_MONITOR_PODS"] - @@rsPromMonitorPodsNamespaceLength = ENV["TELEMETRY_RS_PROM_MONITOR_PODS_NS_LENGTH"] - @@rsPromMonitorPodsLabelSelectorLength = ENV["TELEMETRY_RS_PROM_LABEL_SELECTOR_LENGTH"] - @@rsPromMonitorPodsFieldSelectorLength = ENV["TELEMETRY_RS_PROM_FIELD_SELECTOR_LENGTH"] - @@collectAllKubeEvents = ENV["AZMON_CLUSTER_COLLECT_ALL_KUBE_EVENTS"] - @@osmNamespaceCount = ENV["TELEMETRY_OSM_CONFIGURATION_NAMESPACES_COUNT"] - - def initialize - super + def initialize (kubernetesApiClient=nil, + applicationInsightsUtility=nil, + extensionUtils=nil, + env=nil, + telemetry_flush_interval=nil) + super() + require "yaml" require "yajl/json_gem" require "yajl" @@ -38,6 +25,31 @@ def initialize require_relative "omslog" require_relative "extension_utils" + @kubernetesApiClient = kubernetesApiClient == nil ? KubernetesApiClient : kubernetesApiClient + @applicationInsightsUtility = applicationInsightsUtility == nil ? ApplicationInsightsUtility : applicationInsightsUtility + @extensionUtils = extensionUtils == nil ? ExtensionUtils : extensionUtils + @env = env == nil ? ENV : env + @TELEMETRY_FLUSH_INTERVAL_IN_MINUTES = telemetry_flush_interval == nil ? Constants::TELEMETRY_FLUSH_INTERVAL_IN_MINUTES : telemetry_flush_interval + + # these defines were previously at class scope Moving them into the constructor so that they can be set by unit tests + @@configMapMountPath = "/etc/config/settings/log-data-collection-settings" + @@promConfigMountPath = "/etc/config/settings/prometheus-data-collection-settings" + @@osmConfigMountPath = "/etc/config/osm-settings/osm-metric-collection-configuration" + @@AzStackCloudFileName = "/etc/kubernetes/host/azurestackcloud.json" + + + @@rsPromInterval = @env["TELEMETRY_RS_PROM_INTERVAL"] + @@rsPromFieldPassCount = @env["TELEMETRY_RS_PROM_FIELDPASS_LENGTH"] + @@rsPromFieldDropCount = @env["TELEMETRY_RS_PROM_FIELDDROP_LENGTH"] + @@rsPromK8sServiceCount = @env["TELEMETRY_RS_PROM_K8S_SERVICES_LENGTH"] + @@rsPromUrlCount = @env["TELEMETRY_RS_PROM_URLS_LENGTH"] + @@rsPromMonitorPods = @env["TELEMETRY_RS_PROM_MONITOR_PODS"] + @@rsPromMonitorPodsNamespaceLength = @env["TELEMETRY_RS_PROM_MONITOR_PODS_NS_LENGTH"] + @@rsPromMonitorPodsLabelSelectorLength = @env["TELEMETRY_RS_PROM_LABEL_SELECTOR_LENGTH"] + @@rsPromMonitorPodsFieldSelectorLength = @env["TELEMETRY_RS_PROM_FIELD_SELECTOR_LENGTH"] + @@collectAllKubeEvents = @env["AZMON_CLUSTER_COLLECT_ALL_KUBE_EVENTS"] + @@osmNamespaceCount = @env["TELEMETRY_OSM_CONFIGURATION_NAMESPACES_COUNT"] + @ContainerNodeInventoryTag = "oneagent.containerInsights.CONTAINER_NODE_INVENTORY_BLOB" @insightsMetricsTag = "oneagent.containerInsights.INSIGHTS_METRICS_BLOB" @MDMKubeNodeInventoryTag = "mdm.kubenodeinventory" @@ -64,8 +76,8 @@ def configure(conf) def start if @run_interval super - if !ENV["NODES_CHUNK_SIZE"].nil? && !ENV["NODES_CHUNK_SIZE"].empty? && ENV["NODES_CHUNK_SIZE"].to_i > 0 - @NODES_CHUNK_SIZE = ENV["NODES_CHUNK_SIZE"].to_i + if !@env["NODES_CHUNK_SIZE"].nil? && !@env["NODES_CHUNK_SIZE"].empty? && @env["NODES_CHUNK_SIZE"].to_i > 0 + @NODES_CHUNK_SIZE = @env["NODES_CHUNK_SIZE"].to_i else # this shouldnt happen just setting default here as safe guard $log.warn("in_kube_nodes::start: setting to default value since got NODES_CHUNK_SIZE nil or empty") @@ -73,8 +85,8 @@ def start end $log.info("in_kube_nodes::start : NODES_CHUNK_SIZE @ #{@NODES_CHUNK_SIZE}") - if !ENV["NODES_EMIT_STREAM_BATCH_SIZE"].nil? && !ENV["NODES_EMIT_STREAM_BATCH_SIZE"].empty? && ENV["NODES_EMIT_STREAM_BATCH_SIZE"].to_i > 0 - @NODES_EMIT_STREAM_BATCH_SIZE = ENV["NODES_EMIT_STREAM_BATCH_SIZE"].to_i + if !@env["NODES_EMIT_STREAM_BATCH_SIZE"].nil? && !@env["NODES_EMIT_STREAM_BATCH_SIZE"].empty? && @env["NODES_EMIT_STREAM_BATCH_SIZE"].to_i > 0 + @NODES_EMIT_STREAM_BATCH_SIZE = @env["NODES_EMIT_STREAM_BATCH_SIZE"].to_i else # this shouldnt happen just setting default here as safe guard $log.warn("in_kube_nodes::start: setting to default value since got NODES_EMIT_STREAM_BATCH_SIZE nil or empty") @@ -112,19 +124,19 @@ def enumerate @nodeInventoryE2EProcessingLatencyMs = 0 nodeInventoryStartTime = (Time.now.to_f * 1000).to_i - if ExtensionUtils.isAADMSIAuthMode() + if @extensionUtils.isAADMSIAuthMode() $log.info("in_kube_nodes::enumerate: AAD AUTH MSI MODE") if @kubeperfTag.nil? || !@kubeperfTag.start_with?(Constants::EXTENSION_OUTPUT_STREAM_ID_TAG_PREFIX) - @kubeperfTag = ExtensionUtils.getOutputStreamId(Constants::PERF_DATA_TYPE) + @kubeperfTag = @extensionUtils.getOutputStreamId(Constants::PERF_DATA_TYPE) end if @insightsMetricsTag.nil? || !@insightsMetricsTag.start_with?(Constants::EXTENSION_OUTPUT_STREAM_ID_TAG_PREFIX) - @insightsMetricsTag = ExtensionUtils.getOutputStreamId(Constants::INSIGHTS_METRICS_DATA_TYPE) + @insightsMetricsTag = @extensionUtils.getOutputStreamId(Constants::INSIGHTS_METRICS_DATA_TYPE) end if @ContainerNodeInventoryTag.nil? || !@ContainerNodeInventoryTag.start_with?(Constants::EXTENSION_OUTPUT_STREAM_ID_TAG_PREFIX) - @ContainerNodeInventoryTag = ExtensionUtils.getOutputStreamId(Constants::CONTAINER_NODE_INVENTORY_DATA_TYPE) + @ContainerNodeInventoryTag = @extensionUtils.getOutputStreamId(Constants::CONTAINER_NODE_INVENTORY_DATA_TYPE) end if @tag.nil? || !@tag.start_with?(Constants::EXTENSION_OUTPUT_STREAM_ID_TAG_PREFIX) - @tag = ExtensionUtils.getOutputStreamId(Constants::KUBE_NODE_INVENTORY_DATA_TYPE) + @tag = @extensionUtils.getOutputStreamId(Constants::KUBE_NODE_INVENTORY_DATA_TYPE) end $log.info("in_kube_nodes::enumerate: using perf tag -#{@kubeperfTag} @ #{Time.now.utc.iso8601}") $log.info("in_kube_nodes::enumerate: using insightsmetrics tag -#{@insightsMetricsTag} @ #{Time.now.utc.iso8601}") @@ -136,8 +148,9 @@ def enumerate # Initializing continuation token to nil continuationToken = nil $log.info("in_kube_nodes::enumerate : Getting nodes from Kube API @ #{Time.now.utc.iso8601}") + # KubernetesApiClient.getNodesResourceUri is a pure function, so call it from the actual module instead of from the mock resourceUri = KubernetesApiClient.getNodesResourceUri("nodes?limit=#{@NODES_CHUNK_SIZE}") - continuationToken, nodeInventory = KubernetesApiClient.getResourcesAndContinuationToken(resourceUri) + continuationToken, nodeInventory = @kubernetesApiClient.getResourcesAndContinuationToken(resourceUri) $log.info("in_kube_nodes::enumerate : Done getting nodes from Kube API @ #{Time.now.utc.iso8601}") nodesAPIChunkEndTime = (Time.now.to_f * 1000).to_i @nodesAPIE2ELatencyMs = (nodesAPIChunkEndTime - nodesAPIChunkStartTime) @@ -151,7 +164,7 @@ def enumerate #If we receive a continuation token, make calls, process and flush data until we have processed all data while (!continuationToken.nil? && !continuationToken.empty?) nodesAPIChunkStartTime = (Time.now.to_f * 1000).to_i - continuationToken, nodeInventory = KubernetesApiClient.getResourcesAndContinuationToken(resourceUri + "&continue=#{continuationToken}") + continuationToken, nodeInventory = @kubernetesApiClient.getResourcesAndContinuationToken(resourceUri + "&continue=#{continuationToken}") nodesAPIChunkEndTime = (Time.now.to_f * 1000).to_i @nodesAPIE2ELatencyMs = @nodesAPIE2ELatencyMs + (nodesAPIChunkEndTime - nodesAPIChunkStartTime) if (!nodeInventory.nil? && !nodeInventory.empty? && nodeInventory.key?("items") && !nodeInventory["items"].nil? && !nodeInventory["items"].empty?) @@ -165,9 +178,9 @@ def enumerate @nodeInventoryE2EProcessingLatencyMs = ((Time.now.to_f * 1000).to_i - nodeInventoryStartTime) timeDifference = (DateTime.now.to_time.to_i - @@nodeInventoryLatencyTelemetryTimeTracker).abs timeDifferenceInMinutes = timeDifference / 60 - if (timeDifferenceInMinutes >= Constants::TELEMETRY_FLUSH_INTERVAL_IN_MINUTES) - ApplicationInsightsUtility.sendMetricTelemetry("NodeInventoryE2EProcessingLatencyMs", @nodeInventoryE2EProcessingLatencyMs, {}) - ApplicationInsightsUtility.sendMetricTelemetry("NodesAPIE2ELatencyMs", @nodesAPIE2ELatencyMs, {}) + if (timeDifferenceInMinutes >= @TELEMETRY_FLUSH_INTERVAL_IN_MINUTES) + @applicationInsightsUtility.sendMetricTelemetry("NodeInventoryE2EProcessingLatencyMs", @nodeInventoryE2EProcessingLatencyMs, {}) + @applicationInsightsUtility.sendMetricTelemetry("NodesAPIE2ELatencyMs", @nodesAPIE2ELatencyMs, {}) @@nodeInventoryLatencyTelemetryTimeTracker = DateTime.now.to_time.to_i end # Setting this to nil so that we dont hold memory until GC kicks in @@ -175,7 +188,7 @@ def enumerate rescue => errorStr $log.warn "in_kube_nodes::enumerate:Failed in enumerate: #{errorStr}" $log.debug_backtrace(errorStr.backtrace) - ApplicationInsightsUtility.sendExceptionTelemetry(errorStr) + @applicationInsightsUtility.sendExceptionTelemetry(errorStr) end end # end enumerate @@ -188,7 +201,7 @@ def parse_and_emit_records(nodeInventory, batchTime = Time.utc.iso8601) containerNodeInventoryEventStream = Fluent::MultiEventStream.new insightsMetricsEventStream = Fluent::MultiEventStream.new kubePerfEventStream = Fluent::MultiEventStream.new - @@istestvar = ENV["ISTEST"] + @@istestvar = @env["ISTEST"] #get node inventory nodeInventory["items"].each do |item| # node inventory @@ -299,49 +312,79 @@ def parse_and_emit_records(nodeInventory, batchTime = Time.utc.iso8601) # Adding telemetry to send node telemetry every 10 minutes timeDifference = (DateTime.now.to_time.to_i - @@nodeTelemetryTimeTracker).abs timeDifferenceInMinutes = timeDifference / 60 - if (timeDifferenceInMinutes >= Constants::TELEMETRY_FLUSH_INTERVAL_IN_MINUTES) - properties = getNodeTelemetryProps(item) - properties["KubernetesProviderID"] = nodeInventoryRecord["KubernetesProviderID"] - capacityInfo = item["status"]["capacity"] - - ApplicationInsightsUtility.sendMetricTelemetry("NodeMemory", capacityInfo["memory"], properties) + if (timeDifferenceInMinutes >= @TELEMETRY_FLUSH_INTERVAL_IN_MINUTES) begin - if (!capacityInfo["nvidia.com/gpu"].nil?) && (!capacityInfo["nvidia.com/gpu"].empty?) - properties["nvigpus"] = capacityInfo["nvidia.com/gpu"] + properties = getNodeTelemetryProps(item) + properties["KubernetesProviderID"] = nodeInventoryRecord["KubernetesProviderID"] + capacityInfo = item["status"]["capacity"] + + ApplicationInsightsUtility.sendMetricTelemetry("NodeMemory", capacityInfo["memory"], properties) + begin + if (!capacityInfo["nvidia.com/gpu"].nil?) && (!capacityInfo["nvidia.com/gpu"].empty?) + properties["nvigpus"] = capacityInfo["nvidia.com/gpu"] + end + + if (!capacityInfo["amd.com/gpu"].nil?) && (!capacityInfo["amd.com/gpu"].empty?) + properties["amdgpus"] = capacityInfo["amd.com/gpu"] + end + rescue => errorStr + $log.warn "Failed in getting GPU telemetry in_kube_nodes : #{errorStr}" + $log.debug_backtrace(errorStr.backtrace) + ApplicationInsightsUtility.sendExceptionTelemetry(errorStr) end - if (!capacityInfo["amd.com/gpu"].nil?) && (!capacityInfo["amd.com/gpu"].empty?) - properties["amdgpus"] = capacityInfo["amd.com/gpu"] + # Telemetry for data collection config for replicaset + if (File.file?(@@configMapMountPath)) + properties["collectAllKubeEvents"] = @@collectAllKubeEvents end - rescue => errorStr - $log.warn "Failed in getting GPU telemetry in_kube_nodes : #{errorStr}" - $log.debug_backtrace(errorStr.backtrace) - ApplicationInsightsUtility.sendExceptionTelemetry(errorStr) - end - # Telemetry for data collection config for replicaset - if (File.file?(@@configMapMountPath)) - properties["collectAllKubeEvents"] = @@collectAllKubeEvents - end + #telemetry about prometheus metric collections settings for replicaset + if (File.file?(@@promConfigMountPath)) + properties["rsPromInt"] = @@rsPromInterval + properties["rsPromFPC"] = @@rsPromFieldPassCount + properties["rsPromFDC"] = @@rsPromFieldDropCount + properties["rsPromServ"] = @@rsPromK8sServiceCount + properties["rsPromUrl"] = @@rsPromUrlCount + properties["rsPromMonPods"] = @@rsPromMonitorPods + properties["rsPromMonPodsNs"] = @@rsPromMonitorPodsNamespaceLength + properties["rsPromMonPodsLabelSelectorLength"] = @@rsPromMonitorPodsLabelSelectorLength + properties["rsPromMonPodsFieldSelectorLength"] = @@rsPromMonitorPodsFieldSelectorLength + end + # telemetry about osm metric settings for replicaset + if (File.file?(@@osmConfigMountPath)) + properties["osmNamespaceCount"] = @@osmNamespaceCount + end + ApplicationInsightsUtility.sendMetricTelemetry("NodeCoreCapacity", capacityInfo["cpu"], properties) + telemetrySent = true - #telemetry about prometheus metric collections settings for replicaset - if (File.file?(@@promConfigMountPath)) - properties["rsPromInt"] = @@rsPromInterval - properties["rsPromFPC"] = @@rsPromFieldPassCount - properties["rsPromFDC"] = @@rsPromFieldDropCount - properties["rsPromServ"] = @@rsPromK8sServiceCount - properties["rsPromUrl"] = @@rsPromUrlCount - properties["rsPromMonPods"] = @@rsPromMonitorPods - properties["rsPromMonPodsNs"] = @@rsPromMonitorPodsNamespaceLength - properties["rsPromMonPodsLabelSelectorLength"] = @@rsPromMonitorPodsLabelSelectorLength - properties["rsPromMonPodsFieldSelectorLength"] = @@rsPromMonitorPodsFieldSelectorLength - end - # telemetry about osm metric settings for replicaset - if (File.file?(@@osmConfigMountPath)) - properties["osmNamespaceCount"] = @@osmNamespaceCount + # Telemetry for data collection config for replicaset + if (File.file?(@@configMapMountPath)) + properties["collectAllKubeEvents"] = @@collectAllKubeEvents + end + + #telemetry about prometheus metric collections settings for replicaset + if (File.file?(@@promConfigMountPath)) + properties["rsPromInt"] = @@rsPromInterval + properties["rsPromFPC"] = @@rsPromFieldPassCount + properties["rsPromFDC"] = @@rsPromFieldDropCount + properties["rsPromServ"] = @@rsPromK8sServiceCount + properties["rsPromUrl"] = @@rsPromUrlCount + properties["rsPromMonPods"] = @@rsPromMonitorPods + properties["rsPromMonPodsNs"] = @@rsPromMonitorPodsNamespaceLength + properties["rsPromMonPodsLabelSelectorLength"] = @@rsPromMonitorPodsLabelSelectorLength + properties["rsPromMonPodsFieldSelectorLength"] = @@rsPromMonitorPodsFieldSelectorLength + end + # telemetry about osm metric settings for replicaset + if (File.file?(@@osmConfigMountPath)) + properties["osmNamespaceCount"] = @@osmNamespaceCount + end + @applicationInsightsUtility.sendMetricTelemetry("NodeCoreCapacity", capacityInfo["cpu"], properties) + telemetrySent = true + rescue => errorStr + $log.warn "Failed in getting telemetry in_kube_nodes : #{errorStr}" + $log.debug_backtrace(errorStr.backtrace) + @applicationInsightsUtility.sendExceptionTelemetry(errorStr) end - ApplicationInsightsUtility.sendMetricTelemetry("NodeCoreCapacity", capacityInfo["cpu"], properties) - telemetrySent = true end end if telemetrySent == true @@ -385,7 +428,7 @@ def parse_and_emit_records(nodeInventory, batchTime = Time.utc.iso8601) rescue => errorStr $log.warn "Failed to retrieve node inventory: #{errorStr}" $log.debug_backtrace(errorStr.backtrace) - ApplicationInsightsUtility.sendExceptionTelemetry(errorStr) + @applicationInsightsUtility.sendExceptionTelemetry(errorStr) end $log.info "in_kube_nodes::parse_and_emit_records:End #{Time.now.utc.iso8601}" end @@ -414,7 +457,7 @@ def run_periodic $log.info("in_kube_nodes::run_periodic.enumerate.end #{Time.now.utc.iso8601}") rescue => errorStr $log.warn "in_kube_nodes::run_periodic: enumerate Failed to retrieve node inventory: #{errorStr}" - ApplicationInsightsUtility.sendExceptionTelemetry(errorStr) + @applicationInsightsUtility.sendExceptionTelemetry(errorStr) end end @mutex.lock @@ -428,8 +471,8 @@ def getNodeInventoryRecord(item, batchTime = Time.utc.iso8601) begin record["CollectionTime"] = batchTime #This is the time that is mapped to become TimeGenerated record["Computer"] = item["metadata"]["name"] - record["ClusterName"] = KubernetesApiClient.getClusterName - record["ClusterId"] = KubernetesApiClient.getClusterId + record["ClusterName"] = @kubernetesApiClient.getClusterName + record["ClusterId"] = @kubernetesApiClient.getClusterId record["CreationTimeStamp"] = item["metadata"]["creationTimestamp"] record["Labels"] = [item["metadata"]["labels"]] record["Status"] = "" diff --git a/source/plugins/ruby/in_kube_nodes_test.rb b/source/plugins/ruby/in_kube_nodes_test.rb new file mode 100644 index 000000000..8f4984c6c --- /dev/null +++ b/source/plugins/ruby/in_kube_nodes_test.rb @@ -0,0 +1,171 @@ +require 'minitest/autorun' + +require 'fluent/test' +require 'fluent/test/driver/input' +require 'fluent/test/helpers' + +require_relative 'in_kube_nodes.rb' + +class InKubeNodesTests < Minitest::Test + include Fluent::Test::Helpers + + def setup + Fluent::Test.setup + end + + def create_driver(conf = {}, kubernetesApiClient=nil, applicationInsightsUtility=nil, extensionUtils=nil, env=nil, telemetry_flush_interval=nil) + Fluent::Test::Driver::Input.new(Fluent::Plugin::Kube_nodeInventory_Input.new(kubernetesApiClient=kubernetesApiClient, + applicationInsightsUtility=applicationInsightsUtility, + extensionUtils=extensionUtils, + env=env)).configure(conf) + end + + # Collection time of scrapped data will always be different. Overwrite it in any records returned by in_kube_ndes.rb + def overwrite_collection_time(data) + if data.key?("CollectionTime") + data["CollectionTime"] = "~CollectionTime~" + end + if data.key?("Timestamp") + data["Timestamp"] = "~Timestamp~" + end + return data + end + + def test_basic_single_node + kubeApiClient = Minitest::Mock.new + appInsightsUtil = Minitest::Mock.new + extensionUtils = Minitest::Mock.new + env = {} + env["NODES_CHUNK_SIZE"] = "200" + + kubeApiClient.expect(:==, false, [nil]) + appInsightsUtil.expect(:==, false, [nil]) + extensionUtils.expect(:==, false, [nil]) + + # isAADMSIAuthMode() is called multiple times and we don't really care how many time it is called. This is the same as mocking + # but it doesn't track how many times isAADMSIAuthMode is called + def extensionUtils.isAADMSIAuthMode + false + end + + nodes_api_response = eval(File.open("test/unit-tests/canned-api-responses/kube-nodes.txt").read) + kubeApiClient.expect(:getResourcesAndContinuationToken, [nil, nodes_api_response], ["nodes?limit=200"]) + kubeApiClient.expect(:getClusterName, "/cluster-name") + kubeApiClient.expect(:getClusterId, "/cluster-id") + + config = "run_interval 999999999" # only run once + + d = create_driver(config, kubernetesApiClient=kubeApiClient, applicationInsightsUtility=appInsightsUtil, extensionUtils=extensionUtils, env=env) + d.instance.start + d.instance.enumerate + d.run(timeout: 99999) # Input plugins decide when to run, so we have to give it enough time to run + + + expected_responses = { ["oneagent.containerInsights.KUBE_NODE_INVENTORY_BLOB", overwrite_collection_time({"CollectionTime"=>"2021-08-17T20:24:18Z", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"aks-nodepool1-24816391-vmss000000", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"})] => true, + ["mdm.kubenodeinventory", overwrite_collection_time({"CollectionTime"=>"2021-08-17T20:24:18Z", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"aks-nodepool1-24816391-vmss000000", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"})] => true, + ["oneagent.containerInsights.CONTAINER_NODE_INVENTORY_BLOB", overwrite_collection_time({"CollectionTime"=>"2021-08-17T20:24:18Z", "Computer"=>"aks-nodepool1-24816391-vmss000000", "OperatingSystem"=>"Ubuntu 18.04.5 LTS", "DockerVersion"=>"containerd://1.4.4+azure"})] => true, + ["oneagent.containerInsights.LINUX_PERF_BLOB", overwrite_collection_time({"Timestamp"=>"2021-08-17T20:24:18Z", "Host"=>"aks-nodepool1-24816391-vmss000000", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ObjectName"=>"K8SNode", "InstanceName"=>"None/aks-nodepool1-24816391-vmss000000", "json_Collections"=>"[{\"CounterName\":\"cpuAllocatableNanoCores\",\"Value\":1900000000.0}]"})] => true, + ["oneagent.containerInsights.LINUX_PERF_BLOB", overwrite_collection_time({"Timestamp"=>"2021-08-17T20:24:18Z", "Host"=>"aks-nodepool1-24816391-vmss000000", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ObjectName"=>"K8SNode", "InstanceName"=>"None/aks-nodepool1-24816391-vmss000000", "json_Collections"=>"[{\"CounterName\":\"memoryAllocatableBytes\",\"Value\":4787511296.0}]"})] => true, + ["oneagent.containerInsights.LINUX_PERF_BLOB", overwrite_collection_time({"Timestamp"=>"2021-08-17T20:24:18Z", "Host"=>"aks-nodepool1-24816391-vmss000000", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ObjectName"=>"K8SNode", "InstanceName"=>"None/aks-nodepool1-24816391-vmss000000", "json_Collections"=>"[{\"CounterName\":\"cpuCapacityNanoCores\",\"Value\":2000000000.0}]"})] => true, + ["oneagent.containerInsights.LINUX_PERF_BLOB", overwrite_collection_time({"Timestamp"=>"2021-08-17T20:24:18Z", "Host"=>"aks-nodepool1-24816391-vmss000000", "Computer"=>"aks-nodepool1-24816391-vmss000000", "ObjectName"=>"K8SNode", "InstanceName"=>"None/aks-nodepool1-24816391-vmss000000", "json_Collections"=>"[{\"CounterName\":\"memoryCapacityBytes\",\"Value\":7291510784.0}]"})] => true} + + d.events.each do |tag, time, record| + cleaned_record = overwrite_collection_time record + if expected_responses.key?([tag, cleaned_record]) + expected_responses[[tag, cleaned_record]] = true + else + assert(false, "got unexpected record") + end + end + + expected_responses.each do |key, val| + assert(val, "expected record not emitted: #{key}") + end + + # make sure all mocked methods were called the expected number of times + kubeApiClient.verify + appInsightsUtil.verify + extensionUtils.verify + end + + # Sometimes customer tooling creates invalid node specs in the Kube API server (its happened more than once). + # This test makes sure that it doesn't creash the entire input plugin and other nodes are still collected + def test_malformed_node_spec + kubeApiClient = Minitest::Mock.new + appInsightsUtil = Minitest::Mock.new + extensionUtils = Minitest::Mock.new + env = {} + env["NODES_CHUNK_SIZE"] = "200" + + kubeApiClient.expect(:==, false, [nil]) + appInsightsUtil.expect(:==, false, [nil]) + extensionUtils.expect(:==, false, [nil]) + + # isAADMSIAuthMode() is called multiple times and we don't really care how many time it is called. This is the same as mocking + # but it doesn't track how many times isAADMSIAuthMode is called + def extensionUtils.isAADMSIAuthMode + false + end + + # Set up the KubernetesApiClient Mock. Note: most of the functions in KubernetesApiClient are pure (access no + # state other than their arguments), so there is no need to mock them (this test file would be far longer and + # more brittle). Instead, in_kube_nodes bypasses the mock and directly calls these functions in KubernetesApiClient. + # Ideally the pure functions in KubernetesApiClient would be refactored into their own file to reduce confusion. + nodes_api_response = eval(File.open("test/unit-tests/canned-api-responses/kube-nodes-malformed.txt").read) + kubeApiClient.expect(:getResourcesAndContinuationToken, [nil, nodes_api_response], ["nodes?limit=200"]) + kubeApiClient.expect(:getClusterName, "/cluster-name") + kubeApiClient.expect(:getClusterName, "/cluster-name") + kubeApiClient.expect(:getClusterId, "/cluster-id") + kubeApiClient.expect(:getClusterId, "/cluster-id") + + def appInsightsUtil.sendExceptionTelemetry(exception) + if exception.to_s != "undefined method `[]' for nil:NilClass" + raise "an unexpected exception has occured" + end + end + + # This test doesn't care if metric telemetry is sent properly. Looking for an unnecessary value would make it needlessly rigid + def appInsightsUtil.sendMetricTelemetry(a, b, c) + end + + config = "run_interval 999999999" # only run once + + d = create_driver(config, kubernetesApiClient=kubeApiClient, applicationInsightsUtility=appInsightsUtil, extensionUtils=extensionUtils, env=env, telemetry_flush_interval=0) + d.instance.start + + d.instance.enumerate + d.run(timeout: 99999) #TODO: is this necessary? + + expected_responses = { + ["oneagent.containerInsights.KUBE_NODE_INVENTORY_BLOB", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"correct-node", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"correct-node", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"}] => false, + ["mdm.kubenodeinventory", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"correct-node", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"correct-node", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"}] => false, + ["oneagent.containerInsights.CONTAINER_NODE_INVENTORY_BLOB", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"correct-node", "OperatingSystem"=>"Ubuntu 18.04.5 LTS", "DockerVersion"=>"containerd://1.4.4+azure"}] => false, + ["oneagent.containerInsights.LINUX_PERF_BLOB", {"Timestamp"=>"~Timestamp~", "Host"=>"correct-node", "Computer"=>"correct-node", "ObjectName"=>"K8SNode", "InstanceName"=>"None/correct-node", "json_Collections"=>"[{\"CounterName\":\"cpuAllocatableNanoCores\",\"Value\":1000000.0}]"}] => false, + ["oneagent.containerInsights.LINUX_PERF_BLOB", {"Timestamp"=>"~Timestamp~", "Host"=>"correct-node", "Computer"=>"correct-node", "ObjectName"=>"K8SNode", "InstanceName"=>"None/correct-node", "json_Collections"=>"[{\"CounterName\":\"memoryAllocatableBytes\",\"Value\":444.0}]"}] => false, + ["oneagent.containerInsights.LINUX_PERF_BLOB", {"Timestamp"=>"~Timestamp~", "Host"=>"correct-node", "Computer"=>"correct-node", "ObjectName"=>"K8SNode", "InstanceName"=>"None/correct-node", "json_Collections"=>"[{\"CounterName\":\"cpuCapacityNanoCores\",\"Value\":2000000.0}]"}] => false, + ["oneagent.containerInsights.LINUX_PERF_BLOB", {"Timestamp"=>"~Timestamp~", "Host"=>"correct-node", "Computer"=>"correct-node", "ObjectName"=>"K8SNode", "InstanceName"=>"None/correct-node", "json_Collections"=>"[{\"CounterName\":\"memoryCapacityBytes\",\"Value\":555.0}]"}] => false, + + # these records are for the malformed node (it doesn't have limits or requests set so there are no PERF records) + ["oneagent.containerInsights.KUBE_NODE_INVENTORY_BLOB", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"malformed-node", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"malformed-node", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"}] => false, + ["mdm.kubenodeinventory", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"malformed-node", "ClusterName"=>"/cluster-name", "ClusterId"=>"/cluster-id", "CreationTimeStamp"=>"2021-07-21T23:40:14Z", "Labels"=>[{"agentpool"=>"nodepool1", "beta.kubernetes.io/arch"=>"amd64", "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", "beta.kubernetes.io/os"=>"linux", "failure-domain.beta.kubernetes.io/region"=>"westus2", "failure-domain.beta.kubernetes.io/zone"=>"0", "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", "kubernetes.azure.com/mode"=>"system", "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", "kubernetes.azure.com/os-sku"=>"Ubuntu", "kubernetes.azure.com/role"=>"agent", "kubernetes.io/arch"=>"amd64", "kubernetes.io/hostname"=>"malformed-node", "kubernetes.io/os"=>"linux", "kubernetes.io/role"=>"agent", "node-role.kubernetes.io/agent"=>"", "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", "storageprofile"=>"managed", "storagetier"=>"Premium_LRS", "topology.kubernetes.io/region"=>"westus2", "topology.kubernetes.io/zone"=>"0"}], "Status"=>"Ready", "KubernetesProviderID"=>"azure", "LastTransitionTimeReady"=>"2021-07-21T23:40:24Z", "KubeletVersion"=>"v1.19.11", "KubeProxyVersion"=>"v1.19.11"}] => false, + ["oneagent.containerInsights.CONTAINER_NODE_INVENTORY_BLOB", {"CollectionTime"=>"~CollectionTime~", "Computer"=>"malformed-node", "OperatingSystem"=>"Ubuntu 18.04.5 LTS", "DockerVersion"=>"containerd://1.4.4+azure"}] => false + } + + d.events.each do |tag, time, record| + cleaned_record = overwrite_collection_time record + if expected_responses.key?([tag, cleaned_record]) + expected_responses[[tag, cleaned_record]] = true + end + # don't do anything if an unexpected record was emitted. Since the node spec is malformed, there will be some partial data. + # we care more that the non-malformed data is still emitted + end + + expected_responses.each do |key, val| + assert(val, "expected record not emitted: #{key}") + end + + kubeApiClient.verify + appInsightsUtil.verify + extensionUtils.verify + end +end diff --git a/source/plugins/utils/oms_common.rb b/source/plugins/ruby/oms_common.rb similarity index 100% rename from source/plugins/utils/oms_common.rb rename to source/plugins/ruby/oms_common.rb diff --git a/source/plugins/utils/omslog.rb b/source/plugins/ruby/omslog.rb similarity index 100% rename from source/plugins/utils/omslog.rb rename to source/plugins/ruby/omslog.rb diff --git a/test/unit-tests/canned-api-responses/kube-nodes-malformed.txt b/test/unit-tests/canned-api-responses/kube-nodes-malformed.txt new file mode 100644 index 000000000..bb4c61ca5 --- /dev/null +++ b/test/unit-tests/canned-api-responses/kube-nodes-malformed.txt @@ -0,0 +1,1674 @@ +{ + "kind"=>"NodeList", + "apiVersion"=>"v1", + "metadata"=>{ + "selfLink"=>"/api/v1/nodes", + "resourceVersion"=>"5974879" + }, + "items"=>[ + { + "metadata"=>{ + "name"=>"malformed-node", + "selfLink"=>"/api/v1/nodes/malformed-node", + "uid"=>"fe073f0a-e6bf-4d68-b4e5-ffaa42b91528", + "resourceVersion"=>"5974522", + "creationTimestamp"=>"2021-07-21T23:40:14Z", + "labels"=>{ + "agentpool"=>"nodepool1", + "beta.kubernetes.io/arch"=>"amd64", + "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "beta.kubernetes.io/os"=>"linux", + "failure-domain.beta.kubernetes.io/region"=>"westus2", + "failure-domain.beta.kubernetes.io/zone"=>"0", + "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", + "kubernetes.azure.com/mode"=>"system", + "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", + "kubernetes.azure.com/os-sku"=>"Ubuntu", + "kubernetes.azure.com/role"=>"agent", + "kubernetes.io/arch"=>"amd64", + "kubernetes.io/hostname"=>"malformed-node", + "kubernetes.io/os"=>"linux", + "kubernetes.io/role"=>"agent", + "node-role.kubernetes.io/agent"=>"", + "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "storageprofile"=>"managed", + "storagetier"=>"Premium_LRS", + "topology.kubernetes.io/region"=>"westus2", + "topology.kubernetes.io/zone"=>"0" + }, + "annotations"=>{ + "node.alpha.kubernetes.io/ttl"=>"0", + "volumes.kubernetes.io/controller-managed-attach-detach"=>"true" + }, + "managedFields"=>[ + { + "manager"=>"kube-controller-manager", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:20Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "f:node.alpha.kubernetes.io/ttl"=>{} + } + } + } + }, + { + "manager"=>"kubelet", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:24Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "."=>{}, + "f:volumes.kubernetes.io/controller-managed-attach-detach"=>{} + }, + "f:labels"=>{ + "."=>{}, + "f:agentpool"=>{}, + "f:beta.kubernetes.io/arch"=>{}, + "f:beta.kubernetes.io/instance-type"=>{}, + "f:beta.kubernetes.io/os"=>{}, + "f:failure-domain.beta.kubernetes.io/region"=>{}, + "f:failure-domain.beta.kubernetes.io/zone"=>{}, + "f:kubernetes.azure.com/cluster"=>{}, + "f:kubernetes.azure.com/mode"=>{}, + "f:kubernetes.azure.com/node-image-version"=>{}, + "f:kubernetes.azure.com/os-sku"=>{}, + "f:kubernetes.azure.com/role"=>{}, + "f:kubernetes.io/arch"=>{}, + "f:kubernetes.io/hostname"=>{}, + "f:kubernetes.io/os"=>{}, + "f:node.kubernetes.io/instance-type"=>{}, + "f:storageprofile"=>{}, + "f:storagetier"=>{}, + "f:topology.kubernetes.io/region"=>{}, + "f:topology.kubernetes.io/zone"=>{} + } + }, + "f:spec"=>{ + "f:providerID"=>{} + }, + "f:status"=>{ + "f:addresses"=>{ + "."=>{}, + "k:{\"type\":\"Hostname\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"InternalIP\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + } + }, + "f:allocatable"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:capacity"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:conditions"=>{ + "."=>{}, + "k:{\"type\":\"DiskPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"MemoryPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PIDPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"Ready\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + }, + "f:config"=>{}, + "f:daemonEndpoints"=>{ + "f:kubeletEndpoint"=>{ + "f:Port"=>{} + } + }, + "f:images"=>{}, + "f:nodeInfo"=>{ + "f:architecture"=>{}, + "f:bootID"=>{}, + "f:containerRuntimeVersion"=>{}, + "f:kernelVersion"=>{}, + "f:kubeProxyVersion"=>{}, + "f:kubeletVersion"=>{}, + "f:machineID"=>{}, + "f:operatingSystem"=>{}, + "f:osImage"=>{}, + "f:systemUUID"=>{} + } + } + } + }, + { + "manager"=>"kubectl-label", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:53Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:labels"=>{ + "f:kubernetes.io/role"=>{}, + "f:node-role.kubernetes.io/agent"=>{} + } + } + } + }, + { + "manager"=>"node-problem-detector", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-08-10T18:10:02Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:status"=>{ + "f:conditions"=>{ + "k:{\"type\":\"ContainerRuntimeProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FilesystemCorruptionProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FreezeScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentContainerdRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentDockerRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentKubeletRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentUnregisterNetDevice\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KernelDeadlock\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KubeletProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PreemptScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"ReadonlyFilesystem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RebootScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RedeployScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"TerminateScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + } + } + } + } + ] + }, + "spec"=>{ + "providerID"=>"azure:///subscriptions/3b875bf3-0eec-4d8c-bdee-25c7ccc1f130/resourceGroups/mc_davidaks16_davidaks16_westus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-24816391-vmss/virtualMachines/0" + }, + "status"=>{ + "conditions"=>[ + { + "type"=>"FrequentDockerRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentDockerRestart", + "message"=>"docker is functioning properly" + }, + { + "type"=>"FilesystemCorruptionProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsOK", + "message"=>"Filesystem is healthy" + }, + { + "type"=>"KernelDeadlock", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KernelHasNoDeadlock", + "message"=>"kernel has no deadlock" + }, + { + "type"=>"FrequentContainerdRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentContainerdRestart", + "message"=>"containerd is functioning properly" + }, + { + "type"=>"FreezeScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-11T23:25:04Z", + "reason"=>"NoFreezeScheduled", + "message"=>"VM has no scheduled Freeze event" + }, + { + "type"=>"FrequentUnregisterNetDevice", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentUnregisterNetDevice", + "message"=>"node is functioning properly" + }, + { + "type"=>"TerminateScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoTerminateScheduled", + "message"=>"VM has no scheduled Terminate event" + }, + { + "type"=>"ReadonlyFilesystem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsNotReadOnly", + "message"=>"Filesystem is not read-only" + }, + { + "type"=>"RedeployScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRedeployScheduled", + "message"=>"VM has no scheduled Redeploy event" + }, + { + "type"=>"KubeletProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KubeletIsUp", + "message"=>"kubelet service is up" + }, + { + "type"=>"PreemptScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:11:11Z", + "reason"=>"NoPreemptScheduled", + "message"=>"VM has no scheduled Preempt event" + }, + { + "type"=>"RebootScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRebootScheduled", + "message"=>"VM has no scheduled Reboot event" + }, + { + "type"=>"ContainerRuntimeProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"ContainerRuntimeIsUp", + "message"=>"container runtime service is up" + }, + { + "type"=>"FrequentKubeletRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentKubeletRestart", + "message"=>"kubelet is functioning properly" + }, + { + "type"=>"MemoryPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientMemory", + "message"=>"kubelet has sufficient memory available" + }, + { + "type"=>"DiskPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasNoDiskPressure", + "message"=>"kubelet has no disk pressure" + }, + { + "type"=>"PIDPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientPID", + "message"=>"kubelet has sufficient PID available" + }, + { + "type"=>"Ready", + "status"=>"True", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:24Z", + "reason"=>"KubeletReady", + "message"=>"kubelet is posting ready status. AppArmor enabled" + } + ], + "addresses"=>[ + { + "type"=>"Hostname", + "address"=>"malformed-node" + }, + { + "type"=>"InternalIP", + "address"=>"10.240.0.4" + } + ], + "daemonEndpoints"=>{ + "kubeletEndpoint"=>{ + "Port"=>10250 + } + }, + "nodeInfo"=>{ + "machineID"=>"17a654260e2c4a9bb3a3eb4b4188e4b4", + "systemUUID"=>"7ff599e4-909e-4950-a044-ff8613af3af9", + "bootID"=>"02bb865b-a469-43cd-8b0b-5ceb4ecd80b0", + "kernelVersion"=>"5.4.0-1051-azure", + "osImage"=>"Ubuntu 18.04.5 LTS", + "containerRuntimeVersion"=>"containerd://1.4.4+azure", + "kubeletVersion"=>"v1.19.11", + "kubeProxyVersion"=>"v1.19.11", + "operatingSystem"=>"linux", + "architecture"=>"amd64" + }, + "images"=>[ + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021-1" + ], + "sizeBytes"=>331689060 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021" + ], + "sizeBytes"=>330099815 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021-hotfix" + ], + "sizeBytes"=>271471426 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021" + ], + "sizeBytes"=>269703297 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03262021" + ], + "sizeBytes"=>264732875 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.19.0" + ], + "sizeBytes"=>166352383 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210623.2" + ], + "sizeBytes"=>147750148 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210524.1" + ], + "sizeBytes"=>146446618 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210427.1" + ], + "sizeBytes"=>136242776 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.5" + ], + "sizeBytes"=>101794833 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.47.0" + ], + "sizeBytes"=>101445696 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.3.0_v0.0.5" + ], + "sizeBytes"=>101194562 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210623.2" + ], + "sizeBytes"=>96125176 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210524.1" + ], + "sizeBytes"=>95879501 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/exechealthz:1.2_v0.0.5" + ], + "sizeBytes"=>94348102 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.2" + ], + "sizeBytes"=>93537927 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/acc/sgx-attestation:2.0" + ], + "sizeBytes"=>91841669 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.4.0" + ], + "sizeBytes"=>91324193 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.2.0" + ], + "sizeBytes"=>89103171 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.1-rc3" + ], + "sizeBytes"=>86839805 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0" + ], + "sizeBytes"=>86488586 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210427.1" + ], + "sizeBytes"=>86120048 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0" + ], + "sizeBytes"=>81252495 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0" + ], + "sizeBytes"=>79586703 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.4.0" + ], + "sizeBytes"=>78795016 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.2.0" + ], + "sizeBytes"=>76527179 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.1.8" + ], + "sizeBytes"=>75025803 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix" + ], + "sizeBytes"=>73533889 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.3.1" + ], + "sizeBytes"=>72242894 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.8" + ], + "sizeBytes"=>70622822 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/nvidia/k8s-device-plugin:v0.9.0" + ], + "sizeBytes"=>67291599 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.1" + ], + "sizeBytes"=>66415836 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc7" + ], + "sizeBytes"=>65965658 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.1" + ], + "sizeBytes"=>64123775 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.3" + ], + "sizeBytes"=>63581323 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8" + ], + "sizeBytes"=>63154716 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.2" + ], + "sizeBytes"=>61626312 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.18.1" + ], + "sizeBytes"=>60500885 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.17.2" + ], + "sizeBytes"=>58419768 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8_hotfix", + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8post2" + ], + "sizeBytes"=>56368756 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:282543237a1aa3f407656290f454b7068a92e1abe2156082c750d5abfbcad90c", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.2" + ], + "sizeBytes"=>56310724 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.19.0" + ], + "sizeBytes"=>55228749 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.1" + ], + "sizeBytes"=>54692048 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc3" + ], + "sizeBytes"=>50803639 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19" + ], + "sizeBytes"=>49759361 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.5" + ], + "sizeBytes"=>49704644 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.21" + ], + "sizeBytes"=>49372390 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:a64d3538b72905b07356881314755b02db3675ff47ee2bcc49dd7be856e285d5", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526" + ], + "sizeBytes"=>49322942 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" + ], + "sizeBytes"=>48108311 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kubernetes-dashboard:v1.10.1" + ], + "sizeBytes"=>44907744 + } + ], + "config"=>{} + } + }, + { + "metadata"=>{ + "name"=>"correct-node", + "selfLink"=>"/api/v1/nodes/correct-node", + "uid"=>"fe073f0a-e6bf-4d68-b4e5-ffaa42b91528", + "resourceVersion"=>"5974522", + "creationTimestamp"=>"2021-07-21T23:40:14Z", + "labels"=>{ + "agentpool"=>"nodepool1", + "beta.kubernetes.io/arch"=>"amd64", + "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "beta.kubernetes.io/os"=>"linux", + "failure-domain.beta.kubernetes.io/region"=>"westus2", + "failure-domain.beta.kubernetes.io/zone"=>"0", + "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", + "kubernetes.azure.com/mode"=>"system", + "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", + "kubernetes.azure.com/os-sku"=>"Ubuntu", + "kubernetes.azure.com/role"=>"agent", + "kubernetes.io/arch"=>"amd64", + "kubernetes.io/hostname"=>"correct-node", + "kubernetes.io/os"=>"linux", + "kubernetes.io/role"=>"agent", + "node-role.kubernetes.io/agent"=>"", + "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "storageprofile"=>"managed", + "storagetier"=>"Premium_LRS", + "topology.kubernetes.io/region"=>"westus2", + "topology.kubernetes.io/zone"=>"0" + }, + "annotations"=>{ + "node.alpha.kubernetes.io/ttl"=>"0", + "volumes.kubernetes.io/controller-managed-attach-detach"=>"true" + }, + "managedFields"=>[ + { + "manager"=>"kube-controller-manager", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:20Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "f:node.alpha.kubernetes.io/ttl"=>{} + } + } + } + }, + { + "manager"=>"kubelet", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:24Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "."=>{}, + "f:volumes.kubernetes.io/controller-managed-attach-detach"=>{} + }, + "f:labels"=>{ + "."=>{}, + "f:agentpool"=>{}, + "f:beta.kubernetes.io/arch"=>{}, + "f:beta.kubernetes.io/instance-type"=>{}, + "f:beta.kubernetes.io/os"=>{}, + "f:failure-domain.beta.kubernetes.io/region"=>{}, + "f:failure-domain.beta.kubernetes.io/zone"=>{}, + "f:kubernetes.azure.com/cluster"=>{}, + "f:kubernetes.azure.com/mode"=>{}, + "f:kubernetes.azure.com/node-image-version"=>{}, + "f:kubernetes.azure.com/os-sku"=>{}, + "f:kubernetes.azure.com/role"=>{}, + "f:kubernetes.io/arch"=>{}, + "f:kubernetes.io/hostname"=>{}, + "f:kubernetes.io/os"=>{}, + "f:node.kubernetes.io/instance-type"=>{}, + "f:storageprofile"=>{}, + "f:storagetier"=>{}, + "f:topology.kubernetes.io/region"=>{}, + "f:topology.kubernetes.io/zone"=>{} + } + }, + "f:spec"=>{ + "f:providerID"=>{} + }, + "f:status"=>{ + "f:addresses"=>{ + "."=>{}, + "k:{\"type\":\"Hostname\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"InternalIP\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + } + }, + "f:allocatable"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:capacity"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:conditions"=>{ + "."=>{}, + "k:{\"type\":\"DiskPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"MemoryPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PIDPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"Ready\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + }, + "f:config"=>{}, + "f:daemonEndpoints"=>{ + "f:kubeletEndpoint"=>{ + "f:Port"=>{} + } + }, + "f:images"=>{}, + "f:nodeInfo"=>{ + "f:architecture"=>{}, + "f:bootID"=>{}, + "f:containerRuntimeVersion"=>{}, + "f:kernelVersion"=>{}, + "f:kubeProxyVersion"=>{}, + "f:kubeletVersion"=>{}, + "f:machineID"=>{}, + "f:operatingSystem"=>{}, + "f:osImage"=>{}, + "f:systemUUID"=>{} + } + } + } + }, + { + "manager"=>"kubectl-label", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:53Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:labels"=>{ + "f:kubernetes.io/role"=>{}, + "f:node-role.kubernetes.io/agent"=>{} + } + } + } + }, + { + "manager"=>"node-problem-detector", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-08-10T18:10:02Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:status"=>{ + "f:conditions"=>{ + "k:{\"type\":\"ContainerRuntimeProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FilesystemCorruptionProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FreezeScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentContainerdRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentDockerRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentKubeletRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentUnregisterNetDevice\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KernelDeadlock\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KubeletProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PreemptScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"ReadonlyFilesystem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RebootScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RedeployScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"TerminateScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + } + } + } + } + ] + }, + "spec"=>{ + "providerID"=>"azure:///subscriptions/3b875bf3-0eec-4d8c-bdee-25c7ccc1f130/resourceGroups/mc_davidaks16_davidaks16_westus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-24816391-vmss/virtualMachines/0" + }, + "status"=>{ + "capacity"=>{ + "attachable-volumes-azure-disk"=>"8", + "cpu"=>"2m", + "ephemeral-storage"=>"666", + "hugepages-1Gi"=>"0", + "hugepages-2Mi"=>"0", + "memory"=>"555", + "pods"=>"30" + }, + "allocatable"=>{ + "attachable-volumes-azure-disk"=>"8", + "cpu"=>"1m", + "ephemeral-storage"=>"333", + "hugepages-1Gi"=>"0", + "hugepages-2Mi"=>"0", + "memory"=>"444", + "pods"=>"30" + }, + "conditions"=>[ + { + "type"=>"FrequentDockerRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentDockerRestart", + "message"=>"docker is functioning properly" + }, + { + "type"=>"FilesystemCorruptionProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsOK", + "message"=>"Filesystem is healthy" + }, + { + "type"=>"KernelDeadlock", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KernelHasNoDeadlock", + "message"=>"kernel has no deadlock" + }, + { + "type"=>"FrequentContainerdRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentContainerdRestart", + "message"=>"containerd is functioning properly" + }, + { + "type"=>"FreezeScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-11T23:25:04Z", + "reason"=>"NoFreezeScheduled", + "message"=>"VM has no scheduled Freeze event" + }, + { + "type"=>"FrequentUnregisterNetDevice", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentUnregisterNetDevice", + "message"=>"node is functioning properly" + }, + { + "type"=>"TerminateScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoTerminateScheduled", + "message"=>"VM has no scheduled Terminate event" + }, + { + "type"=>"ReadonlyFilesystem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsNotReadOnly", + "message"=>"Filesystem is not read-only" + }, + { + "type"=>"RedeployScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRedeployScheduled", + "message"=>"VM has no scheduled Redeploy event" + }, + { + "type"=>"KubeletProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KubeletIsUp", + "message"=>"kubelet service is up" + }, + { + "type"=>"PreemptScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:11:11Z", + "reason"=>"NoPreemptScheduled", + "message"=>"VM has no scheduled Preempt event" + }, + { + "type"=>"RebootScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRebootScheduled", + "message"=>"VM has no scheduled Reboot event" + }, + { + "type"=>"ContainerRuntimeProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"ContainerRuntimeIsUp", + "message"=>"container runtime service is up" + }, + { + "type"=>"FrequentKubeletRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentKubeletRestart", + "message"=>"kubelet is functioning properly" + }, + { + "type"=>"MemoryPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientMemory", + "message"=>"kubelet has sufficient memory available" + }, + { + "type"=>"DiskPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasNoDiskPressure", + "message"=>"kubelet has no disk pressure" + }, + { + "type"=>"PIDPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientPID", + "message"=>"kubelet has sufficient PID available" + }, + { + "type"=>"Ready", + "status"=>"True", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:24Z", + "reason"=>"KubeletReady", + "message"=>"kubelet is posting ready status. AppArmor enabled" + } + ], + "addresses"=>[ + { + "type"=>"Hostname", + "address"=>"correct-node" + }, + { + "type"=>"InternalIP", + "address"=>"10.240.0.4" + } + ], + "daemonEndpoints"=>{ + "kubeletEndpoint"=>{ + "Port"=>10250 + } + }, + "nodeInfo"=>{ + "machineID"=>"17a654260e2c4a9bb3a3eb4b4188e4b4", + "systemUUID"=>"7ff599e4-909e-4950-a044-ff8613af3af9", + "bootID"=>"02bb865b-a469-43cd-8b0b-5ceb4ecd80b0", + "kernelVersion"=>"5.4.0-1051-azure", + "osImage"=>"Ubuntu 18.04.5 LTS", + "containerRuntimeVersion"=>"containerd://1.4.4+azure", + "kubeletVersion"=>"v1.19.11", + "kubeProxyVersion"=>"v1.19.11", + "operatingSystem"=>"linux", + "architecture"=>"amd64" + }, + "images"=>[ + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021-1" + ], + "sizeBytes"=>331689060 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021" + ], + "sizeBytes"=>330099815 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021-hotfix" + ], + "sizeBytes"=>271471426 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021" + ], + "sizeBytes"=>269703297 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03262021" + ], + "sizeBytes"=>264732875 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.19.0" + ], + "sizeBytes"=>166352383 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210623.2" + ], + "sizeBytes"=>147750148 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210524.1" + ], + "sizeBytes"=>146446618 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210427.1" + ], + "sizeBytes"=>136242776 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.5" + ], + "sizeBytes"=>101794833 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.47.0" + ], + "sizeBytes"=>101445696 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.3.0_v0.0.5" + ], + "sizeBytes"=>101194562 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210623.2" + ], + "sizeBytes"=>96125176 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210524.1" + ], + "sizeBytes"=>95879501 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/exechealthz:1.2_v0.0.5" + ], + "sizeBytes"=>94348102 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.2" + ], + "sizeBytes"=>93537927 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/acc/sgx-attestation:2.0" + ], + "sizeBytes"=>91841669 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.4.0" + ], + "sizeBytes"=>91324193 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.2.0" + ], + "sizeBytes"=>89103171 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.1-rc3" + ], + "sizeBytes"=>86839805 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0" + ], + "sizeBytes"=>86488586 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210427.1" + ], + "sizeBytes"=>86120048 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0" + ], + "sizeBytes"=>81252495 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0" + ], + "sizeBytes"=>79586703 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.4.0" + ], + "sizeBytes"=>78795016 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.2.0" + ], + "sizeBytes"=>76527179 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.1.8" + ], + "sizeBytes"=>75025803 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix" + ], + "sizeBytes"=>73533889 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.3.1" + ], + "sizeBytes"=>72242894 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.8" + ], + "sizeBytes"=>70622822 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/nvidia/k8s-device-plugin:v0.9.0" + ], + "sizeBytes"=>67291599 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.1" + ], + "sizeBytes"=>66415836 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc7" + ], + "sizeBytes"=>65965658 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.1" + ], + "sizeBytes"=>64123775 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.3" + ], + "sizeBytes"=>63581323 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8" + ], + "sizeBytes"=>63154716 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.2" + ], + "sizeBytes"=>61626312 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.18.1" + ], + "sizeBytes"=>60500885 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.17.2" + ], + "sizeBytes"=>58419768 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8_hotfix", + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8post2" + ], + "sizeBytes"=>56368756 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:282543237a1aa3f407656290f454b7068a92e1abe2156082c750d5abfbcad90c", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.2" + ], + "sizeBytes"=>56310724 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.19.0" + ], + "sizeBytes"=>55228749 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.1" + ], + "sizeBytes"=>54692048 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc3" + ], + "sizeBytes"=>50803639 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19" + ], + "sizeBytes"=>49759361 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.5" + ], + "sizeBytes"=>49704644 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.21" + ], + "sizeBytes"=>49372390 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:a64d3538b72905b07356881314755b02db3675ff47ee2bcc49dd7be856e285d5", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526" + ], + "sizeBytes"=>49322942 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" + ], + "sizeBytes"=>48108311 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kubernetes-dashboard:v1.10.1" + ], + "sizeBytes"=>44907744 + } + ], + "config"=>{} + } + } + ] +} \ No newline at end of file diff --git a/test/unit-tests/canned-api-responses/kube-nodes.txt b/test/unit-tests/canned-api-responses/kube-nodes.txt new file mode 100644 index 000000000..ed411c2e5 --- /dev/null +++ b/test/unit-tests/canned-api-responses/kube-nodes.txt @@ -0,0 +1,851 @@ +{ + "kind"=>"NodeList", + "apiVersion"=>"v1", + "metadata"=>{ + "selfLink"=>"/api/v1/nodes", + "resourceVersion"=>"5974879" + }, + "items"=>[ + { + "metadata"=>{ + "name"=>"aks-nodepool1-24816391-vmss000000", + "selfLink"=>"/api/v1/nodes/aks-nodepool1-24816391-vmss000000", + "uid"=>"fe073f0a-e6bf-4d68-b4e5-ffaa42b91528", + "resourceVersion"=>"5974522", + "creationTimestamp"=>"2021-07-21T23:40:14Z", + "labels"=>{ + "agentpool"=>"nodepool1", + "beta.kubernetes.io/arch"=>"amd64", + "beta.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "beta.kubernetes.io/os"=>"linux", + "failure-domain.beta.kubernetes.io/region"=>"westus2", + "failure-domain.beta.kubernetes.io/zone"=>"0", + "kubernetes.azure.com/cluster"=>"MC_davidaks16_davidaks16_westus2", + "kubernetes.azure.com/mode"=>"system", + "kubernetes.azure.com/node-image-version"=>"AKSUbuntu-1804gen2containerd-2021.07.03", + "kubernetes.azure.com/os-sku"=>"Ubuntu", + "kubernetes.azure.com/role"=>"agent", + "kubernetes.io/arch"=>"amd64", + "kubernetes.io/hostname"=>"aks-nodepool1-24816391-vmss000000", + "kubernetes.io/os"=>"linux", + "kubernetes.io/role"=>"agent", + "node-role.kubernetes.io/agent"=>"", + "node.kubernetes.io/instance-type"=>"Standard_DS2_v2", + "storageprofile"=>"managed", + "storagetier"=>"Premium_LRS", + "topology.kubernetes.io/region"=>"westus2", + "topology.kubernetes.io/zone"=>"0" + }, + "annotations"=>{ + "node.alpha.kubernetes.io/ttl"=>"0", + "volumes.kubernetes.io/controller-managed-attach-detach"=>"true" + }, + "managedFields"=>[ + { + "manager"=>"kube-controller-manager", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:20Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "f:node.alpha.kubernetes.io/ttl"=>{} + } + } + } + }, + { + "manager"=>"kubelet", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:24Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:annotations"=>{ + "."=>{}, + "f:volumes.kubernetes.io/controller-managed-attach-detach"=>{} + }, + "f:labels"=>{ + "."=>{}, + "f:agentpool"=>{}, + "f:beta.kubernetes.io/arch"=>{}, + "f:beta.kubernetes.io/instance-type"=>{}, + "f:beta.kubernetes.io/os"=>{}, + "f:failure-domain.beta.kubernetes.io/region"=>{}, + "f:failure-domain.beta.kubernetes.io/zone"=>{}, + "f:kubernetes.azure.com/cluster"=>{}, + "f:kubernetes.azure.com/mode"=>{}, + "f:kubernetes.azure.com/node-image-version"=>{}, + "f:kubernetes.azure.com/os-sku"=>{}, + "f:kubernetes.azure.com/role"=>{}, + "f:kubernetes.io/arch"=>{}, + "f:kubernetes.io/hostname"=>{}, + "f:kubernetes.io/os"=>{}, + "f:node.kubernetes.io/instance-type"=>{}, + "f:storageprofile"=>{}, + "f:storagetier"=>{}, + "f:topology.kubernetes.io/region"=>{}, + "f:topology.kubernetes.io/zone"=>{} + } + }, + "f:spec"=>{ + "f:providerID"=>{} + }, + "f:status"=>{ + "f:addresses"=>{ + "."=>{}, + "k:{\"type\":\"Hostname\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"InternalIP\"}"=>{ + "."=>{}, + "f:address"=>{}, + "f:type"=>{} + } + }, + "f:allocatable"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:capacity"=>{ + "."=>{}, + "f:attachable-volumes-azure-disk"=>{}, + "f:cpu"=>{}, + "f:ephemeral-storage"=>{}, + "f:hugepages-1Gi"=>{}, + "f:hugepages-2Mi"=>{}, + "f:memory"=>{}, + "f:pods"=>{} + }, + "f:conditions"=>{ + "."=>{}, + "k:{\"type\":\"DiskPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"MemoryPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PIDPressure\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"Ready\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + }, + "f:config"=>{}, + "f:daemonEndpoints"=>{ + "f:kubeletEndpoint"=>{ + "f:Port"=>{} + } + }, + "f:images"=>{}, + "f:nodeInfo"=>{ + "f:architecture"=>{}, + "f:bootID"=>{}, + "f:containerRuntimeVersion"=>{}, + "f:kernelVersion"=>{}, + "f:kubeProxyVersion"=>{}, + "f:kubeletVersion"=>{}, + "f:machineID"=>{}, + "f:operatingSystem"=>{}, + "f:osImage"=>{}, + "f:systemUUID"=>{} + } + } + } + }, + { + "manager"=>"kubectl-label", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-07-21T23:40:53Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:metadata"=>{ + "f:labels"=>{ + "f:kubernetes.io/role"=>{}, + "f:node-role.kubernetes.io/agent"=>{} + } + } + } + }, + { + "manager"=>"node-problem-detector", + "operation"=>"Update", + "apiVersion"=>"v1", + "time"=>"2021-08-10T18:10:02Z", + "fieldsType"=>"FieldsV1", + "fieldsV1"=>{ + "f:status"=>{ + "f:conditions"=>{ + "k:{\"type\":\"ContainerRuntimeProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FilesystemCorruptionProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FreezeScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentContainerdRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentDockerRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentKubeletRestart\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"FrequentUnregisterNetDevice\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KernelDeadlock\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"KubeletProblem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"PreemptScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"ReadonlyFilesystem\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RebootScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"RedeployScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + }, + "k:{\"type\":\"TerminateScheduled\"}"=>{ + "."=>{}, + "f:lastHeartbeatTime"=>{}, + "f:lastTransitionTime"=>{}, + "f:message"=>{}, + "f:reason"=>{}, + "f:status"=>{}, + "f:type"=>{} + } + } + } + } + } + ] + }, + "spec"=>{ + "providerID"=>"azure:///subscriptions/3b875bf3-0eec-4d8c-bdee-25c7ccc1f130/resourceGroups/mc_davidaks16_davidaks16_westus2/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-24816391-vmss/virtualMachines/0" + }, + "status"=>{ + "capacity"=>{ + "attachable-volumes-azure-disk"=>"8", + "cpu"=>"2", + "ephemeral-storage"=>"129900528Ki", + "hugepages-1Gi"=>"0", + "hugepages-2Mi"=>"0", + "memory"=>"7120616Ki", + "pods"=>"30" + }, + "allocatable"=>{ + "attachable-volumes-azure-disk"=>"8", + "cpu"=>"1900m", + "ephemeral-storage"=>"119716326407", + "hugepages-1Gi"=>"0", + "hugepages-2Mi"=>"0", + "memory"=>"4675304Ki", + "pods"=>"30" + }, + "conditions"=>[ + { + "type"=>"FrequentDockerRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentDockerRestart", + "message"=>"docker is functioning properly" + }, + { + "type"=>"FilesystemCorruptionProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsOK", + "message"=>"Filesystem is healthy" + }, + { + "type"=>"KernelDeadlock", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KernelHasNoDeadlock", + "message"=>"kernel has no deadlock" + }, + { + "type"=>"FrequentContainerdRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentContainerdRestart", + "message"=>"containerd is functioning properly" + }, + { + "type"=>"FreezeScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-11T23:25:04Z", + "reason"=>"NoFreezeScheduled", + "message"=>"VM has no scheduled Freeze event" + }, + { + "type"=>"FrequentUnregisterNetDevice", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentUnregisterNetDevice", + "message"=>"node is functioning properly" + }, + { + "type"=>"TerminateScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoTerminateScheduled", + "message"=>"VM has no scheduled Terminate event" + }, + { + "type"=>"ReadonlyFilesystem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"FilesystemIsNotReadOnly", + "message"=>"Filesystem is not read-only" + }, + { + "type"=>"RedeployScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRedeployScheduled", + "message"=>"VM has no scheduled Redeploy event" + }, + { + "type"=>"KubeletProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"KubeletIsUp", + "message"=>"kubelet service is up" + }, + { + "type"=>"PreemptScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:11:11Z", + "reason"=>"NoPreemptScheduled", + "message"=>"VM has no scheduled Preempt event" + }, + { + "type"=>"RebootScheduled", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoRebootScheduled", + "message"=>"VM has no scheduled Reboot event" + }, + { + "type"=>"ContainerRuntimeProblem", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"ContainerRuntimeIsUp", + "message"=>"container runtime service is up" + }, + { + "type"=>"FrequentKubeletRestart", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:25:56Z", + "lastTransitionTime"=>"2021-08-10T18:10:01Z", + "reason"=>"NoFrequentKubeletRestart", + "message"=>"kubelet is functioning properly" + }, + { + "type"=>"MemoryPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientMemory", + "message"=>"kubelet has sufficient memory available" + }, + { + "type"=>"DiskPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasNoDiskPressure", + "message"=>"kubelet has no disk pressure" + }, + { + "type"=>"PIDPressure", + "status"=>"False", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:14Z", + "reason"=>"KubeletHasSufficientPID", + "message"=>"kubelet has sufficient PID available" + }, + { + "type"=>"Ready", + "status"=>"True", + "lastHeartbeatTime"=>"2021-08-17T19:28:21Z", + "lastTransitionTime"=>"2021-07-21T23:40:24Z", + "reason"=>"KubeletReady", + "message"=>"kubelet is posting ready status. AppArmor enabled" + } + ], + "addresses"=>[ + { + "type"=>"Hostname", + "address"=>"aks-nodepool1-24816391-vmss000000" + }, + { + "type"=>"InternalIP", + "address"=>"10.240.0.4" + } + ], + "daemonEndpoints"=>{ + "kubeletEndpoint"=>{ + "Port"=>10250 + } + }, + "nodeInfo"=>{ + "machineID"=>"17a654260e2c4a9bb3a3eb4b4188e4b4", + "systemUUID"=>"7ff599e4-909e-4950-a044-ff8613af3af9", + "bootID"=>"02bb865b-a469-43cd-8b0b-5ceb4ecd80b0", + "kernelVersion"=>"5.4.0-1051-azure", + "osImage"=>"Ubuntu 18.04.5 LTS", + "containerRuntimeVersion"=>"containerd://1.4.4+azure", + "kubeletVersion"=>"v1.19.11", + "kubeProxyVersion"=>"v1.19.11", + "operatingSystem"=>"linux", + "architecture"=>"amd64" + }, + "images"=>[ + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021-1" + ], + "sizeBytes"=>331689060 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021" + ], + "sizeBytes"=>330099815 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021-hotfix" + ], + "sizeBytes"=>271471426 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021" + ], + "sizeBytes"=>269703297 + }, + { + "names"=>[ + "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03262021" + ], + "sizeBytes"=>264732875 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.19.0" + ], + "sizeBytes"=>166352383 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210623.2" + ], + "sizeBytes"=>147750148 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210524.1" + ], + "sizeBytes"=>146446618 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/hcp-tunnel-front:master.210427.1" + ], + "sizeBytes"=>136242776 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.5" + ], + "sizeBytes"=>101794833 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:0.47.0" + ], + "sizeBytes"=>101445696 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.3.0_v0.0.5" + ], + "sizeBytes"=>101194562 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210623.2" + ], + "sizeBytes"=>96125176 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210524.1" + ], + "sizeBytes"=>95879501 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/exechealthz:1.2_v0.0.5" + ], + "sizeBytes"=>94348102 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.8.9.2" + ], + "sizeBytes"=>93537927 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/acc/sgx-attestation:2.0" + ], + "sizeBytes"=>91841669 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.4.0" + ], + "sizeBytes"=>91324193 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.2.0" + ], + "sizeBytes"=>89103171 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.1-rc3" + ], + "sizeBytes"=>86839805 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0" + ], + "sizeBytes"=>86488586 + }, + { + "names"=>[ + "mcr.microsoft.com/aks/hcp/tunnel-openvpn:master.210427.1" + ], + "sizeBytes"=>86120048 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.3.0" + ], + "sizeBytes"=>81252495 + }, + { + "names"=>[ + "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.4.0" + ], + "sizeBytes"=>79586703 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.4.0" + ], + "sizeBytes"=>78795016 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.2.0" + ], + "sizeBytes"=>76527179 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.1.8" + ], + "sizeBytes"=>75025803 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix" + ], + "sizeBytes"=>73533889 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.3.1" + ], + "sizeBytes"=>72242894 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.8" + ], + "sizeBytes"=>70622822 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/nvidia/k8s-device-plugin:v0.9.0" + ], + "sizeBytes"=>67291599 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.1" + ], + "sizeBytes"=>66415836 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc7" + ], + "sizeBytes"=>65965658 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/azure-npm:v1.2.1" + ], + "sizeBytes"=>64123775 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.3" + ], + "sizeBytes"=>63581323 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8" + ], + "sizeBytes"=>63154716 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/cni:v3.8.9.2" + ], + "sizeBytes"=>61626312 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.18.1" + ], + "sizeBytes"=>60500885 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.17.2" + ], + "sizeBytes"=>58419768 + }, + { + "names"=>[ + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8_hotfix", + "mcr.microsoft.com/containernetworking/networkmonitor:v1.1.8post2" + ], + "sizeBytes"=>56368756 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:282543237a1aa3f407656290f454b7068a92e1abe2156082c750d5abfbcad90c", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.2" + ], + "sizeBytes"=>56310724 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/calico/node:v3.19.0" + ], + "sizeBytes"=>55228749 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526.1" + ], + "sizeBytes"=>54692048 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/dashboard:v2.0.0-rc3" + ], + "sizeBytes"=>50803639 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19" + ], + "sizeBytes"=>49759361 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.5" + ], + "sizeBytes"=>49704644 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.21" + ], + "sizeBytes"=>49372390 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:a64d3538b72905b07356881314755b02db3675ff47ee2bcc49dd7be856e285d5", + "mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.19.11-hotfix.20210526" + ], + "sizeBytes"=>49322942 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" + ], + "sizeBytes"=>48108311 + }, + { + "names"=>[ + "mcr.microsoft.com/oss/kubernetes/kubernetes-dashboard:v1.10.1" + ], + "sizeBytes"=>44907744 + } + ], + "config"=>{} + } + } + ] +} \ No newline at end of file diff --git a/test/unit-tests/run_go_tests.sh b/test/unit-tests/run_go_tests.sh new file mode 100755 index 000000000..7036531fd --- /dev/null +++ b/test/unit-tests/run_go_tests.sh @@ -0,0 +1,12 @@ +set -e + +OLD_PATH=$(pwd) +SCRIPTPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )" +cd $SCRIPTPATH/../../source/plugins/go/src +echo "# Runnign go generate" +go generate + +echo "# Running go test ." +go test . + +cd $OLD_PATH diff --git a/test/unit-tests/run_ruby_tests.sh b/test/unit-tests/run_ruby_tests.sh new file mode 100755 index 000000000..824346eee --- /dev/null +++ b/test/unit-tests/run_ruby_tests.sh @@ -0,0 +1,13 @@ +# this script will exit with an error if any commands exit with an error +set -e + +# NOTE: to run a specific test (instead of all) use the following arguments: --name test_name +# ex: run_ruby_tests.sh --name test_basic_single_node + +OLD_PATH=$(pwd) +SCRIPTPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )" +# cd $SCRIPTPATH/../../source/plugins/ruby +echo "# Running ruby $SCRIPTPATH/test_driver.rb $1 $2" +ruby $SCRIPTPATH/test_driver.rb $1 $2 + +cd $OLD_PATH diff --git a/test/unit-tests/test_driver.rb b/test/unit-tests/test_driver.rb new file mode 100644 index 000000000..32687cc99 --- /dev/null +++ b/test/unit-tests/test_driver.rb @@ -0,0 +1,13 @@ +$in_unit_test = true + +script_path = __dir__ +# go to the base directory of the repository +Dir.chdir(File.join(__dir__, "../..")) + +Dir.glob(File.join(script_path, "../../source/plugins/ruby/*_test.rb")) do |filename| + require_relative filename +end + +Dir.glob(File.join(script_path, "../../build/linux/installer/scripts/*_test.rb")) do |filename| + require_relative filename +end