[Bug] SIGSEGV: segmentation violation in banyand/query.(*topNQueryProcessor).Rev.func1()

[Almot77]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

BanyanDB (apache/skywalking-banyandb)

What happened

Server crash on segmentation fault in docker. I use :latest SW and BYDB images (new releases).

What you expected to happen

{"level":"info","module":"STREAM-SEGMENT.SCHEDULER.RETENTION","name":"retention","now":"2024-05-13T14:49:52Z","time":"2024-05-13T14:49:52Z","message":"start"}
{"level":"info","module":"STREAM","group":"stream-browser_error_log","time":"2024-05-13T14:49:52Z","message":"creating a tsdb"}
{"level":"info","module":"STREAM-BROWSER_ERROR_LOG","path":"/tmp/stream-data/stream/stream-browser_error_log","time":"2024-05-13T14:49:52Z","message":"initialized"}
{"level":"info","module":"STREAM-BROWSER_ERROR_LOG.SCHEDULER.RETENTION","name":"retention","now":"2024-05-13T14:49:52Z","time":"2024-05-13T14:49:52Z","message":"start"}
{"level":"info","module":"STREAM","group":"stream-zipkin_span","time":"2024-05-13T14:49:52Z","message":"creating a tsdb"}
{"level":"info","module":"STREAM-ZIPKIN_SPAN","path":"/tmp/stream-data/stream/stream-zipkin_span","time":"2024-05-13T14:49:52Z","message":"initialized"}
{"level":"info","module":"STREAM-ZIPKIN_SPAN.SCHEDULER.RETENTION","name":"retention","now":"2024-05-13T14:49:52Z","time":"2024-05-13T14:49:52Z","message":"start"}
{"level":"error","module":"QUERY.TOPN.MEASURE-MINUTE.ENDPOINT_RESP_TIME_MINUTE_TOPN","error":"failed to query measure: unmarshal tag value: unsupported tag value type","req":{"groups":["measure-minute"], "name":"endpoint_resp_time_minute_topn", "timeRange":{"begin":"2024-05-13T14:20:00Z", "end":"2024-05-13T14:51:00Z"}, "topN":10, "agg":"AGGREGATION_FUNCTION_MEAN", "conditions":[{"name":"service_id", "op":"BINARY_OP_EQ", "value":{"str":{"value":"cGhwLW1zay1sZWdhY3k=.1"}}}], "fieldValueSort":"SORT_DESC"},"time":"2024-05-13T14:50:30Z","message":"fail to close the topn plan"}
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1251163]

goroutine 356 [running]:
github.com/apache/skywalking-banyandb/banyand/query.(*topNQueryProcessor).Rev.func1()
        /src/banyand/query/processor_topn.go:126 +0x23
panic({0x13d6e40?, 0x259ebd0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/apache/skywalking-banyandb/banyand/query.(*topNQueryProcessor).Rev(0xc000010fd8, {{0x156fd20, 0xc0085ce280}, {0x15f10a0, 0x5}, 0x17cf13deadbc52eb, 0x0})
        /src/banyand/query/processor_topn.go:133 +0xfd6
github.com/apache/skywalking-banyandb/pkg/bus.(*Bus).Subscribe.func1({0x1a71de0, 0xc000010fd8}, 0xc0001d09c0)
        /src/pkg/bus/bus.go:274 +0xfa
created by github.com/apache/skywalking-banyandb/pkg/bus.(*Bus).Subscribe in goroutine 1
        /src/pkg/bus/bus.go:270 +0x28f

How to reproduce

Docker compose docker compose --profile banyandb up -d

version: '3.8'
services:
  elasticsearch:
    profiles:
      - "elasticsearch"
    image: itbgk/elasticsearch-oss:7.9.2
    container_name: skywalking-elasticsearch
    ports:
      - "9200:9200"
    networks:
      - skywalking
    volumes:
      - elastic-sw:/usr/share/elasticsearch/data
    healthcheck:
      test: [ "CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1" ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: always
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  banyandb:
    profiles:
      - "banyandb"
    image: ${BANYANDB_IMAGE:-apache/skywalking-banyandb:latest}
    container_name: banyandb
    restart: always
    networks:
      - skywalking
    expose:
      - 17912
    ports:
      - 17913:17913
    volumes:
      - banyandb-stream-data:/tmp/stream-data
      - banyandb-measure-data:/tmp/measure-data

    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: [ "CMD", "sh", "-c", "nc -nz 127.0.0.1 17912" ]
      interval: 5s
      timeout: 60s
      retries: 120

  oap-base: &oap-base
    profiles: [ "none" ]
    image: ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
    ports:
      - "11800:11800"
      - "12800:12800"
      - "9099:9090"
      - "3100:3100"
    networks:
      - skywalking
    healthcheck:
      test: [ "CMD-SHELL", "curl http://localhost:12800/internal/l7check" ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
#    restart: always
    environment: &oap-env
      TZ: Europe/Moscow
      SW_HEALTH_CHECKER: default
      SW_OTEL_RECEIVER: default
      SW_OTEL_RECEIVER_ENABLED_OC_RULES: vm
      SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES: vm
      SW_TELEMETRY: prometheus
      JAVA_OPTS: "-Xms2048m -Xmx2048m"
      SW_CORE_RECORD_DATA_TTL: 2 # https://skywalking.apache.org/docs/main/next/en/setup/backend/ttl/
      SW_CORE_METRICS_DATA_TTL: 2
      SW_DCS_MAX_INBOUND_MESSAGE_SIZE: 5000000000

  oap-es:
    <<: *oap-base
    profiles:
      - "elasticsearch"
    container_name: skywalking-server # rename to something else if switching to BanyanDB
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      <<: *oap-env
      SW_STORAGE: elasticsearch
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200
      SW_CORE_RECORD_DATA_TTL: 2 # https://skywalking.apache.org/docs/main/next/en/setup/backend/ttl/
      SW_CORE_METRICS_DATA_TTL: 2
      SW_DCS_MAX_INBOUND_MESSAGE_SIZE: 5000000000

  oap-bdb:
    <<: *oap-base
    profiles:
      - "banyandb"
    container_name: skywalking-server-bdb # rename to oap if switching to Elasticsearch
    depends_on:
      banyandb:
        condition: service_healthy
    environment:
      <<: *oap-env
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
      SW_CORE_RECORD_DATA_TTL: 14 # https://skywalking.apache.org/docs/main/next/en/setup/backend/ttl/
      SW_CORE_METRICS_DATA_TTL: 14
      SW_DCS_MAX_INBOUND_MESSAGE_SIZE: 5000000000

  ui:
    image: ${UI_IMAGE:-ghcr.io/apache/skywalking/ui:latest}
    container_name: skywalking-ui
    ports:
      - "1010:8080"
    networks:
      - skywalking
    restart: always
    environment:
      <<: *oap-env
      SW_OAP_ADDRESS: http://skywalking-server-bdb:12800
      SW_ZIPKIN_ADDRESS: http://skywalking-server-bdb:9412

volumes:
  elastic-sw:
  banyandb-stream-data:
    external: true
  banyandb-measure-data:
    external: true

networks:
  skywalking:

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Your configuration is not well formatted. Please correct them. And what does SW_STORAGE: elasticsearch mean? I think we don't need Elasticsearch when you use BanyanDB.

And we don't have banyandb-helm 0.2 release, how do you deploy the database?

[lujiajing1126]

After checking the code, it seems error is not handled properly.

[wu-sheng]

@lujiajing1126 What is the case of occurring this error?

[Almot77]

Your configuration is not well formatted. Please correct them. And what does SW_STORAGE: elasticsearch mean? I think we don't need Elasticsearch when you use BanyanDB.

And we don't have banyandb-helm 0.2 release, how do you deploy the database?

s/bus.go:270 +

It`s docker-compose.yml file.
I run SW with selected db profile: elastic or banyandb.

Right way to run it:
docker compose --profile banyandb up -d

[wu-sheng]

Are you using docker quick start? We haven't upgraded it to latest. It needs v10 oap and latest banyandb 0.6.

[wu-sheng]

This error is easy to fix, @Almot77 but we want to know how you could trigger it, as we have run many tests to verify features.

[hanahmily]

The nil error is fixed by https://github.com/apache/skywalking-banyandb/pull/445/files#diff-695073ea8dec3fcdaae77a3fcfb4eabc7290daade34399aee1de429999d7b476R124

But the error below is a bit tricky.

{"level":"error","module":"QUERY.TOPN.MEASURE-MINUTE.ENDPOINT_RESP_TIME_MINUTE_TOPN","error":"failed to query measure: unmarshal tag value: unsupported tag value type","req":{"groups":["measure-minute"], "name":"endpoint_resp_time_minute_topn", "timeRange":{"begin":"2024-05-13T14:20:00Z", "end":"2024-05-13T14:51:00Z"}, "topN":10, "agg":"AGGREGATION_FUNCTION_MEAN", "conditions":[{"name":"service_id", "op":"BINARY_OP_EQ", "value":{"str":{"value":"cGhwLW1zay1sZWdhY3k=.1"}}}], "fieldValueSort":"SORT_DESC"},"time":"2024-05-13T14:50:30Z","message":"fail to close the topn plan"}

@Almot77 Would you pls use the last banyandb image built from apache/skywalking-banyandb#445 to output more context about this error?

If you have an appropriate docker environment, issuing make docker.build is all you need.