ImportChangesFiles test segfaults on linux-arm64 and darwin-arm64

[UniversalSuperBox]

Detailed Description

When running Aptly's test suite on linux-arm64 or darwin-arm64, a segmentation fault is hit during the ImportChangesFiles test. The trace goes back through goleveldb into snappy. This appears to be related to golang/snappy#62, the suggested fix is to update to snappy 0.0.4 or higher. goleveldb updated to this version of snappy in v1.0.1-0.20210819022825-2ae1ddf74ef7 (syndtr/goleveldb#365).

Possible Implementation

Fixed by updating goleveldb:

aptly % go get github.com/syndtr/goleveldb/leveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7
go: upgraded github.com/syndtr/goleveldb v1.0.1-0.20190923125748-758128399b1d => v1.0.1-0.20210819022825-2ae1ddf74ef7
aptly % go test ./deb/ -check.f 'ImportChangesFiles'
ok      github.com/aptly-dev/aptly/deb  0.227s

Your Environment

darwin-arm64 on an M1 Mac (where else?) and linux-amd64 via Podman Machine on the same host (it's running Fedora Core OS in a VM)

Full output

Output of "go test -v ./deb/ -gocheck.v=true -check.f 'ImportChangesFiles'"

# go test -v ./deb/ -gocheck.v=true -check.f 'ImportChangesFiles'
=== RUN   Test
Loading repository test for changes file hardlink_0.2.1_amd64.changes...
unexpected fault address 0x7564200a2c785b
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7564200a2c785b pc=0x469a24]

goroutine 36 [running]:
runtime.throw({0x663dbd?, 0x469880?})
        /usr/lib/go-1.18/src/runtime/panic.go:992 +0x50 fp=0x40007bc620 sp=0x40007bc5f0 pc=0x46ca0
runtime.sigpanic()
        /usr/lib/go-1.18/src/runtime/signal_unix.go:825 +0x1a4 fp=0x40007bc650 sp=0x40007bc620 pc=0x5e284
github.com/golang/snappy.encodeBlock({0x4000232002, 0xe39, 0xe39}, {0x40007b0000, 0xc13, 0xd28})
        /root/go/pkg/mod/github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x354 fp=0x40007c46f0 sp=0x40007bc660 pc=0x469a24
github.com/golang/snappy.Encode({0x4000232000?, 0x458b38?, 0x40001067b8?}, {0x40007b0000?, 0x4000106801?, 0x467c3c?})
        /root/go/pkg/mod/github.com/golang/snappy@v0.0.2/encode.go:39 +0x1dc fp=0x40007c4780 sp=0x40007c46f0 pc=0x468fdc
github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0x4000194900, 0x4000194958, 0x4000106868?)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/table/writer.go:170 +0x118 fp=0x40007c47f0 sp=0x40007c4780 pc=0x471ea8
github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0x4000194900)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/table/writer.go:221 +0x40 fp=0x40007c4830 sp=0x40007c47f0 pc=0x472280
github.com/syndtr/goleveldb/leveldb/table.(*Writer).Close(0x4000194900)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/table/writer.go:294 +0x44 fp=0x40007c4910 sp=0x40007c4830 pc=0x4726a4
github.com/syndtr/goleveldb/leveldb.(*tWriter).finish(0x4000201140)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/table.go:578 +0x60 fp=0x40007c49c0 sp=0x40007c4910 pc=0x495f50
github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0xffffb7128f18?, {0x76aec8, 0x4000129000})
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/table.go:408 +0x194 fp=0x40007c4a70 sp=0x40007c49c0 pc=0x494864
github.com/syndtr/goleveldb/leveldb.(*Transaction).flush(0x40000cc000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_transaction.go:97 +0x108 fp=0x40007c4bb0 sp=0x40007c4a70 pc=0x4856c8
github.com/syndtr/goleveldb/leveldb.(*Transaction).Commit(0x40000cc000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_transaction.go:203 +0xb0 fp=0x40007c4ca0 sp=0x40007c4bb0 pc=0x486430
github.com/aptly-dev/aptly/database/goleveldb.(*transaction).Commit(0x5a3f00?)
        /code/database/goleveldb/transaction.go:49 +0x24 fp=0x40007c4cc0 sp=0x40007c4ca0 pc=0x49f314
github.com/aptly-dev/aptly/deb.ImportPackageFiles(0x4000110dd0?, {0x400002c840, 0x3, 0x766a58?}, 0x0, {0x769b60, 0xb03f10}, {0x76a6f0, 0x400006e060}, 0x40000c00f0, ...)
        /code/deb/import.go:230 +0x168 fp=0x40007c5030 sp=0x40007c4cc0 pc=0x4dadd8
github.com/aptly-dev/aptly/deb.ImportChangesFiles({0x4000128600?, 0x5, 0x40002c35e8?}, {0x766a58, 0x400007c050}, 0x0?, 0x0?, 0x0?, 0x0, {0x769b60, ...}, ...)
        /code/deb/changes.go:375 +0xf0c fp=0x40007c5400 sp=0x40007c5030 pc=0x4d3f1c
github.com/aptly-dev/aptly/deb.(*ChangesSuite).TestImportChangesFiles(0x4000212200, 0x0?)
        /code/deb/changes_test.go:124 +0x71c fp=0x40007c5700 sp=0x40007c5400 pc=0x5066ec
runtime.call16(0x400007e2a0, 0x4000010008, 0x0, 0x0, 0x0, 0x10, 0x40007c5c40)
        /usr/lib/go-1.18/src/runtime/asm_arm64.s:507 +0x7c fp=0x40007c5720 sp=0x40007c5700 pc=0x7805c
runtime.reflectcall(0x657720?, 0x40002b4000?, 0x2?, 0x66af8a?, 0x0?, 0x12?, 0x657720?)
        <autogenerated>:1 +0x34 fp=0x40007c5760 sp=0x40007c5720 pc=0x7bad4
reflect.Value.call({0x61e6e0?, 0x4000212200?, 0x400005ddf8?}, {0x6636f2, 0x4}, {0x400005df00, 0x1, 0xe?})
        /usr/lib/go-1.18/src/reflect/value.go:556 +0x5e4 fp=0x40007c5dd0 sp=0x40007c5760 pc=0xdcd24
reflect.Value.Call({0x61e6e0?, 0x4000212200?, 0x400020e0c0?}, {0x400005df00, 0x1, 0x1})
        /usr/lib/go-1.18/src/reflect/value.go:339 +0x98 fp=0x40007c5e50 sp=0x40007c5dd0 pc=0xdc538
gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0x40002b4000)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:775 +0x544 fp=0x40007c5f60 sp=0x40007c5e50 pc=0x4c2bf4
gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1()
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:669 +0x94 fp=0x40007c5fd0 sp=0x40007c5f60 pc=0x4c1ee4
runtime.goexit()
        /usr/lib/go-1.18/src/runtime/asm_arm64.s:1259 +0x4 fp=0x40007c5fd0 sp=0x40007c5fd0 pc=0x7a204
created by gopkg.in/check%2ev1.(*suiteRunner).forkCall
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:666 +0x268

goroutine 1 [chan receive]:
testing.(*T).Run(0x4000280b60, {0x6637c2?, 0xa45dd32e559?}, 0x6ab510)
        /usr/lib/go-1.18/src/testing/testing.go:1487 +0x33c
testing.runTests.func1(0x0?)
        /usr/lib/go-1.18/src/testing/testing.go:1839 +0x74
testing.tRunner(0x4000280b60, 0x4000107cb8)
        /usr/lib/go-1.18/src/testing/testing.go:1439 +0x110
testing.runTests(0x400021a6e0?, {0xabe320, 0x1, 0x1}, {0xfc00000000000000?, 0x1dc90?, 0xad14e0?})
        /usr/lib/go-1.18/src/testing/testing.go:1837 +0x3e8
testing.(*M).Run(0x400021a6e0)
        /usr/lib/go-1.18/src/testing/testing.go:1719 +0x510
main.main()
        _testmain.go:53 +0x1e8

goroutine 34 [chan receive]:
gopkg.in/check%2ev1.(*suiteRunner).runTest(...)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:813
gopkg.in/check%2ev1.(*suiteRunner).run(0x4000212380)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:618 +0x1d8
gopkg.in/check%2ev1.Run({0x61e6e0?, 0x4000212200?}, 0xffffd8f57934?)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/run.go:92 +0x2c
gopkg.in/check%2ev1.RunAll(0x400004ae48?)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/run.go:84 +0x88
gopkg.in/check%2ev1.TestingT(0x4000280d00)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/run.go:72 +0x2b4
github.com/aptly-dev/aptly/deb.Test(0x0?)
        /code/deb/debian_test.go:11 +0x20
testing.tRunner(0x4000280d00, 0x6ab510)
        /usr/lib/go-1.18/src/testing/testing.go:1439 +0x110
created by testing.(*T).Run
        /usr/lib/go-1.18/src/testing/testing.go:1486 +0x328

goroutine 35 [select]:
gopkg.in/check%2ev1.(*resultTracker)._loopRoutine(0x40002921b0)
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:464 +0x74
created by gopkg.in/check%2ev1.(*resultTracker).start
        /root/go/pkg/mod/gopkg.in/check.v1@v1.0.0-20201130134442-10cb98267c6c/check.go:444 +0x64

goroutine 38 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x40001121c0)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/util/buffer_pool.go:206 +0x9c
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/util/buffer_pool.go:237 +0x18c

goroutine 39 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0x40002b41e0)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/session_util.go:189 +0x398
created by github.com/syndtr/goleveldb/leveldb.newSession
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/session.go:93 +0x280

goroutine 21 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x40002e0000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_compaction.go:91 +0x120
created by github.com/syndtr/goleveldb/leveldb.openDB
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db.go:148 +0x3f8

goroutine 22 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x40002e0000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_state.go:101 +0x84
created by github.com/syndtr/goleveldb/leveldb.openDB
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db.go:149 +0x43c

goroutine 23 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x40002e0000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_compaction.go:836 +0x490
created by github.com/syndtr/goleveldb/leveldb.openDB
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db.go:155 +0x4a4

goroutine 24 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x40002e0000)
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db_compaction.go:773 +0xcc
created by github.com/syndtr/goleveldb/leveldb.openDB
        /root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190923125748-758128399b1d/leveldb/db.go:156 +0x4e8

goroutine 25 [chan receive]:
github.com/aptly-dev/aptly/console.(*standardProgressWorker).run(0x40000aa128)
        /code/console/progress.go:203 +0x54
created by github.com/aptly-dev/aptly/console.(*Progress).Start
        /code/console/progress.go:59 +0x7c
FAIL    github.com/aptly-dev/aptly/deb  0.154s
FAIL

[r4co0n]

Since aptly is running a modified and locally maintained version of goleveldb, as to be found in /database/goleveldb/ inherited from the original maintainer, I think we would need to backport the goleveldb patch you mentioned to try to fix the test error you encountered.

In the long run, we should discover whether we can use upstream goleveldb, and patch it to what we need it to be, or better yet, try to make aptly only use what is provided upstream anyways. (I said it on this bug tracker, and I'll do so again: I am not a Go developer, but I can read source files and I know a thing or two about software development in general.)

Maybe someone familiar with goleveldb can give us hints on how to progress with getting rid of our local goleveldb package?

[r4co0n]

I think I was severly wrong in my comment above, sorry about that. The relevant goleveldb version can be bumped here.

[neolynx]

should be fixed in CU builds, please test and confirm !

[neolynx]

I can confirm this does also affect linux arm64 systems. the updated dependencies fix the segfault in snappy.

[neolynx]

aptly 1.6.1 released, should fix this issue !