Segmentation fault (core dumped) when putting items in Index

[kamcio181]

Hello,

first of all I really appreciate your work and tool you have created. Currently I am trying to create a PoC using keyvi but I am facing following issue.
I have decided to put some data in one batch in order to create as few files as possible. I am using python so I cannot verify the issue by calling C++ code directly but it seems that error returned Segmentation fault (core dumped) is related to C++ part.
Basically I am trying to create 100300 items Index. Interesting part is that when I try to create i.e. 200300 items Index the issue is not occurring. You can find repro script below.
Any help will be appreciated.

from keyvi._core import Index
def run_repro():
    index = Index('repro')
    data = [(str(i), 'test') for i in range(100300)] # 
    index.MSet(data)
    index.Flush()
if __name__ == '__main__':
    run_repro()

best regards,
Kamil

[hendrikmuhs]

Thanks for the report, I can reproduce it.

From a quick look I see a memory corruption as the issue, I will take a closer look tomorrow.

[hendrikmuhs]

@kamcio181

I debugged your issue and was able to create a standalone example that triggers the crash in boost parallel stable sort. This sort is used only in the index compiler for > 10000 keys.

I opened boostorg/sort#54.

[kamcio181]

@hendrikmuhs
Thank you for quick response and information.

[hendrikmuhs]

@kamcio181

I switched the parallel sorting algorithm to the other one provided in boost sort and released a new version 0.5.2. The issue is fixed, however I created an enhancement issue as follow up: #219.

I hope this fixes it for you, don't hesitate to report further problems.