Latest SEnginx master causes 'segfault's - but older SEnginx V1.5.13 is fine

[peterbowey]

@InfoHunter

Pulling the latest SEnginx, I notice regular 'segfault's showing in the system logs:

May  3 21:03:10 server1 kernel: [83732.093146] nginx[37442]: segfault at 45555d ip 000000000045e8b1 sp 00007fff63554bc0 error 7 in nginx[400000+849000]
May  3 21:03:10 server1 kernel: nginx[37442]: segfault at 45555d ip 000000000045e8b1 sp 00007fff63554bc0 error 7 in nginx[400000+849000]
May  3 21:03:41 server1 kernel: [83762.969395] nginx[37448]: segfault at 45555d ip 000000000045e8b1 sp 00007fff63554bc0 error 7 in nginx[400000+849000]
May  3 21:03:41 server1 kernel: nginx[37448]: segfault at 45555d ip 000000000045e8b1 sp 00007fff63554bc0 error 7 in nginx[400000+849000]

Reverting to the older SEnginx v1.5.13 + changes [@4e896e9], no event errors or issues are seen.
Notes: The older SEnginx results in clean logs and no 'faults' for days.... [14 days on last run and using SEnginx v1.5.13].

Notes: the older SEnginx v1.5.13 fallback includes the older essential 'modules' within "neusoft". The only extra 'third-party' modules used on both test cases were: 1) ngx_pagespeed, 2) ngx-cache-purge-2.1 and 3) nginx-upload-progress-module-0.9

Sorry I have not done a full SEnginx debug analysis and report here, but my system is a live production server. My web client's do not like the server with broken data streams - for long periods.

[InfoHunter]

Hi Peter,

Did you enable the "dynamic resolve" functionality in proxy_pass directive in your config file when running the latest SEnginx ?

[peterbowey]

@InfoHunter
Hi Paul,

Via: ""dynamic resolve" functionality in proxy_pass directive"
No I did not. But thinking about the master SEnginx 'segfault' - I would agree that some 'new' func() was missing - in my build. I had thought that perhaps one of the 'third-party' modules was now a dependency. Typically - that would lead to a 'segfault'

Notes: Globally, I am compiling in - all the 'neusoft' modules - but none of the clearly marked '3rd-party'.

[peterbowey]

@InfoHunter

Paul, what I might do here is run a 'diff -uNr old new > SEnginx-change.patch' and see if can spot the change that might trigger this 'segfault'

1. 'old' being = SEnginx v1.5.13
2. 'new' being = SEnginx @ master

Some new function() is out of scope - or has a dependency on 'other'??

Again, note: globally, I am compiling in - all the 'neusoft' modules - but none of the included '3rd-party' modules.

[InfoHunter]

Most changes between v1.5.13 and master are related to some enhancements to the "dynamic resolve" feature. Those enhancements are now under testing and is not ready for production uses. I think there might be some bugs that caused the 'segfault'.

Could you please provide me more information about this segfault ? For example: configuration file / the core dump file / error.log ... That would be very helpful. Thanks.

[peterbowey]

@InfoHunter

I will see what I can do, however the production 'nginx.conf' is large (but with comments).
I will 'later' schedule are core dump to help, but it is a production server and so needs to be done outside of normal human 'traffic' times.

At this moment, the server is running 'clean' using SEnginx 1.5.13 + the changes [@4e896e9]

[InfoHunter]

@peterbowey

OK, at your convenience. Thanks a lot !

[InfoHunter]

@peterbowey

Hi Peter,

Do you still have the binary file - 'nginx' which caused the segfault in your environment ? If so, would you please send me a copy of it ? Thanks

[peterbowey]

@InfoHunter

Hi Paul,

Do you want the actual Fedora *.rpm package or just the the [@master] SEnginx-1.5.14.tar.gz?
Realize that my version of the *.tar will not have the packaged 'neusoft' modules' - as I compile those as separate file modules, so they exist as external *.tar.gz's.

As my build machine is separate from production, I can easily re-create the scenario above...[anyway].

[InfoHunter]

@peterbowey

Just a single binary file(the executable 'nginx' file), don't need the whole rpm package. But the whole package is also fine, I can extract the binary file by myself.

[peterbowey]

@InfoHunter

I found the original 'nginx' binary as I used to report the 'segfault' above:

Using a 'hard' download link to my own web site, here it is:
(optionally added the 'config' too)

1. ELF 64-bit LSB executable, x86-64 "nginx" binary: http://www.pbcomp.com.au/downloads/nginx
2. "nginx.conf" config: http://www.pbcomp.com.au/downloads/nginx.conf

Notes: My records indicate that should be the actual binary I last used - as last compiled from SEnginx master. I did run a compile on 2 earlier bin's based on SEnginx master, but this link was the last and purest (SEnginx wise).

[peterbowey]

@InfoHunter

Paul, I may have to turn off my web server IPSET geo-country blacklisting if you are pulling this from 'Asia'.
Is that required at your IP geo?

[InfoHunter]

@peterbowey

Yeah, I can't even resolve your site's domain.

[InfoHunter]

But if I use a proxy to connect your site, it kicked back a 403 forbidden error.