Green Cell UPS 2000VA fails to shutdown UPS at FSD

[got3nks]

I'm correctly connecting to the UPS using this driver:

[greencell]
        driver = "nutdrv_qx"
        default.battery.voltage.high = 25.60
        default.battery.voltage.low = 20.80
        port = "auto"
        desc = "Green Cell UPS 2000VA"

root@rpi:~# upsc greencell

battery.charge: 100
battery.voltage: 28.00
battery.voltage.high: 25.60
battery.voltage.low: 20.80
device.type: ups
driver.name: nutdrv_qx
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: Q1 0.07
driver.version.internal: 0.28
input.frequency: 50.0
input.voltage: 240.5
input.voltage.fault: 245.5
output.voltage: 240.5
ups.beeper.status: enabled
ups.delay.shutdown: 30
ups.delay.start: 180
ups.load: 1
ups.productid: 0000
ups.status: OL
ups.type: offline / line interactive
ups.vendorid: 0001

The command upsdrvctl shutdown correctly sends the command to the UPS

Network UPS Tools - UPS driver controller 2.7.4
Network UPS Tools - Generic Q* USB/Serial driver 0.28 (2.7.4)
USB communication driver 0.33
Using protocol: Q1 0.07
Initiating UPS shutdown
instcmd(shutdown.stop, [NULL])
instcmd: SUCCEED
instcmd(shutdown.return, [NULL])
instcmd: SUCCEED
Shutting down in 30 seconds

But it fails to shutdown the UPS, and doesnt set the FSD flag either. After more digging I found out my UPS won't go into shutdown mode because the shutdown.return is called too soon after the shutdown.stop.

/usr/bin/upscmd -u <user> -p <pass> greencell shutdown.stop && sleep "3" && /usr/bin/upscmd -u <user> -p <password> greencell shutdown.return works fine while /usr/bin/upscmd -u <user> -p <pass> greencell shutdown.stop && /usr/bin/upscmd -u <user> -p <password> greencell shutdown.return (notice the lack of sleep between the commands) won't put the UPS into FSD and won't shutdown the UPS.

I thought I could solve this editing the /lib/systemd/system-shutdown/nutshutdown script, replacing /sbin/upsdrvctl shutdown with my commands. But of course the nut-server service is already stopped when the nutshutdown script is executed resulting in a failure Error: Connection failure: Connection refused.

Any idea on how to solve this / a workaround?

[jimklimov]

Just to clarify: when you call upsdrvctl shutdown, is the original driver daemon still running? It may be holding the device node for the USB port then and not actually let the command pass to the device. Gotta run with debug or dig into code to check if that is the case (I'd expect some libusb errors) or why it reports "SUCCESS".

The upsdrvctl (and the driver program it invokes) talk to the device, and do not tell upsd to set the FSD flag. In fact, as you found, this is a last-moment activity to cut the power (many devices do not have a timeout, or not one easily configured - e.g. many CPS devices account in whole 60 seconds, so 59 == 0 for the timeout) and daemons including upsd AND the usual driver daemon are assumed stopped by then, remaining filesystems flushed and remounted read-only, etc.

You might be after upsmon -c fsd which tells upsd to tell drivers to tell UPSes to turn off (for "master"/"primary" UPSes as assigned to this upsmon deployment). It should arrange interaction with other upsmon clients connected to the upsd instances involved in the shutdown, so those systems have a chance to cleanly shut down first (if their remaining power source count becomes too small per their config).

[got3nks]

I tried to execute upsdrvctl shutdown in normal run-time conditions, so yes the nut-server and nut-monitor services are all running (and so the driver daemon is running too, I guess).

Of course, I tried to execute the upsmon -c fsd command. All secondary hosts correctly shut down, the primary shuts down BUT the UPS is still on after the 30 seconds timer. This is the issue I'm trying to solve and from my findings it depends on the shutdown.stop call the upsdrvctl command is issuing, without any delay before the shutdown.return.

[got3nks]

Maybe I can use the /usr/lib/nut/nutdrv_qx driver directly to issue a shutdown.return (without the shutdown.stop) in my /lib/systemd/system-shutdown/nutshutdown script?

[jimklimov]

Yes, that is among options. Using upsdrvctl normally allows to avoid system scripts hardcoded for particular driver names, but otherwise it is not forbidden.

Check if your upsmon is configured to leave a KILLFLAG file and actually does (allowed to write there, location still is mounted when late-shutdown scripts are handled). IIRC upsmon -K is used to check that locally managed UPS poweroff was requested and checks that such file exists.

[got3nks]

I edited the /lib/systemd/system-shutdown/nutshutdown script:

#!/bin/sh
echo "nutshutdown called" | sudo tee /dev/kmsg

powerdownflag=$(/sbin/upsmon -K 2>&1)
RETVAL=$?
echo "nutshutdown flag: $powerdownflag" | sudo tee /dev/kmsg
echo "nutshutdown return code: $RETVAL" | sudo tee /dev/kmsg
if [ $RETVAL -eq 0 ]; then
        /sbin/upsdrvctl shutdown 2>&1 | sudo tee /dev/kmsg
        echo "nutshutdown executed UPS shutdown" | sudo tee /dev/kmsg
fi

mount -o remount,rw /
dmesg > /root/shutdown-log
mount -o remount,ro /

And I can see it's being called, and the POWERDOWNFLAG was succesfully set (upsmon -K has return code of 0), as it's been logged in /root/shutdown-log (and shown in the stdout during the final shutdown phase of the OS):

[24333.501326] nutshutdown called
[24336.563560] nutshutdown flag: 
               UPS: greencell@localhost (master) (power value 1)
               Using power down flag file /etc/killpower
               Power down flag is set
               Network UPS Tools upsmon 2.7.4
[24336.608092] nutshutdown return code: 0
[24339.973531] Using protocol: Q1 0.07
               Initiating UPS shutdown
[24339.975007] instcmd(shutdown.stop, [NULL])
[24339.986175] instcmd: SUCCEED
               instcmd(shutdown.return, [NULL])
[24340.006297] instcmd: SUCCEED
               Shutting down in 30 seconds
[24340.007620] Network UPS Tools - Generic Q* USB/Serial driver 0.28 (2.7.4)
               USB communication driver 0.33
               Network UPS Tools - UPS driver controller 2.7.4
[24340.052987] nutshutdown executed UPS shutdown

Can you tell me how to issue a shutdown.return command through /usr/lib/nut/nutdrv_qx? I think the -k flag will do the same as the upsdrvctl wrapper i.e. first shutdown.stop then immediately shutdown.return, which won't work with my UPS.

[jimklimov]

Sorry, can't quickly say, and not sure OTOH it is possible now (more so in old NUT v2.7.4). Maybe the driver model needs to be extended to post such custom commands to devices via driver CLI :\

[got3nks]

As a workarond, this /lib/systemd/system-shutdown/nutshutdown will do the trick for me:

#!/bin/sh
powerdownflag=$(/sbin/upsmon -K 2>&1)
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
        /sbin/upsdrvctl start 2>&1 | sudo tee /dev/kmsg
        sleep "1"
        /sbin/upsd 2>&1 | sudo tee /dev/kmsg
        sleep "3"
        battery_status=$(/usr/bin/upsc greencell ups.status 2>/dev/null)
        if echo "$battery_status" | grep -q "OL" 2>/dev/null; then
           echo "UPS is OL - reboot" | sudo tee /dev/kmsg
           sudo reboot -f  | sudo tee /dev/kmsg
        else
           echo "UPS is not OL - shutdown UPS" | sudo tee /dev/kmsg
           /usr/bin/upscmd -u <user> -p <pass> greencell shutdown.return 2>&1 | sudo tee /dev/kmsg
           sleep "120"
           echo "UPS did not power off, maybe power returned after we issued the shutdown" | sudo tee /dev/kmsg
           sudo reboot -f | sudo tee /dev/kmsg
        fi
fi

[got3nks]

UPDATE: I compiled NUT v2.8.0 from source code and installed it. Found out the subdriver = "hunnox" works fine with the standard upsdrvctl shutdown.

[greencell]
        driver = "nutdrv_qx"
        port = "auto"
        vendorid = "0001"
        productid = "0000"
        product = "MEC0003"
        vendor = "MEC"
        bus = "001"
        subdriver = "hunnox"
        desc = "Green Cell UPS"

BUT the Green Cell UPS has a power race issue. If the line power is back between the shutdown command is issued and the UPS actually shuts down (the UPS has an hard coded delay of 30 secs), the UPS wont turn the load on after the shutdown.

To fix the issue, I had to implement some kind of workaround in the /lib/systemd/system-shutdown/nutshutdown script again. Maybe it can be helpful to others so I'll post it here.

#!/bin/sh

[ -x "/usr/local/ups/sbin/upsmon" ] && [ -x "/usr/local/ups/sbin/upsdrvctl" ] || exit

get_ups_status() {
  echo "$(/usr/local/ups/bin/upsc greencell ups.status 2>/dev/null)"
}

if /usr/local/ups/sbin/upsmon -K >/dev/null 2>&1; then

  mount -o remount,rw /
  /usr/local/ups/sbin/upsdrvctl start 2>&1 | sudo tee /dev/kmsg
  /bin/sleep "2"
  /usr/local/ups/sbin/upsd 2>&1 | sudo tee /dev/kmsg
  /bin/sleep "2"
  mount -o remount,ro /

  echo "UPS status $(get_ups_status)" | sudo tee /dev/kmsg

  if echo "$(get_ups_status)" | grep -q "OL" 2>/dev/null; then
     echo "UPS is OL - reboot" | sudo tee /dev/kmsg
     /bin/sleep "10"
     sudo /bin/systemctl reboot --force --force | sudo tee /dev/kmsg
  else
     echo "UPS is not OL - shutdown UPS" | sudo tee /dev/kmsg
     /usr/local/ups/sbin/upsdrvctl shutdown
     i=1
     while [ "$i" -le 60 ]; do
       if echo "$(get_ups_status)" | grep -q "OL" 2>/dev/null; then
          echo "UPS is back OL - STOP UPS shutdown and REBOOT" | sudo tee /dev/kmsg
          /usr/local/ups/bin/upscmd -u <user> -p <pass> -w greencell shutdown.stop 2>&1 | sudo tee /dev/kmsg
          /bin/sleep "10"
          sudo /bin/systemctl reboot --force --force | sudo tee /dev/kmsg
          exit 0
       else
          echo "UPS status $(get_ups_status)" | sudo tee /dev/kmsg
       fi
       /bin/sleep "2"
       i=$(( i + 1 ))
     done
     echo "UPS did not power off? We will reboot" | sudo tee /dev/kmsg
     /bin/sleep "120"
     sudo /bin/systemctl reboot --force --force | sudo tee /dev/kmsg
  fi
fi

exit 0

After the reboot, on this device (primary) I'm running a python script which will check the battery status and will wake up all the secondary devices with WOL magic packets.

[jimklimov]

Looks great, thanks! I'll re-post it into the Wiki eventually.

Such sort of workarounds (sleep long to drain battery and turn UPS off this way, or reboot after say 30min if still alive) were popular a couple of decades ago. One potential problem with current event/timing driven frameworks (systemd, SMF) may be that they would enforce a shutdown timeout (e.g. 90s or 5m) and kill anything that remains anyway.

[got3nks]

The issue I tried to solve with the custom script is issuing a shutdown.stop and reboot the primary device during the shutdown phase if the line power is back before the UPS actually goes down (at the very last moment as possible), because my UPS wouldn't turn the load back on later. I tested it and it worked. ;)

[jimklimov]

Looking at the script itself, a couple of concerns:

1. Did you check how systemd reacts to a shutdown script returning non-zero exit code? (e.g. in first lines where you check that NUT programs are present)
2. in get_ups_status() I think you can do without the echo; it might however help to generalize with upsc -l to list devices on the localhost, and query their statuses instead of hardcoding
3. remount,rw during shutdown seems a bit risky. If that is done for pipe-files between upsd and driver, you might be better served with e.g. export NUT_STATEPATH=/dev/shm (maybe also NUT_ALTPIDPATH) to use a tmpfs (if still mounted) for that. Generally it is also not "a given" that the typical /var/state/ups is in / (often /var is separate).

[got3nks]

Looking at the script itself, a couple of concerns:

1. Did you check how systemd reacts to a shutdown script returning non-zero exit code? (e.g. in first lines where you check that NUT programs are present)

The original nutshutdown script does the same:

#!/bin/sh

# This script requires both nut-server (drivers)
# and nut-client (upsmon) to be present locally
# and on mounted filesystems
[ -x "/usr/local/ups/sbin/upsmon" ] && [ -x "/usr/local/ups/sbin/upsdrvctl" ] || exit

2. in `get_ups_status()` I think you can do without the `echo`; it might however help to generalize with `upsc -l` to list devices on the localhost, and query their statuses instead of hardcoding

Well, I made a very customized script for my needs, it has to be generalized for common usage.

3. `remount,rw` during shutdown seems a bit risky. If that is done for pipe-files between `upsd` and driver, you might be better served with e.g. `export NUT_STATEPATH=/dev/shm` (maybe also `NUT_ALTPIDPATH`) to use a tmpfs (if still mounted) for that. Generally it is also not "a given" that the typical `/var/state/ups` is in `/` (often `/var` is separate).

Yes without mounting the fs as rw the driver wouldn't start because of writing problems in the /var/state/ups directory. I didn't know about the NUT_STATEPATH var, I'll try that. Thanks a lot!