rate of change / customizeable parsers

[holmanb]

I have a question regarding intended scope and extensibility of plot.

Rate of Change

the example script for plotting tx/rx counters is but one example of this

Plotting counters of various types is a common activity that I see myself using plot for. However, it currently doesn't provide the ability to handle rate of change - the burden of calculating rate of change presently depends on some external program that usually does something like the monitor() function in the example script.

Since this seems (at least to me) like a common use case, it would be nice to add an option such as:

-r plot rate of change

which maybe does something like:

(val - old_val) /  sleep_time

to get a unit of "counters per second"

Generic Parsing

Presently plot expects the input value that it receives (stdout or file) to be a number represented as a string that can be converted to a double via strtod() (input.c:39).

Many counters exposed to userspace by the linux kernel are exported as a file that is not simply a file with a number in it, but has either 1. multiple counters or 2. surrounding text. Many files exposed via /proc/ are examples of this.

It would be really nice if plot supported a generic parsing functionality for extracting numbers from text. What I have in mind is:

-p regex:parse-string|<custom parser name>

where one could pass in PCRE regex to specify an extraction or optionally a name for a parser in C that they manually write

[holmanb]

The following is what I had in mind for a generic parser implementation:

disclaimer I haven't tested or attempted any of this, but I'd be happy to attempt an intial implementation if this would be a welcome addition to plot

this could possibly be instrumented by adding something like:

struct parser {
    char * name;
    int (*parse_function)(char*, char*);
};

then add struct parser to input in plot.h

struct input {
	char buf[INBUF];
	FILE *src;
        struct parser;
	size_t rem;
	size_t size;
};

add something to the options parsing code to look up (via the name field in struct parser) the parser from an array of function pointers and assign the parser for each input based on user args (default to a parser that doesn't do anything).

and then in input.c:read_numbers() put a temporary buffer into fread() and run in->parser->parse_function() after fread() to put the parsed output into &in->buf[in-rem]

basically the default parser (which does nothing), and the regex parser could be implemented, but additional parsers could be added so that users can add custom parsers if they want to

parser.h

int noop(char *, char *);
int regex(char *, char *);
int my_custom_parser(char *, char *);

parser.c

// default parser doesn't parse
int noop(char *in, char *out){
     out = in;
     return 0;
}

...
other parsers not included

[annacrombie]

Rate of Change

I think this is a great idea. Definitely within the scope of plot, and it shouldn't be difficult to implement.

Generic Parsing

Overall, I really like your proposal. After reviewing input.c I think that it should probably be mostly rewritten if we add this feature. The code is pretty brittle as it is now.

I think that linking to a regex library is out of scope, but I understand its use. I'm wondering what you had in mind for the mechanism that enabled user extensibility? Would it be suckless style where you just modify the source, or is there a more user-friendly option?

[holmanb]

Excellent, thanks for the feedback.

For my purposes, modifying the source (suckless style) to add custom parsers would be perfectly fine. I don't require regex support. Additionally if anyone really does want that, this interface would allow them to add it if they need it (which keeps linking regex or other bloat out of plot's codebase).

What I was implying in my previous comment is that users could define their own parser function definition in parser.c and a function prototype in parser.h which could be looked up (from a global array of struct parsers perhaps) for a given input file (-i) based on user request for that parser (-p parser_name).

This could be implemented a few different ways of course, but the snippets I shared above were an attempt at describing an implementation that would be minimally invasive to the existing code. It sounds like you are willing to rewrite parts of input.c in favor of robustness, so there are likely better designs to satisfy this request.

[annacrombie]

modifying the source (suckless style) to add custom parsers would be perfectly fine

great, I think that is the best option.

After thinking about it for a bit, I realized that most "parsers" could be implemented with a grep/sed pipeline. You could already do this easily for one input, but it gets messy fast.

Just for a test, I tried to make a plot of my cpu usage based on /proc/stat. The simplest I could come up with was this:

while true; do
        awk 'NR==1 {v=$2+$3+$4+$7+$8+$9+$10+$11; print v/(v+$5+$6)}' /proc/stat
        sleep 0.1
done | plot -f -

It plots a graph of my overall cpu non-idle ratio, which combined with the -r option, would be a useful graph of cpu usage.

I feel like rewriting this snippet in C as a "parser" would not be very useful though, since there probably aren't very many other files (if any) I would want to handle this way. I was wondering if you had any concrete examples of a common pattern that you would want a built-in parser for.

Another idea, though i'm not sure it is a good one, is to just accept shell command lines as arguments and run them internally with popen(3). E.g. something like

 plot -f -p "awk 'NR==1 {v=$2+$3+$4+$7+$8+$9+$10+$11; print v/(v+$5+$6)}'" /proc/stat

Obviously there would be more to it than that, but it seems very flexible, albeit not very performant.

[annacrombie]

Another fun one:

while true; do 
        awk '/MemFree/ {memf=$2}
             /MemTotal/ {memt=$2}
             END {print (memt-memf)*100/memt}' \
                /proc/meminfo
        sleep 0.1
done | plot -f -cr -

[holmanb]

As you've demonstrated, this could be implemented today by filtering w/unix cli utilities and sending output to stdin.

I have done multi-input testing by creating multiple pipes (or files), filtering values via sed/awk/grep, and specifying those pipes as input options via -i. It isn't very elegant, and its pretty heavyweight for what really needs to be accomplished. Hence this discussion :)

Any parsers that I imagine writing in C would likely be based on seek() and/or strtok() and would likely be pretty lightwight as a result. Unfortunately I don't really have any use case that would be very reusable.

I would lean away from the popen() option, since with that you'll be stuck either maintaining a long running process per parser, or you have to popen() once per fread(). Both options are messy and the second option is a bit resource intensive.

[annacrombie]

I think I agree with you. I feel like if we redesign the input system well enough this feature (and perhaps others) will basically fall into place. I probably won't have time work on this for a bit, so go ahead if you want.

[holmanb]

I have some ideas for improving the input system. I should hopefully have something in a couple of days for review.

[annacrombie]

Sorry it took awhile to merge your PR. I didn't realize that it was only a cleanup, and not a full implementation of these features. I guess that means this stays open for now

[AdrianDC]

The recent changes for this MR broke the average feature quiet heavily.
Bounds are evaluated at giant values, and the points count is not limited either as it was.

seq 1 100 | plot -a 2
seq 1 100 | plot -a 10 -b 0:100

# seq 1 100 | plot -a 10 -b 0:100
     100.00 ┤                                                                                
      95.65 ┤                                                                 ╭╮             
      91.30 ┤                                                                 ││             
      86.96 ┤                                                                ╭╯│             
      82.61 ┤                                                                │ │             
      78.26 ┤                                                                │ │             
      73.91 ┤                                                               ╭╯ │             
      69.57 ┤                                                               │  │             
      65.22 ┤                                                              ╭╯  │             
      60.87 ┤                                                              │   │             
      56.52 ┤    ╭╮                                                        │   │╭╮           
      52.17 ┤    ││                                                        │   │││           
      47.83 ┤    ││                                                        │   │││           
      43.48 ┤   ╭╯│                                                        │   ╰╯│           
      39.13 ┤   │ │                                                        │     │           
      34.78 ┤  ╭╯ │                                                        │     │           
      30.43 ┤  │  │                                                        │     │           
      26.09 ┤ ╭╯  │                                                        │     │           
      21.74 ┤ │   │                                                        │     │           
      17.39 ┤╭╯   │                                                        │     │           
      13.04 ┤│    │                                                        │     │           
       8.70 ┤│    │                                                        │     │           
       4.35 ┼╯    │                                                        │     │           
       0.00 ┤     ╰───────╮╭───────────────────────────────────────────────╯     ╰───────╮╭──

[annacrombie]

Thanks for noticing, I'll take a look

[annacrombie]

Ok, so I have a wip fix in #18. After looking at the code, I decided to overhaul it, partially in preparation for the rate of change feature in this issue, but also because I wanted to add more data processing features like simple moving averages etc.