Spine: Invalid output from script using regex

[erusenger]

Hello all,

I have a non-prod instance of cacti (recently upgraded to 1.2.26 after running into the issue below) and recently changed to running SPINE (1.2.25) as the poller.


I have a couple of .py scripts that run and each returns multiple values (both are data input as script/command).
Both work fine on the php poller, but one of them gives the following error when using spine:

Code: Select all

2024-05-10 21:15:03 - POLLER: Poller[Main Poller] PID[2995] WARNING: Invalid output! MULTI DS[rxagg] Graphs[rxaggtest] Encountered [U] Expected[Rxagg:value]
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037561440320] WARNING: Device[<hostname>] polling sleeping while waiting for 1 Threads to End
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037561440320] WARNING: Device[<hostname>] polling sleeping while waiting for 1 Threads to End
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037394978560] WARNING: Invalid Response, Device[<hostname>] HT[1] DS[rxagg] Graphs[rxaggtest] SCRIPT: /usr/bin/python3 /usr/share/cacti/scripts/total_CX_throughput.py '<host_ip>', output: U
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037394978560] Device[<hostname>] DS[rxagg] Graphs[rxaggtest] ERROR: Empty result [<host_ip>]: '/usr/bin/python3 /usr/share/cacti/scripts/total_CX_throughput.py '<host_ip>''  

This only happens for one of the two scripts. The other is apparently fine.
As soon as I switch the poller to php, both scripts start acting normally.

Both scripts output correctly when run from command line:

Code: Select all

txagg:21.9 rxagg:11.12 count:5

Code: Select all

scgopr:-1570 scgopt:5

Both scripts have the same ownership.
Only issue is that spine doesn't like one of them.


The command for both of the data input methods for both scripts is essentially the same:

Code: Select all

<python_binary_path>/python3 <cacti_path>/cacti/scripts/<scriptname>.py <host>

Both scripts do very similar functions, and both of them execute in less than 3 seconds depending on host.
The only difference is that one of them uses a regex function (the non-working one) and the other does not.
I have tried several other scripts and none of them that use regex in the script seem to work, which I find odd.

I am also not 100% certain that the data-source is not the issue.

I am not sure what is going wrong here, but for the time being this is non-prod so I can mess with it at will.
Any guidance is very much appreciated.

Thank you!

[erusenger]

I should mention I am running centOS 7.9.2009

[TheWitness]

It could be that the user account running that script does not have permission. Not that Spine does not load the user profile. So, you have to include the full path to python if you call it that way.

[erusenger]

Hello, thank you for your reply!

Both data input methods have the Python3 full path in the command.
I also tried the base python binary, the explicit 3 and 3.6 binarys. Both with and without their full paths (re-creating the input method, with a new source, with a new template each time).

I would assume that running the PHP poller would also break if the path were to be broken on one.

I also tried modifying the scripts to have a different output string in case spine didn't like something in the string.
I also tried several different scripts that have the same regex function and they all were broken on spine but OK on the PHP poller.

I'll likely just write a new script that doesn't

Code: Select all

import re

and go from there.

[erusenger]

As an update:

I wrote a new script that imports re and does some other stuff, then outputs two variables in name:value pairs.
I commented out the portion where re is used, but the module is still imported.

Put this in the path/cacti/scripts dir and ran it and it worked fine.

I un-commented the regex statement and kept the variable names the same, verifying that the output is normal when running from command line, leaving all cacti configs the same, and it broke (keeping the same filename etc).
Re-commented again, tested on command line to ensure it was working, and cacti is again working with it.

Seems to be pointing more towards the RE import being the issue.

[macan]

Is it broken only with "import re" or any other import?

I don't know python .. is it posible to import with full path?

[erusenger]

Re is part of the python package so it should be native.
I am also importing sys, subprocess, and argparse. All of which are working properly.

Importing RE breaks when RE is called (so maybe it is RE that is breaking and non the import, but if there is no statement in the script that uses RE then I don't think it ever properly gets called beyond just checking that the library exists).
Having the "import re" in there without RE ever being called doesn't break anything, it is just once RE is called.

I would have also assumed that once python is called that it is no longer running under the spine process but rather spine is calling python > Python executes script w/ command > script returns string and passes string back to spine, at least that is what I would assume.
Though, that is a lot of assumptions. I'm a bit out of my depth here.

[erusenger]

So I was able to get the script to return valid values:

Code: Select all

 2024-05-22 08:10:03 - SPINE: Poller[Main Poller] PID[31575] PT[140303005861632] Device[**hostname**] HT[1] DS[CXintaggDS] Graphs[CX **hostname** - agg traffic] TT[199.09] SCRIPT: /usr/bin/python3 /usr/share/cacti/scripts/allint.py 'host.ip.v.4', output: txagg:0.00 rxagg:0.00 count:0 

but the values are all 0s. As soon as I switch to PHP it works again.
If I hard-code a value, it returns it properly, so the scrip is definitely running but it is not doing so properly.

Still baffled. I also tried making a .py script that calls the other script but that also returns nothing.

I am going to drop this and attempt a different way of doing the same thing.
With that said, if anyone knows how to aggregate all graph of a given type that a given host has, in a way that can be templated and automated and is dynamic, please let me know.

This script got SNMP indexes, parsed UP interfaces, parsed which ones were of a given type, grabbed the Tx/Rx octets, and provided a total of all interfaces. In short, it showed how much traffic a host was passing, and how many interfaces were UP/UP.

[erusenger]

Still looking for a solution to this.
Aggregate templates are a no-go as there are 400+ devices for which to create specific graphs.
Really not understanding why spine is adverse to this specific query when it is working elsewhere, and working under the PHP poller.