Phantom Host records with no name/ip showing in Cacti db

[KevinNoall]

I have a bizarre issue where I have some entries with no name/ip/information. These originally were valid host records, but ov er time they seem to have gotten messed up. I've tried deleting the records, but they continue to re-appear. I've tried deleting all the graphs/etc and then deleting the record, but that does not seem to help. This is on the current 1.2.26 release multi-site install with remote pollers but I've not been able to find a pattern to it.
Unfortunately I do not know what these devices originally were and I'm just trying to get rid of them alltogether.
Any advice would be appreciated.

[TheWitness]

Is your Cacti internet facing? Delete the records and then download and install the audit plugin. You'll be able to trace things then.

[KevinNoall]

No, it is not exposed to the internet, it is an internal only connection. I've installed the Audit plugin to see what shows up.

[TheWitness]

Keep us posted.

[KevinNoall]

Of course, after installing the audit plugin this issue has stopped happening. So nothing to report at the moment.

[KevinNoall]

And no sooner than I posted that nothing had shown up, and I got a blank entry. Audit plugin does not show any rogue creations or anything at all, just a new blank entry. This ties in with log entries complaining of missing datasource IDs as well, and when i check the database those entries are tied to the blank host record.

[TheWitness]

Look for CLI script calls and host.php posts. It could also be automation. Are you running automation? Review the Cacti log for when it happened.

[KevinNoall]

This is still intermittent, but I do not see anything referencing scripts, or hosts.php in the log file. I'm on Low log level, do I need to turn that up?

[cigamit]

Ya, I am wondering if this has something to do with the automation also. In Console >> Automation >> Networks, do you have anything enabled in there?

[KevinNoall]

After deleting the record earlier this morning it has not shown up in the UI. It still persists in the host table in the db though, with the deleted flag on.
Is this typical behavior to have the record persist in this state? How long before it is actually deleted?
I see several other entries with the deleted flag set to ON, and their last updated shows current timestamp.
+-----+---------+----------+-----------+---------+---------+---------------------+
| id | deleted | disabled | poller_id | site_id | site_id | last_updated |
+-----+---------+----------+-----------+---------+---------+---------------------+
| 19 | on | | 11 | 14 | 14 | 2024-03-26 10:52:14 |
| 30 | on | | 11 | 1 | 1 | 2024-03-26 10:52:07 |
| 59 | on | | 11 | 1 | 1 | 2024-03-26 10:52:16 |
| 156 | on | | 11 | 1 | 1 | 2024-03-26 10:52:17 |
| 166 | on | on | 11 | 1 | 1 | 2024-03-26 10:52:13 |
| 202 | on | | 11 | 14 | 14 | 2024-03-26 10:52:18 |
+-----+---------+----------+-----------+---------+---------+---------------------+

Code: Select all

set @myid = 1465; SELECT * FROM data_local where id = @myid; SELECT * FROM data_template_data WHERE local_data_id = @myid; SELECT local_data_id,poller_id,host_id,hostname,last_updated FROM poller_item where local_data_id = @myid;

Query OK, 0 rows affected (0.000 sec)

Empty set (0.000 sec)

Empty set (0.000 sec)

+---------------+-----------+---------+----------+---------------------+
| local_data_id | poller_id | host_id | hostname | last_updated |
+---------------+-----------+---------+----------+---------------------+
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
| 1465 | 1 | 59 | | 2024-03-26 09:54:15 |
+---------------+-----------+---------+----------+---------------------+

I also noted that the poller_id entry does not match, in the local_data_id table it's 1, but in host above its 11.

I also have seen that as I delete graphs, the respective data sources are also 'deleted' via the gui, but those DS then are added to the log...
POLLER: Poller[Main Poller] PID[28288] WARNING: Poller Output Table not Empty. Issues: 248, DS[Missing Datasource 1466, Missing Datasource 2341, Missing Datasource 7569, Missing Datasource 7568, Missing Datasource 1465, Missing Datasource 4855, Missing Datasource 825, Missing Datasource 826, Missing Datasource 5426, Missing Datasource 4856, Missing Datasource 7567, Missing Datasource 4356], Additional Issues Remain. Only showing first 20 he log entry for 'Missing Data Sources' which is now up to 244 missing.

The missing DS seems to be increasing with any graph that is deleted, regardless of visible host or not. It's almost like things are not getting deleted completely?

[KevinNoall]

Apologies, I don't use Automation and I went in and verified that there is nothing in there.

[TheWitness]

Better run the repair_database.php script in the cli directory.

[KevinNoall]

Should I run that on each remote poller as well?

[TheWitness]

After you run it on the main poller, do a full sync.