MQTT open files keeps increasing

Mosquitto MQTT Broker
1

Hi,

I installed Mosquitto a few months ago to communicate between instances of Node-Red environments.

A few days ago the MQTT messages stopped. I rebooted my Debian 11 server on which Mosquitto is installed and it worked again. The issue happened again about 24 hours later, and so on.

I investigated and found an error that implied there were too many open files, so I added the following line to /usr/lib/systemd/system/mosquitto.service:


[Service]

LimitNOFILE=49152

then ran these…


sudo systemctl daemon-reload

sudo service mosquitto stop

sudo service mosquitto start

I ran the code below, after a restart to see the count of open connections:


sudo netstat -natp | grep ESTABLISHED.*mosquitto | wc -l

It stood at 348, which makes sense as I probably have about that many nodes in NR communicating between the Node-Red environments (that is, between the Debian 11 server and 3x Pis + the Debian server itself)

The issue happened again, this time maybe only 1 hour later. someone on the Node-Red forum suggested doing the following:


sudo service mosquitto stop

sudo rm /var/lib/mosquitto/mosquitto.db

sudo service mosquitto start

I did the above and the problem has not happened in over 24 hours. It appears to have gone away, however when I checked earlier today for the number of open files, it had increased to 3,046.

I ran this to see the list of connections


sudo netstat -ntp | grep ESTABLISHED.*mosquitto

But the results are pretty dull, just a very long list …

image
image906×675 8.94 KB

Can someone please direct me as to how I can dig deeper to find out why more and more open files are being created, or confirm if this is expected behaviour?

… I am splitting messages up to overcome the embedded media restriction for new users

2

I am running:

  • v 2.0.11 of the MQTT client on 1x Pi connecting to the Debian Server running Mosquitto

  • v1.5.71 of the MQTT client on 2x Pis connecting to the Debian Server running Mosquitto

When running this on the Debian 11 server


apt-cache policy mosquitto-clients

I get the following:

image

3

The setup on the PIs, in Node-Red is:

From the Debian Server

image

4

image
image650×536 13.7 KB

5

To the Debian Server

image

6

image
image647×556 13.7 KB

The setup in the Debian Server, in Node-Red is:

7

From the Pis

image

8

image
image652×545 13.9 KB

9

To the Pis

image

10

image
image656×575 14.1 KB

There is no security setup on any of the Node-Red environments for any of the MQTT nodes.

To be clear, the setup is working perfectly fine now and was working perfectly fine for months. The issue (MQTT message abruptly stopping) happened around the time I upgrade to v2.2.2 of Node-Red a few days ago.

I admittedly had not been checking the number of open connections, so can not comment on what it has been over time. Only what I can see now.

I suspect (hope) the setup will keep working fine now, but I would like to know if it is usual for the number of open files to increase and if not, how can I diagnose the root-cause.

Appreciate this is a lengthy post. Trying to share as much upfront as possible.

Thanks in advance for any help!

11

That took quite some time to post… I hope someone can look at it and give me a nudge in the right direction… thanks again!

I have been reading up on open files and found this…

"The too many open files error is normally because you have too many open sockets (Sockets consume file handles the same as open files).

In this case you have hit the maximum number of incoming client connections, which is probably the default 1024 limit."

a) I am wondering is some of my settings are not suitable e.g. I have set QOS to 2.
b) also, I’m thinking to upgrade the clients on the 2x Pis running the older versions to the same version, as I suspect that will be recommended… I’ll hold off until I hear something, in case it’s better to test the setup as it is

12

Hello! Thanks for the masses of details, that’s a great effort you’ve made.

The output of netstat is very useful here - what we can see is that you have many open connections from a client to the broker, and that all of the connections are (in that screenshot) from the same computer.

Each one of those ESTABLISHED lines is effectively saying there is a network connection open. If either the broker or the client were to close their side of the connection, it would no longer be listed as ESTABLISHED. Also, if each MQTT client isn’t sending regular ping messages to the broker then the broker will close the connection after the keepalive timer for that client expires (assuming that you don’t have keepalive=0, which means infinity).

In other words, it looks at first glance as those they may be “healthy” connections that haven’t been closed by the client. I haven’t seen behaviour in the broker that would be causing this itself.

Where to look - check your clients have a keepalive value set (30-60 seconds is probably a fair bet), check that you’re telling your clients to disconnect when they’ve finished. Make sure you aren’t accidentally creating new clients without realising it, maybe in a loop that never removes the client when it has finished.

Let us know how you get on.

Cheers,

Roger