Mosquitto API : Unable to connect more than 340 clients to the broker from the same binary file

Mosquitto MQTT Broker Issues
1

Hello,

I am trying to create and connect 1024 Mosquitto clients using Mosquitto API libmosquitto inside a c file (For a test).

I wrote a basic test c file, where I have a 1024 iterations loop which, for each itearation, creates a Mosquitto client calling mosquitto_new function, then connects it calling mosquitto_connect and mosquitto_loop_start functions.

From the 341th created element, I get a “Lookup error” (inside the connect callback) after calling mosquitto_connect function.

I am using a 64 bits Red Hat Enterprise Linux 8.6.

I tried the following solutions :

  1) I checked the system limitations, and tried to increase the open files limitations to 16384  : 

     core file size          (blocks, -c) unlimited
     data seg size           (kbytes, -d) unlimited
     scheduling priority             (-e) 0
     file size               (blocks, -f) unlimited
     pending signals                 (-i) 30912
     max locked memory       (kbytes, -l) 64
     max memory size         (kbytes, -m) unlimited
     open files                      (-n) 16384
     pipe size            (512 bytes, -p) 8
     POSIX message queues     (bytes, -q) 819200
     real-time priority              (-r) 0
     stack size              (kbytes, -s) 8192
     cpu time               (seconds, -t) unlimited
     max user processes              (-u) 30912
     virtual memory          (kbytes, -v) unlimited
     file locks                      (-x) unlimited

     When I increase the open files limitations, I can connect all my 1024 elements but I have still issues from the 341th, seems there is no dialog with the broker, the clients from 341th get disconnected by the broker after 60 seconds timeout. 
     Howerver no problem for the first 340 first elements.

  2) I tried to use 2 loops with 1024 iteations each, on for creation, another for connection and loop start.

     In that case, I can create all my 1024 elements but then I cannot even connect the first one.

  3) I tried to create 2 binary files, which create and connect 256 clients each. This works, the server accepts the 512 connections. But for my test I need to create and connect the clients from the same binary file.

I compiled both broker and API from sources. The broker, and the binary file which creates and connects clients both run on the same local machine.
The broker doesn’t seem to have limitations. Seems there is another unknown limitation somewhere then prevents my binary file to create and connect more than 340 clients.

Is there another system limitation, or maybe a limitation in the api itself that could prevent more than 340 clients to be generated and connected from the same binary file?

Thank you in advance for your help.

2

Hi Romain,
I think you are running into an implementation limit inside the libmosquitto. Internally in the mosquitto_loop inside the libmosquitto is using the select/pselect system calls to work on the active sockets. And the select/pselect system calls are internally using a fixed size bitmap to mark the active file descriptors. On typical linux systems this bitmap is limited to 1024. This means select/pselect cannot handle any file descriptor with a handle higher than 1024. In effect you cannot handle this many connection using select/pselect.
Additionally the mosquitto_loop_start function will start an additional thread for each connection. So you might run into limits on the number of threads as well.
The broker shares a lot of code with the lib, but is internally using epoll (or poll, if no epoll available). Decision is made by config dring compile time of the broker. select is used by the broker only in case poll/epoll is not avaialable on the system (very rarely today) or explicitly deactivated.
You maybe able to get around the problems by implementing your own event loop and use the external event loop functions mosquitto_loop_read, mosquitto_loop_write,…
Unfortunately I don’t have any complete example for that. Found a small example for a single connection based on poll here, but cannot garantee it’s correct and complete. You would have to work yourself into use of poll. And But this way you should be able to handle multiple connection with a thread and single poll fd pair. Similar to the way the broker is handling several thousands of connetcions. You may implement multiple independent threads, each with it’s own poll fd struct pair (e.g. each thread handling 100 connections).
Or you may event implement something based on epoll.
Would be great, if you would be willing to share part of the implementation as an example into the eclipse mosquitto example (e.g. without your specific callback handling the messages).

3

Hi Norbert,

Sorry for not having replied earlier.

Thank you a lot for all this information.

Of course, in case I implement this model, I guess it could be possible to share part of the implementation as an example.

However, I tried several models, maybe I will use a more classic and simpler model with less clients in the binary file.

I will let you know in case I implement something for this issue.

Thank you.