Mosquitto is slow when used in localhost mode

Mosquitto MQTT Broker Issues
1

Hello,

We are doing speed and load tests using the broker Mosquitto. The tests consists of continuously sending messages both ways from a client A to a client B using the broker.

We notice a speed issue when we use the broker in localhost with qos=0. There is a huge slowdown in messages speed.

When the broker is used in remote mode (Broker on a server, clients on another machine), we reach around 12000 messages/s.
When the broker is used in local mode (Broker and clients on localhost), we reach only around 400 messages/s.

We have no speed issue with qos=1 or qos=2 in localhost.
Technically, we hope qos=0 being faster.

We tried modifying the following parameters in the moquitto.conf file without success :

max_queued_messages 0
queue_qos0_messages true
set_tcp_nodelay true

The machine where the broker is installed runs Red Hat Enterprise Linux 8.6 with 64 bits architecture.

We have the following limitations configuration on the machine :

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63164
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63164
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

We tried with Mosquitto 2.0.15 and also latest Mosuiqtto 2.0.18 version.

Are there any other parameters or limitations that should be checked? Or something else in the parameters or maybe the API?

Any help would be appreciated.

Thanks you.

2

Hello,

We are still having this issue : Mosquitto is slow when used in localhost mode.

Anybody would have any idea?

Thank you in advance.

3

Hi,

Sorry for not having spotted this sooner. Could you give a hint as to how you are doing the testing so I can try and reproduce it? 400 messages/s is painfully slow, I’m really surprised and wonder at the cause.

I’m interested in exactly reproducing it, so your full config (or if it produces the same result with no config that would be fine), plus which tools you are using to do the testing and how you are running them.

I have done some testing here using two Python scripts for publish and subscribing, with the subscriber printing out how many messages it receives per second and the publisher publishing in a loop with a 100us pause.

Publisher:

#!/usr/bin/python3

import paho.mqtt.client as mqtt
import time

mqttc = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
mqttc.connect("localhost", 1883, 60)

mqttc.loop_start()

while True:
    time.sleep(0.0001)
    mqttc.publish("topic", "payload")

Subscriber:

#!/usr/bin/python3

import paho.mqtt.client as mqtt
import time

count = 0

def on_connect(mqttc, obj, flags, reason_code, properties):
    mqttc.subscribe("#")

def on_message(mqttc, obj, msg):
    global count
    count += 1

mqttc = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.connect("localhost", 1883, 60)
mqttc.subscribe("#")

mqttc.loop_start()

last_count = 0
while True:
    time.sleep(1)
    new_count = count
    print(f"{new_count - last_count}")
    last_count = new_count

The msgs/s results I got were:

5591
5746
5683
5682
5725
5675
5770
5774
5735
5765
5689
5630
5695
5490

Regards,

Roger

4

Hi Roger,

Thank you for your reply.

Regarding our tests :

  - We have a client A which sends a message to the client B. The client B, after receiving the message from client A, replies to the client A. We repeat this operation continuously to mesure the number of transactions per second.
  - To do so, we use a topic "client_A" for the client A to send the messages, the client B subscibes to this topic "client_A". 
    Also, the client B sends the messages to its own topic "client_B", the client A subscribes to this topic "client_B". 
	Let's call it a channel (2 topics per channel).
  - To test several configurations, in our tests, we launched 1 to N client_A instances, with always only one client B. 
    We also used 1 to 3 channels between the client_B and each instance of client A. 
	Ex : 1 client A, 1 client B, 1 channel - 1 client A, 1 client B, 2 channels, 2 client A, 1 client B, 3 channels, and so on (With always only one client B).
  
    We gradually inscread the numer of client A and channels and noticed the issue for the following configurations : 
	 
	    - 1  client A, 1 client B, 3 channels
		- 2  client A, 1 client B, 3 channels
		- 5  client A, 1 client B, 2 channels
		- 10 client A, 1 client B, 1 channels
  
  - Please note when the client B receives a message, the receive callback is triggered, and a thread is created to reply to the instance of client_A which sent the message. 
    The threads allow several messages to be processed and replied simultaneously. 
	We had the same problem even without the usage of threads.

When we tested these different configurations, we had no speed issue with the other qos settings (1 and 2). However when we set qos to 0, as I mentioned before, we have an important speed slowdown when we use localhost mode. When the broker is used in remote mode (on another machine), there is no problem, even with qos set to 0.

Our test uses Mosquitto C API.

Regarding the configuration, we tried modifying the following parameters in the moquitto.conf file without success :

  max_queued_messages 0
  queue_qos0_messages true
  set_tcp_nodelay true

We also tried to modify a value from the API with mosquitto_int_option function, to set MOSQ_OPT_SEND_MAXIMUM and MOSQ_OPT_RECEIVE_MAXIMUM to 65535. Seems it doesn’t help either.

I cannot enclose the conf file to this message but if necessary I can still send it to you (I guess it will not be necessary since we modified only the 3 parameters above frm the default configuration file).

I hope this information can help you to reproduce the issue.

Btw, we thought about something : Is there any possibility of bug in sockets usage inside the API since it seems to work fine in remote mode but not in localhost mode?

Thank you.