Very high times on large amounts of data

Mosquitto MQTT Broker
1

Hello,
I am programming firstly in C# in .NET 8 with the mqttnet library, let me explain my goal, I want to use MQTT to update a large amount of data in a graphical interface called HMI in the electrical industry, how many data I mean? so far I am testing with 10k topics, for the moment I am only sending the date in the payload property (DateTime.Now.ToString(“yyyy-MM-dd HH:mm:ss.fff”)).
The structure of the topics is the following test/device/param, for now device and param are numerical data, for example test/100/29, the topics start in test/1/1 and end in test/100/100.
Reading the documentation, I understood that MQTT is very fast to process large amount of data, but I am not seeing times that convince me, since between the topical test/1/1 and the test/1/1 of the next round, I am finding that the difference is more than 2 minutes.
I get this time by scheduling everything linear, now, after adding tasks, where the tasks in C# are supposed to work as different threads, the times between each pass is reduced to between 1 minute and 30 seconds, but for me it is not enough, since as they are systems that should be real time, the most that is allowed is 1 or 2 seconds between each update, and how it is important information, of course I am using QoS 2.
So, I decided to write it and if you can help me to clear the doubts that if MQTT can help me with my problem, if the problem is the programming language or some configuration that I may be missing?
The settings I have enabled are:

  • max_connections -1
  • message_size_limit 268435455
  • persistence true
  • persistence_location C:\mosquitto\data

Thanks for reading and for the help.

2

Hello,

If I understand correctly, you’re publishing small messages to 10000 different topics using QoS 2 and timing how long it takes?

I’ve written a quick bit of Python to replicate that behaviour. This sends a message after it connects, then after each message is successfully published it sends another one, all the way from test/1/1 to test/100/100, via test/1/2 etc. It then prints the time taken. For me this was around 0.89 seconds when running locally - so no network speed consideration.

#!/usr/bin/python3

import paho.mqtt.client as mqtt
import time

def do_publish(obj):
    if obj['param'] == 100:
        if obj['device'] == 100:
            obj['stop'] = time.time()
            mqttc.disconnect()
            return
        else:
            obj['device'] += 1
        obj['param'] = 1
    else:
        obj['param'] += 1

    mqttc.publish(f"test/{obj['device']}/{obj['param']}", "a"*20, qos=2)

def on_connect(mqttc, obj, flags, rc):
    do_publish(obj)

def on_publish(mqttc, obj, mid):
    do_publish(obj)

data = {
    'device': 1,
    'param': 0
}
mqttc = mqtt.Client(userdata=data)
mqttc.on_publish = on_publish
mqttc.on_connect = on_connect
mqttc.connect("localhost", 1883, 60)

data['start'] = time.time()

mqttc.loop_forever()

print(f"{data['stop']-data['start']}")

I’ve also got a variant where my program sends the messages to the Python library as quickly as it can, so multiple messages will be in flight at once. This gives me a result of 0.53 seconds for all 10000 messages.

There are some caveats - I’m doing my testing on Linux and performance on Windows isn’t quite as good - but I’m talking on the order of a second or so difference at the upper end, nowhere near what you are seeing. Also, I’ve recently discovered that the Paho Python client doesn’t limit its outgoing messages as well as it ought to, and this can cause the broker to kick it out. I’ve only seen the problem on Windows. My first example program should work perfectly though.

#!/usr/bin/python3

import paho.mqtt.client as mqtt
import time

def do_publish(obj):
    if obj['param'] == 100:
        if obj['device'] == 100:
            return
        else:
            obj['device'] += 1
        obj['param'] = 1
    else:
        obj['param'] += 1

    mqttc.publish(f"test/{obj['device']}/{obj['param']}", "a"*20, qos=2)

def on_publish(mqttc, obj, mid):
    if mid == 10000:
        obj['stop'] = time.time()
        mqttc.disconnect()

data = {
    'device': 1,
    'param': 0
}
mqttc = mqtt.Client(userdata=data)
mqttc.on_publish = on_publish
mqttc.connect("localhost", 1883, 60)

data['start'] = time.time()
data['stop'] = None

mqttc.loop_start()

while data['device'] != 100 or data['param'] != 100:
    do_publish(data)

while data['stop'] is None:
    time.sleep(0.1)

print(f"{data['stop']-data['start']}")

Please could you try testing out these versions yourself? If you’re get similar values to me, then it’s definitely a client problem. If you still get the same sort of numbers you were getting before, I’d be interested to hear more to figure out why.

Regards,

Roger

3

Hello,

I appreciate your reply. I used your code on my Windows PC and you are right, the times are a bit longer compared to the ones you tell me are on Linux. Currently we work strictly with C# and in Windows environments, developing in .NET 8 to make it cross-platform, but with the mqttnet library I can’t get the times to decrease, the most I have achieved by removing tasks and disabling debugging, is to reduce it on average to 40 seconds.
I do not know if specifically it is the Windows by itself, or there is some extra configuration that has to be implemented in the broker or can be improved in something using it through a docker?

Thank you.

4

I’ve done some more reading around the mqttnet client, and it seems that there are two points that are relevant. First, it is not thread safe, so if you are using any thread components it is not guaranteed to work correctly. Second, it does not support any form of flow control for sending messages and will always try and send as many messages as it has. This is non-compliant behaviour that is explicitly not allowed in MQTT v5.0, and is typically not allowed in MQTT v3.1.1 either. This is something that needs fixing on the library level unfortunately.

5

Generally the solution is to throw disks at a problem, with or without CPUs/memory dedicated to those disks to split the problem. Distributed …refers to the incredible amount of spacebar counter structured and unstructured information that humans and machines generate—petabytes every day, according to PwC. The correct answer is Big data. Extremely large data sets that could, when analyzed, reveal useful information and patterns are called Big-Data. It refers to data that is so large, fast or complex that it is difficult or impossible to process, using traditional methods.

6

Thank you all for your answers, I will take them into account.

Thank you again.