Hi, i have mosquitto up and running on my Odroid based Home Assistant. I receive messages from my shelly power monitor so that works all fine. I can view those with the mqtt explorer etc.
One thing i am wondering about is retention of the data and cleaning / deletion - so how that actually works.
As far as i understand the MqTT broker is like a queue manager.
It receives messages and stores those under topics and other clients can get that info by subscription.
What i do not see in the configuration of MqTT on Home Assistant is how to control / manage the retention of these messages ? How does that actually work ? Can you set a max number of events after which it will roll over or LIFO delete … or oage … or … etc … Or does it go on until my emmc card (storage) is full ? And is it persistent ? etc . where can i read how this works ?
The retain message feature does not persist data. It is only there as a “last known state”.
This is helpful if a sensor only publishes very rarely and a subscriber does not want to wait for a new message. With retain set to true, a subscriber gets the last message on that topic after the sub.
To configure this you don´t need to configure the broker, but the “configuration” of your publisher. The publisher decides, if a message will be retained or not.
The retain will always only store one message per topic. So there is no queueing involved.
To help you better, it would be interesting to know, what you want to achieve.
Maybe the QoS level is of interest for you, but this is also not persisting data, but making sure a subscriber is getting all missed messages after a reconnect.
Hi Tizian thanks. I just want to understand how it works on the broker side.
There are a large amount of events coming in there an i want to understand how i can control and manage retention / disposal / claeanup of the data on the Broker side to prevent it from flooding the system storage.
So will it store and keep messages as long as there is storage space ? Or can i set it to for example max number of messages after which it will start with removal of the oldest etc etc … so what are the options we have here
so by default retain is using memory space, but this can be changed to use disc space with the configuration parameter “persistence true”. This will then write into a file called mosquitto.db
This will store all QoS and retain messages.
I´m not sure about your use case, but for “retain” to kill your hardware, it will only do this, if you have many topics. Because for each topic, only one message is stored.
If you are using an emmc I don´t think you want to turn on persistence, because I think these cards are pretty limited in terms of write cycles, before corrupting.
What you can do is limiting the max usage of memory for the broker.
This will then affect the broker as a whole and is not only for retain messages.
I think as long as you know how many topics you have, and know the size of your messages, you can just do the math, if your memory will hold up or not.
Then in context of retain, you should be fine.
100 topics will not be such a big deal. The message rate does not come into play in terms of the retain.
It will always max store 100 messages.
dont get that last point … every topic has one value every 10 seconds … and it is not only showing the last value… so i guess i have 6x60x24 messages per topic per day ?
Yes, but the retain only keeps the last message on each topic.
It will overwrite the older messages.
Retain does not persist data. It is only there for connecting sub Clients to have a message on connect.
it is not intended for any kind of data persistence.
So the math is:
100 x message size
But I feel like, retain is not what you want to achieve. I´m not sure yet what, but for persistence as in kind of a data base, MQTT does not do that. It is not a message queue in a sense, that data is persisted, only forwarded.
Hi yeah i am used to for example mqseries which is a queue manager where you have queues that can contain large amount of messages untill they are purged etc… and subscribers listen to those queues and can get stuff out of there… but those quesues are kind of persistent … they keep stacking up data
So mqtt is just a broker in that sense. What i want to achive is that i persistently store the messages that are coming from my mqtt enabled energy monitoring devices (and now send to my mqtt on my home asssistant n2+) so that i can do analysis on them.
Gues that means i have to forward those to a database (like influxdb) to get that persistency as a minimum otherwise the messages are lost after system restart
MQTT is a message transport protocol and does not provide out of the box storing.
And as you said, adding Influx or sth similar as storage would achieve what you need.
