Hello JMH,
If you are using the QoS2 level for your messaging, then the broker will keep all the messages in memory, which should be delivered to the subscriber, since it is currently not connected.
This is why the memory usage will keep increasing over time.
If you want to use this level of QoS it is important to know the hardware limits your broker has and adapt accordingly.
In the Mosquitto Pro we have a feature, which allows to store messages not only in memory, but also in disc. This helps with memory capacity problems.