MQTT is a lightweight messaging protocol widely used in IoT applications where devices need to exchange data efficiently. Its publish/subscribe architecture allows an ESP32 to send sensor readings to an MQTT broker, which distributes messages to subscribed applications without requiring the device to manage individual connections.
However, implementing reliable MQTT communication involves more than establishing a connection. Developers must consider message delivery guarantees, topic structures, connection handling, and security. This article explores these aspects and explains how to build a reliable MQTT client on ESP32 using ESP-IDF.
How MQTT Publish/Subscribe Works
MQTT uses three main components:
- Publisher: Sends messages to named topics.
- Broker: Receives messages and forwards them to matching subscribers.
- Subscriber: Subscribes to topics to receive messages.
Publishers do not need to know which clients receive their messages. This separation makes it easier to add subscribers without modifying the publisher's firmware.
MQTT topics support two wildcard characters. The plus sign (+) matches one topic level, while the hash (#) matches all remaining levels and must appear at the end of a subscription filter. Wildcards are used in subscriptions, not published topic names. Topic names are case-sensitive. Filters beginning with a wildcard do not match broker topics whose first level begins with $, such as $SYS.
An MQTT client establishes a connection using CONNECT and receives CONNACK from the broker. It then subscribes using SUBSCRIBE/SUBACK and exchanges messages through PUBLISH. PINGREQ/PINGRESP packets maintain an idle connection. The broker can close a silent connection after approximately 1.5 times the negotiated Keep Alive interval.
MQTT vs. HTTP
HTTP is suitable for on-demand queries, firmware downloads, and configuration endpoints. MQTT is better suited to frequent, lightweight updates that multiple applications need to receive.
A persistent broker connection allows MQTT messages to be distributed to several subscribers without requiring separate application-level requests from the device. However, MQTT also requires managing a broker and connection state. Actual bandwidth and battery consumption depend on network conditions and device workload.
Choosing the Right MQTT Quality of Service
MQTT supports three Quality of Service (QoS) levels:
- QoS 0 (At Most Once): Messages are published without acknowledgment. Suitable for frequent telemetry when occasional message loss is acceptable.
- QoS 1 (At Least Once): The broker acknowledges receipt through PUBACK. Retransmissions can produce duplicate messages, so subscribers must handle repeated messages safely.
- QoS 2 (Exactly Once): Uses a four-step exchange involving PUBLISH, PUBREC, PUBREL, and PUBCOMP to prevent duplicate delivery at the MQTT protocol level. It requires additional messages and state.
QoS applies separately between the publisher and broker and between the broker and subscriber. The broker cannot deliver messages at a higher QoS than the publication or subscription permits.
Even QoS 2 cannot guarantee that an application performs a physical action exactly once. Important commands should use command IDs, acknowledgments, deduplication, and device-state verification.
Retained Messages, Persistent Sessions, and Last Will
Retained Messages
A retained publication allows the broker to store the last-known value for a topic. New subscribers receive this value immediately instead of waiting for another update. This is useful for configuration and device status, although retained information can become outdated. Including timestamps helps identify stale data.
To remove a retained value, publish an empty retained message to the same topic.
Persistent Sessions
Persistent sessions preserve subscriptions and certain in-flight QoS messages across disconnections.
MQTT 3.1.1 uses Clean Session to control this behavior, while MQTT 5 separates Clean Start from Session Expiry Interval. Retained publications belong to topics rather than individual client sessions.
ESP-MQTT normally uses MQTT 3.1.1 unless MQTT 5 support is enabled and selected.
Last Will and Testament
The Last Will and Testament (LWT) feature allows the broker to publish a predefined message when a device disconnects unexpectedly.
An ESP32 can register a retained offline message during connection setup and publish a retained online message after successfully connecting. The broker publishes the will when it detects the unexpected disconnection, often after the Keep Alive timeout.
Since detection can take time, LWT is useful for device status reporting rather than safety-critical heartbeat monitoring.
ESP32 MQTT Implementation Using ESP-IDF
ESP-IDF provides the ESP-MQTT component for implementing MQTT communication on ESP32.
The implementation requires an ESP32 connected to Wi-Fi, a valid IP address, and an accessible MQTT broker.
The ESP-MQTT configuration API is available in ESP-IDF v5.x. In these versions, the MQTT dependency must be added to the project's main component CMake configuration using PRIV_REQUIRES mqtt.
Starting with ESP-IDF v6.0, ESP-MQTT is a separate component and must be added as an external dependency. Earlier ESP-IDF v4.x projects used a different, flat configuration structure.
For bench testing, an MQTT broker can be configured on a trusted, isolated local network. The broker address and port must match the actual broker configuration.
The MQTT client configuration includes the broker URI, a unique client ID, and Last Will settings. The event handler manages connection, disconnection, incoming data, and communication errors.
When the MQTT client connects successfully, it can publish its retained online status and subscribe to command topics. When using the default clean session, subscriptions should be established again after reconnection because the broker does not preserve them.
The MQTT_EVENT_DATA event provides explicit lengths for the received topic and payload. Neither pointer is guaranteed to be null-terminated, so applications should not treat the received data as ordinary C strings. Larger messages may arrive in fragments identified by current_data_offset and total_data_len. These fragments should be reassembled within a defined size limit, and the complete command should be validated before processing.
Publishing Sensor Readings
An ESP32 MQTT application can publish sensor readings as JSON payloads. For temperature monitoring, the payload can contain a temperature value in degrees Celsius.
The publishing function formats the sensor reading, checks for payload truncation, and sends the message to the configured telemetry topic.
In this implementation, the message is published using QoS 1 without retaining it. The returned message ID indicates that ESP-MQTT accepted the publication, not that a subscriber acted on it. MQTT_EVENT_PUBLISHED indicates broker acknowledgment for QoS 1 messages.
For commands that change physical device states, an application-level acknowledgment is still necessary.
Since esp_mqtt_client_publish() can block during network activity, it should not be called from time-critical paths. ESP-MQTT also provides esp_mqtt_client_enqueue(), which hands transmission to the MQTT task. A FreeRTOS queue between sampling and communication tasks can provide another way to maintain predictable timing.
Designing MQTT Topics and Payloads
MQTT topics should reflect how applications consume information rather than how firmware organizes functions. A hierarchy such as site/area/device/message-type helps organize communication.
Telemetry, reported status, commands, and acknowledgments should use separate topics so that access permissions remain narrow.
JSON telemetry payloads can include timestamps, sensor readings, and sequence numbers. Sequence numbers help receivers identify missing or repeated readings, while timestamps require a valid clock, usually synchronized through SNTP.
Units and schema versions should also be documented.
MQTT transports payload bytes without interpreting them. JSON is convenient for inspection, while CBOR and Protocol Buffers can reduce message size.
Handling Reconnection and Offline Messages
When Wi-Fi connectivity is interrupted, the application must determine how to handle pending sensor readings.
If newer readings make older ones irrelevant, they can be discarded. If every sample matters, a bounded queue in persistent storage can be used, with a defined policy for handling a full queue. The ESP-MQTT RAM outbox alone does not preserve readings across power loss.
Each device must have a unique MQTT client ID. If another device connects using the same ID, the broker disconnects the existing client. After reconnecting, verify whether the broker restored the session rather than assuming previous subscriptions remain active.
Large deployments can overload a broker if devices reconnect simultaneously. Bounded exponential backoff with random jitter helps distribute reconnection attempts when the library's retry policy does not already provide it.
Monitoring dropped messages, outbox occupancy, connection failures, and publishing delays helps identify communication problems. Network callbacks should remain short, with heavier processing delegated to application tasks.
Securing MQTT Before Deployment
For production applications, MQTT communication should use TLS and verify both the broker certificate and hostname.
In ESP-IDF, certificate verification can be configured through the broker verification certificate setting or by attaching the certificate bundle. Encryption without certificate verification does not protect against broker impersonation.
Other important security practices include:
- Give each device its own credentials or client certificate.
- Avoid sharing passwords across an entire device fleet.
- Configure broker access-control lists (ACLs) to restrict devices to the topics they require.
- Avoid broad wildcard permissions for command topics.
- Store credentials securely and provide a way to rotate or revoke them.
- Restrict broker access, log administrative changes, and rate-limit abusive clients.
TLS requires additional RAM and connection time, so its impact should be measured on the target hardware. Certificate validity checks also require a reliable clock. MQTT does not provide authorization by itself; these permissions must be configured on the broker.
Testing MQTT Communication
A successful MQTT publication is only the first step in testing an application. Before deployment, verify that subscribers receive messages on the intended topics and that repeated QoS 1 commands do not cause physical actions to execute multiple times.
The ESP32 should reconnect correctly after Wi-Fi loss and handle pending readings according to the application's requirements. Retained status messages and Last Will publication should also be tested.
Additional checks should include malformed and oversized command payloads, unique client IDs, broker ACL restrictions, and certificate verification.
Long-running tests should monitor memory usage and communication error counters to identify stability issues.
Final Thoughts
MQTT provides an efficient communication mechanism for connecting ESP32 devices with applications that consume their data. Its publish/subscribe architecture makes it useful for IoT applications involving frequent sensor updates and remote device monitoring.
However, reliable communication requires careful attention to delivery guarantees, topic design, reconnection handling, message validation, and security.
As IoT applications move from development to commercial deployment, hardware reliability becomes equally important. PCB layout, component selection, manufacturing quality, and assembly all contribute to the performance of the final device.
At PCBCool, PCB manufacturing, component sourcing, assembly, and testing support the development of reliable electronic hardware for industrial and IoT applications.
For more technical details, you can also read the complete guide: Building Reliable MQTT Communication on ESP32 with ESP-IDF