Manufacturers increasingly need to make data from machines, controllers, and other shop-floor sources available to multiple applications. A scalable IIoT data pipeline can help meet this need by separating data collection from distribution, while preserving the context required for analysis. This approach is especially relevant where legacy equipment, different data formats, and limited operational technology (OT) resources must work alongside newer information technology (IT) systems.
The central design principle is to avoid tying data collection too closely to any one application. Instead, data can be gathered, standardized where possible, and moved into shared infrastructure that supports multiple consumers. That makes it easier for analytics and other systems to use production data without each one repeatedly querying equipment directly.
Why manufacturing data needs a pipeline approach
Industrial Internet of Things (IIoT) applications depend on more than access to raw device readings. Data needs enough structure and context to be interpreted consistently across equipment and applications. Standards such as MTConnect provide a controlled vocabulary, data types, and relationships that help describe manufacturing information in a common way.
In a typical MTConnect implementation, an Adapter connects to equipment or its API and translates proprietary data into MTConnect representations. An Agent adds context and provides data to clients through a standard interface. This supports interoperability, but the Agent may be deployed on or near a machine, where computing and memory resources are constrained. If many applications make large or complex requests directly, the collection layer can become difficult to scale.
There is also a practical difference between time-sensitive control and applications that can tolerate more delay. A control system may need to respond quickly to current events, while operations management or historical analysis may use data that is processed and reviewed later. A pipeline architecture can help keep these needs distinct rather than forcing every use case through the same path.
How a scalable IIoT data pipeline separates responsibilities
The proposed architecture divides the pipeline into two functions: moving data away from the shop floor and distributing it to applications at scale. This separation lets shop-floor collection remain focused while shared infrastructure manages downstream delivery and storage.
Move data from equipment with lightweight messaging
MQTT is a lightweight publish-subscribe protocol suited to transporting messages between devices and services. In this model, a shop-floor source publishes data to a broker, and interested consumers subscribe to relevant topics. This can reduce the need for multiple applications to request the same information directly from an MTConnect Agent or other equipment-facing system. MQTT also supports Quality of Service levels and connections using Transport Layer Security (TLS), where these features are required.
A broker alone does not necessarily provide the history that applications need. Messages may have limited retention, and data can be unavailable to a consumer that connects after publication. For intermittent applications or analysis that requires recent history, the architecture therefore adds a separate system for persistent, scalable distribution.
Distribute and retain streams with a data backbone
Apache Kafka provides a platform for collecting, storing, and distributing streams of data. Producers write records to topics, and consumer applications can read from those topics independently. Its append-only log allows data to remain available according to configured retention, so a disconnected or newly added consumer can retrieve a portion of the recent stream rather than relying only on messages delivered in real time.
This arrangement also separates producers from consumers: equipment-facing services can publish data without needing to know which applications will use it. As workload grows, the architecture is designed to support adding server and storage resources. The exact capacity and response times still depend on implementation and workload; evaluating those limits at the intended scale remains important.
Bringing standardized and ad-hoc data together
A useful pipeline must accommodate more than data that already follows a manufacturing standard. In the described implementation, MTConnect data from a machine tool was collected through its REST API. A Kafka Connector organized requests into tasks, retrieved intervals of data from the Agent, tracked the last collected sequence number, and placed the results in a Kafka topic.
A separate sourceāa Beckhoff PLC providing power dataāpublished through MQTT. Although this data was not MTConnect-compliant, it entered the same broader pipeline and was associated with the machineās data environment. Bringing both streams together made it possible to align and synchronize them for further analysis.
The point is not that every source must use an identical format at origin. Rather, collection and transport can provide a common route into the data architecture, while source-specific knowledge and information models help preserve meaning. OEM knowledge can be important when translating PLC tags, units, and other equipment-specific details into standardized representations.
Design considerations for IT and OT teams
Introducing middleware into a production environment requires attention to operational constraints as well as data flow. Solutions placed within OT environments may be harder to patch and maintain than those on the IT side, while time-critical applications may have requirements that influence where processing belongs. Architects should assess the needs of each use case before deciding which functions run near equipment and which can be handled by shared infrastructure.
Key considerations include:
- Data meaning and format: Use standards and equipment knowledge to make readings interpretable across systems.
- Time alignment: Synchronize timestamps when events from multiple sources need to be compared or their sequence established.
- Continuity and recovery: Plan how the pipeline handles interruptions and how consumers can access retained data.
- Traceability and security: Preserve data provenance and protect equipment and connections from unauthorized access.
- Deployment and scale: Account for legacy or embedded systems, multiple destinations, storage needs, and availability requirements.
One implementation combined Eclipse Mosquitto as an MQTT broker, Apache Kafka and Kafka Connect for collection and stream distribution, MongoDB for persistent storage, and Docker to simplify deployment. These components illustrate one way to assemble the functions; the architecture is based on separating responsibilities, not on requiring every manufacturer to select the same tools.
Practical implications for manufacturers
A scalable IIoT data pipeline can give manufacturers a more flexible foundation for using shop-floor information across applications. By moving collection away from repeated direct requests, retaining streams for later consumers, and accommodating both standardized and ad-hoc sources, the approach can support a broader analytics ecosystem while respecting existing equipment interfaces.
For engineering and operations teams, the practical next step is to map data sources, consumers, timing requirements, and OT constraints before choosing components. The architecture offers a way to connect heterogeneous data and make it available for streaming or persistent analysis, but its performance at hundreds or thousands of sources requires further evaluation in the target environment. Careful testing of workload, latency, and bandwidth is therefore essential when moving from a pipeline design to a production-scale deployment.
Access Official Technical PDF