Communication Architectures
for Substation Automation

Dennis K. Holstein, Publisher
September 1999

A new class of software has risen to address the challenge of interoperability. Called middleware, this software is a layer between the application code and the networking code provided by the communication processor. Middleware insulates the application programmer from the raw networking code, providing an easier way to communicate.

Communication architectures

Using middleware that is appropriate for substation automation is the key to your success. There are three main types of middleware architectures:

Point-to-point architectures

Point-to-point communication is the simplest form of communication. Perhaps the most familiar example of point-to-point communication device is between Intelligent Electronic Devices (IEDs) within the substation. To use it, the initiating IED (calling IED) must know the address of the responding IED. Once the connection has been established, the two IED can have a two-way communication dialog.

However, a point-to-point connection is not very useful if one IED wants to talk to several IEDs simultaneously. It is designed to support one-to-one communications.

Client-server architectures

Client-server networks have one special “server” node that can connect simultaneously to many client nodes. Thus, client-server architecture is a many-to-one architecture.

Client-server architectures work well when all the nodes on the network need to access centralized information. Examples are substation database of configuration parameters, transaction processing between two relay IEDs, and external file servers for downloading to the substation IED.

However, if the information is being generated at multiple IEDs, client-server architectures require that all the information be sent to the server before it becomes accessible to the clients. This is inefficient. It also adds an unknown delay (and therefore indeterminism) to the system, because the receiving client does not know when new information has been added to the server.

Publish-subscribe architectures

These systems are good at distributing large quantities of time-critical information quickly, even in the presence of unreliable delivery mechanisms, and they can handle very complex data flow patterns.

When communicating through publish-subscribe middleware, the application software in data sources and data sinks are kept independent of each other. Perhaps most importantly, a publish-subscribe middleware layer handles connections, failures, and changes in the network, eliminating the need to handle exceptions. The application software simply requests the data it wants, and the middleware delivers it.

Using a publish-subscribe architecture for real-time substation applications

Publish-subscribe architectures are well suited to distributed-real-time communication. Because there is no need to request data, and because the data transfers are done directly between the publisher and the subscriber, a publish-subscribe architecture makes much more efficient use of the network resources than client-server architectures.

Because the data can be sent from the publisher to the subscriber as soon as it becomes available, publish-subscribe architectures provide low latency delivery. Moreover, they can take advantage of the new multicast capabilities of modern communication networks.

In fact, when configured properly, a publish-subscribe architecture can support all the data transfer requirements of distributed-real-time systems including:

In short, publish-subscribe is the best architecture for most substation real-time distributed designs.

However, publish-subscribe (as we have defined it) is not enough. Substation real-time systems have other needs, including:

Trade-off reliability against delivery delay

Real-time application programmers must be able to trade-off delivery reliability against delivery delay. Any communication protocol designed to provide guaranteed delivery over a less-than-reliable medium would find itself retrying failed transmissions. Of course, each retry takes time, so a guarantee of reliable delivery destroys timing determinism.

The ability to trade-off the reliability of delivery for greater determinism is crucial for multicasting GOOSE state change messages. Sending the most recent GOOSE is much more important than resending old updates that were previously dropped by the network, which will probably be out-of-date when they are delivered anyway. In this example the best policy would be to send the latest update, discarding any earlier updates.

If on the other hand, a sequence of device operation commands is required, a communication processor must receive every step in the command sequence in the proper order, even if resending a command in the sequence delays transmission of the next command. This can only be guaranteed with a reliable delivery.

Protection applications might require some intermediate policy, such as retrying for 100 ms, and then moving on. Unfortunately, current networking protocols don’t allow users to specify this policy. For example, TCP retries a dropped packet for several minutes – refusing to accept any subsequent packets until the current packet is successfully delivered. Thus, TCP cannot provide deterministic timing.

Communication protocol requirements

For example, the communication processor software cannot pause the execution of a high-priority task just because that task issues a publication request. The software must be completely event-driven, to avoid the latencies and the inefficiencies of polling. It must also be fully reentrant, because there might be many threads running at different priority levels that need simultaneous accessing to networking services.

What is needed

Non-real-time applications running over TCP/IP are easily implemented. Real-time applications running over UDP/IP require a network data delivery service that is small and fast for use in embedded real-time applications. UDP and IP are quite simple, and are reasonably fast. If properly implemented, the network data delivery service middleware should add only minimal overhead to the underlying network communication stack. And, it should be much more efficient than TCP, DCOM or CORBA.

In addition to simplifying the system configuration, this distributed approach provides for graceful degradation. The loss of any particular node (or any part of the network) does not stop data transmission between unaffected nodes in the rest of the system.

The good news

EPRI’s working groups on the Utility Communication Architecture – UCA 2.0, initially developed such a model. UCA 2.0’s Reporting Model has been submitted to the IEEE and IEC for consideration in the development of several standards. IEEE projects include PSRC’s development of Standard Communication Test Scenarios (PC37.115), and Substations Committee’s development of Requirements and Build- to specification for Substation Integrated Protection, Control, and Data Acquisition (P1525). IEC Technical Committee 57 projects include IEC 61850 for a Substation Automation System.