There are several popular patterns for creating a replicated set of AMPS instances. One popular pattern is a “cascading mesh”, or a set of instances that receives publishes in one set of instances and then distributes those messages to other sets of instances. This blog post describes a common approach to creating a mesh of this type, explains how the configuration works, and discusses the tradeoffs involved in this approach.

Cascading Mesh Topology

The sketch below shows the topology of the cascading mesh.

The first tier of the mesh has two instances that receive publishes from applications. These instances are part of the “IncomingMessages” group, and replicate both to each other and to the next tier in the mesh.

The second tier of the mesh provides services to subscribers. This tier has two instances that are part of the “LiveProdSubscriptions” group. These instances replicate to each other and also the next tier in the mesh.

The third tier of the mesh provides a non-production environment for application development and testing. This tier has two instances that are part of the “DevUAT” group. These instances replicate to each other and to the last tier in the mesh.

The final tier in the mesh provides an environment for archival and audit. This tier has two instances that are part of the “ArchiveAudit” group. These instances replicate to each other, but do not replicate anywhere else.

This topology is designed to meet the following requirements:

• All messages are available on all tiers.

• The mesh can survive the loss of a server on any tier (and, in fact, could survive the loss of one server on each tier) and still deliver messages.

• Different types of usage are isolated to different instances. Only approved, production-ready applications will be allowed to access the “LiveProdSubscriptions” instances.

An archive of the configuration for these instances is available for download.

AMPS Replication

For a basic introduction to AMPS replication, see From Zero to Fault-Tolerance Hero and the AMPS User Guide section on High Availability.

For the purposes of creating a cascading mesh, the following aspects of AMPS replication are most important:

• AMPS replication is always point-to-point, from an originating instance to a receiving instance.

• By default, AMPS replication does not replicate messages that arrive over a replication connection. An instance in a mesh uses the PassThrough directive to further replicate messages that arrive at the instance via replication. This isn’t necessary if there are only two instances in use, but is typically required for a multi-instance installation to replicate correctly.

  PassThrough in a given Destination specifies that when a message is received via replication from an instance in a matching Group, it is eligible to be replicated by this instance to this Destination. If no PassThrough is present in the Destination configuration, replicated messages will not be further replicated by this instance. Notice that the PassThrough setting applies to the Group name of the instance the message is received from, and not any other instance that the message may have been replicated through.

• An outgoing message stream from an originating instance is specified by adding a Replication/Destination to the configuration of the outgoing instance, one per outgoing message stream.

• AMPS duplicate detection uses the name of the publisher and the sequence number assigned by the publisher. If the same message is received more than once (for example, over different replication paths), only the first copy received is written to the transaction log. Any other copy will be discarded. Notice that, if there are multiple paths to an instance, there is no guarantee as to which path the first message will take. This is especially important in a failover situation or if there is network congestion, but can also be true even when the network, hardware, and AMPS instances are all performing as expected.

• AMPS replication validation, by default, ensures that both sides of a replication connection provide a complete replica of the messages being replicated. Since this topology intentionally does not replicate messages from the lower-priority environments to the higher priority environments, the configuration will need to relax some of the validation rules.

The diagram above shows one possible arrangement of replication connections for this mesh. To make this application more resilient, each instance in a given tier will be configured to fail over replication to either of the instances in the next tier.

Determining the Outgoing Destinations

To be sure that every message reaches every instance in the mesh, each instance in the mesh replicates to the other instance at the same tier.

Each instance also insures that messages are replicated to the next tier. Since the instances in the next tier will also replicate to each other, the configuration uses a single Destination with the addresses of both the instances in the next tier as failover equivalents. Once a message reaches an instance in the next tier, that instance will be sure that the message is replicated to the other server in the tier.

Determining the PassThrough Configuration

For a mesh like this, where the intent is for each message to reach every instance in the mesh, it’s important that each instance pass through messages from every upstream instance.

Since the intent is to pass through every message, the easiest way to specify PassThrough is to provide a regular expression that matches any group. AMPS configurations, by convention, typically use the regular expression .* (any quantity of any character) to match anything.

If it were necessary to be more explicit, though, for every instance, the configuration would pass through the group names for every instance from which it could receive an incoming message.

For example, as shown in the following diagram, when a message is initially published to Incoming-A, the Live-B instance could receive the first copy of that message from the Incoming-B instance in the IncomingMessages group, from the Incoming-A instance in the IncomingMessages group, or from the Live-A instance in the LiveProdSubscriptions group. A PassThrough configuration for an outgoing Destination from this instance must specify at least IncomingMessages|LiveProdSubscriptions for the Live-B instance to replicate the message further. For convenience, an instance would typically specify .* to match any incoming Group.

Likewise, the Dev-A instance could receive a replicated message from the Live-Ainstance in the LiveProdSubscriptions group, from the Live-B instance in the LiveProdSubscriptions group, or from the Dev-B instance in the DevUAT group. A PassThrough configuration for an outgoing Destination from this instance should specify at least LiveProdSubscriptions|DevUAT. Again, for convenience, an instance would typically specify .* to match any incoming Group.

Sync or Async Acknowledgement?

One other choice that needs to be made for each replication destination is the type of acknowledgement to be used for that destination. The acknowledgement type controls when the instance of AMPS considers a message to be safely persisted and acknowledges that persistence.

With sync acknowledgement, an instance of AMPS will wait for the replication destination to acknowledge that a message has been persisted before considering the message to be safely persisted and acknowledging the message as persisted (to either a publisher or an upstream replication instance).

With async acknowledgement, an instance of AMPS will consider the message to be safely persisted when it is persisted to the local transaction log.

The acknowledgement type doesn’t affect how quickly AMPS replicates messages. However, because async acknowledgement may acknowledge a message before the message has been received by a downstream instance, it is important to be sure that a message source (either a publisher or an upstream instance) does not fail over between two AMPS instances that replicate over an async connection.

For this mesh, this means that we can use async replication between tiers, but we need to use sync replication within a tier. For the IncomingMessages tier, a publisher may fail over between Incoming-A and Incoming-B, so those instances must use sync acknowledgement to replicate to each other. In the other tiers, a replication connection may fail over between an -A instance and a -B instance, so those connections must use sync acknowledgement.

Since a connection will not fail over from one tier to another, and since every instance fully replicates the message stream (thanks to the PassThrough configuration discussed in the previous section), it’s reasonable to use async acknowledgement between the tiers. This can reduce the storage needs for the applications that publish to the IncomingMessages instances, since messages could potentially be acknowledged to those publishers before the messages are replicated to all of the downstream instances.

Validation Rules

As mentioned earlier, by default AMPS replication validation tries to ensure that messages published to a replicated topic on any instance will be delivered to all other replicated instances. In many cases, this is exactly what replication is intended to do.

This replicated mesh, though, does not replicate messages from a lower-priority environment to a higher-priority environment. Because of this, the configuration will need to relax some of the validation checks.

Between tiers (for example, replication between Live-A and either Dev-A or Dev-B), messages are only replicated in one direction. The replicate validation check ensures that topics that are replicated to a given instance are also replicated from a given instance. To allow messages to be replicated in only one direction, the configuration excludes the replicate validation check.

Within a tier (for example, replication between Live-A and Live-B), every message will be fully replicated, so there is no need to disable the replicate validation check within a tier. However, the cascade validation check ensures that every destination that messages are replicated to enforces the same replication checks to its own destinations. This means that, within a tier, the configurations need to exclude the cascade validation check.

Because the configurations exclude the cascade validation check within a tier, this means that the tier that replicates to each tier must also exclude the cascade validation check.

So, for each tier, we need to relax replication validation as follows:

• Within a tier, the configurations will exclude the cascade validation check.

• Between tiers, the configurations will exclude the replicate and cascade validation checks.

Putting The Plan Together

To summarize the steps we go through to put the plan together, we have the following steps for creating a cascading mesh:

• Sketch out the mesh:

  • Each instance should replicate to every other instance in the same tier.

  • Each instance should have an outgoing connection that can fail over to any of the instances on the next tier.

• For every instance, ensure that the passthrough configuration is either .*, or includes the Group name for every incoming replication connection.

• If there are any replication connections where it is not possible for either a publisher or a replication connection to fail over from one side of the connection to the other side of the connection (in this example, connections between tiers), consider whether using async acknowledgement might reduce the resource needs for publishers by providing acknowledgement more quickly.

For example, the Replication section of the Incoming-A instance would be as follows in the example configuration:

<Replication><Destination><Name>Incoming-B</Name><Group>IncomingMessages</Group><PassThrough>.*</PassThrough><Topic><Name>.*</Name><MessageType>json</MessageType><ExcludeValidation>cascade</ExcludeValidation></Topic><SyncType>sync</SyncType><Transport><InetAddr>localhost:4002</InetAddr><!-- Incoming-B --><Type>amps-replication</Type></Transport></Destination><Destination><Group>LiveProdSubscriptions</Group><PassThrough>.*</PassThrough><Topic><Name>.*</Name><MessageType>json</MessageType><ExcludeValidation>replicate,cascade</ExcludeValidation></Topic><SyncType>async</SyncType><Transport><InetAddr>localhost:4101</InetAddr><!-- Live-A --><InetAddr>localhost:4102</InetAddr><!-- Live-B --><Type>amps-replication</Type></Transport></Destination></Replication>

The destination configuration for Live-A follows a similar pattern. Notice that since the LiveProdSubscriptions group is not intended for new messages to be published, the instance does not specify replication back to any instance in the IncomingMessages group.

The Replication for Live-A looks like this:

<Replication><Destination><Group>LiveProdSubscriptions</Group><PassThrough>.*</PassThrough><Topic><Name>.*</Name><MessageType>json</MessageType><ExcludeValidation>cascade</ExcludeValidation></Topic><SyncType>sync</SyncType><Transport><InetAddr>localhost:4102</InetAddr><!-- Live-B --><Type>amps-replication</Type></Transport></Destination><Destination><Group>DevUAT</Group><PassThrough>.*</PassThrough><Topic><Name>.*</Name><MessageType>json</MessageType><ExcludeValidation>replicate,cascade</ExcludeValidation></Topic><SyncType>async</SyncType><Transport><InetAddr>localhost:4201</InetAddr><!-- Dev-A --><InetAddr>localhost:4202</InetAddr><!-- Dev-B --><Type>amps-replication</Type></Transport></Destination></Replication>

Try it Yourself

An archive of the configuration for these instances is available for download. These configuration files are set up to run on a single system, but can easily be adapated to a multi-server configuration (by adding the host name for each instance to the replication InetAddr).

AMPS bookmark subscriptions provide a way for applications to resume subscriptions in the event of a disconnection or failure of either the application or server. With a bookmark subscription, the application side manages the correct point to resume the subscription by setting a BookmarkStore on the client, and then discarding messages from the store once the application is done processing them. The clients offer both a memory-based bookmark store (which does not persist when the application restarts) and a file-based bookmark store (which is persisted on the file system).

Those options work well for many AMPS deployments. In some deployments, though, no filesystem is available, but the application still needs to resume a subscription when it is restarted. This post explains how current versions of the AMPS client can use a topic in the AMPS State-of-the-World to store the point at which the application should be resumed. The post assumes a good working knowledge of resumable subscriptions (covered in detail in the AMPS User Guide and the AMPS Java Developer Guide), and also assumes some familiarity with the implementations of the AMPS clients.

In older versions of the AMPS clients, you could do this by creating your own BookmarkStore (typically by wrapping a MemoryBookmarkStore to do the heavy lifting), as described on the 60East blog at No Filesystem? No Problem! Keeping State in AMPS and Keeping State in AMPS, Rebooted blog posts.

Current versions of the AMPS client libraries make this much simpler. The AMPS client libraries now include a new interface, RecoveryPointAdapter, that allows you to implement only the storage for a bookmark store. Even better, the clients include a SOWRecoveryPointAdapter to store bookmark state in AMPS without having to develop an adapter at all.

In this blog post, we’ll cover what a recovery point is, what a recovery point adapter is, and how to use the built-in recovery point adapter to restore bookmark subscription state from a Topic in the State-of-the-World. If all you need to do is store recovery points in AMPS, feel free to skip ahead to Bookmark Subscription Recovery from a SOW.

What’s a Recovery Point?

A recovery point is a bookmark, or set of bookmarks, that the application provides to AMPS when a subscription is restarted so that the AMPS server can find the correct point in the local transaction log to resume that subscription. In the AMPS client libraries, a recovery point is represented by a combination of the subscription identifier and a string that includes the current stable recovery point for that application.

A helpful way to think about the contents of the recovery point string is “one or more bookmarks that provide a useful place to resume a subscription”. This can include one or more of:

• The last bookmark discarded by the application
• For a set of publishers, the last bookmark discarded by the application for each of those publishers
• The last bookmark AMPS has provided as being fully persisted
• A timestamp indicating the last time the store produced a recovery point

Depending on the failover situation, one (or all) of these may be the correct point to recover from. For example, if the application goes for a long stretch of time without receiving messages (and all previously received messages have been discarded), it may be most helpful to start from the last bookmark that AMPS has provided as being fully persisted rather than the last message that the application discarded. If the bookmark store knows that there are no messages for this subscription between those two points, this could save a large amount of work on the server side to replay journals with no messages that match the subscription.

Likewise, some applications may be offline longer than the retention period for the transaction log. In those cases, none of the bookmarks seen by the application (discarded or not) will be present in the AMPS instance, and it would be most helpful to restart at the timestamp, which will be before the beginning of the transaction log. If no timestamp were provided, no bookmark in the recovery point would be present, and AMPS would restart the subscription from the end of the transaction log rather than the beginning of the transaction log.

The MemoryBookmarkStore includes logic for managing these situations and determining the best recovery point to use for a given subscription. One of the advantages of a RecoveryPointAdapter is that your application doesn’t need to figure out the best recovery point for the subscription. The MemoryBookmarkStore provides the best current recovery point to your adapter when the recovery point changes.

What’s a Recovery Point Adapter?

A recovery point adapter has two responsibilities to the MemoryBookmarkStore that uses the adapter:

• Persist recovery points provided by the Store. The adapter itself chooses how and where the recovery points are persisted. When the MemoryBookmarkStore updates the recovery point for a subscription – for example, when a message is discarded or a persisted ack arrives from AMPS – the store calls the adapter with the current recovery point.

• Produce the current set of recovery points when requested to by the Store. This happens when a subscription is created or restored.

The exact interface that a recovery point adapter uses to provide the recovery points for a set of subscriptions varies depending on the programming language, but it is intended to be easy to implement. For example, the Java language recovery point adapter provides a set of recovery points using the standard java.util.Iterable<RecoveryPoint> and java.util.Iterator<RecoveryPoint> interfaces.

When the MemoryBookmarkStore needs to update the recovery point for a subscription, the store calls the update method of the adapter with the recovery point that needs to be persisted.

Notice that the adapter doesn’t need to concern itself with deciding what the correct recovery point for a subscription should be. All of that logic is in the Store itself: the adapter just needs to store the recovery points provided by the Store and provide those recovery points to the Store when necessary.

Bookmark Subscription Recovery from a SOW

The SOWRecoveryPointAdapter in the AMPS clients uses a SOW topic in an AMPS instance for storage and retrieval of recovery points.

Recovering from a SOW with a recovery point adapter is a matter of:

• Configuring AMPS with a SOW topic for recovery points

• Creating an AMPS client to use for storing and retrieving recovery points (separate from the client that will be used for the application)

• Installing the recovery point adapter when the Store is constructed

To be able to store recovery points in a SOW topic, the AMPS configuration needs to define the topic. The SOW needs to be able to store the recovery point for each distinct subscription for each distinct client. By default, the SOW recovery point adapter produces a JSON document that includes a clientName field with the name of the client and a subId field with the identifier for the subscription (the names and message format are customizable, see the API documentation for the interface). This means that the configuration for the SOW topic should be along the lines of:

<SOW><!-- other topics, views, etc. here --><Topic><Name>/ADMIN/bookmark_store</Name><MessageType>json</MessageType><Key>/clientName</Key><Key>/subId</Key><!-- Storage/persistence configuration here.        In most cases, this topic should be        persisted to a file, but that is not        a requirement.  --><FileName>./sow/%n.sow</FileName></Topic></SOW>

This creates a topic named /ADMIN/bookmark_store in the instance.

To use this topic to save and restore recovery points, you create an AMPS client that connects to the instance that stores the topic, and then use that client to create a SOWRecoveryPointAdapter for the Store to use. This client must be a different client than the one used by the application: it is reserved for recovery point storage.

It’s often convenient to wrap this creation in a separate method. For example, the following Java method sets up an HAClient to use a SOWRecoveryPointAdapter that will use the instance at tcp://bookmark-storage-host:9007 to store recovery state:

// Call setSowBackedBookmarkStore before calling connectAndLogon on the// HAClient that will use the SOWBookmarkStore.publicvoidsetSowBackedBookmarkStore(HAClientclient_)throwsAMPSException{// Construct a new client (to help ensure naming uniqueness,// this name is based on the name of the client the adapter will// be saving state for).HAClientrecoveryPointStorage=newHAClient(String.format("recoveryPointStorage-%s",client_.getName()));// Substitute this with a production-ready ServerChooser// and an appropriate ReconnectDelayStrategy. Notice that// the bookmark store does not need to be on the same// instance as the client that works with actual data.DefaultServerChoosersc=newDefaultServerChooser();sc.add("tcp://bookmark-storage-host:9007/amps/json");recoveryPointStorage.setServerChooser(sc);// Connect the recovery point storage clientrecoveryPointStorage.connectAndLogon();// Create a recovery point adapter. SOWRecoveryPointAdapteradapter=newSOWRecoveryPointAdapter(recoveryPointStorage,client_.getName(),// name of the client this adapter is trackingtrue,// close recovery point client when Adapter is closed// Adjust these options as necessarytrue,// include last update timestamp (journal removal protection)false// do not throw exceptions);// Set the bookmark store using the adapter.// In this case, initialize the store to expect up to// 5 simultaneous subscriptions (as a reasonable default)client_.setBookmarkStore(newMemoryBookmarkStore(5,adapter));}

That’s all that’s needed to enable the SOW recovery point adapter for an HAClient.

Reducing Bandwidth for Updates

The SOWRecoveryPointAdapter will save the recovery point to the State-of-the-World each time a new recovery point is created. For an application that is actively processing messages, this could produce a relatively high volume of updates to the State-of-the-World topic.

To reduce the bandwidth required to maintain the recovery point, the AMPS clients include a ConflatingRecoveryPointAdapter that wraps another adapter and passes along recovery points to that adapter at the specified interval. The interval consists of a number of updates and a time period: when the time period expires, or the number of updates is reached, the ConflatingRecoveryPointAdapter calls the wrapped adapter with the recovery point to be persisted.

For example, to persist the recovery point to the AMPS server every 100 updates or 250ms, the example above could be modified to set the bookmark store this way:

// The adapter object is a SOWRecoveryPointAdapter// constructed as above.ConflatingRecoveryPointAdapterconflater=newConflatingRecoveryPointAdapter(adapter,100,// Number of updates250,// Update interval in milliseconds50// Check thresholds every 50ms);// Provide the conflating adapter to the MemoryBookmarkStore// instead of providing the SOWRecoveryPointAdapter directly.client_.setBookmarkStore(newMemoryBookmarkStore(5,conflater));

The ConflatingRecoveryPointAdapter starts its own background thread to manage timeouts and updates. When the adapter is closed, it writes all updates to the underlying adapter before closing that adapter.

Setting the threshold for updates is a matter of how much tolerance the application has for receiving messages that have already been processed in the event that the application fails before an updated recovery point is delivered to AMPS.

Other Considerations

To the AMPS server, the topic that stores the recovery points is the same as any other topic in the State of the World. The topic can be replicated to other instances of AMPS to improve reliability and allow the client that saves the recovery point to fail over. As mentioned earlier, the topic can be included in the same instance that has application data, or can be stored in a completely different instance of AMPS.

Like a MemoryBookmarkStore itself, an application that uses a SOWRecoveryPointAdapter must be able to handle some overlap in subscriptions (duplicate messages) in the event that the application fails (or the connection fails over) with messages that have not been discarded.