BLOG
The MCC: A Decentralised and Consortium-Aware Technology for Simplifying Distributed Architectures
A comprehensive guide to understanding and implementing the MCC (Mesh Communication Component) in your distributed system architecture, including its basic services, security features, DHT technology, as well as service discovery and tunneling functionality.
Published: 03.03.2023
Author: Maksim Sukhotin, Network Developer at Staex GmbH
Introduction
The MCC is a generic component that simplifies the creation and deployment of distributed architectures.
In order to achieve this, it provides two basic services:
In the MESH ecosystem, MCC would have the role of the communication enabler. Allowing edge nodes to be deployed anywhere while always keeping connectivity to the core services (such as the API). Nodes can now be anywhere in the network, even behind NATs, and still be accessible. There is no need to deploy everything on the same node. And since the service discovery allows more than one provider per service, it is very easy to add load balancing and redundancy in services.
The Zen of MCC
MCC aspires to be a truly decentralised, consortium-aware technology. This means that several organisations can share a single network, and still be independent from each other. Each organisation would control their own nodes and be able to choose what to share and what not to share with other organisations. This allows each member of the consortium to be independent and to freely choose what level of cooperation and involvement it wants to have.
In order to achieve this, a number of obstacles have to be solved. First of all, it is important to find a technology that would not generate very large messages, in order to avoid MTU issues. Also we need a technology that would be fast and easy to implement. For these reasons we have chosen Elliptic Curve Cryptography (ECC) as our cryptography solution. We will therefore use ECC to sign messages and derive encryption keys using elliptic curve Diffie-Hellman (ECDH).
In ECC there is a number of curves that can be used for cryptographic purposes, and we have chosen to use the Curve25519 due to its multiple advantages (it is patent-free, it is very fast, it is designed for ECDH, there are no known backdoors or cryptographic flaws on it, and keys are just 256bit long while still providing high security).
Each MCC node has an ID which is also a public key and a point on the elliptic curve. All messages in the network are encrypted. We can represent an encryption key as F(x, y), where x is the sender's private key, and y is the receiver's public key. This way a key is unique for any pair of nodes.
DHT
MCC is based on a Distributed Hash Table (DHT). A DHT is simply a key-value store that instead of storing all keys in the memory of a single machine, it uses a deterministic algorithm to spread it across a number of machines. This way, keys can be always retrieved, no matter where in the network they were stored. We have chosen Kademlia as the underlying DHT technology.
Kademlia, however, defines a 1-to-1 key/value lookup, as specified by its paper, and it would be desirable for us to have a 1-to-many discovery, so that many providers can be registered for the same service, allowing for fault-tolerance and scalability. In order to achieve this we have implemented an extension to the Kademlia protocol.
The MCC exposes a REST API for the Apps and the Service Providers to use. MCC nodes talk to each other using a custom Protobuf-based UDP protocol.
Author: Maksim Sukhotin, Network Developer at Staex GmbH
Introduction
The MCC is a generic component that simplifies the creation and deployment of distributed architectures.
In order to achieve this, it provides two basic services:
- Service discovery: Service providers can register their services in the network and then clients can query them from any node of the network. The service discovery allows many providers to register for any given service, thus allowing redundancy and load-balancing between services.
- Tunneling: In order to simplify connectivity to services, no matter if they are behind NAT/firewalls and/or other network configurations, MCC provides a tunnel to access services. This way clients can send their requests to the MCC network and they will be routed to its destination, independently of where the source and destination are.
In the MESH ecosystem, MCC would have the role of the communication enabler. Allowing edge nodes to be deployed anywhere while always keeping connectivity to the core services (such as the API). Nodes can now be anywhere in the network, even behind NATs, and still be accessible. There is no need to deploy everything on the same node. And since the service discovery allows more than one provider per service, it is very easy to add load balancing and redundancy in services.
The Zen of MCC
- Ready like a warrior: (AKA batteries included), MCC should be easy to deploy and use, no complex configuration required, it just works, always.
- Transparent like water: MCC should try to be as transparent as possible, requiring existing applications with no or almost no modification to be compatible with MCC.
- Flexible like bamboo: Distributed systems are complex and always evolving, although it is impossible to adapt to every single use case (present or future), MCC should strive to be as flexible as possible.
- Simple like a monk: MCC should be as simple as possible, and we must resist the temptation to add more features than needed.
MCC aspires to be a truly decentralised, consortium-aware technology. This means that several organisations can share a single network, and still be independent from each other. Each organisation would control their own nodes and be able to choose what to share and what not to share with other organisations. This allows each member of the consortium to be independent and to freely choose what level of cooperation and involvement it wants to have.
In order to achieve this, a number of obstacles have to be solved. First of all, it is important to find a technology that would not generate very large messages, in order to avoid MTU issues. Also we need a technology that would be fast and easy to implement. For these reasons we have chosen Elliptic Curve Cryptography (ECC) as our cryptography solution. We will therefore use ECC to sign messages and derive encryption keys using elliptic curve Diffie-Hellman (ECDH).
In ECC there is a number of curves that can be used for cryptographic purposes, and we have chosen to use the Curve25519 due to its multiple advantages (it is patent-free, it is very fast, it is designed for ECDH, there are no known backdoors or cryptographic flaws on it, and keys are just 256bit long while still providing high security).
Each MCC node has an ID which is also a public key and a point on the elliptic curve. All messages in the network are encrypted. We can represent an encryption key as F(x, y), where x is the sender's private key, and y is the receiver's public key. This way a key is unique for any pair of nodes.
DHT
MCC is based on a Distributed Hash Table (DHT). A DHT is simply a key-value store that instead of storing all keys in the memory of a single machine, it uses a deterministic algorithm to spread it across a number of machines. This way, keys can be always retrieved, no matter where in the network they were stored. We have chosen Kademlia as the underlying DHT technology.
Kademlia, however, defines a 1-to-1 key/value lookup, as specified by its paper, and it would be desirable for us to have a 1-to-many discovery, so that many providers can be registered for the same service, allowing for fault-tolerance and scalability. In order to achieve this we have implemented an extension to the Kademlia protocol.
The MCC exposes a REST API for the Apps and the Service Providers to use. MCC nodes talk to each other using a custom Protobuf-based UDP protocol.
For these reasons, DHT acts as a network and storage layer for the MCC, therefore the MCC stack would look like this:
Service discovery
The service discovery service, as mentioned above, runs on top of the DHT. Whenever a new provider registers a service, the node will go to the DHT and write this service in the corresponding keys.
An endpoint identifier is assigned per each service instance. It can be thought of as the address of a service inside the MCC network. A group of endpoint IDs can be associated with a human readable service name. In a typical use case, a node in the network requests available endpoints by a service name, and then connects to one of them using a tunnel.
Tunneling
Service providers and service consumers can communicate with each other via the MCC network. Each of them connects to the closest MCC node, and sends its traffic through the virtual network interface. The network encrypts, signs, and sends packets to the destination.
In order to fully utilise the bandwidth, MCC uses Google BBR congestion control algorithm to control the traffic flow. It performs greatly for network connections with packet losses.
NAT traversal
NAT is a technique that allows multiple devices inside a network to talk to the outside world using only one public IP address. Without NAT, every single device inside the network would need a publicly routable IP address. However, if one wants to contact a node behind a NAT, it might be a tricky task. It becomes even more complicated if both nodes are behind different NATs.
The NAT traversal is an integral part of the MCC protocol. The NAT traversal requires no additional central servers and it scales as the network scales. The network can detect the NAT status of each node and adjust to it, and even use nodes as a relay for the communication between two NATed nodes. This way it is ensured that there is always a way to connect two nodes.
Also, MCC uses the UDP hole punching technique to connect NATed nodes directly. In order to do that, the network looks up for a relay node that can communicate with the receiver. Then, the sender sends a message to the relay node, and the relay node passes a message to the receiver. After that, the receiver tries to reply directly to the sender bypassing the relay node. In case of success, these two nodes can continue further communication directly. However, this might not work due to the network setup, and in that case nodes will continue communicating via the relay.
Summary
MCC is a great tool for simplifying and rebuilding your network in a distributed manner. Features like a distributed storage, service discovery and secure tunnels make it usable for many use cases.
The service discovery service, as mentioned above, runs on top of the DHT. Whenever a new provider registers a service, the node will go to the DHT and write this service in the corresponding keys.
An endpoint identifier is assigned per each service instance. It can be thought of as the address of a service inside the MCC network. A group of endpoint IDs can be associated with a human readable service name. In a typical use case, a node in the network requests available endpoints by a service name, and then connects to one of them using a tunnel.
Tunneling
Service providers and service consumers can communicate with each other via the MCC network. Each of them connects to the closest MCC node, and sends its traffic through the virtual network interface. The network encrypts, signs, and sends packets to the destination.
In order to fully utilise the bandwidth, MCC uses Google BBR congestion control algorithm to control the traffic flow. It performs greatly for network connections with packet losses.
NAT traversal
NAT is a technique that allows multiple devices inside a network to talk to the outside world using only one public IP address. Without NAT, every single device inside the network would need a publicly routable IP address. However, if one wants to contact a node behind a NAT, it might be a tricky task. It becomes even more complicated if both nodes are behind different NATs.
The NAT traversal is an integral part of the MCC protocol. The NAT traversal requires no additional central servers and it scales as the network scales. The network can detect the NAT status of each node and adjust to it, and even use nodes as a relay for the communication between two NATed nodes. This way it is ensured that there is always a way to connect two nodes.
Also, MCC uses the UDP hole punching technique to connect NATed nodes directly. In order to do that, the network looks up for a relay node that can communicate with the receiver. Then, the sender sends a message to the relay node, and the relay node passes a message to the receiver. After that, the receiver tries to reply directly to the sender bypassing the relay node. In case of success, these two nodes can continue further communication directly. However, this might not work due to the network setup, and in that case nodes will continue communicating via the relay.
Summary
MCC is a great tool for simplifying and rebuilding your network in a distributed manner. Features like a distributed storage, service discovery and secure tunnels make it usable for many use cases.
Are you interested in seeing what MCC is capable of?
Get notified about our releases
Enter your email to stay up-to-date about Staex developments.
