Topics

Design proposal for Segregated Sub-Zones

Mark Oldfield
 

Hi everyone,


As a follow up to Richard's email 'An approach to short-term privacy/autonomy with a path to global interoperability' we would like to share Kat's initial design with you so you have more background on the proposed approach for segregated sub-zones. This email is also going out to the corda-dev@groups.io mailing list.


This is work in progress so we cannot guarantee that this will be delivered or delivered exactly as described in this design. The specific aspects of the design that will be made available on the Corda Network are subject to the policy decisions of that network.


The majority of this document should be relevant and readable by anyone but please note there are some details that are implementation specific and address the Corda Network.


If you have more detailed comments/questions please get back to us and Kat will reply to you early in the New Year.


Regards


Mark



Segregated Sub-Zones

Overview

There are a number of common requirements raised by customers when presented with the option of joining the Corda network

  • The ability to hive off members of the business network into a private enclave.
  • The ability to cloak the members of the business network.
  • The ability to operate "exclusive" notary services.
    • That is where the notary is not whitelisted globally and is operated to the standard deemed acceptable by the BNO, not the Zone operator and members (as per those in the global whitelist)
  • The ability to operate "private" notary service.
    • That is where the notary can exist in the global whitelist but is restricted to notarisation of specific State types
  • The ability to operate non whitelisted, "less universally trusted", notaries.
  • The ability to operate their own Compatibility Zone.

Some of these positions are orthogonal to the design and goals of Corda, which are:

  • A compatibility zone must operate at a level of "maximal compatibility", all nodes must trust all elements of the network. This makes it impossible for parties to run their own notaries unless they are also willing to prove they are operated to a standard consistent with a public global network. This is because the Zone operator should refuse to 'whitelist' notaries that could lower the overall integrity of the network's 'uniqueness' guarantees for transactions.
  • Whilst self-deployed private networks would allow users to "roll their own" Compatibility Zone, this prevents nodes joining the wider network as merging of networks is extremely difficult once you factor in the fact these disparate zones would be controlled by different trust roots. Whilst BNO's may have no issue with the users of their service being locked onto a private platform, in the long run this may harm those users as they have no access to the opportunities provided by a global network.
    • A core tenet of both Corda and the "Corda Network's" design is that users will gain disproportionate value from being able to combine states from different applications in the future in ways that cannot be predicted today. Therefore, we place great emphasis on ensuring this future on-network interoperability is never sacrificed, even though this is not a top priority for users today.
  • Privacy of membership of a zone is not part of the core design. Joining a zone means having your membership known to all current and future members.
    • We anticipate a small number of sub-zones, all of which are very large. Knowledge of an identities membership of one of these sub-zones, just as for the zone as a whole, shouldn't be a sensitive fact.
      • Concerns around the creation of actually private sub-zones are discussed later.
    • However, there is nothing in this design that precludes the creation and operation of many thousands of sub-zones. Indeed, any tooling built will be designed to cater for such scale.

Segregated sub-zones operated as part of a compatibility zone seek to address the issues surrounding the masking of business network membership (including membership of a meta-network overseen by some BNG) and the desire to operate a bespoke notary services as part of a business network.

This is done by allowing multiple network map services to be run within a Compatibility Zone under the control of the zone operator whilst deferring elements of their configuration to the entity requesting the services existence. Those elements would be, in general, the means of the operation and whitelisting of a number of exclusive notary services. The zone still operates a single Doorman service to maintain uniqueness of identity across the entire compatibility zone whilst allowing for sub-zones to exist that operate their own notaries without the requirement for universal trust of the notarisation service. Membership is inherently masked, nodes will not see the members of any zone except the one they are a member of. Finally, the trust root is shared ensuring networks are mergeable in the future without introducing an asymmetry in identity lookup and that also allows for exclusive notaries to be operated within a BNOs network.

NOTE: A Glossary of terms can be found at the end of this document.

Aims of this document

This design document outlines a mechanism to achieve some of the user objectives stated above without compromising the underlying design of the platform or unnecessarily preventing people from subsequently being able to 'merge' into Corda Network or another widely-adopted zone. The key to understanding what follows is to observe the following:

  • The question of whether two nodes can, in principle, transact and are, therefore, members of the same zone is determined by whether they share the same trust root and the same set of network parameters.
  • And one of these parameters is the set of trusted notaries.
  • However, Corda's design allows for, and Corda 4 will ship with support for, evolution of network parameters over time, which will be implemented by having every transaction commit to the version of the network parameters that prevailed at its time of notarisation.
  • This opens up the possibility of nodes that share the same network parameters today but which had different network parameters in the past.
  • This design proposal goes a step further and observes that this also means it is possible for different groups of nodes today to have different network parameters but to converge in the future to the same set.
  • Therefore, the loose intuition the reader should have when reviewing this document is the idea of disjoint groups of nodes sharing a common root of trust but conforming to different network parameters. And since, as a matter of technical detail, network parameters are published by a network map, this observation has the implication that it would be possible for a zone operator to operate multiple network maps (each publishing different network parameters) under a common trust root.
  • In so doing, this would allow disjoint groups of nodes to be visible in segregated network maps, with their own parameters (and hence trusted notaries) whilst preserving the option to migrate (en masse, as it will turn out) to a different network map (perhaps one denoted as the "global" network map) in the future with full compatibility assurances.

Background

Current state of the Network Services Architecture

To operate a compatibility zone, the following elements are deployed (R3 Network Services 0.2.1 and earlier)

Note, this diagram does conflate logical services with more abstract concepts in places for clarity.

To Summarise

  • The Network Map, Doorman, and Revocation service are independently configured, long running services
    • Specifically excluding the configuration of entities such as a Norman
      • A server running both the Network Map service and Doorman service where the two entities use the same database and schema.
    • Additionally excluding operating these entities in local signing mode
      • id est where there is no HSM being used to store the keys and certs that define the zones PKI and where there is thus no signing service to interact with.
  • These services communicate with one another through shared access of a single SQL data source
  • The signing services are isolated from outside interference by only initiating communication with their configured source of truth.
    • In the case of the Network Map and CSRs this is the Network Map and Doorman databases
    • In the case of the CRL, this is the revocation service.
    • It is impossible to communicate directly with the signers, they can't be "made" to sign something on demand, rather they pull data from a single pathway and put it back. This minimises their contact points with the outside and thus decreases any opportunity to compromise them.
    • These services are executed as one-shot operations the repeated execution of which is handled by individually configured cron jobs.
    • Both the CSR and NetMap are driven by lookups into the SQL database
    • The CRL Signer however initiates a socket connection with the CRL service
      • It is a known "feature" of the ENM software that this really needs changing into an encrypted ssl socket connection. Discussion of this is out of scope for this design document but should be corrected as part fo the general 1.0 feature work.

Currently, on registration of a signed NodeInfo the Network Map checks that node is registered with the Doorman service by inspecting the cert_signing_request table directly, it is this feature that has tied the two services together.

Private Network Maps

Private network maps as defined within the Zone Services toolchain are a solution to those BNOs who wish for the members (customers) of that business network to be cloaked, yet they are happy in general to use the notaries of a global compatibility zone. The requirement for this feature came about when the number of business networks on the public network is small (or one), making association obvious to anyone looking. By initially cloaking their existence on the public network, business networks can wait until sufficient numbers exist such that correlation between node and network is impossible.

What's wrong with Private Network Maps

The feature is cumbersome to interact with due to the lack of tooling and requires large amounts of manual intervention by the operators of a zone to operate. It also requires a multi-phase out of band mechanism by which membership can be obtained; adding a node to a Private Network Map is not a first class operation for the zone.

Additionally, it suffers from a flaw that creates an asymmetry in the identity look up mechanism. Nodes within a private network can find those on the public network whilst the inverse is not true.

Finally the issue exists that, without removing the requirement for notaries be whitelisted by the zone operator, BNO operated exclusive notaries are impossible.

Design Goals

  • Separate the network map and doorman into discrete, independent, services.
  • Take over the management of the Network Parameters
    • Split them to allow for deferred ownership of portions of them that can be applied to a particular segregated sub-zone.
      • For the purposes of supporting segregated sub-zones this effectively means:
        • delegating the listing of which notaries are whitelisted. It is intended that some mechanism is established between the zone operator and BNO for acquisition of the notary NodeInfo's which can then be added to the ENM software suite.
        • delegation of the minimum platform version
    • Facilitate their generation and signing for any given sub-zone as needed by the operators of a compatibility zone.
  • Support the deployment of multiple network map services within a single Compatibility Zone.
    • This makes the definition of a Compatibility Zone the zone operators trust root and identity service (Doorman).

      It seems to be manageable this must be a requirement/constraint places upon the system. Whilst it would be possible to serve completely unique parameters to every segregated sub zone the burden of management on the operator of the Zone would be large. This operation is greatly simplified if it is assumed they only need manage one set of general parameters.

    • A network map service should be deployable either as a co-located service on a single JVM or configured as a standalone binary
  • New tooling to support the operational side of running multiple network maps.
    • Centralised General Network Parameter management.
    • Collating per-segregated network parameter elements (Notary lists).
    • Building per-segregated network Network Parameters and deploying them.
    • Management of the configurations of all segregated networks.
    • Management of the signing services required to sign all network parameters.

Non-goals (out of scope)

  • Complete rationalisation of the esoteric, heterogeneous, Network Services architecture.
  • Discussion of productisation of the Network Services suite of tools (the Enterprise Network Manager)
  • How the Network map services should be presented to the internet (Behind NGINX etc).
  • Customisable Doorman workflow
    • For a BNO they may wish to customise their users interaction with the Doorman. Whilst this is a feature intended for addition at some point doing so as part of this work is considered out of scope.

Note: This doesn't imply these aren't important or required or valuable to debate, only that it isn't appropriate to do so in the context of the discussion and design of Segregated Sub-Zones

Requirements

Constraints

These are essentially internal requirements that constrain any proposed solutions to align with the delivery and success of the Corda project.

  • No node (including notaries) can ever exist within more than one Network Map.
  • Segregated sub-zones should be completely distinct entities from one another, there must exist no asymmetry of identification.
  • Segregated sub-zones must use the same Trust Root as the main, public, part of the Compatibility Zone.
    • Corda / R3 will not add support for multiple trust roots. Adding support for multiple trust roots to Corda would violate several fundamental assumptions in the architecture and so is not planned/contemplated at this time
  • Future mergeability of networks, either the joining of two segregated networks or the merging of one segregated network into the public network, must always be an option.
    • NOTE: The use of the term "public" network is subjective, in effect this is the largest segregated zone that is operated by the one operator.

  • Identity for the entire Compatibility Zone should be governed by a single Doorman.
  • NO WEB TECH Beyond the simple REST endpoints that form part of the existing NM protocol.

Hard Requirements (deliverable in the first phase)

  • There is a need for segregated sub-zones to be operable within a compatibility zone
  • There is a need for BNO to host and operate exclusive notaries
  • System should be scalable up to the management of many segregated sub-zones.
    • For example, we should scale to easily support the deployment and management of 10000 sub-zones (the public sub-zone and any number of segregated zones)
  • The Network Parameters governing the compatibility zone should operate on a ratcheting mechanism where segregated either use the version of the global parameters when they were bootstrapped or, when changed, adopt the global values save the elements that uniquely define the segregation (notaries, whitelists etc)
  • Merging two networks is a destructive action, one will be utterly consumed by the other, there will be no support of partial merging of some elements.
  • Documentation must be clear and accurate.
  • Operational side of the management of a Compatibility Zone with a number of segregated networks must be supported with excellent tooling.
    • Ideally such tooling would be "opinionated" in that it guides users toward best practice.
  • Should be invisible to Corda Nodes (no changes needed to the Corda / Ent codebases) (int)
  • BACKWARD COMPATIBLE with existing configurations (int)
  • Any new inter-process communication must be encrypted and authenticated (sec)
    • Equally all payloads should be encrypted and authenticated
  • Data (configuration files) must be version controlled (ops)

Soft Requirements (can be delivered in a second phase)

  • Segregated sub-zone membership should be controllable and secret. Security through obscurity (via some UUID url as per Private Networks) is unacceptable
    • This is a phase two issue given it requires changes in the node to work
    • Nodes would need some method of authenticating with the service before registering and downloading the map, possibly through the use of some key issued by the zone operator and given to the BNO to distribute.
  • The Network Map should be shippable as a discrete entity where a BNO could opt to run it "on prem" themselves whilst still deferring to the Doorman of a global Compatibility Zone.
    • This would require the generation of certificate for that BNO that could be used to sign their local Network Map
  • Separation of the network parameters into finer grained elements that could be signed by different entities (still chaining to the trust root) or distribution via extension to the protocol. This would allow segregated sub-zones to serve their own non-common parameters without the need to modify the global ones at all.
    • So, rather than one Network Parameters section, have multiple elements included at the top level of the Network Map alongside the list of NodeInfos and the Parameters as it currently exists
    • note that some of this is being undertaken already by adding in the concept of parameters whose change will be automatically applied without restart by the node.

    • By adding additional sections, sub-zone operators could include their own custom sections that might be useful.

Additional Requirement Decisions

Whilst there are a number of concrete requirements, there are several areas we could tackle if it's determined that they're sufficiently important.

Security / Privacy

The level of privacy / secrecy / access control to a segregated sub-zone is currently defined as a soft requirement in the context of operating only at the same privacy level of the existing Private Network map solution. What this means is, for all intents and purposes, there is no security associated with them, merely an obfuscation of their existence through the use of UUID's as URL's

For ENM 1.0 it is (mostly) accepted this is an acceptable solution. The real driving force of the use of segregated sub-zones within a compatibility zone for a BNO is the ability to deploy and manage their own notaries. Indeed, if privacy / access is of no concern to such operators a more human readable sub-zone name could be generated.

However, if secrecy is an issue then any solution will necessitate some changes within the node as it would mean a change / extension to the existing Network Map protocol, which is currently unauthenticated. Whilst an entity registering must have been permissioned by the Zones Doorman, retrieving the membership is open to all comers.

There are a number of solutions that could be put in place

  1. Token Header, assumes SSL (currently terminated at NGINX)
    • Used for auto enrolment in a PNM already
  2. Mutual Authentication based on the TLS connection, would require each segregated sub-zone to be issued it's own certificate for communication with the Network Map
  3. Signed Requests in the same way a node signs a CSR
  4. Out of band authentication based on a white / black list of some heuristic (Legal name, originating ip etc)

Pending Decision

  1. Can this be put off till "phase 2"?
    • RECOMMENDATION: YES (It requires node changes so won't be available till Corda 5)
  2. If not, what should be done?
    • RECOMMENDATION: Defer to security team

Decision Taken

Defer to a later phase of implementation

Membership Uniqueness

It is a stated requirement that a node exists in exactly one sub-zone of a compatibility zone, either the public one or some segregated sub-zone. However, there isn't anything currently in place (or proposed for phase 1) that actually enforces this. A node could be registered with the compatibility zone and issued a certificate. IT would then be trivial, assuming the URL's were known, for the node to join each network in turn, adding itself to all network maps.

Setting aside malicious intent if we wish to prevent this we would need to add a check across the sub-zone harem that rejected a registration. However, this would mean extending the protocol to allow a node to remove itself explicitly from a Network Map as currently that process is governed by the

Pending Decision

RECOMMENDATION: Defer till phase 2 given it requires node side changes

Decision Taken

Defer to a later phase of implementation

Target Solution

Before discussing some details, we envisage a Compatibility Zone and its segregated sub-zones would look something like this:


Cut the tie between Network Map and Doorman

Currently the Network Map and Doorman services are inter-linked through being forced to share a data engine due to the manner in which the Network Map checks weather a registering node "belongs" to the compatibility zone. Simply put, it checks weather the node is registered with the Doorman.

We can replace this check with a simple inspection of the Certificate chain used to sign the Node Info being submitted. If it chains back to the trust root of the Compatibility Zone and no level of that hierarchy has been revoked ( node -> doorman -> intermediate CA => Root) then we can accept that is is registered with our Doorman.

With this tie gone, the two services are no longer compelled to share the same database schema.

This enables two things

  1. The two services can actually be deployed as true distinct elements
  2. There is nothing to stop multiple network maps being configured to run on different URL's with separate schemas from one another. It is this that really enables segregated networks to operate as part of one compatibility zone.

Segregated Sub-Zone Identification

In the same way that a Private Network is identified by a UUID we propose segregated sub-zones are equally identified in such a way. For each network created (in response to some out of band agreement with the BNO) a segregated sub-zones ID can be generated and stored [see later]

Network Map Configuration

The current Network Map is configured thusly

...
address = "<some-server>:<some-port>"
...
networkMap {
  cacheTimeout = 600000
  versionInfoValidation {
    minimumPlatformVersion = 1
    newPKIOnly = false
  }
  localSigner {
    keystorePath = ${basedir}"/certificates/caKeystore.jks"
    keystorePassword = "password"
    signInterval = 10000
  }
}
...

URL

Currently the Network Map REST endpoints are served from the configured address of the server with /network-mapappended. This needs to be made configurable for segregated sub-zones to operate, especially if they are being run on the same Network Map server.

For the above example this would be

<some-server>:<some-port>/network-map 

An optional option segregatedSubZone section will be added to those network maps being configured as segregated sub-zones. This can be used to define elements of the network. Most importantly, configuring the segregated sub-zone as a specific instance

...
address = "<some-server>:<some-port>"
...
networkMap {
  cacheTimeout = 600000
  segregateNetwork {
    // where this has been established as the identifier of the network out of band
    id = 6fbc61f5-14ef-42d0-8bc7-487b578624c1
  }
}

This would cause the Network Map service to listen on the following URL

<some-server>:<some-port>/6fbc61f5-14ef-42d0-8bc7-487b578624c1/network-map 

with a node configured

networkServices = {
     ...
     networkMapURL = "<some-server>:<some-port>/6fbc61f5-14ef-42d0-8bc7-487b578624c1"
     ...
}

Making this, from the node's perspective, a non-change.

Co-Hosted Segregated Sub-Zone Network Map services

Whilst the simplest deployment of a segregated sub-zone would be to configure a second Network Map server alongside the global, public, part of the Compatibility Zone, that won't scale. It would be much better to enable individual Network Map servers (those that operate the Network Map Service) to be configured to host multiply configured services alongside one another, only listening on different URLs.

We make no distinction which is "better" (multiple servers or co-located services) and prefer to leave that to the decision of the operations team (a mixed environment is clearly possible)

The Network Map server would be modified to facilitate this co-hosting

Question: Does this need to be multi threaded, with each service listening on it's own http endpoint bound to a port or can we simply multiplex on a single http listener based on the URL

After discussion we think it better to multiplex on a single thread and split on endpoint

Configuration

The existing configuration of no Segregated Sub-Zone and a single Network Map service representing a public compatibility zone must continue to work "as is". To configure multiple services within a single server support the provision of a list of configurations.

networkMaps [
    // The public network
    map1  {
        cacheTimeout = 600000  
    },
    map2  {    
        cacheTimeout = 600000
        segregatedSubZone {
            // where this has been established as the identifier of the network out of band
            id = 6fbc61f5-14ef-42d0-8bc7-487b578624c1
        }
    }
]

Note: This is a suggestion only, implementation details are beyond the scope of this document

The individual REST endpoints would reside at

http://<someserver>/network-map
http://<someserver>/6fbc61f5-14ef-42d0-8bc7-487b578624c1/network-map

With the server appropriately forwarding requests to the correct service

Configuration management

Given the nature of the proposed system, many servers and services deployed across many virtual machines all with different but very similar configurations we propose adding a configuration manager service as part of the ENM software suite. This would, on demand, allow components (Network Map Servers, Doormen, Revocation Services) to be configured to download their actual configuration from a central source, identified by some unique identifier.

This configuration would be generated "on the fly" using information stored within the ENM system. Likewise, the same system would be used to generate the Network Parameter files.

ENM components would be invoked with a minimal configuration file that indicates where they should look for their config (the server address) and what identifiers they should use when talking to it

This communication would of course be authenticated and encrypted as per the requirements of the security team

Network Parameters

Whilst it would be possible for each segregated sub-zone alongside the public, default, sub-zone to support their own unique and distinct Network Parameters, given the requirements for future mergability and operational manageability, we propose a single set of common Network Parameters be managed by a Zone Operator with sub-zone specific elements merged into created the actually served parameters file.

These sub-zone specific attributes are, in effect, being deferred to the BNO. This is needed to allow these third parties to define elements of the sub-zone to match their desired operational parameters. Therefore, the Network Parameters for a sub-zone (segregated and public) will be split into three distinct elements

  1. The epoch / version number
  2. Common components, those defined by the zone operator that are applied to all segregated sub-zones a
  3. Sub Zone specific components
    • Notary White Lists etc

Thus, the signing of a zones parameters becomes a three step process

  1. Retrieve the unique epoch for the new parameters (See later)
  2. Merge the common and unique components with the above to form the actual file
  3. Using the appropriate configuration, sign and deploy to the correct network map (see tooling)

NOTE: The acquisition of Zone-Specific parameters is out of scope for this document. It is assumed that a segregated sub-zone operator can in some "out of band" operation provide signed NodeInfo files to the Compatibility Zone operator. The provided tooling from Network Service should allow for the storage of these and visualisation, but acquisition is left to the Zone Operator.

This is to prevent the ENM being opinionated as to how a Zone operator should interact with their users / customers. Whatever single system we put in place may not cover all use cases. Better that the ENM remain agnostic to the gathering of the NodeInfos and present a simple interface for the Zone operator to leverage with their own bespoke tooling that meets their exact needs.

Parameter Loading

It is intended that sub-zone owners will have no direct interaction with the ENM software. The acquisition of deferred parameter objects (NodeInfos for notaries) should be a separate process an operator can then load into the ENM's configuration manager. This is to prevent abuse of the system and allow for the building of a workflow around the gathering and application of NodeInfo's.

Which Parameters to Defer

Segregated sub-zones exist primarily for two reasons.

  • Firstly, to facilitate the deployment and operation of exclusive notaries by some BNO.
  • Secondly, to allow for the realistic scenario where an established zone cannot force an upgrade in the minimum platform version, yet there is a need for a new deployment to use features in newer Corda versions.

As such, these are the only two parameters targeted for ease of tooling. Theoretically, the whole set of parameters could be deferred by simply allowing any hand crafted set to be applied to a sub-zone but the tooling we provide and the relationships between zone operator and sub-zone owner will be based around the deferment of these two elements.

Automatically managed by the ENM

  • epoch

Global, managed by the compatibility zone operator

  • maxMessageSize
  • maxTransactionSize
  • eventHorizonDays
  • whitelistContracts
  • other (package ownership for example)

Per sub-zone

  • minimumPlatformVersion
  • notaries

NOTE: As mentioned above, it is intentional that the mechanism by which these deferred parameters are gathered from the BNO's is left to the discretion of the Zone Operator. The ENM seeks to remain un-opinionated regarding the security and contractual lifecycle of gathering those entities and will simply provide a convenient way to input them once gathered.

Epoch Ratchet

It is intended that the zone as a whole is always governed by a ever incrementing, unique, epoch value. This will be vital to the merging of segregated sub-zones into the public zone at some future date.

Consider the following sequence of events

  1. The Zone is commissioned, consisting of only the public sub-zone (p-sz), this is governed by Network Parameters with Epoch 1
  2. A segregated sub-zone is added (sz1), it takes the common components of the global parameters and applies its own whitelist. This is now Epoch 2.
  3. A value is changed / added to the set of common parameters
  4. A flag day is run to adopt that new set of common parameters on the public sub-zone, this is now Epoch 3,
  5. sz1 doesn't want to adopt the new parameters so remains on the old parameters, remaining at Epoch 2.
  6. A second segregated sub-zone is added (sz2). This has no choice but to adopt the second version of the common components forming Epoch 4.
  7. sz1 switches to the new common properties becoming epoch 5.
  8. A new notary is added to the public sub-zone, this is Epoch 6.
  9. sz2 merges into the public sub-zone, the public sub-zone moves to Epoch 7

NOTE: This process must be automated as much as possible by the tooling to make it easy and fault free

To summarise the above

Step Epoch Common Param Version p-sz s-sz1 s-sz2
1 1 1 (1, 1)
2 2 1 (1,1) (2,1)
4 3 2 (3,2) (2,1)
6 4 2 (3,2) (2,1) (4,2)
7 5 2 (3,2) (5,2) (4,2)
8 6 2 (6,2) (5,2) (4,2)
9 7 2 (7,2) (5,2)

If two or more sub-zones adopt a change to the common parameters at any time, each zone would receive a unique version number. No two sub-zones can have the same epoch number.

All zones have the option, when the common parameters are changed, to remain as they are or adopt the new ones ( forcing a flag day). Any change to a zones parameters introduces a new epoch into the zone as a whole. It is this reason central management of the configurations and zone properties is delivered as part of this work.

Tooling - Zone Management

  1. Segregated Sub-Zone Management
    1. Creation of new networks
    2. Management / visualisation of existing networks (members, deployment, etc)
    3. On-boarding of externally sourced delegated parameters (Node Info files)
  2. Network service configurations
  3. Signing service configurations
    • Some form of configuration service may be required to facilitate pull based server configuration when managing the deployment of N services across m servers with y data engines
    • Configuration management
  4. Network Parameter management
    1. Centrally managed component
    2. Management of the distinct, per-network, elements
    3. Epoch management / creation (must be unique and ever incrementing)
    4. Parameter file generation, signing, and deployment
  5. Flag Day management
  6. Configuration Distribution

Configuration Management

As discussed previously to enable the simple management of the zone we are proposing the creation of a configuration management tool that will give a single point of configuration for the disparate components of the Compatibility Zone as a whole.

Network Merging

The intention is to allow for the following

  1. A segregated sub-zone to merge into the public sub-zone
  2. Two segregated sub-zones to "merge", effectively creating a new sub-zone

In either case the operation is the same, 1 network is destroyed with all its members moving to the new by changing the constituent nodes (including notaries) node configuration to refer to the appropriate Network Map REST service.

This would be a multi-step process

  1. Decommission the network that is merging, shutdown the network map service
  • This could be a parallel effort with the provision of the new network given the time taken to commission and test new infrastructure. However, there should be a business decision taken that the network being merged from is no at the end of its life with appropriate controls put in place.
  1. The network being merged into conducts a flag day that bumps it's Epoch value, ensuring it is currently the highest in the whole zone.
  2. The merging nodes join the Network Map service of the network
  3. The notaries from the old nodes also join the NM service of the network
  4. A Notary change operation must be run on all unconsumed states from the "old" notaries, moving them to notaries trusted by the remaining network.
    • The purpose of the notary whitelist is to constrain which notaries can own created states and thus notarise transactions that refer to them.
    • Transactions are signed with the set of parameters that governed them at the time attached, including the set of trusted notaries.
    • The Notary Change transaction will embed the current Network Parameters of the network being merged into which will of course list the notary being moved to as globally valid.
    • Historic validation of transactions can be done by examining the attached parameters file.
    • The epoch ratchet exists to ensure that walking down the transaction path that the parameters governing newer transactions are always valid. id est it is impossible to attempt to use an old set of parameters that include a now untrusted notary in a transaction because the epoch of the parameters would be older than the current one.

There is an amount of work here to be done by several other feature teams

  • The Notaries and Perf team to change the transaction validation steps to check epoch versioning
  • The V&M Team to ensure Network Parameters are attached to transactions

Where these changes require new Network Parameters, such as a list of the "last trusted zone parameters" the ENM tooling must support these to make operation of the zone painless.

Implementation plan

Rough outline of implementation steps (Testing and Demoing omitted for brevity)

Segregated Sub-Zones

  1. Separate the NM & Doorman functionally
    • Should be able to deploy each separately and have that work
  2. Prune the Schemas
  3. Migration script for existing systems once the above is in place
    • Will require extensive testing to ensure no production system is broken
  4. Get Multiple Services running inside a single server
  5. Work on config extraction from some service that manages services across the zone.
  6. Signing service changes to work across multiple Network Maps / Network Parameters

Compatibility Zone Management

  1. Define schema for parameter and configuration storage
  2. Tooling for generating entities within the above
    • Common params
    • Global epoch
    • local entity storage
    • --> Generates the configs and deploys them
  3. Back end integration with GIT
  4. Injection of Node Infos into the system
  5. Epoch visualisation and tracking
  6. Zone visualisation and tracking
    • Whose on what where etc
  7. Signing service management tooling
  8. ENM component changes to retrieve config from central server

Glossary

  • Compatibility Zone - Defined by a trust root and global uniqueness of identity, formed of one or more sub-zones.
  • Public Sub-Zone - The default sub-zone for the compatibility zone, governed by the Zone operator.
  • Segregated Sub-Zone - Zones run along side the default public sub-zone, certain elements of their governance are deferred to a third party, a BNG. Intended as a managed commercial arrangement between the zone operator and the BNG.
  • BNG - Business Network Governor.
  • BNO - Business Network Operator.
  • Network Map Server - A JAR running inside a VM, operates one oe more Network Map services
  • Network Map Service - Represents the identity service for a sub-zone to which all members publish their NodeInfo files. One or more services are managed by a Network Map Server.
  • Private Notary - A Notary Service that restricts who can access its services. This doesn't preclude it's identity being public (if obfuscated)
  • Exclusive Notary - A Notary Service operated without universal trust, rather than being vetted by the by zones operator and governed by the rules and standard of the zone, it is operated (desired to be operated) under the purview of some other entity. In the scope of this document exclusive notaries can only exist when deployed as part of a segregated sub-zone.
  • ENM - Enterprise Network Manager - The Suite of tools formed by the Doorman, Network Map, and various utilities that are used to operate a Compatibility Zone.
  • Norman - The combined Network Map and Doorman services running as a single entity
    • NOTE: This is relegated to internal testing and is no longer a supported production deployment.


Mark Oldfield R3

Head of Architecture

M: +44 7710 360877