The VisualSVN Distributed File System (VDFS) follows the classic master/slave replication architecture which ensures the following:

  • a master repository is always readable and writable — even if some of the corresponding slave repositories are unreachable,
  • a slave repository is writable if there is connectivity to the corresponding master repository,
  • replication is resilient to temporary connectivity issues,
  • read access to the slave repository does not require connectivity to the corresponding master repository.

However, in case of a disaster that renders master server and its repositories permanently inoperable, the replication will halt and slave repositories will become read only. In the event of such hardware or software failure, you can perform a disaster recovery procedure that is briefly described below.

The main steps required for the disaster recovery are the following:

  1. Promote the most up to date slave repository to act as a temporary master. This is a quick operation and all the available slave repositories become writable after this step.
  2. Recover the lost repository from the temporary master. This operation could take significant time depending on the size of repositories.
  3. Promote the recovered repository back to its master role. After this step, the replication system returns to its original state.

Prerequisites and assumptions

For explanatory reasons, we assume the following:

  • There are 3 instances of VisualSVN Server Enterprise Edition installed in the corresponding geographically distributed sites:
    • ‘Berlin-SVN’ hosts a single master VDFS repository ‘MyRepo’,
    • ‘Rome-SVN’ and ‘NY-SVN’ servers host slave VDFS repositories co-named ‘MyRepo’,
  • The ‘Berlin-SVN’ server is completely lost due to a disaster and the replacement server is named ‘Berlin-SVN2’.
  • All VisualSVN Server instances are in the same Active Directory domain named ‘example.com’.
  • End users on all three sites contact the VisualSVN Server using the ‘svn.example.com’ hostname and are routed to the closest VisualSVN Server instance using the DNS settings.
  • All VisualSVN Server instances are at the same server version of 3.4 or newer.
  • Windows Firewall and security settings on Berlin-SVN2, Rome-SVN and NY-SVN servers allow remote VisualSVN Server administration using PowerShell and VisualSVN Server Manager console.

We also assume that all PowerShell cmdlets required for disaster recovery are executed remotely and the target server for all cmdlets is identified by the standard CimSession parameter.

Promote the most up to date slave repository to act as a temporary master

Because master repository on Berlin-SVN server is completely lost, you should choose the most up-to-date slave repository and promote it to be the temporary master. You can examine which of remaining slave repositories is up-to-date by comparing the youngest revision number using VisualSVN Server Manager or the Get-SvnRepository PowerShell cmdlet.

Let’s assume that the slave repository on Rome-SVN server is the most up-to-date. This means that you can safely promote it to act as a temporary master. After this promotion, repositories on both Rome-SVN and NY-SVN become writable again.

The following steps are required to promote a repository on Rome-SVN server to act as a temporary master:

  1. Make sure that VDFS firewall rule is enabled on Rome-SVN server to allow incoming connections to the VDFS service. Follow the steps specified in the article KB73: Enabling the inbound firewall rule for a master VDFS service.
  2. Make sure that NY-SVN server is included into the VisualSVN Replication Partners group on the Rome-SVN server. For more information refer to the article KB71: Granting general access to the master server VDFS service.
  3. Suspend both remaining slave repositories to prevent accidental desynchronization until the procedure is completed and all the configuration checks are done:
    PS C:\> Suspend-SvnRepository -Name MyRepo -CimSession Rome-SVN
    PS C:\> Suspend-SvnRepository -Name MyRepo -CimSession NY-SVN
    This step provides additional safety and prevents your repositories from becoming out of sync due to accidental commit and configuration mistakes: both repositories are read-only until the Resume-SvnRepository command is executed for them.
  4. Promote slave VDFS repository located on Rome-SVN server to act as a temporary master:
    PS C:\> Switch-SvnRepository -Name MyRepo -Role Master -CimSession Rome-SVN
  5. Allow NY-SVN server to replicate the temporary master repository located on the Rome-SVN server:
    PS C:\> Set-SvnRepositoryReplication MyRepo -Replicators NY-SVN$ -CimSession Rome-SVN
  6. Enable repository replication for the temporary master repository (replication is initially disabled in the temporary master repository, hence you need to enable it manually):
    PS C:\> Set-SvnRepositoryReplication MyRepo -Enabled $true -CimSession Rome-SVN
  7. Configure slave VDFS repository located on the NY-SVN server to begin replication from the Rome-SVN server:
    PS C:\> Set-SvnRepository MyRepo -MasterServer Rome-SVN -CimSession NY-SVN
  8. Resume repositories on Rome-SVN and NY-SVN server to make them writable:
    PS C:\> Resume-SvnRepository -Name MyRepo -CimSession Rome-SVN
    PS C:\> Resume-SvnRepository -Name MyRepo -CimSession NY-SVN
  9. Reconfigure DNS settings in Berlin to route all local users to the closest site.

On completion of the above steps, the replication system will be partially recovered and repositories on Rome-SVN and NY-SVN servers will be writable again. Users in Berlin will be also able to work with one of the servers at the remote site (but data transfer speed will be slower due to WAN latency).

Recover the lost repository from the temporary master

As the original master repository is completely lost, you need to create a new slave VDFS repository on the replacement Berlin-SVN2 server and perform initial replication from the temporary master. During the initial replication phase end users from Berlin should be still redirected to one of the available operable repositories (on Rome-SVN or NY-SVN servers).

The following steps are required to create a new slave VDFS repository on the replacement Berlin-SVN2 server:

  1. Make sure that Berlin2-SVN server is included into the VisualSVN Replication Partners group on the Rome-SVN server. For more information refer to the article KB71: Granting general access to the master server VDFS service.
  2. Allow Berlin2-SVN server to replicate the temporary master repository located on the Rome-SVN server:
    PS C:\> Set-SvnRepositoryReplication MyRepo -Replicators Berlin2-SVN$ -CimSession Rome-SVN
  3. Begin recovering the lost repository from the temporary master by creating a new slave VDFS repository on Berlin-SVN2 server:
    PS C:\> New-SvnRepository MyRepo -Type VdfsSlave -MasterServer Rome-SVN -CimSession Berlin-SVN2

The new slave repository will immediately start replicating data from the Rome-SVN server. Depending on the size of the repository it may take some time to complete the initial replication. When the initial replication of the slave repository in Berlin is finished, you can alter your DNS settings and reroute users in Berlin to work with the local slave repository on ‘Berlin-SVN2’.

Promote the recovered repository back to its master role

As the lost repository is recovered from the temporary master, you need to promote the recovered repository on Berlin-SVN2 server back to its master role and demote the temporary master on Rome-SVN to be a slave. Following these steps will fully recover the multisite repository replication system to the state as it was before the disaster.

  1. Make sure that VDFS firewall rule is enabled on Berlin-SVN2 server to allow incoming connections to the VDFS service. Follow the steps specified in the article KB73: Enabling the inbound firewall rule for a master VDFS service.
  2. Make sure that Rome-SVN and NY-SVN servers are included into the VisualSVN Replication Partners group on the Berlin-SVN2 server. For more information refer to the article KB71: Granting general access to the master server VDFS service.
  3. Suspend the temporary master repository on the Rome-SVN server:
    PS C:\> Suspend-SvnRepository -Name MyRepo -CimSession Rome-SVN
    This step is required to make sure that your repositories will be in sync after switching the temporary and recovered master repositories.
  4. Make sure that the slave repository located on Berlin-SVN2 is up to date and the initial replication process has actually completed:
    PS C:\> Sync-SvnRepository MyRepo -PassThru -CimSession Berlin-SVN2
    This step ensures that the slave repository on Berlin-SVN2 is in sync with the temporary master on the Rome-SVN server (that is suspended).
  5. Suspend both available slave repositories to prevent accidental commits until the procedure is completed and all the configuration checks are done:
    PS C:\> Suspend-SvnRepository -Name MyRepo -CimSession Berlin-SVN2
    PS C:\> Suspend-SvnRepository -Name MyRepo -CimSession NY-SVN
  6. Promote the recovered repository on Berlin-SVN2 to its original master role:
    PS C:\> Switch-SvnRepository MyRepo -Role Master -CimSession Berlin-SVN2
  7. Grant the Rome-SVN and NY-SVN server permissions to replicate the recovered master repository located on the Berlin-SVN2 computer:
    PS C:\> Set-SvnRepositoryReplication MyRepo -Replicators Rome-SVN$, NY-SVN$ -CimSession Berlin-SVN2
  8. Enable repository replication in the recovered master repository on the Berlin-SVN2 server (replication is initially disabled after the switching to the master role, hence you need to enable it manually):
    PS C:\> Set-SvnRepositoryReplication MyRepo -Enabled $true -CimSession Berlin-SVN2
  9. Demote temporary master repository on the Rome-SVN server back to its original slave role:
    PS C:\> Switch-SvnRepository MyRepo -Role Slave -MasterServer Berlin-SVN2 -CimSession Rome-SVN
  10. Resume replication for all three repositories:
    PS C:\> Resume-SvnRepository MyRepo -CimSession Berlin-SVN2
    PS C:\> Resume-SvnRepository MyRepo -CimSession Rome-SVN
    PS C:\> Resume-SvnRepository MyRepo -CimSession NY-SVN

Your replication system will be fully recovered after this step and all three repositories will be both readable and writable.

Please make sure that your DNS settings are reverted back to its original state and that users located in Berlin are working with the local repository (which is currently recovered to its original master role).

See also

KB84: Comparing VDFS with master-master replication solutions
KB68: Getting started with Multisite Repository Replication
Last Modified: