<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.10">
<meta name="author" content="Perforce Professional Services">
<title>Server Deployment Package (SDP) for Perforce Helix: SDP Failover Guide (for Unix and Windows)</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/* Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
/* Uncomment @import statement to use as custom stylesheet */
/*@import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700";*/
article,aside,details,figcaption,figure,footer,header,hgroup,main,nav,section{display:block}
audio,video{display:inline-block}
audio:not([controls]){display:none;height:0}
html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}
a{background:none}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
abbr[title]{border-bottom:1px dotted}
b,strong{font-weight:bold}
dfn{font-style:italic}
hr{-moz-box-sizing:content-box;box-sizing:content-box;height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,*::before,*::after{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;font-weight:400;font-style:normal;line-height:1;position:relative;cursor:auto;tab-size:4;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.center{margin-left:auto;margin-right:auto}
.stretch{width:100%}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0;direction:ltr}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:0}
p{font-family:inherit;font-weight:400;font-size:1em;line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em;height:0}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{font-size:1em;line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0;font-size:1em}
ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit}
ul.square{list-style-type:square}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
abbr,acronym{text-transform:uppercase;font-size:90%;color:rgba(0,0,0,.8);border-bottom:1px dotted #ddd;cursor:help}
abbr{text-transform:none}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote cite{display:block;font-size:.9375em;color:rgba(0,0,0,.6)}
blockquote cite::before{content:"\2014 \0020"}
blockquote cite a,blockquote cite a:visited{color:rgba(0,0,0,.6)}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
h1{font-size:2.75em}
h2{font-size:2.3125em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:solid 1px #dedede}
table thead,table tfoot{background:#f7f8f7}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt{background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{display:table-cell;line-height:1.6}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table}
.clearfix::after,.float-group::after{clear:both}
:not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;-webkit-border-radius:4px;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed;word-wrap:break-word}
:not(pre)>code.nobreak{word-wrap:normal}
:not(pre)>code.nowrap{white-space:nowrap}
pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed}
pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit}
pre>code{display:block}
pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal}
em em{font-style:normal}
strong strong{font-weight:400}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;-webkit-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em white inset;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em #fff inset;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menuref{color:#000}
.menuseq b:not(.caret),.menuref{font-weight:inherit}
.menuseq{word-spacing:-.02em}
.menuseq b.caret{font-size:1.25em;line-height:.8}
.menuseq i.caret{font-weight:bold;text-align:center;width:.45em}
b.button::before,b.button::after{position:relative;top:-1px;font-weight:400}
b.button::before{content:"[";padding:0 3px 0 2px}
b.button::after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin-left:auto;margin-right:auto;margin-top:0;margin-bottom:0;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table}
#header::after,#content::after,#footnotes::after,#footer::after{clear:both}
#content{margin-top:1.25em}
#content::before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf}
#header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px}
#header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:-ms-flexbox;display:-webkit-flex;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span::before{content:"\00a0\2013\00a0"}
#header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark::before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber::after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media screen and (min-width:768px){#toctitle{font-size:1.375em}
body.toc2{padding-left:15em;padding-right:0}
#toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
#toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
#toc.toc2>ul{font-size:.9em;margin-bottom:0}
#toc.toc2 ul ul{margin-left:0;padding-left:1em}
#toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
body.toc2.toc-right{padding-left:0;padding-right:15em}
body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}}
@media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
#toc.toc2{width:20em}
#toc.toc2 #toctitle{font-size:1.375em}
#toc.toc2>ul{font-size:.95em}
#toc.toc2 ul ul{padding-left:1.25em}
body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:100%;background:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:rgba(255,255,255,.8);line-height:1.44}
#content{margin-bottom:.625em}
.sect1{padding-bottom:.625em}
@media screen and (min-width:768px){#content{margin-bottom:1.25em}
.sect1{padding-bottom:1.25em}}
.sect1:last-child{padding-bottom:0}
.sect1+.sect1{border-top:1px solid #e7e7e9}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
details>summary:first-of-type{cursor:pointer;display:list-item;outline:none;margin-bottom:.75em}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock.fit-content>caption.title{white-space:nowrap;width:0}
.paragraph.lead>p,#preamble>.sectionbody>[class="paragraph"]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)}
table.tableblock #preamble>.sectionbody>[class="paragraph"]:first-of-type p{font-size:inherit}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6)}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border-style:solid;border-width:1px;border-color:#e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;-webkit-border-radius:4px;border-radius:4px}
.exampleblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child{margin-bottom:0}
.sidebarblock{border-style:solid;border-width:1px;border-color:#dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;-webkit-border-radius:4px;border-radius:4px}
.sidebarblock>:first-child{margin-top:0}
.sidebarblock>:last-child{margin-bottom:0}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock>.content>pre{-webkit-border-radius:4px;border-radius:4px;word-wrap:break-word;overflow-x:auto;padding:1em;font-size:.8125em}
@media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}}
@media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}}
.literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class="highlight"],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8}
.literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)}
.listingblock>.content{position:relative}
.listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5}
.listingblock:hover code[data-lang]::before{display:block}
.listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5}
.listingblock.terminal pre .command:not([data-prompt])::before{content:"$"}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;-webkit-border-radius:4px;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.prettyprint{background:#f7f7f8}
pre.prettyprint .linenums{line-height:1.45;margin-left:2em}
pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0}
pre.prettyprint li code[data-lang]::before{opacity:1}
pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none}
table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal}
table.linenotable td.code{padding-left:.75em}
table.linenotable td.linenos{border-right:1px solid currentColor;opacity:.35;padding-right:.5em}
pre.pygments .lineno{border-right:1px solid currentColor;opacity:.35;display:inline-block;margin-right:.75em}
pre.pygments .lineno::before{content:"";margin-right:-.125em}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right}
.verseblock{margin:0 1em 1.25em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none}
.quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0}
.quoteblock.abstract{margin:0 1em 1.25em;display:block}
.quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center}
.quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf}
.quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0}
.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem}
.quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;text-align:left;margin-right:0}
table.tableblock{max-width:100%;border-collapse:separate}
p.tableblock:last-child{margin-bottom:0}
td.tableblock>.content>:last-child{margin-bottom:-1.25em}
td.tableblock>.content>:last-child.sidebarblock{margin-bottom:0}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all>thead>tr>.tableblock,table.grid-all>tbody>tr>.tableblock{border-width:0 1px 1px 0}
table.grid-all>tfoot>tr>.tableblock{border-width:1px 1px 0 0}
table.grid-cols>*>tr>.tableblock{border-width:0 1px 0 0}
table.grid-rows>thead>tr>.tableblock,table.grid-rows>tbody>tr>.tableblock{border-width:0 0 1px}
table.grid-rows>tfoot>tr>.tableblock{border-width:1px 0 0}
table.grid-all>*>tr>.tableblock:last-child,table.grid-cols>*>tr>.tableblock:last-child{border-right-width:0}
table.grid-all>tbody>tr:last-child>.tableblock,table.grid-all>thead:last-child>tr>.tableblock,table.grid-rows>tbody>tr:last-child>.tableblock,table.grid-rows>thead:last-child>tr>.tableblock{border-bottom-width:0}
table.frame-all{border-width:1px}
table.frame-sides{border-width:0 1px}
table.frame-topbot,table.frame-ends{border-width:1px 0}
table.stripes-all tr,table.stripes-odd tr:nth-of-type(odd),table.stripes-even tr:nth-of-type(even),table.stripes-hover tr:hover{background:#f8f8f7}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{display:table-cell;line-height:1.6;background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
ol>li p,ul>li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none}
ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em}
ul.unstyled,ol.unstyled{margin-left:0}
ul.checklist{margin-left:.625em}
ul.checklist li>p:first-child>.fa-square-o:first-child,ul.checklist li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em}
ul.checklist li>p:first-child>input[type="checkbox"]:first-child{margin-right:.25em}
ul.inline{display:-ms-flexbox;display:-webkit-box;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em}
ul.inline>li{margin-left:1.25em}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top}
.colist td:not([class]):first-child img{max-width:none}
.colist td:not([class]):last-child{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:solid 4px #fff;-webkit-box-shadow:0 0 0 1px #ddd;box-shadow:0 0 0 1px #ddd}
.imageblock.left{margin:.25em .625em 1.25em 0}
.imageblock.right{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
.gist .file-data>table{border:0;background:#fff;width:100%;margin-bottom:0}
.gist .file-data>table td.line-data{width:99%}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background:#00fafa}
.black{color:#000}
.black-background{background:#000}
.blue{color:#0000bf}
.blue-background{background:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background:#fa00fa}
.gray{color:#606060}
.gray-background{background:#7d7d7d}
.green{color:#006000}
.green-background{background:#007d00}
.lime{color:#00bf00}
.lime-background{background:#00fa00}
.maroon{color:#600000}
.maroon-background{background:#7d0000}
.navy{color:#000060}
.navy-background{background:#00007d}
.olive{color:#606000}
.olive-background{background:#7d7d00}
.purple{color:#600060}
.purple-background{background:#7d007d}
.red{color:#bf0000}
.red-background{background:#fa0000}
.silver{color:#909090}
.silver-background{background:#bcbcbc}
.teal{color:#006060}
.teal-background{background:#007d7d}
.white{color:#bfbfbf}
.white-background{background:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background:#fafa00}
span.icon>.fa{cursor:default}
a span.icon>.fa{cursor:inherit}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);-webkit-border-radius:100px;border-radius:100px;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]::after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,span.alt{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background:#fffef7;border-color:#e0e0dc;-webkit-box-shadow:0 1px 4px #e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@page{margin:1.25cm .75cm}
@media print{*{-webkit-box-shadow:none!important;box-shadow:none!important;text-shadow:none!important}
html{font-size:80%}
a{color:inherit!important;text-decoration:underline!important}
a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
abbr[title]::after{content:" (" attr(title) ")"}
pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
thead{display:table-header-group}
svg{max-width:100%}
p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
#toc,.sidebarblock,.exampleblock>.content{background:none!important}
#toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important}
body.book #header{text-align:center}
body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em}
body.book #header .details{border:0!important;display:block;padding:0!important}
body.book #header .details span:first-child{margin-left:0!important}
body.book #header .details br{display:block}
body.book #header .details br+span::before{content:none!important}
body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
.listingblock code[data-lang]::before{display:block}
#footer{padding:0 .9375em}
.hide-on-print{display:none!important}
.print-only{display:block!important}
.hide-for-print{display:none!important}
.show-for-print{display:inherit!important}}
@media print,amzn-kf8{#header>h1:first-child{margin-top:1.25rem}
.sect1{padding:0!important}
.sect1+.sect1{border:0}
#footer{background:none}
#footer-text{color:rgba(0,0,0,.6);font-size:.9em}}
@media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}}
</style>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
</head>
<body class="book">
<div id="header">
<h1>Server Deployment Package (SDP) for Perforce Helix: SDP Failover Guide (for Unix and Windows)</h1>
<div class="details">
<span id="author" class="author">Perforce Professional Services</span><br>
<span id="email" class="email"><a href="mailto:consulting@perforce.com">consulting@perforce.com</a></span><br>
<span id="revnumber">version v2020.1,</span>
<span id="revdate">2020-09-12</span>
</div>
<div id="toc" class="toc">
<div id="toctitle">Table of Contents</div>
<ul class="sectlevel1">
<li><a href="#_preface">Preface</a></li>
<li><a href="#_overview">1. Overview</a>
<ul class="sectlevel2">
<li><a href="#_planning">1.1. Planning</a></li>
</ul>
</li>
<li><a href="#_planned_failover">2. Planned Failover</a>
<ul class="sectlevel2">
<li><a href="#_prerequisites">2.1. Prerequisites</a>
<ul class="sectlevel3">
<li><a href="#_pre_failover_checklist">2.1.1. Pre Failover Checklist</a>
<ul class="sectlevel4">
<li><a href="#_swarm_triggers">2.1.1.1. Swarm Triggers</a></li>
<li><a href="#_other_triggers">2.1.1.2. Other Triggers</a></li>
<li><a href="#_other_replicas_p4target">2.1.1.3. Other Replica’s P4TARGET</a></li>
<li><a href="#_proxies">2.1.1.4. Proxies</a></li>
<li><a href="#_brokers">2.1.1.5. Brokers</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#_failing_over">2.2. Failing over</a></li>
<li><a href="#_post_failover">2.3. Post Failover</a>
<ul class="sectlevel3">
<li><a href="#_moving_of_checkpoints">2.3.1. Moving of Checkpoints</a></li>
<li><a href="#_check_on_replication">2.3.2. Check on Replication</a></li>
</ul>
</li>
<li><a href="#_failing_over_on_windows">2.4. Failing over on Windows</a>
<ul class="sectlevel3">
<li><a href="#_post_failover_on_windows">2.4.1. Post Failover on Windows</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#_unplanned_failover">3. Unplanned Failover</a>
<ul class="sectlevel2">
<li><a href="#_post_unplanned_failover">3.1. Post Unplanned Failover</a>
<ul class="sectlevel3">
<li><a href="#_resetting_downstream_replicas">3.1.1. Resetting Downstream Replicas</a></li>
</ul>
</li>
<li><a href="#_unplanned_failover_on_windows">3.2. Unplanned Failover on Windows</a></li>
</ul>
</li>
<li><a href="#_old_style_failover">4. Old style failover</a></li>
</ul>
</div>
</div>
<div id="content">
<div class="sect1">
<h2 id="_preface">Preface</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The Server Deployment Package (SDP) is the implementation of Perforce’s recommendations for operating and managing a production Perforce Helix Core Version Control System. It is intended to provide the Helix Core administration team with tools to help:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>High Availability (HA)</p>
</li>
<li>
<p>Disaster Recovery (DR)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This guide is intended to provide instructions for failover in an SDP environment using built in Helix Core features.</p>
</div>
<div class="paragraph">
<p>For more details see:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://www.perforce.com/manuals/p4sag/Content/P4SAG/failover.html#Failover">Sysadmin Guide - Failover</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><strong>Please Give Us Feedback</strong></p>
</div>
<div class="paragraph">
<p>Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to <a href="mailto:consulting@perforce.com">consulting@perforce.com</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_overview">1. Overview</h2>
<div class="sectionbody">
<div class="paragraph">
<p>We need to consider planned vs unplanned failover. Planned may be due to upgrading the core Operating System or some other dependency in your infrastructure, or a similar activity.</p>
</div>
<div class="paragraph">
<p>Unplanned covers risks you are seeking to mitigate with failover:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>loss of a machine, or some machine related hardware failure</p>
</li>
<li>
<p>loss of a VM cluster</p>
</li>
<li>
<p>failure of storage</p>
</li>
<li>
<p>loss of a data center or machine room</p>
</li>
<li>
<p>etc…​</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>See also <a href="https://www.perforce.com/manuals/cmdref/Content/CmdRef/p4_failover.html">p4 failover in Command Reference Guide</a></p>
</div>
<div class="paragraph">
<p>Please refer to the following sections:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="SDP_Guide.Unix.html#_planning_for_ha_and_dr">SDP Guide: Planning for HA and DR</a></p>
</li>
<li>
<p><a href="SDP_Guide.Unix.html#_pre_requisites_for_failover">SDP Guide: Pre-requisites for Failover</a></p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="_planning">1.1. Planning</h3>
<div class="paragraph">
<p>HA failover should not require a P4PORT change for end users. Depending on your topology, you can avoid changing P4PORT by having users set P4PORT to an alias which can be easily changed centrally.</p>
</div>
<div class="paragraph">
<p>For example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>p4d-bos-01</code>, a master/commit-server in Boston, pointed to by a DNS name like <code>perforce</code> or <code>perforce.p4demo.com</code>.</p>
</li>
<li>
<p><code>p4d-bos-02</code>, a standby replica in Boston, not pointed to by a DNS until failover, at which time it gets pointed to by <code>perforce</code>/<code>perforce.p4demo.com</code>.</p>
</li>
<li>
<p>Changing the Perforce broker configuration to target the backup machine.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>See <a href="SDP_Guide.Unix.html#_server_host_naming_conventions">SDP Guide: Server Host Naming Conventions</a></p>
</div>
<div class="paragraph">
<p>Other advanced networking options might be possible if you talk to your local networking gurus (virtual IPs etc).</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_planned_failover">2. Planned Failover</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In this instance you can run <code>p4 failover</code> with the active participation of its upstream server.</p>
</div>
<div class="paragraph">
<p>We are going to provide examples with the following assumptions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ServerID <code>master_1</code> is current commit, running on machine <code>p4d-bos-01</code></p>
</li>
<li>
<p>ServerID <code>p4d_ha_bos</code> is HA server</p>
</li>
<li>
<p>DNS alias <code>perforce</code> is set to <code>p4d-bos-01</code></p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="_prerequisites">2.1. Prerequisites</h3>
<div class="paragraph">
<p>You need to ensure:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>you are running p4d 2018.2 or later for your commit and all replica instances, preferably 2020.1+</p>
<div class="literalblock">
<div class="content">
<pre>source /p4/common/bin/p4_vars 1
p4 info | grep version</pre>
</div>
</div>
</li>
<li>
<p>your failover target server instance is of type <code>standby</code> or <code>forwarding-standby</code></p>
<div class="paragraph">
<p>On HA machine:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 info
:
ServerID: p4d_ha_bos
Server services: standby
Replica of: perforce:1999
:</pre>
</div>
</div>
</li>
<li>
<p>it has Options mandatory set in its server spec</p>
<div class="literalblock">
<div class="content">
<pre>p4 server -o p4d_ha_bos | grep Options
Options: mandatory</pre>
</div>
</div>
</li>
<li>
<p>you have a valid <code>license</code> installed in <code>/p4/1/root</code> (<instance> root)</p>
<div class="paragraph">
<p>On HA machine:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /p4/1/license</pre>
</div>
</div>
</li>
<li>
<p>Monitoring is enabled - so the following works:</p>
<div class="literalblock">
<div class="content">
<pre>p4 monitor show -al</pre>
</div>
</div>
</li>
<li>
<p>DNS changes are possible so that downstream replicas can seamlessly connect to HA server</p>
</li>
<li>
<p>Current <code>pull</code> status is valid</p>
<div class="literalblock">
<div class="content">
<pre>p4 pull -lj</pre>
</div>
</div>
</li>
<li>
<p>You have a valid <code>offline_db</code> for the HA instance</p>
<div class="paragraph">
<p>Check that the sizes of the <code>db.*</code> are similar - compare output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls -lhSr /p4/1/offline_db/db.* | tail
ls -lhSr /p4/1/root/db.* | tail</pre>
</div>
</div>
<div class="paragraph">
<p>Check the current journal counter and compare against live journal counter:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/p4/1/bin/p4d_1 -r /p4/1/offline_db -jd - db.counters | grep journal
p4 counters | grep journal</pre>
</div>
</div>
</li>
<li>
<p>Check all defined triggers will work (see next section) - including Swarm triggers</p>
</li>
<li>
<p>Check authentication will work (e.g. LDAP configuration)</p>
</li>
<li>
<p>Check firewall for HA host - ensure that other replicas will be able to connect on the appropriate port to the instance (using the DNS alias)</p>
</li>
</ol>
</div>
<div class="sect3">
<h4 id="_pre_failover_checklist">2.1.1. Pre Failover Checklist</h4>
<div class="paragraph">
<p>It is important to perform these checks before any planned failover, and also
to make sure they are considered prior any unplanned failover.</p>
</div>
<div class="sect4">
<h5 id="_swarm_triggers">2.1.1.1. Swarm Triggers</h5>
<div class="paragraph">
<p>If Swarm is installed, ensure:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Swarm trigger is installed on HA machine (could be executed from a checked in depot file)</p>
<div class="paragraph">
<p>Typically installed (via package) to <code>/opt/perforce/swarm-triggers/bin/swarm-trigger.pl</code></p>
</div>
<div class="paragraph">
<p>But can be installed anywhere on the filesystem</p>
</div>
<div class="paragraph">
<p>Execute the trigger to ensure that any required Perl modules are installed:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>perl swarm-trigger.pl</pre>
</div>
</div>
<div class="paragraph">
<p>Note that things like <code>JSON.pm</code> can often be installed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install libjson-perl
sudo yum install perl-JSON</pre>
</div>
</div>
</li>
<li>
<p>Swarm trigger configuration file has been copied over from commit server to appropriate place.</p>
<div class="paragraph">
<p>This defaults to one of:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/etc/perforce/swarm-trigger.conf
/opt/perforce/etc/swarm-trigger.conf
swarm-trigger.conf (in the same directory as trigger script)</pre>
</div>
</div>
</li>
</ol>
</div>
</div>
<div class="sect4">
<h5 id="_other_triggers">2.1.1.2. Other Triggers</h5>
<div class="paragraph">
<p>Checklist:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Make sure that the appropriate version of <code>perl</code>, <code>python</code>, <code>ruby</code> etc are installed in locations as referenced by <code>p4 triggers</code> entries.</p>
</li>
<li>
<p>Make sure that any relevant modules have also been installed (e.g. <code>P4Python</code> or <code>P4Perl</code>)</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="_other_replicas_p4target">2.1.1.3. Other Replica’s P4TARGET</h5>
<div class="paragraph">
<p>Review the settings for other replicas and also to check live replicas on the source server of the failover (<code>p4 servers -J</code>)</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 configure show allservers | grep P4TARGET</pre>
</div>
</div>
<div class="paragraph">
<p>Make sure the above settings are using correct DNS alias (which will be redirected).</p>
</div>
</div>
<div class="sect4">
<h5 id="_proxies">2.1.1.4. Proxies</h5>
<div class="paragraph">
<p>These are typically configured via <code>/p4/common/bin/p4_1.vars</code> settings:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>export PROXY_TARGET=</pre>
</div>
</div>
<div class="paragraph">
<p>Ensure the target is the correct DNS alias (which will be redirected).</p>
</div>
</div>
<div class="sect4">
<h5 id="_brokers">2.1.1.5. Brokers</h5>
<div class="paragraph">
<p>These are typically configured via <code>/p4/common/config/p4_1.broker.cfg</code></p>
</div>
<div class="paragraph">
<p>Ensure the config file correctly identifies the appropriate <code>target</code> server using correct DNS alias (which will be redirected).</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_failing_over">2.2. Failing over</h3>
<div class="paragraph">
<p>The basic actions are the same for Unix and Windows, but there are extra steps required as noted in <a href="#_failing_over_on_windows">Section 2.4, “Failing over on Windows”</a></p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Run <code>p4 failover</code> in reporting mode on HA machine:</p>
<div class="literalblock">
<div class="content">
<pre>p4 failover</pre>
</div>
</div>
<div class="paragraph">
<p>Successful output looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Checking if failover might be possible ...
Checking for archive file content not transferred ...
Verifying content of recently update archive files ...
After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.</pre>
</div>
</div>
</li>
<li>
<p>Perform failover:</p>
<div class="literalblock">
<div class="content">
<pre>p4 failover --yes</pre>
</div>
</div>
<div class="paragraph">
<p>Output should be something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Starting failover process ...
Refusing new commands on server from which failover is occurring ...
Giving commands already running time to complete ...
Stalling commands on server from which failover is occurring ...
Waiting for 'journalcopy' to complete its work ...
Waiting for 'pull -L' to complete its work ...
Waiting for 'pull -u' to complete its work ...
Checking for archive file content not transferred ...
Verifying content of recently updated archive files ...
Stopping server from which failover is occurring ...
Moving latest journalcopy'd journal into place as the active journal ...
Updating configuration of the failed-over server ...
Restarting this server ...</pre>
</div>
</div>
<div class="paragraph">
<p>During this time, if you run commands against the master, you may see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Server currently in failover mode, try again after failover has completed</pre>
</div>
</div>
</li>
<li>
<p>Change the DNS entries so downstream replicas (and users) will connect to the new master (that was previously HA)</p>
</li>
<li>
<p>Validate that your downstream replicas are communicating with your new master</p>
<div class="paragraph">
<p>On each replica machine:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 pull -lj</pre>
</div>
</div>
<div class="paragraph">
<p>Or against the new master:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 servers -J</pre>
</div>
</div>
<div class="paragraph">
<p>Check output of <code>p4 info</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>:
Server address: p4d-bos-02
:
ServerID: master_1
Server services: commit-server
:</pre>
</div>
</div>
</li>
<li>
<p>Make sure the old server spec (<code>p4d_ha_bos</code>) has correctly had its <code>Options:</code> field set to <code>nomandatory</code> (otherwise all replication would stop!)</p>
</li>
</ol>
</div>
</div>
<div class="sect2">
<h3 id="_post_failover">2.3. Post Failover</h3>
<div class="sect3">
<h4 id="_moving_of_checkpoints">2.3.1. Moving of Checkpoints</h4>
<div class="paragraph">
<p>After failing over, on Unix there will be journals which need to be copied/moved and renamed due to the SDP structure.</p>
</div>
<div class="paragraph">
<p>For example, an HA server might have stored its journals in <code>/p4/1/checkpoints.ha_bos</code> (assuming it was create by <code>mkrep.sh</code> with serverid <code>p4d_ha_bos</code>):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/p4/1/checkpoints.ha_bos/p4_1.ha_bos.jnl.123
/p4/1/checkpoints.ha_bos/p4_1.ha_bos.jnl.124</pre>
</div>
</div>
<div class="paragraph">
<p>As a result of failover, these files need to be copied/moved to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/p4/1/checkpoints/p4_1.jnl.123
/p4/1/checkpoints/p4_1.jnl.124</pre>
</div>
</div>
<div class="paragraph">
<p>The reason different directories is used is because in some installations the <code>/hxdepots</code> filesystem is shared on NFS between commit server and HA server, and we don’t want to risk overwriting of these files.</p>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
If these files are not present, then normal SDP crontab tasks such as <code>daily_checkpoint.sh</code> will fail as they won’t be able to find the required journals to be applied to the <code>offline_db</code>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>The following command will rotate journal and replay any missing ones to <code>offline_db</code> (it is both fairly quick and safe to run without placing much load on the server host as it doesn’t do any checkpointing):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/p4/common/bin/rotate_journal.sh 1</pre>
</div>
</div>
<div class="paragraph">
<p>If it fails, then check <code>/p4/1/logs/checkpoint.log</code> for details - it may have an error such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Replay journal /p4/1/checkpoints/p4_1.jnl.123 to offline db.</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Perforce server error:
open for read: /p4/1/checkpoints/p4_1.jnl.123: No such file or directory</pre>
</div>
</div>
<div class="paragraph">
<p>This indicates missing journals which will need to be moved/copied as above.</p>
</div>
</div>
<div class="sect3">
<h4 id="_check_on_replication">2.3.2. Check on Replication</h4>
<div class="paragraph">
<p>We recommend that you connect to all your replicas/proxies/brokers and make sure that they are successfully working after failover.</p>
</div>
<div class="paragraph">
<p>It is surprisingly common to find forgotten configuration details meaning that they are attempting to connect to old server for example!</p>
</div>
<div class="paragraph">
<p>For proxies and brokers - you probably just need to run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 info</pre>
</div>
</div>
<div class="paragraph">
<p>For downstream replicas of any type, we recommend logging on to the host and running:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 pull -lj</pre>
</div>
</div>
<div class="paragraph">
<p>and checking for any errors.</p>
</div>
<div class="paragraph">
<p>We also recommend the following is executed on both HA server and all replicas and the output examined for any unexpected errors:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep -A4 error: /p4/1/logs/log</pre>
</div>
</div>
<div class="paragraph">
<p>Or you can review the contents of <code>/p4/1/logs/errors.csv</code> if you have enabled structured logging.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_failing_over_on_windows">2.4. Failing over on Windows</h3>
<div class="paragraph">
<p>The basic steps are the same as for on Unix, but with some extra steps at the end.</p>
</div>
<div class="paragraph">
<p>After the <code>p4 failover --yes</code> command has completed its work (on the HA server machine):</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Review the settings for the Windows service (examples are for instance <code>1</code>) - note below <strong>-S</strong> is uppercase:</p>
<div class="literalblock">
<div class="content">
<pre>p4 set -S p4_1</pre>
</div>
</div>
<div class="paragraph">
<p>Example results:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>C:\p4\1>p4 set -S p4_1
:
P4JOURNAL=c:\p4\1\logs\journal (set -S)
P4LOG=c:\p4\1\logs\p4d_ha_aws.log (set -S)
P4NAME=p4d_ha_aws (set -S)
P4PORT=1666 (set -S)
P4ROOT=c:\p4\1\root (set -S)
:</pre>
</div>
</div>
</li>
<li>
<p>Change the value of <code>P4NAME</code> and <code>P4LOG</code> to correct value for <code>master</code>:</p>
<div class="literalblock">
<div class="content">
<pre>p4 set -S p4_1 P4NAME=master
p4 set -S p4_1 P4LOG=c:\p4\1\logs\master.log</pre>
</div>
</div>
<div class="paragraph">
<p>And re-check the output of <code>p4 set -S p4_1</code></p>
</div>
</li>
<li>
<p>Restart the service:</p>
<div class="literalblock">
<div class="content">
<pre>c:\p4\common\bin\svcinst stop -n p4_1
c:\p4\common\bin\svcinst start -n p4_1</pre>
</div>
</div>
</li>
<li>
<p>Run <code>p4 configure show</code> to check that the output is as expected for the above values.</p>
</li>
</ol>
</div>
<div class="sect3">
<h4 id="_post_failover_on_windows">2.4.1. Post Failover on Windows</h4>
<div class="paragraph">
<p>This is slightly different for the Windows and Linux SDP, since there is currently no equivalent of <code>mkrep.sh</code> for Windows, and replication topologies on Windows are typically smaller and simpler.</p>
</div>
<div class="paragraph">
<p>Thus it is Windows HA instances are likely to have journals rotated into <code>c:\p4\1\checkpoints</code> instead of something like <code>c:\p4\1\checkpoints.ha_bos</code>.</p>
</div>
<div class="paragraph">
<p>However, it is still worth ensuring things like:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>offline_db</code> on HA is up-to-date</p>
</li>
<li>
<p>all triggers (including any Swarm triggers) are appropriately configured</p>
</li>
<li>
<p><code>daily_backup.bat</code> will work appropriate after failover</p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_unplanned_failover">3. Unplanned Failover</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In this case there is no active participation of upstream server, so there is an increased risk of lost data.</p>
</div>
<div class="paragraph">
<p>We assume we are still failing over to the HA machine, so:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Failover target is <code>standby</code> or <code>forwarding-standby</code></p>
</li>
<li>
<p>Server spec still has <code>Options:</code> set to <code>mandatory</code></p>
</li>
<li>
<p>Original master is not running</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The output of <code>p4 failover</code> on the DR machine might be:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Checking if failover might be possible ...
Server ID must be specified in the '-s' or --serverid' argument for a failover without the participation of the server from which failover is occurring.
Checking for archive file content not transferred ...
Verifying content of recently update archive files ...
After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.</pre>
</div>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Execute <code>p4 failover</code> with the extra parameter to specify server we are failing over from:</p>
<div class="literalblock">
<div class="content">
<pre>p4 failover --serverid master_1 --yes</pre>
</div>
</div>
</li>
</ol>
</div>
<div class="paragraph">
<p>Expected output is somewhat shorter than for planned failover:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Starting failover process ...
Waiting for 'pull -L' to complete its work ...
Checking for archive file content not transferred ...
Verifying content of recently updated archive files ...
Moving latest journalcopy'd journal into place as the active journal ...
Updating configuration of the failed-over server ...
Restarting this server ...</pre>
</div>
</div>
<div class="sect2">
<h3 id="_post_unplanned_failover">3.1. Post Unplanned Failover</h3>
<div class="paragraph">
<p>This is similar to <a href="#_post_failover">Section 2.3, “Post Failover”</a> with the exception of the next section below.</p>
</div>
<div class="sect3">
<h4 id="_resetting_downstream_replicas">3.1.1. Resetting Downstream Replicas</h4>
<div class="paragraph">
<p>In an unplanned failover scenario it is possible that there is a journal synchronisation problem with downstream replicas.</p>
</div>
<div class="paragraph">
<p>The output of <code>p4 pull -lj</code> may indicate an error, and/or there may be errors in the log:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep -A4 error: /p4/1/logs/log | less</pre>
</div>
</div>
<div class="paragraph">
<p>If you need to reset the replica to restart from the beginning of the current journal it is attempting to pull, then the process is:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Stop the replica:</p>
<div class="literalblock">
<div class="content">
<pre>/p4/1/bin/p4d_1_init stop</pre>
</div>
</div>
</li>
<li>
<p>Remove the <code>state</code> file:</p>
<div class="literalblock">
<div class="content">
<pre>cd /p4/1/root
mv state save/</pre>
</div>
</div>
</li>
<li>
<p>Restart the replica</p>
<div class="literalblock">
<div class="content">
<pre>/p4/1/bin/p4d_1_init start</pre>
</div>
</div>
</li>
<li>
<p>Recheck the log for errors as above.</p>
</li>
</ol>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_unplanned_failover_on_windows">3.2. Unplanned Failover on Windows</h3>
<div class="paragraph">
<p>The extra steps required are basically the same as in <a href="#_failing_over_on_windows">Section 2.4, “Failing over on Windows”</a> as well as the steps in <a href="#_post_unplanned_failover">Section 3.1, “Post Unplanned Failover”</a></p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_old_style_failover">4. Old style failover</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This does not use the <code>p4 failover</code> command (so is valid for pre-2018.2 p4d versions)</p>
</div>
<div class="paragraph">
<p>See: <a href="https://community.perforce.com/s/article/2495">KB - Failing over to a Replica</a></p>
</div>
</div>
</div>
</div>
<div id="footer">
<div id="footer-text">
Version v2020.1<br>
Last updated 2020-09-12 15:46:26 +0100
</div>
</div>
</body>
</html> | # | Change | User | Description | Committed | |
|---|---|---|---|---|---|
| #20 | 29611 | tom_tyler |
Updated Version to release SDP 2023.1.29610. Re-generated docs. |
||
| #19 | 29442 | tom_tyler |
Updated Version to release SDP 2022.2.29441. Re-generated docs. |
||
| #18 | 29414 | robert_cowham | Added a failback section | ||
| #17 | 29251 | tom_tyler |
Updated Version to release SDP 2022.2.29250. Re-generated docs. |
||
| #16 | 28988 | tom_tyler |
Updated Version to release SDP 2022.1.28987. Re-generated docs. |
||
| #15 | 28840 | tom_tyler |
Updated Version to release SDP 2022.1.28839. Re-generated docs. |
||
| #14 | 28411 | tom_tyler |
Updated Version to release SDP 2021.2.28410. Re-generated docs. |
||
| #13 | 28239 | tom_tyler |
Updated Version to release SDP 2021.1.28238. Re-generated docs. |
||
| #12 | 28099 | robert_cowham | Updated Failover Guide to add a couple of post failover checks. | ||
| #11 | 27920 | tom_tyler |
Updated Version to release SDP 2020.1.27919. Re-generated docs. |
||
| #10 | 27725 | tom_tyler | Re-generated HTML and PDF from adoc files. | ||
| #9 | 27722 | tom_tyler |
Refinements to @27712: * Resolved one out-of-date file (verify_sdp.sh). * Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc). * Updated revdate/revnumber in *.adoc files. * Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md. * Bumped version numbers on scripts with Version= def'n. * Generated HTML, PDF, and doc/gen files: - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility. - HTML for Perl scripts is generated with pod2html. - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh. #review-27712 |
||
| #8 | 27709 | robert_cowham | Note check for serverlocks. Fix typo in path in failover. | ||
| #7 | 27213 | tom_tyler | Regenerated docs. | ||
| #6 | 26851 | robert_cowham |
Fix typo in tmpfs /etc/fstab entry which stopped it working in the doc.
Mention in pre-requisites for failover and failover guide the need to review OS Config for your failover server. Document Ubuntu 2020.04 LTS and CentOS/RHEL 8 support. Note performance has been observed to be better with CentOS. Document pull.sh and submit.sh in main SDP guide (remove from Unsupported doc). Update comments in triggers to reflect that they are reference implementations, not just examples. No code change. |
||
| #5 | 26747 | robert_cowham |
Update with some checklists for failover to ensure valid. Update to v2020.1 Add Usage sections where missing to Unix guide Refactor the content in Unix guide to avoid repetition and make things read more sensibly. |
||
| #4 | 26728 | robert_cowham | Change filetype to +w for make. | ||
| #3 | 26727 | robert_cowham |
Add section on server host naming conventions Clarify HA and DR, and update links across docs Fix doc structure for Appendix numbering |
||
| #2 | 26660 | robert_cowham |
New common Failover Guide - removed old one. This is based on 'p4 failover' so requires 2018.2 or later. |
||
| #1 | 26654 | robert_cowham |
First draft of new Failover Guide using "p4 failover" Linked from SDP Unix Guide |