<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="generator" content="Asciidoctor 2.0.17"> <meta name="author" content="Perforce Professional Services"> <title>Server Deployment Package (SDP) for Perforce Helix: SDP Failover Guide (for Unix and Windows)</title> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"> <style> /*! Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */ /* Uncomment the following line when using as a custom stylesheet */ /* @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"; */ html{font-family:sans-serif;-webkit-text-size-adjust:100%} a{background:none} a:focus{outline:thin dotted} a:active,a:hover{outline:0} h1{font-size:2em;margin:.67em 0} b,strong{font-weight:bold} abbr{font-size:.9em} abbr[title]{cursor:help;border-bottom:1px dotted #dddddf;text-decoration:none} dfn{font-style:italic} hr{height:0} mark{background:#ff0;color:#000} code,kbd,pre,samp{font-family:monospace;font-size:1em} pre{white-space:pre-wrap} q{quotes:"\201C" "\201D" "\2018" "\2019"} small{font-size:80%} sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline} sup{top:-.5em} sub{bottom:-.25em} img{border:0} svg:not(:root){overflow:hidden} figure{margin:0} audio,video{display:inline-block} audio:not([controls]){display:none;height:0} fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em} legend{border:0;padding:0} button,input,select,textarea{font-family:inherit;font-size:100%;margin:0} button,input{line-height:normal} button,select{text-transform:none} button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer} button[disabled],html input[disabled]{cursor:default} input[type=checkbox],input[type=radio]{padding:0} button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0} textarea{overflow:auto;vertical-align:top} table{border-collapse:collapse;border-spacing:0} *,::before,::after{box-sizing:border-box} html,body{font-size:100%} body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;line-height:1;position:relative;cursor:auto;-moz-tab-size:4;-o-tab-size:4;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased} a:hover{cursor:pointer} img,object,embed{max-width:100%;height:auto} object,embed{height:100%} img{-ms-interpolation-mode:bicubic} .left{float:left!important} .right{float:right!important} .text-left{text-align:left!important} .text-right{text-align:right!important} .text-center{text-align:center!important} .text-justify{text-align:justify!important} .hide{display:none} img,object,svg{display:inline-block;vertical-align:middle} textarea{height:auto;min-height:50px} select{width:100%} .subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em} div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0} a{color:#2156a5;text-decoration:underline;line-height:inherit} a:hover,a:focus{color:#1d4b8f} a img{border:0} p{line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility} p aside{font-size:.875em;line-height:1.35;font-style:italic} h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em} h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0} h1{font-size:2.125em} h2{font-size:1.6875em} h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em} h4,h5{font-size:1.125em} h6{font-size:1em} hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em} em,i{font-style:italic;line-height:inherit} strong,b{font-weight:bold;line-height:inherit} small{font-size:60%;line-height:inherit} code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)} ul,ol,dl{line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit} ul,ol{margin-left:1.5em} ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0} ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit} ul.square{list-style-type:square} ul.circle{list-style-type:circle} ul.disc{list-style-type:disc} ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0} dl dt{margin-bottom:.3125em;font-weight:bold} dl dd{margin-bottom:1.25em} blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd} blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)} @media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2} h1{font-size:2.75em} h2{font-size:2.3125em} h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em} h4{font-size:1.4375em}} table{background:#fff;margin-bottom:1.25em;border:1px solid #dedede;word-wrap:normal} table thead,table tfoot{background:#f7f8f7} table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left} table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)} table tr.even,table tr.alt{background:#f8f8f7} table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{line-height:1.6} h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em} h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400} .center{margin-left:auto;margin-right:auto} .stretch{width:100%} .clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table} .clearfix::after,.float-group::after{clear:both} :not(pre).nobreak{word-wrap:normal} :not(pre).nowrap{white-space:nowrap} :not(pre).pre-wrap{white-space:pre-wrap} :not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed} pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed} pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit} pre>code{display:block} pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal} em em{font-style:normal} strong strong{font-weight:400} .keyseq{color:rgba(51,51,51,.8)} kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;border-radius:3px;box-shadow:0 1px 0 rgba(0,0,0,.2),inset 0 0 0 .1em #fff;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap} .keyseq kbd:first-child{margin-left:0} .keyseq kbd:last-child{margin-right:0} .menuseq,.menuref{color:#000} .menuseq b:not(.caret),.menuref{font-weight:inherit} .menuseq{word-spacing:-.02em} .menuseq b.caret{font-size:1.25em;line-height:.8} .menuseq i.caret{font-weight:bold;text-align:center;width:.45em} b.button::before,b.button::after{position:relative;top:-1px;font-weight:400} b.button::before{content:"[";padding:0 3px 0 2px} b.button::after{content:"]";padding:0 2px 0 3px} p a>code:hover{color:rgba(0,0,0,.9)} #header,#content,#footnotes,#footer{width:100%;margin:0 auto;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em} #header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table} #header::after,#content::after,#footnotes::after,#footer::after{clear:both} #content{margin-top:1.25em} #content::before{content:none} #header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0} #header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf} #header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px} #header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:flex;flex-flow:row wrap} #header .details span:first-child{margin-left:-.125em} #header .details span.email a{color:rgba(0,0,0,.85)} #header .details br{display:none} #header .details br+span::before{content:"\00a0\2013\00a0"} #header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)} #header .details br+span#revremark::before{content:"\00a0|\00a0"} #header #revnumber{text-transform:capitalize} #header #revnumber::after{content:"\00a0"} #content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem} #toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em} #toc>ul{margin-left:.125em} #toc ul.sectlevel0>li>a{font-style:italic} #toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0} #toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none} #toc li{line-height:1.3334;margin-top:.3334em} #toc a{text-decoration:none} #toc a:active{text-decoration:underline} #toctitle{color:#7a2518;font-size:1.2em} @media screen and (min-width:768px){#toctitle{font-size:1.375em} body.toc2{padding-left:15em;padding-right:0} #toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto} #toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em} #toc.toc2>ul{font-size:.9em;margin-bottom:0} #toc.toc2 ul ul{margin-left:0;padding-left:1em} #toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em} body.toc2.toc-right{padding-left:0;padding-right:15em} body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}} @media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0} #toc.toc2{width:20em} #toc.toc2 #toctitle{font-size:1.375em} #toc.toc2>ul{font-size:.95em} #toc.toc2 ul ul{padding-left:1.25em} body.toc2.toc-right{padding-left:0;padding-right:20em}} #content #toc{border:1px solid #e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;border-radius:4px} #content #toc>:first-child{margin-top:0} #content #toc>:last-child{margin-bottom:0} #footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em} #footer-text{color:hsla(0,0%,100%,.8);line-height:1.44} #content{margin-bottom:.625em} .sect1{padding-bottom:.625em} @media screen and (min-width:768px){#content{margin-bottom:1.25em} .sect1{padding-bottom:1.25em}} .sect1:last-child{padding-bottom:0} .sect1+.sect1{border-top:1px solid #e7e7e9} #content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400} #content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em} #content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible} #content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none} #content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221} details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em} details{margin-left:1.25rem} details>summary{cursor:pointer;display:block;position:relative;line-height:1.6;margin-bottom:.625rem;outline:none;-webkit-tap-highlight-color:transparent} details>summary::-webkit-details-marker{display:none} details>summary::before{content:"";border:solid transparent;border-left:solid;border-width:.3em 0 .3em .5em;position:absolute;top:.5em;left:-1.25rem;transform:translateX(15%)} details[open]>summary::before{border:solid transparent;border-top:solid;border-width:.5em .3em 0;transform:translateY(15%)} details>summary::after{content:"";width:1.25rem;height:1em;position:absolute;top:.3em;left:-1.25rem} .admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic} table.tableblock.fit-content>caption.title{white-space:nowrap;width:0} .paragraph.lead>p,#preamble>.sectionbody>[class=paragraph]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)} .admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%} .admonitionblock>table td.icon{text-align:center;width:80px} .admonitionblock>table td.icon img{max-width:none} .admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase} .admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6);word-wrap:anywhere} .admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0} .exampleblock>.content{border:1px solid #e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;border-radius:4px} .exampleblock>.content>:first-child{margin-top:0} .exampleblock>.content>:last-child{margin-bottom:0} .sidebarblock{border:1px solid #dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;border-radius:4px} .sidebarblock>:first-child{margin-top:0} .sidebarblock>:last-child{margin-bottom:0} .sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center} .exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0} .literalblock pre,.listingblock>.content>pre{border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em} @media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}} @media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}} .literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class=highlight],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8} .literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)} .listingblock>.content{position:relative} .listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5} .listingblock:hover code[data-lang]::before{display:block} .listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5} .listingblock.terminal pre .command:not([data-prompt])::before{content:"$"} .listingblock pre.highlightjs{padding:0} .listingblock pre.highlightjs>code{padding:1em;border-radius:4px} .listingblock pre.prettyprint{border-width:0} .prettyprint{background:#f7f7f8} pre.prettyprint .linenums{line-height:1.45;margin-left:2em} pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0} pre.prettyprint li code[data-lang]::before{opacity:1} pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none} table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none} table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal} table.linenotable td.code{padding-left:.75em} table.linenotable td.linenos,pre.pygments .linenos{border-right:1px solid;opacity:.35;padding-right:.5em;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none} pre.pygments span.linenos{display:inline-block;margin-right:.75em} .quoteblock{margin:0 1em 1.25em 1.5em;display:table} .quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em} .quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify} .quoteblock blockquote{margin:0;padding:0;border:0} .quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)} .quoteblock blockquote>.paragraph:last-child p{margin-bottom:0} .quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right} .verseblock{margin:0 1em 1.25em} .verseblock pre{font-family:"Open Sans","DejaVu Sans",sans-serif;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility} .verseblock pre strong{font-weight:400} .verseblock .attribution{margin-top:1.25rem;margin-left:.5ex} .quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic} .quoteblock .attribution br,.verseblock .attribution br{display:none} .quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)} .quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none} .quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0} .quoteblock.abstract{margin:0 1em 1.25em;display:block} .quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center} .quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf} .quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0} .quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem} .quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;font-size:.85rem;text-align:left;margin-right:0} p.tableblock:last-child{margin-bottom:0} td.tableblock>.content{margin-bottom:1.25em;word-wrap:anywhere} td.tableblock>.content>:last-child{margin-bottom:-1.25em} table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede} table.grid-all>*>tr>*{border-width:1px} table.grid-cols>*>tr>*{border-width:0 1px} table.grid-rows>*>tr>*{border-width:1px 0} table.frame-all{border-width:1px} table.frame-ends{border-width:1px 0} table.frame-sides{border-width:0 1px} table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0} table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0} table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0} table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0} table.stripes-all>*>tr,table.stripes-odd>*>tr:nth-of-type(odd),table.stripes-even>*>tr:nth-of-type(even),table.stripes-hover>*>tr:hover{background:#f8f8f7} th.halign-left,td.halign-left{text-align:left} th.halign-right,td.halign-right{text-align:right} th.halign-center,td.halign-center{text-align:center} th.valign-top,td.valign-top{vertical-align:top} th.valign-bottom,td.valign-bottom{vertical-align:bottom} th.valign-middle,td.valign-middle{vertical-align:middle} table thead th,table tfoot th{font-weight:bold} tbody tr th{background:#f7f8f7} tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold} p.tableblock>code:only-child{background:none;padding:0} p.tableblock{font-size:1em} ol{margin-left:1.75em} ul li ol{margin-left:1.5em} dl dd{margin-left:1.125em} dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0} li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em} ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none} ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em} ul.unstyled,ol.unstyled{margin-left:0} li>p:empty:only-child::before{content:"";display:inline-block} ul.checklist>li>p:first-child{margin-left:-1em} ul.checklist>li>p:first-child>.fa-square-o:first-child,ul.checklist>li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em} ul.checklist>li>p:first-child>input[type=checkbox]:first-child{margin-right:.25em} ul.inline{display:flex;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em} ul.inline>li{margin-left:1.25em} .unstyled dl dt{font-weight:400;font-style:normal} ol.arabic{list-style-type:decimal} ol.decimal{list-style-type:decimal-leading-zero} ol.loweralpha{list-style-type:lower-alpha} ol.upperalpha{list-style-type:upper-alpha} ol.lowerroman{list-style-type:lower-roman} ol.upperroman{list-style-type:upper-roman} ol.lowergreek{list-style-type:lower-greek} .hdlist>table,.colist>table{border:0;background:none} .hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none} td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em} td.hdlist1{font-weight:bold;padding-bottom:1.25em} td.hdlist2{word-wrap:anywhere} .literalblock+.colist,.listingblock+.colist{margin-top:-.5em} .colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top} .colist td:not([class]):first-child img{max-width:none} .colist td:not([class]):last-child{padding:.25em 0} .thumb,.th{line-height:0;display:inline-block;border:4px solid #fff;box-shadow:0 0 0 1px #ddd} .imageblock.left{margin:.25em .625em 1.25em 0} .imageblock.right{margin:.25em 0 1.25em .625em} .imageblock>.title{margin-bottom:0} .imageblock.thumb,.imageblock.th{border-width:6px} .imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em} .image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0} .image.left{margin-right:.625em} .image.right{margin-left:.625em} a.image{text-decoration:none;display:inline-block} a.image object{pointer-events:none} sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super} sup.footnote a,sup.footnoteref a{text-decoration:none} sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline} #footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em} #footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0} #footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em} #footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em} #footnotes .footnote:last-of-type{margin-bottom:0} #content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0} div.unbreakable{page-break-inside:avoid} .big{font-size:larger} .small{font-size:smaller} .underline{text-decoration:underline} .overline{text-decoration:overline} .line-through{text-decoration:line-through} .aqua{color:#00bfbf} .aqua-background{background:#00fafa} .black{color:#000} .black-background{background:#000} .blue{color:#0000bf} .blue-background{background:#0000fa} .fuchsia{color:#bf00bf} .fuchsia-background{background:#fa00fa} .gray{color:#606060} .gray-background{background:#7d7d7d} .green{color:#006000} .green-background{background:#007d00} .lime{color:#00bf00} .lime-background{background:#00fa00} .maroon{color:#600000} .maroon-background{background:#7d0000} .navy{color:#000060} .navy-background{background:#00007d} .olive{color:#606000} .olive-background{background:#7d7d00} .purple{color:#600060} .purple-background{background:#7d007d} .red{color:#bf0000} .red-background{background:#fa0000} .silver{color:#909090} .silver-background{background:#bcbcbc} .teal{color:#006060} .teal-background{background:#007d7d} .white{color:#bfbfbf} .white-background{background:#fafafa} .yellow{color:#bfbf00} .yellow-background{background:#fafa00} span.icon>.fa{cursor:default} a span.icon>.fa{cursor:inherit} .admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default} .admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c} .admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111} .admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900} .admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400} .admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000} .conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold} .conum[data-value] *{color:#fff!important} .conum[data-value]+b{display:none} .conum[data-value]::after{content:attr(data-value)} pre .conum[data-value]{position:relative;top:-.125em} b.conum *{color:inherit!important} .conum:not([data-value]):empty{display:none} dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility} h1,h2,p,td.content,span.alt,summary{letter-spacing:-.01em} p strong,td.content strong,div.footnote strong{letter-spacing:-.005em} p,blockquote,dt,td.content,span.alt,summary{font-size:1.0625rem} p{margin-bottom:1.25rem} .sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em} .exampleblock>.content{background:#fffef7;border-color:#e0e0dc;box-shadow:0 1px 4px #e0e0dc} .print-only{display:none!important} @page{margin:1.25cm .75cm} @media print{*{box-shadow:none!important;text-shadow:none!important} html{font-size:80%} a{color:inherit!important;text-decoration:underline!important} a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important} a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em} abbr[title]{border-bottom:1px dotted} abbr[title]::after{content:" (" attr(title) ")"} pre,blockquote,tr,img,object,svg{page-break-inside:avoid} thead{display:table-header-group} svg{max-width:100%} p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3} h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid} #header,#content,#footnotes,#footer{max-width:none} #toc,.sidebarblock,.exampleblock>.content{background:none!important} #toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important} body.book #header{text-align:center} body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em} body.book #header .details{border:0!important;display:block;padding:0!important} body.book #header .details span:first-child{margin-left:0!important} body.book #header .details br{display:block} body.book #header .details br+span::before{content:none!important} body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important} body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always} .listingblock code[data-lang]::before{display:block} #footer{padding:0 .9375em} .hide-on-print{display:none!important} .print-only{display:block!important} .hide-for-print{display:none!important} .show-for-print{display:inherit!important}} @media amzn-kf8,print{#header>h1:first-child{margin-top:1.25rem} .sect1{padding:0!important} .sect1+.sect1{border:0} #footer{background:none} #footer-text{color:rgba(0,0,0,.6);font-size:.9em}} @media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}} </style> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css"> </head> <body class="book"> <div id="header"> <h1>Server Deployment Package (SDP) for Perforce Helix: SDP Failover Guide (for Unix and Windows)</h1> <div class="details"> <span id="author" class="author">Perforce Professional Services</span><br> <span id="email" class="email"><a href="mailto:consulting@perforce.com">consulting@perforce.com</a></span><br> <span id="revnumber">version v2023.1,</span> <span id="revdate">2023-08-07</span> </div> <div id="toc" class="toc"> <div id="toctitle">Table of Contents</div> <ul class="sectlevel1"> <li><a href="#_preface">Preface</a></li> <li><a href="#_overview">1. Overview</a> <ul class="sectlevel2"> <li><a href="#_planning">1.1. Planning</a></li> </ul> </li> <li><a href="#_planned_failover">2. Planned Failover</a> <ul class="sectlevel2"> <li><a href="#_prerequisites">2.1. Prerequisites</a> <ul class="sectlevel3"> <li><a href="#_pre_failover_checklist">2.1.1. Pre Failover Checklist</a> <ul class="sectlevel4"> <li><a href="#_confirm_trust_established_with_replicas">2.1.1.1. Confirm Trust Established with Replicas</a></li> <li><a href="#_swarm_triggers">2.1.1.2. Swarm Triggers</a></li> <li><a href="#_other_triggers">2.1.1.3. Other Triggers</a></li> <li><a href="#_other_replicas_p4target">2.1.1.4. Other Replica’s P4TARGET</a></li> <li><a href="#_proxies">2.1.1.5. Proxies</a></li> <li><a href="#_ha_server_os_configuration">2.1.1.6. HA Server OS Configuration</a></li> </ul> </li> </ul> </li> <li><a href="#_failing_over">2.2. Failing over</a></li> <li><a href="#_post_failover">2.3. Post Failover</a> <ul class="sectlevel3"> <li><a href="#_validation_of_sdp_config">2.3.1. Validation of SDP Config</a></li> <li><a href="#_moving_of_checkpoints">2.3.2. Moving of Checkpoints</a></li> <li><a href="#_check_daily_checkpoint_sh_runs_successfully">2.3.3. Check daily_checkpoint.sh runs successfully</a></li> <li><a href="#_check_on_replication">2.3.4. Check on Replication</a></li> </ul> </li> <li><a href="#_failing_over_on_windows">2.4. Failing over on Windows</a> <ul class="sectlevel3"> <li><a href="#_post_failover_on_windows">2.4.1. Post Failover on Windows</a></li> </ul> </li> <li><a href="#_preparation_for_failing_back_setup_original_commit_server_as_new_failover_replica">2.5. Preparation for Failing Back - Setup original commit server as new Failover replica</a></li> </ul> </li> <li><a href="#_unplanned_failover">3. Unplanned Failover</a> <ul class="sectlevel2"> <li><a href="#_post_unplanned_failover">3.1. Post Unplanned Failover</a> <ul class="sectlevel3"> <li><a href="#_resetting_downstream_replicas">3.1.1. Resetting Downstream Replicas</a></li> </ul> </li> <li><a href="#_unplanned_failover_on_windows">3.2. Unplanned Failover on Windows</a></li> </ul> </li> </ul> </div> </div> <div id="content"> <div class="sect1"> <h2 id="_preface">Preface</h2> <div class="sectionbody"> <div class="paragraph"> <p>The Server Deployment Package (SDP) is the implementation of Perforce’s recommendations for operating and managing a production Perforce Helix Core Version Control System. It is intended to provide the Helix Core administration team with tools to help:</p> </div> <div class="ulist"> <ul> <li> <p>High Availability (HA)</p> </li> <li> <p>Disaster Recovery (DR)</p> </li> </ul> </div> <div class="paragraph"> <p>This guide is intended to provide instructions for failover in an SDP environment using built in Helix Core features.</p> </div> <div class="paragraph"> <p>For more details see:</p> </div> <div class="ulist"> <ul> <li> <p><a href="https://www.perforce.com/manuals/p4sag/Content/P4SAG/failover.html#Failover">Sysadmin Guide - Failover</a></p> </li> </ul> </div> <div class="paragraph"> <p><strong>Please Give Us Feedback</strong></p> </div> <div class="paragraph"> <p>Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to <a href="mailto:consulting@perforce.com">consulting@perforce.com</a>.</p> </div> </div> </div> <div class="sect1"> <h2 id="_overview">1. Overview</h2> <div class="sectionbody"> <div class="paragraph"> <p>We need to consider planned vs unplanned failover. Planned may be due to upgrading the core Operating System or some other dependency in your infrastructure, or a similar activity.</p> </div> <div class="paragraph"> <p>Unplanned covers risks you are seeking to mitigate with failover:</p> </div> <div class="ulist"> <ul> <li> <p>loss of a machine, or some machine related hardware failure</p> </li> <li> <p>loss of a VM cluster</p> </li> <li> <p>failure of storage</p> </li> <li> <p>loss of a data center or machine room</p> </li> <li> <p>etc…​</p> </li> </ul> </div> <div class="paragraph"> <p>See also <a href="https://www.perforce.com/manuals/cmdref/Content/CmdRef/p4_failover.html">p4 failover in Command Reference Guide</a></p> </div> <div class="paragraph"> <p>Please refer to the following sections:</p> </div> <div class="ulist"> <ul> <li> <p><a href="SDP_Guide.Unix.html#_planning_for_ha_and_dr">SDP Guide: Planning for HA and DR</a></p> </li> <li> <p><a href="SDP_Guide.Unix.html#_pre_requisites_for_failover">SDP Guide: Pre-requisites for Failover</a></p> </li> </ul> </div> <div class="sect2"> <h3 id="_planning">1.1. Planning</h3> <div class="paragraph"> <p>HA failover should not require a P4PORT change for end users. Depending on your topology, you can avoid changing P4PORT by having users set their P4PORT to a DNS name that is centrally managed.</p> </div> <div class="paragraph"> <p>For example:</p> </div> <div class="ulist"> <ul> <li> <p><code>bos-helix-01</code>, a master/commit-server in Boston, pointed to by a DNS name like <code>perforce</code> or <code>perforce.p4demo.com</code>.</p> </li> <li> <p><code>bos-helix-02</code>, a standby replica in Boston, not pointed to by a DNS until failover. In event of failover, it gets pointed to by <code>perforce</code>/<code>perforce.p4demo.com</code> via a DNS change.</p> </li> </ul> </div> <div class="paragraph"> <p>See <a href="SDP_Guide.Unix.html#_server_host_naming_conventions">SDP Guide: Server Host Naming Conventions</a></p> </div> <div class="paragraph"> <p>Other advanced networking options might be possible if you talk to your local networking gurus (virtual IPs etc).</p> </div> </div> </div> </div> <div class="sect1"> <h2 id="_planned_failover">2. Planned Failover</h2> <div class="sectionbody"> <div class="paragraph"> <p>In this instance you can run <code>p4 failover</code> with the active participation of its upstream server.</p> </div> <div class="paragraph"> <p>We are going to provide examples with the following assumptions:</p> </div> <div class="ulist"> <ul> <li> <p>ServerID <code>master_1</code> is current commit, running on machine <code>p4d-bos-01</code></p> </li> <li> <p>ServerID <code>p4d_ha_bos</code> is HA server</p> </li> <li> <p>DNS alias <code>perforce</code> is set to <code>p4d-bos-01</code></p> </li> </ul> </div> <div class="sect2"> <h3 id="_prerequisites">2.1. Prerequisites</h3> <div class="paragraph"> <p>You need to ensure:</p> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>you are running p4d 2018.2 or later for your commit and all replica instances, preferably 2020.1+</p> <div class="literalblock"> <div class="content"> <pre>source /p4/common/bin/p4_vars 1 p4 info | grep version</pre> </div> </div> </li> <li> <p>your failover target server instance is of type <code>standby</code> or <code>forwarding-standby</code></p> <div class="paragraph"> <p>On HA machine:</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 info : ServerID: p4d_ha_bos Server services: standby Replica of: perforce:1999 :</pre> </div> </div> </li> <li> <p>it has Options mandatory set in its server spec</p> <div class="literalblock"> <div class="content"> <pre>p4 server -o p4d_ha_bos | grep Options Options: mandatory</pre> </div> </div> </li> <li> <p>you have a valid <code>license</code> installed in <code>/p4/1/root</code> (<instance> root)</p> <div class="paragraph"> <p>On HA machine:</p> </div> <div class="literalblock"> <div class="content"> <pre>cat /p4/1/root/license</pre> </div> </div> </li> <li> <p>Monitoring is enabled - so the following works:</p> <div class="literalblock"> <div class="content"> <pre>p4 monitor show -al</pre> </div> </div> </li> <li> <p>DNS changes are possible so that downstream replicas can seamlessly connect to HA server</p> </li> <li> <p>Current <code>pull</code> status is valid</p> <div class="literalblock"> <div class="content"> <pre>p4 pull -lj</pre> </div> </div> </li> <li> <p>You have a valid <code>offline_db</code> for the HA instance</p> <div class="paragraph"> <p>Check that the sizes of the <code>db.*</code> are similar - compare output:</p> </div> <div class="literalblock"> <div class="content"> <pre>ls -lhSr /p4/1/offline_db/db.* | tail ls -lhSr /p4/1/root/db.* | tail</pre> </div> </div> <div class="paragraph"> <p>Check the current journal counter and compare against live journal counter:</p> </div> <div class="literalblock"> <div class="content"> <pre>/p4/1/bin/p4d_1 -r /p4/1/offline_db -jd - db.counters | grep journal p4 counters | grep journal</pre> </div> </div> </li> <li> <p>Check all defined triggers will work (see next section) - including Swarm triggers</p> </li> <li> <p>Check authentication will work (e.g. LDAP configuration)</p> </li> <li> <p>Check firewall for HA host - ensure that other replicas will be able to connect on the appropriate port to the instance (using the DNS alias)</p> </li> </ol> </div> <div class="sect3"> <h4 id="_pre_failover_checklist">2.1.1. Pre Failover Checklist</h4> <div class="paragraph"> <p>It is important to perform these checks before any planned failover, and also to make sure they are considered prior any unplanned failover.</p> </div> <div class="sect4"> <h5 id="_confirm_trust_established_with_replicas">2.1.1.1. Confirm Trust Established with Replicas</h5> <div class="paragraph"> <p>If using <code>ssl</code> on the commit server, then downstream replicas with have already set up <code>P4TRUST</code> entries with that server and its fingerprint.</p> </div> <div class="paragraph"> <p>After failover, you need to check that all replicas have appropriate <code>p4 trust</code> setup with the HA server (which is now the commit server).</p> </div> <div class="paragraph"> <p>If you are using <strong><em>background submit</em></strong> between edge and commit servers, then you need to confirm the following:</p> </div> <div class="ulist"> <ul> <li> <p><code>p4 trust</code> is setup on the HA (new commit) with all edge servers</p> </li> <li> <p>the commit service user is correctly logged on to all edge servers (make sure the configurable <code>auth.id</code> is appropriately set to avoid IP-only tickets) on the HA server before failover</p> </li> </ul> </div> </div> <div class="sect4"> <h5 id="_swarm_triggers">2.1.1.2. Swarm Triggers</h5> <div class="paragraph"> <p>If Swarm is installed, ensure:</p> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Swarm trigger is installed on HA machine (could be executed from a checked in depot file)</p> <div class="paragraph"> <p>Typically installed (via package) to <code>/opt/perforce/swarm-triggers/bin/swarm-trigger.pl</code></p> </div> <div class="paragraph"> <p>But can be installed anywhere on the filesystem</p> </div> <div class="paragraph"> <p>Execute the trigger to ensure that any required Perl modules are installed:</p> </div> <div class="literalblock"> <div class="content"> <pre>perl swarm-trigger.pl</pre> </div> </div> <div class="paragraph"> <p>Note that things like <code>JSON.pm</code> can often be installed with:</p> </div> <div class="literalblock"> <div class="content"> <pre>sudo apt-get install libjson-perl</pre> </div> </div> <div class="paragraph"> <p>or</p> </div> <div class="literalblock"> <div class="content"> <pre>sudo yum install perl-JSON</pre> </div> </div> </li> <li> <p>Swarm trigger configuration file has been copied over from commit server to appropriate place.</p> <div class="paragraph"> <p>This defaults to one of:</p> </div> <div class="literalblock"> <div class="content"> <pre>/etc/perforce/swarm-trigger.conf /opt/perforce/etc/swarm-trigger.conf swarm-trigger.conf (in the same directory as trigger script)</pre> </div> </div> </li> </ol> </div> </div> <div class="sect4"> <h5 id="_other_triggers">2.1.1.3. Other Triggers</h5> <div class="paragraph"> <p>Checklist:</p> </div> <div class="ulist"> <ul> <li> <p>Make sure that the appropriate version of <code>perl</code>, <code>python</code>, <code>ruby</code> etc are installed in locations as referenced by <code>p4 triggers</code> entries.</p> </li> <li> <p>Make sure that any relevant modules and dependencies have also been installed (e.g. <code>P4Python</code> or <code>P4Perl</code>)</p> </li> </ul> </div> </div> <div class="sect4"> <h5 id="_other_replicas_p4target">2.1.1.4. Other Replica’s P4TARGET</h5> <div class="paragraph"> <p>Review the settings for other replicas and also to check live replicas on the source server of the failover (<code>p4 servers -J</code>)</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 configure show allservers | grep P4TARGET</pre> </div> </div> <div class="paragraph"> <p>Make sure the above settings are using correct DNS alias (which will be redirected).</p> </div> </div> <div class="sect4"> <h5 id="_proxies">2.1.1.5. Proxies</h5> <div class="paragraph"> <p>These are typically configured via <code>/p4/common/bin/p4_1.vars</code> settings:</p> </div> <div class="literalblock"> <div class="content"> <pre>export PROXY_TARGET=</pre> </div> </div> <div class="paragraph"> <p>Ensure the target is the correct DNS alias (which will be redirected).</p> </div> </div> <div class="sect4"> <h5 id="_ha_server_os_configuration">2.1.1.6. HA Server OS Configuration</h5> <div class="paragraph"> <p>Check to make sure any performance configuration such as turning off THP (transparent huge pages), and putting serverlocks.dir into a RAM filesystem have also been made to your HA Failover system. See <a href="SDP_Guide.Unix.html#_maximizing_server_performance">SDP Guide: Maximizing Server Performance</a></p> </div> </div> </div> </div> <div class="sect2"> <h3 id="_failing_over">2.2. Failing over</h3> <div class="paragraph"> <p>The basic actions are the same for Unix and Windows, but there are extra steps required as noted in <a href="#_failing_over_on_windows">Section 2.4, “Failing over on Windows”</a></p> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Rotate the active journal on the master:</p> <div class="literalblock"> <div class="content"> <pre>p4 admin journal p4 servers -J # Check all is up-to-date</pre> </div> </div> </li> <li> <p>Run <code>p4 failover</code> in reporting mode on HA machine:</p> <div class="literalblock"> <div class="content"> <pre>p4 failover</pre> </div> </div> <div class="paragraph"> <p>Successful output looks like:</p> </div> <div class="literalblock"> <div class="content"> <pre>Checking if failover might be possible ... Checking for archive file content not transferred ... Verifying content of recently update archive files ... After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.</pre> </div> </div> </li> <li> <p>Perform failover:</p> <div class="literalblock"> <div class="content"> <pre>p4 failover --yes</pre> </div> </div> <div class="paragraph"> <p>Output should be something like:</p> </div> <div class="literalblock"> <div class="content"> <pre>Starting failover process ... Refusing new commands on server from which failover is occurring ... Giving commands already running time to complete ... Stalling commands on server from which failover is occurring ... Waiting for 'journalcopy' to complete its work ... Waiting for 'pull -L' to complete its work ... Waiting for 'pull -u' to complete its work ... Checking for archive file content not transferred ... Verifying content of recently updated archive files ... Stopping server from which failover is occurring ... Moving latest journalcopy'd journal into place as the active journal ... Updating configuration of the failed-over server ... Restarting this server ...</pre> </div> </div> <div class="paragraph"> <p>During this time, if you run commands against the master, you may see:</p> </div> <div class="literalblock"> <div class="content"> <pre>Server currently in failover mode, try again after failover has completed</pre> </div> </div> </li> <li> <p>Change the DNS entries so downstream replicas (and users) will connect to the new master (that was previously HA)</p> </li> <li> <p>Validate that your downstream replicas are communicating with your new master</p> <div class="paragraph"> <p>On each replica machine:</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 pull -lj</pre> </div> </div> <div class="paragraph"> <p>Or against the new master:</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 servers -J</pre> </div> </div> <div class="paragraph"> <p>Check output of <code>p4 info</code>:</p> </div> <div class="literalblock"> <div class="content"> <pre>: Server address: p4d-bos-02 : ServerID: master_1 Server services: commit-server :</pre> </div> </div> </li> <li> <p>Make sure the old server spec (<code>p4d_ha_bos</code>) has correctly had its <code>Options:</code> field set to <code>nomandatory</code> (otherwise all replication would stop!)</p> </li> </ol> </div> </div> <div class="sect2"> <h3 id="_post_failover">2.3. Post Failover</h3> <div class="sect3"> <h4 id="_validation_of_sdp_config">2.3.1. Validation of SDP Config</h4> <div class="olist arabic"> <ol class="arabic"> <li> <p>Run the script:</p> <div class="literalblock"> <div class="content"> <pre>/p4/common/bin/verify_sdp.sh 1</pre> </div> </div> </li> </ol> </div> <div class="paragraph"> <p>(or specify other instance ID).</p> </div> <div class="paragraph"> <p>Ensure that all output is valid and that there are no errors.</p> </div> <div class="paragraph"> <p>Ensure that the following returns 'yes':</p> </div> <div class="literalblock"> <div class="content"> <pre>/p4/common/bin/run_if_master.sh 1 echo yes</pre> </div> </div> <div class="paragraph"> <p>If it does not, then run: <code>bash -xv /p4/common/bin/run_if_master.sh 1 echo yes</code> and check the output. The most likely problems are things like the settings of the following variables in <code>/p4/common/config/p4_1.vars</code>:</p> </div> <div class="literalblock"> <div class="content"> <pre>P4MASTERHOST <== Set to DNS name of new master host P4MASTER_ID <== Make sure this is same as value in /p4/1/root/server.id on the master server machine.</pre> </div> </div> </div> <div class="sect3"> <h4 id="_moving_of_checkpoints">2.3.2. Moving of Checkpoints</h4> <div class="paragraph"> <p>After failing over, on Unix there will be journals which may need to be copied/moved and renamed due to the SDP structure.</p> </div> <div class="paragraph"> <p>For example, an HA server might have stored its journals in <code>/p4/1/checkpoints.ha_bos</code> (assuming it was create by <code>mkrep.sh</code> with serverid <code>p4d_ha_bos</code>):</p> </div> <div class="literalblock"> <div class="content"> <pre>/p4/1/checkpoints.ha_bos/p4_1.ha_bos.jnl.123 /p4/1/checkpoints.ha_bos/p4_1.ha_bos.jnl.124</pre> </div> </div> <div class="paragraph"> <p>As a result of failover, these files need to be copied/moved to:</p> </div> <div class="literalblock"> <div class="content"> <pre>/p4/1/checkpoints/p4_1.jnl.123 /p4/1/checkpoints/p4_1.jnl.124</pre> </div> </div> <div class="admonitionblock tip"> <table> <tr> <td class="icon"> <i class="fa icon-tip" title="Tip"></i> </td> <td class="content"> Easiest way is to install <code>prename</code> on your Linux. Then <code>p4rename -n 's/.ha_bos//' p4_1.*</code> - and re-run without the <code>-n</code> if it looks good. </td> </tr> </table> </div> <div class="paragraph"> <p>The reason different directories are sometimes used is because in some installations the <code>/hxdepots</code> filesystem is shared on NFS between commit server and HA server. When <code>/hxdepots</code> is shared, the <code>journalPrefix</code> must be set differently to avoid path conflicts on the shared storage.</p> </div> <div class="admonitionblock important"> <table> <tr> <td class="icon"> <i class="fa icon-important" title="Important"></i> </td> <td class="content"> If these files are not present, then normal SDP crontab tasks such as <code>daily_checkpoint.sh</code> will fail as they won’t be able to find the required journals to be applied to the <code>offline_db</code>. </td> </tr> </table> </div> <div class="paragraph"> <p>The following command will rotate journal and replay any missing ones to <code>offline_db</code> (it is both fairly quick and safe to run without placing much load on the server host as it doesn’t do any checkpointing):</p> </div> <div class="literalblock"> <div class="content"> <pre>/p4/common/bin/rotate_journal.sh 1</pre> </div> </div> <div class="paragraph"> <p>If it fails, then check <code>/p4/1/logs/checkpoint.log</code> for details - it may have an error such as:</p> </div> <div class="literalblock"> <div class="content"> <pre>Replay journal /p4/1/checkpoints/p4_1.jnl.123 to offline db.</pre> </div> </div> <div class="literalblock"> <div class="content"> <pre>Perforce server error: open for read: /p4/1/checkpoints/p4_1.jnl.123: No such file or directory</pre> </div> </div> <div class="paragraph"> <p>This indicates missing journals which will need to be moved/copied as above.</p> </div> </div> <div class="sect3"> <h4 id="_check_daily_checkpoint_sh_runs_successfully">2.3.3. Check daily_checkpoint.sh runs successfully</h4> <div class="paragraph"> <p>This is an important script for a master server, and it is good to make sure ASAP after failover that it works as expected.</p> </div> <div class="literalblock"> <div class="content"> <pre>nohup /p4/common/bin/daily_checkpoint.sh 1 &</pre> </div> </div> <div class="paragraph"> <p>And check output in: <code>/p4/1/logs/checkpoint.log</code></p> </div> <div class="admonitionblock tip"> <table> <tr> <td class="icon"> <i class="fa icon-tip" title="Tip"></i> </td> <td class="content"> This script can take a while on big repositories. It is OK to have it run in the crontab over night. </td> </tr> </table> </div> </div> <div class="sect3"> <h4 id="_check_on_replication">2.3.4. Check on Replication</h4> <div class="paragraph"> <p>We recommend that you connect to all your replicas/proxies/brokers and make sure that they are successfully working after failover.</p> </div> <div class="paragraph"> <p>It is surprisingly common to find forgotten configuration details meaning that they are attempting to connect to old server for example!</p> </div> <div class="paragraph"> <p>For proxies and brokers - you probably just need to run:</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 info</pre> </div> </div> <div class="paragraph"> <p>For downstream replicas of any type, we recommend logging on to the host and running:</p> </div> <div class="literalblock"> <div class="content"> <pre>p4 pull -lj</pre> </div> </div> <div class="paragraph"> <p>and checking for any errors.</p> </div> <div class="paragraph"> <p>We also recommend the following is executed on both HA server and all replicas and the output examined for any unexpected errors:</p> </div> <div class="literalblock"> <div class="content"> <pre>grep -A4 'Perforce server error:' /p4/1/logs/log</pre> </div> </div> <div class="paragraph"> <p>Or you can review the contents of <code>/p4/1/logs/errors.csv</code> if you have enabled structured logging.</p> </div> </div> </div> <div class="sect2"> <h3 id="_failing_over_on_windows">2.4. Failing over on Windows</h3> <div class="paragraph"> <p>The basic steps are the same as for on Unix, but with some extra steps at the end.</p> </div> <div class="paragraph"> <p>After the <code>p4 failover --yes</code> command has completed its work (on the HA server machine):</p> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Review the settings for the Windows service (examples are for instance <code>1</code>) - note below <strong>-S</strong> is uppercase:</p> <div class="literalblock"> <div class="content"> <pre>p4 set -S p4_1</pre> </div> </div> <div class="paragraph"> <p>Example results:</p> </div> <div class="literalblock"> <div class="content"> <pre>C:\p4\1>p4 set -S p4_1 : P4JOURNAL=c:\p4\1\logs\journal (set -S) P4LOG=c:\p4\1\logs\p4d_ha_aws.log (set -S) P4NAME=p4d_ha_aws (set -S) P4PORT=1666 (set -S) P4ROOT=c:\p4\1\root (set -S) :</pre> </div> </div> </li> <li> <p>Change the value of <code>P4NAME</code> and <code>P4LOG</code> to correct value for <code>master</code>:</p> <div class="literalblock"> <div class="content"> <pre>p4 set -S p4_1 P4NAME=master p4 set -S p4_1 P4LOG=c:\p4\1\logs\master.log</pre> </div> </div> <div class="paragraph"> <p>And re-check the output of <code>p4 set -S p4_1</code></p> </div> </li> <li> <p>Restart the service:</p> <div class="literalblock"> <div class="content"> <pre>c:\p4\common\bin\svcinst stop -n p4_1 c:\p4\common\bin\svcinst start -n p4_1</pre> </div> </div> </li> <li> <p>Run <code>p4 configure show</code> to check that the output is as expected for the above values.</p> </li> </ol> </div> <div class="sect3"> <h4 id="_post_failover_on_windows">2.4.1. Post Failover on Windows</h4> <div class="paragraph"> <p>This is slightly different for the Windows and Linux SDP, since there is currently no equivalent of <code>mkrep.sh</code> for Windows, and replication topologies on Windows are typically smaller and simpler.</p> </div> <div class="paragraph"> <p>Thus it is Windows HA instances are likely to have journals rotated into <code>c:\p4\1\checkpoints</code> instead of something like <code>c:\p4\1\checkpoints.ha_bos</code>.</p> </div> <div class="paragraph"> <p>However, it is still worth ensuring things like:</p> </div> <div class="ulist"> <ul> <li> <p><code>offline_db</code> on HA is up-to-date</p> </li> <li> <p>all triggers (including any Swarm triggers) are appropriately configured</p> </li> <li> <p><code>daily_backup.bat</code> will work appropriate after failover</p> </li> </ul> </div> </div> </div> <div class="sect2"> <h3 id="_preparation_for_failing_back_setup_original_commit_server_as_new_failover_replica">2.5. Preparation for Failing Back - Setup original commit server as new Failover replica</h3> <div class="paragraph"> <p>After failover it is quite common to want to to "failback" to the original machine. This is fairly straight forwards if you are going to use the same server ids (master and failover replica).</p> </div> <div class="paragraph"> <p>The steps are:</p> </div> <div class="ulist"> <ul> <li> <p>Reset the original master to be the failover replica</p> </li> <li> <p>Once it is up and running as a replica, then you can just use <code>p4 failover</code> on it, and then use these same instructions to reset the original HA replica to be running as a failover target.</p> </li> </ul> </div> <div class="paragraph"> <p>The original master p4d service will have been stopped as part of failover, and will not be able to restart without some configuration changes.</p> </div> <div class="paragraph"> <p>Note that for convenience below, we assume SDP instance <code>1</code> and failover serverid <code>p4d_ha_bos</code>.</p> </div> <div class="admonitionblock important"> <table> <tr> <td class="icon"> <i class="fa icon-important" title="Important"></i> </td> <td class="content"> The commands below are run on the <strong>on original master</strong> server which will now become the HA failover replica. </td> </tr> </table> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Change the serverid to replica serverid:</p> <div class="literalblock"> <div class="content"> <pre>cd /p4/1/root mv server.id save/ mv state* save/ echo p4d_ha_bos> server.id</pre> </div> </div> </li> <li> <p>Reset the journal and logs:</p> <div class="literalblock"> <div class="content"> <pre>cd /p4/1/logs mkdir -p save mv journal* save/ mv log log.old</pre> </div> </div> </li> <li> <p>Restart the service and ensure the <code>serviceUser</code> is logged in:</p> <div class="literalblock"> <div class="content"> <pre>sudo systemctl restart p4d_1 /p4/common/bin/p4login -v -service 1 p4 pull -lj</pre> </div> </div> </li> <li> <p>You may wish to ensure that verification of recent archive/depot files has been done, especially if the service was stopped for hours or days since the original failover.</p> <div class="literalblock"> <div class="content"> <pre>nohup /p4/common/bin/p4verify.sh -o MISSING -recent &</pre> </div> </div> </li> </ol> </div> <div class="paragraph"> <p>Please note details for definition of "recent" and how to extend that: <a href="SDP_Guide.Unix.html#_p4verify_sh">p4verify.sh</a></p> </div> <div class="paragraph"> <p>At this stage, the replica should be replicating as normal. If it is not, then please follow <a href="SDP_Guide.Unix.html#_replication_appears_to_be_stalled">standard replication trouble shooting procedures</a>.</p> </div> <div class="admonitionblock important"> <table> <tr> <td class="icon"> <i class="fa icon-important" title="Important"></i> </td> <td class="content"> The Server spec will have had its <code>Options:</code> set to <code>nomandatory</code> - and this should be set back to <code>mandatory</code> when the replica is working correctly. </td> </tr> </table> </div> </div> </div> </div> <div class="sect1"> <h2 id="_unplanned_failover">3. Unplanned Failover</h2> <div class="sectionbody"> <div class="paragraph"> <p>In this case there is no active participation of upstream server, so there is an increased risk of lost data.</p> </div> <div class="paragraph"> <p>We assume we are still failing over to the HA machine, so:</p> </div> <div class="ulist"> <ul> <li> <p>Failover target is <code>standby</code> or <code>forwarding-standby</code></p> </li> <li> <p>Server spec still has <code>Options:</code> set to <code>mandatory</code></p> </li> <li> <p>Original master is not running</p> </li> </ul> </div> <div class="paragraph"> <p>The output of <code>p4 failover</code> on the DR machine might be:</p> </div> <div class="literalblock"> <div class="content"> <pre>Checking if failover might be possible ... Server ID must be specified in the '-s' or --serverid' argument for a failover without the participation of the server from which failover is occurring. Checking for archive file content not transferred ... Verifying content of recently update archive files ... After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.</pre> </div> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Execute <code>p4 failover</code> with the extra parameter to specify server we are failing over from:</p> <div class="literalblock"> <div class="content"> <pre>p4 failover --serverid master_1 --yes</pre> </div> </div> </li> </ol> </div> <div class="paragraph"> <p>Expected output is somewhat shorter than for planned failover:</p> </div> <div class="literalblock"> <div class="content"> <pre>Starting failover process ... Waiting for 'pull -L' to complete its work ... Checking for archive file content not transferred ... Verifying content of recently updated archive files ... Moving latest journalcopy'd journal into place as the active journal ... Updating configuration of the failed-over server ... Restarting this server ...</pre> </div> </div> <div class="sect2"> <h3 id="_post_unplanned_failover">3.1. Post Unplanned Failover</h3> <div class="paragraph"> <p>This is similar to <a href="#_post_failover">Section 2.3, “Post Failover”</a> with the exception of the next section below.</p> </div> <div class="sect3"> <h4 id="_resetting_downstream_replicas">3.1.1. Resetting Downstream Replicas</h4> <div class="paragraph"> <p>In an unplanned failover scenario, if the standby server is not a 'mandatory' standby, it is possible that there is a journal synchronization problem with downstream replicas.</p> </div> <div class="paragraph"> <p>The output of <code>p4 pull -lj</code> may indicate an error, and/or there may be errors in the log:</p> </div> <div class="literalblock"> <div class="content"> <pre>grep -A4 error: /p4/1/logs/log | less</pre> </div> </div> <div class="paragraph"> <p>If you need to reset the replica to restart from the beginning of the current journal it is attempting to pull, then the process is:</p> </div> <div class="olist arabic"> <ol class="arabic"> <li> <p>Stop the replica:</p> <div class="literalblock"> <div class="content"> <pre>sudo systemctl stop p4d_1</pre> </div> </div> </li> <li> <p>Remove the <code>state</code> file:</p> <div class="literalblock"> <div class="content"> <pre>cd /p4/1/root mv state save/.</pre> </div> </div> </li> <li> <p>Restart the replica</p> <div class="literalblock"> <div class="content"> <pre>sudo systemctl start p4d_1</pre> </div> </div> </li> <li> <p>Recheck the log for errors as above.</p> </li> </ol> </div> </div> </div> <div class="sect2"> <h3 id="_unplanned_failover_on_windows">3.2. Unplanned Failover on Windows</h3> <div class="paragraph"> <p>The extra steps required are basically the same as in <a href="#_failing_over_on_windows">Section 2.4, “Failing over on Windows”</a> as well as the steps in <a href="#_post_unplanned_failover">Section 3.1, “Post Unplanned Failover”</a></p> </div> </div> </div> </div> </div> <div id="footer"> <div id="footer-text"> Version v2023.1<br> Last updated 2023-08-07 10:53:17 -0400 </div> </div> </body> </html>
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#14 | 30388 | C. Thomas Tyler |
Released SDP 2024.1.30385 (2024/06/11). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#13 | 30297 | C. Thomas Tyler |
Released SDP 2023.2.30295 (2024/05/08). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#12 | 30043 | C. Thomas Tyler |
Released SDP 2023.2.30041 (2023/12/22). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#11 | 29891 | C. Thomas Tyler |
Released SDP 2023.1.29699 (2023/07/11). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#10 | 29612 | C. Thomas Tyler |
Released SDP 2023.1.29610 (2023/05/25). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#9 | 29443 | C. Thomas Tyler |
Released SDP 2022.2.29441 (2023/02/27). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#8 | 29252 | C. Thomas Tyler |
Released SDP 2022.2.29250 (2022/12/08). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#7 | 28989 | C. Thomas Tyler |
Released SDP 2022.1.28987 (2022/08/25). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#6 | 28858 | C. Thomas Tyler |
Released SDP 2022.1.28855 (2022/05/27). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#5 | 28412 | C. Thomas Tyler |
Released SDP 2021.2.28410 (2021/11/24). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#4 | 28240 | C. Thomas Tyler |
Released SDP 2021.1.28238 (2021/11/12). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#3 | 27921 | C. Thomas Tyler |
Released SDP 2020.1.27919 (2021/07/19). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#2 | 27761 | C. Thomas Tyler |
Released SDP 2020.1.27759 (2021/05/07). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#1 | 27331 | C. Thomas Tyler |
Released SDP 2020.1.27325 (2021/01/29). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
//guest/perforce_software/sdp/dev/doc/SDP_Failover_Guide.html | |||||
#7 | 27213 | C. Thomas Tyler | Regenerated docs. | ||
#6 | 26851 | Robert Cowham |
Fix typo in tmpfs /etc/fstab entry which stopped it working in the doc. Mention in pre-requisites for failover and failover guide the need to review OS Config for your failover server. Document Ubuntu 2020.04 LTS and CentOS/RHEL 8 support. Note performance has been observed to be better with CentOS. Document pull.sh and submit.sh in main SDP guide (remove from Unsupported doc). Update comments in triggers to reflect that they are reference implementations, not just examples. No code change. |
||
#5 | 26747 | Robert Cowham |
Update with some checklists for failover to ensure valid. Update to v2020.1 Add Usage sections where missing to Unix guide Refactor the content in Unix guide to avoid repetition and make things read more sensibly. |
||
#4 | 26728 | Robert Cowham | Change filetype to +w for make. | ||
#3 | 26727 | Robert Cowham |
Add section on server host naming conventions Clarify HA and DR, and update links across docs Fix doc structure for Appendix numbering |
||
#2 | 26660 | Robert Cowham |
New common Failover Guide - removed old one. This is based on 'p4 failover' so requires 2018.2 or later. |
||
#1 | 26654 | Robert Cowham |
First draft of new Failover Guide using "p4 failover" Linked from SDP Unix Guide |