Suggestion: Keep a parse able hardware rule list on a webserver and have the SDP automatically fetch this. As you can get recommendations based on different hardware platforms you can update them and the SDP can dynamically catch these.

Hey Tom!

So when I ran the helix installer I expected that at the end of my run I'd have a p4d and p4broker with(or without) ssl running and be immediately usable. That didn't happen and I'm a bit surprised that the inability to find/run verify_sdp.sh (bogus or not) resulted in the helix installer not:

• setting correct permissions on the systemd service
• enabling cron for the perforce user
• starting p4d/p4broker
• setting up the offline db.

So you're saying this one little hiccup prevented the above from happening (cascaded the problem?)?

I guess my question becomes, how long does a problem need to exist before "temporary" becomes something worth addressing via a supported script? And can that length of time be standardized?

So I'm not familiar with any particular bugs that cause files to not replicate. Given the built in system of tracking what has and hasn't been transferred and the existence of commands to clear that cache and then reschedule, I thought that this was setup this way by design. I'd love to see them improve p4d to handle this natively. But it's been a few years with the status quo, and I don't have any feedback on when that might change.

Today the SDP will send an email alert to the admin. The admin can then manually intervene. So the existing process is monitor and alert only. But this is something that I feel should be: monitor, retry, alert if the retry fails.

[perforce@aws-p4rcgt02 bin]$ ./recreate_offline_db.sh --help
/p4/common/bin/backup_functions.sh: line 221: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 222: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 221: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 222: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 221: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 222: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 221: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 222: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
/p4/common/bin/backup_functions.sh: line 258: /p4/--help/logs/recreate_offline_db.log.2022:02:16-21:12:03: No such file or directory
Some expected dirs are missing or not writable. Aborting.

As the main product (helix core) is moving in the direction of actually checking certificate chains and accepting the OS root CA, more business will opt to using publically signed SSL certs for their perforce infrastructure. Public certs tend to expire more rapidly than the ones issued by the self signed process. To ensure that there isn't an incident affecting the availability of a perforce server due to expired certificates, a server health check should run to warn if a certificate is going to expire soon.

Perhaps not well worded on my part - but I was poking at the /cloud/aws script for snapshots and realized my installation of perforce SDP doesn't have the required utilities to run the script. Perhaps including these utilities as a setup option when first deploying the SDP would be wise?

New find: After deploying a perforce proxy service using the helix installer - if you run this:

./reset_sdp.sh -B -R 2>&1 | tee log.reset_sdp.wipe

At the end of the run, it won't clean up any services for p4broker or p4d_1 in systemd.

New find - after running helix installer for setting up a p4p, the reset_sdp.sh does not move itself to reset_sdp.sh.txt and remove the +x modifier. However that functionality does happen for other server types.

Without over complicating the helix installer - is there an easy way to have a preflight that checks for issues with known workarounds and prompt the user to visit the job/issue on the workshop? Maybe you host on the workshop a dynamic list that is fetched that features the necessary checks for each issue, and is automagically removed from the dynamic list when the associated job is closed?

The binaries for a broker were downloaded, and the p4port for a broker was set as ssl:1666. However no broker service exists. I understand that a broker is standard in an sdp deployment, is that also true for a p4 proxy though? Either way, this bit needs some cleanup.
drwxr-xr-x 2 perforce perforce 90 Nov 15 10:47 .
drwxr-xr-x 3 perforce perforce 53 Nov 15 10:47 ..
lrwxrwxrwx 1 perforce perforce 21 Nov 15 10:47 p4_1 -> /p4/common/bin/p4_bin
lrwxrwxrwx 1 perforce perforce 29 Nov 15 10:47 p4broker_1 -> /p4/common/bin/p4broker_1_bin
-rwxr-xr-x 1 perforce perforce 346 Nov 15 10:47 p4broker_1_init
lrwxrwxrwx 1 perforce perforce 24 Nov 15 10:47 p4p_1 -> /p4/common/bin/p4p_1_bin
-rwxr-xr-x 1 perforce perforce 340 Nov 15 10:47 p4p_1_init
perforce@sdptemplate:/p4/1/bin

Unsure if this is intended for a helix installer deployment of a p4p: There was no crontab setup. Either this is intended or a feature enhancement should be to add the crontab that will do the p4 proxy cleaning script?

Here is a 'redacted' output of my ENV as the 'perforce' user:

AWK=awk
SDP_MAX_STOP_DELAY_P4BROKER=600
CUT=cut
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.m4a=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.oga=01;36:.opus=01;36:.spx=01;36:.xspf=01;36:
P4TRUST=/p4/1/.p4trust
RBHOME=/p4/common/ruby
P4_VERSION=2021.2.2201121
MAILTO=ryan.young@redacted.com
SNAPSHOT_SCRIPT=
LANG=en_CA.UTF-8
MAILFROM=sdpdemo_p4p@redacted.com
SUDO_GID=1000
HISTTIMEFORMAT=%Y/%m/%d %Z %H:%M:%S
HOSTNAME=sdptemplate
SDP_MAX_START_DELAY_P4BROKER=60
OLDPWD=/hxdepots
SSL_PREFIX=ssl:
PROXY_MON_LEVEL=3
DEPOTS=/p4/1/depots
SHAREDDATA=FALSE
LOGS=/p4/1/logs
P4LOG=/p4/1/logs/log
P4SERVICEUSER=svc_sdptemplate
HMS_HOME=/p4/common/hms
SUDO_COMMAND=/bin/su
P4TICKETS=/p4/1/.p4tickets
P4REPLICA=TRUE
which_declare=declare -f
KEEPJNLS=21
P4ROOT=/p4/1/root
SDP_MAX_STOP_DELAY_P4P=600
USER=perforce
P4BROKERPORTNUM=1666
P4PORT=1667
OSUSER=perforce
P4SERVER=p4_1
PWD=/hxdepots/reset
P4DTG_ROOT=/p4/common/dtg
PS=ps
VERIFY_SDP_SKIP_TEST_LIST=
HOME=/home/perforce
TMP=/p4/1/tmp
CHECKPOINTS=/p4/1/checkpoints
SDP_MAX_START_DELAY_P4P=60
SDP_MAX_START_DELAY_P4D=120
SUDO_USER=ryoung
PROXY_PORT=1667
P4CLIB=/p4/common/lib
HISTFILE=~/.hist/history.1799.13242
P4BROKERBIN=/p4/1/bin/p4broker_1
P4CBIN=/p4/common/bin
SDP_INSTANCE=1
P4D_FLAGS=-p 1667 -r /p4/1/root -J /p4/1/logs/journal -L /p4/1/logs/log -q -d --pid-file
P4HOME=/p4/1
KEEPCKPS=21
PERLHOME=/p4/common/perl
P4DBIN=/p4/1/bin/p4d_1
P4P_VERSION=2021.2.2201121
KEEPLOGS=21
P4BROKER_VERSION=2021.2.2201121
SUDO_UID=1000
P4MASTER_ID=master.1
MAIL=/var/spool/mail/ryoung
ID=id
SDP_ALWAYS_LOGIN=0
P4PBIN=/p4/1/bin/p4p_1
TERM=xterm
SHELL=/bin/bash
P4INSTANCE=1
P4PCACHE=/p4/1/cache
SDP_ADMIN_PASSWORD_FILE=/p4/common/config/.p4passwd.p4_1.admin
P4CCFG=/p4/common/config
P4DTG_CFG=p4_1
SERVERID=sdptemplate
PERL5LIB=/p4/common/perl/lib:/p4/common/lib:
P4CSBIN=/p4/common/site/bin
SDP_AUTOMATION_USERS=swarm
PYHOME=/p4/common/python
SHLVL=2
P4CONFIG=/p4/1/.p4config
MANPATH=/p4/common/perl/site/man:/p4/common/perl/man:
DF=df -h
SDPMAIL=mail
P4JOURNAL=/p4/1/logs/journal
P4SSLDIR=/p4/ssl
LOGNAME=perforce
P4USER=p4admin
P4MASTERHOST=sdpdemo.redacted.com
P4LOGS=/p4/1/logs
GREP=grep
P4MASTERPORT=1999
P4PORTNUM=1999
PATH=/p4/1/bin:/p4/common/bin:/p4/common/python/bin:/p4/common/perl/site/bin:/p4/common/perl/bin:/p4/common/ruby/bin:/p4/common/site/bin:/sbin:/bin:/usr/sbin:/usr/bin:.
PROXY_TARGET=redacted:1666
PS1=$USER@${HOSTNAME%%.}:$PWD
SDP_VERSION=Rev. SDP/MultiArch/2021.1/28261 (2021/11/13).
HISTSIZE=1000
P4TMP=/p4/1/tmp
HISTFILESIZE=5000
P4ENVIRO=/dev/null/.p4enviro
P4DTGBIN=/p4/1/bin/p4dtg_1
P4BROKERPORT=ssl:1666
SDP_START_DELAY=2
P4BIN=/p4/1/bin/p4_1
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
eval ${whichdeclare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
}
=/bin/env

So I'm not totally familiar with a review process (not a programmer). here's a list of things that went wrong:

• On a RHEL 8.5 machine VM I created with hostname 'sdptemplate' - I specified a hostname of 'sdpdemop4p' in settings.cfg. The hostname change did not take. The sdp installed and used svc_sdptemplate as the service account name in the environment settings for the OSUSER 'perforce'.

• on a previous attempt to use the helix installer on RHEL 8.5 (from the main branch), p4p_1 would not start due to SELINUX preventing the startup. Had to disable SELINUX.
  Error/warning outputs from the helix installer log:

Warning: No ServerID defined for server of type p4proxy. Defaulting to p4proxy as the ServerID.

• Should this be a warning? If explicitly set as server type p4p, it should be an 'info' no?

Running: /p4/1/bin/p4p_1 -Gc
Perforce proxy error:
P4SSLDIR not defined or does not reference a valid directory.

Warning: Failed to generate SSL Certificates.

Preparing to run Sample Depot configuration script.
Adding perforce to sudoers.
Setting perms on sudoers file, /etc/sudoers.d/perforce.
Running: chmod 0400 /etc/sudoers.d/perforce

Skipping configuration of Sample Depot due to '-no_sd'.

Adding Perforce's packaging key to RPM keyring.
Running: rpm --import https://package.perforce.com/perforce.pubkey
Generating /etc/yum.repos.d/perforce.repo.

Configuring Firewalld services.

sed: can't read /hxdepots/reset/p4p_N.xml.t: No such file or directory

Error: Failed to generate /etc/firewalld/services/p4p_1.xml.

Additional alternative (easier) - when using a settings.cfg file, maybe add some additional text to the comments to note that if you want the demo content, you should keep this case sensitivity as 1

Hey Tom! The current helix installer settings.cfg and man page make reference to this, but you have this open job. Can the helix installer do this and does this job just need to be closed? If not - perhaps removing such references until this functionality exists? By the way, I'd be super stoked to have a run one script to setup a proxy. Looking forward to this!

Side plus - a solution that is more "product baked in" is more cross platform friendly. I know Microsoft is going to go the route of using ssh in the future... but "for now" - You'd be building a cross platform solution out of the gate if you relied on things common to the product like counters keys and properties, and built in perforce commands / bgtask triggers rather than ssh.

I would say that this is not something that should get in the way of progress. But as a customer looking for "a completely integrated solution that is baked into the product" - less infrastructure setup need and more brainless "run this script and the product will take care of itself" is desired. CSO groups are going the route of requiring that SSH keys get rotated, secrets management, etc. Baking something into the product reduces complexity for the customer.

I would steam ahead to get something delivered and a complete product. and version 2.0 of this should have an option that doesn't require SSH.

Idea to "get around" needing SSH access to all the servers in the farm tree:
Create a counter: "Do-Confirm-OfflineDB'. Value is 0 or 1.
When the commit server changes this to a 1, a SDP cron job (run frequently) on all replicas will catch it, do a check and create their own unique counter to say they did the job.
When the commit server has received an answer from a replica in this way, delete the replica's counter and record the result.
When all is answered, proceed.

Also - if we're going to continue along the path of "% free space" - perhaps logging the files that get removed and doing a diff/compare on future runs to see if the same files frequently get deleted? This may encourage the administrator to provision more space because "If the same files are always being deleted and then getting requested for replication (coming back) - it means that they are needed files" - if the files are needed, something should be done to reduce the frequency at which they are deleted.

I'd like to have a discussion around this script/idea (as it's a much needed script). The question I have to ask is this: Is it functionally more useful to specify a % of disk space you want to always have free or is it functionally more useful to have a "number of revisions" kept? If I setup a script to make sure I always have at least 20% free, I could be artificially creating a performance problem by removing revisions my end users need. Those files would be requested "on demand" and the user would have to wait for them to replicate back over, only to wind up being deleted again.

In our view - we see that keeping the latest X revision (usually the head and only the head) is the leanest way to go and solves the performance need for the syncing of data, since we're never accidently deleting a revision that is needed. The trouble with this approach is that provisioning too little storage (to save on storage) and requiring the script/cron to run more frequently eats at CPU cycles. Cachepurge can be a heavy/CPU taxing operation on smaller AWS instance sizes.

I placed an enhancement request to the product a long while ago suggesting that p4 pull should take care of this automatically as part of replication. I hope that goes onto the roadmap, but understand that this script is a good workaround for now. Still - the above discussion poses a challenge of approach. If you can reduce the overhead of running this more frequently "somehow" and set the target of keeping X revisions instead of X percent free space, then administrators can allocate less storage overall and save on storage costs.

You're making me think a little here - we always setup a VPN to edge servers in AWS. Does p4d perform better over SSL over wan (replication of metadata/archives) than over a standard AWS vpn? This could simplify some of our deployments.

I guess my original ask was just to make it easier for a newbie to the SDP to address this situation and I was trying to solve a problem that is "easily solved with training an understanding of the SDP" with a technical fix that addresses this particular situation. Philosophically when someone first touches the SDP and gets into it (without helix installer) one of the first things they play around with is mkdirs.cfg. Addressing the many common scenarios directly in there without requiring the person to go back and forth to read the doc may make it easier for sdp adoption.

So if I am understanding correctly, you're suggesting that the communications between the p4d and p4brokers should all be SSL if you enable SSL. You can then put forth a p4broker that is not SSL (as a second broker/third overall service) that is not SSL and allow users to connect to it. I am about to make a couple of assumptions, and I may very well be wrong so feel free to correct me!

If a user connects to a broker without SSL that is then pointed at an SSL enabled p4d, the broker and the p4d handle the decryption of the data. When the majority of end users are connected in this manner, there "must" be an overhead cost to this rather than simply having the p4d not be SSL enabled. In contrast, I have a small amount of users external to the company (about 15-20 per p4d/edge) that need to be SSL. So those will connect to the SSL enabled broker. The overhead there makes sense because it is for tighter security.

I understand this overhead is "small" in all cases, and I might be asking for "too much". I will say that if I was to migrate an existing p4d to the SDP today we already have cases where users (main office 150+ users) connect to p4d at port 1666 and a broker on another port has SSL for a DMZ like experience. So I would not want to convert my p4d to be SSL because that would be office user impacting.

It becomes functionally useful to have a variable in the SDP mkdirs.cfg / p4_vars that you can set as a bool $OnlyPrimaryBrokerSSL.

As it stands, when setting SSL_PREFIX=ssl: in mkdirs.cfg, all the other variables that result from running mkdirs.sh will take on a result like:
"${SSL_PREFIX}${P4MASTERHOST}:${P4_PORT}

Hence my "all or nothing" when setting up a new SDP, and a lot of manual intervention to get it to be in line with my use case.

I ran upgrade.sh without doing a -n first and ran into an issue because the user running the upgrade didn't have enough access to stop/start the perforce service, so the upgrade script hung and was uncrecoverable. I had to proceed with a manual upgrade after that point.

+1 for this. I ran upgrade.sh without doing a -n first and ran into an issue because the user running the upgrade didn't have enough access to stop/start the perforce service, so the upgrade script hung and was uncrecoverable.