#!/usr/bin/env bash # # Executes the build # # Configuration is done by environment variables: # # - P4WS_HOME: Where to find the web-services stream # # - TEST_HOSTS: The IP addresses or host names of 'eval' environments we expect # to be used for testing. #------------------------------------------------------------------------------ # Configuration #------------------------------------------------------------------------------ # This needs to be the root of the p4ws project. By default, we check it out # under the home directory on the Vagrant environments, but the engineering # tools often need an "open client" that typically puts it under a different # subdirectory. p4ws_dir=${P4WS_HOME:=$HOME/p4ws} ruby_version=ruby-2.2.2 # For the build and buildmaster Salt environments, use this gemset for # most tasks. This gives us a way to reset the Ruby dependencies without having # to reinstall the machine. # # This gemset is typically created by the Salt provisioning process as well. gemset_name=build # This is the user to use for most tasks executed on the build instances. # The engtools system requires our local system user match the Perforce # account, but this isn't necessarily true of the remote Salt minions. build_user=${BUILD_USER:=$USER} # The salt root might be lying under a different user account. salt_root=${SALT_ROOT:=salt/srv} # Most of our build tasks rely on the same gemset for builds, which we can # quickly destroy and recreate on a whim when needed. # This is reference for wrappers on the master instance. wrapper_dir=$HOME/.rvm/wrappers/${ruby_version}@${gemset_name} build_wrapper_dir=/home/$build_user/.rvm/wrappers/${ruby_version}@${gemset_name} # TODO Need better scheme for identifying build minions build_minion=${BUILD_HOST:=build-ubuntu12} #------------------------------------------------------------------------------ # create_omnibus_package(target, package_dir, output_file, tmp_folder) #------------------------------------------------------------------------------ create_omnibus_package () { local target=$1 local package_dir=$2 local output_file=$3 local tmp_folder=$4 echo start creating $target sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd=$package_dir sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle exec omnibus build $target --log-level=debug" runas=$build_user cwd=$package_dir output_dir=`dirname $output_file` echo ls ${output_dir} sudo salt -G 'node_type:build' cmd.run "ls ${output_dir}" runas=$build_user echo push back to master: $output_file sudo salt -G 'node_type:build' cp.push $output_file sudo salt -G 'node_type:build' cmd.run "rm -rf ${tmp_folder}" echo finished creating $target } #------------------------------------------------------------------------------ # copy_package('/path/to/file.deb') # # Transfers the installation file to each 'eval' machine # # Final step is to push the #------------------------------------------------------------------------------ copy_package () { set -v local source_path=$1 local filename=`basename ${source_path}` # Remember that the $SALT_ROOT might actually lie under a different user # account. (It does in the EC setup.) sudo cp $source_path $SALT_ROOT sudo salt -G 'node_type:eval' cp.get_file salt://$filename /var/tmp/$filename if [ -f pkg/$filename ]; then rm pkg/$filename fi cp $SALT_ROOT/$filename pkg sudo rm $SALT_ROOT/$filename echo Package $filename created and moved to pkg } #------------------------------------------------------------------------------ # Main script #------------------------------------------------------------------------------ echo Building p4ws as $USER on `hostname` echo build_user $build_user echo p4ws_dir $p4ws_dir echo wrapper_dir $wrapper_dir # Note: salt remote commands seem to not return error exit codes if something # fails out on a remote. # TODO find a way to fail if remote tasks fail set -ex cd $p4ws_dir/p4ruby $wrapper_dir/bundle install $wrapper_dir/rake clean gem $wrapper_dir/gem install pkg/p4ruby-*.gem cd $p4ws_dir $wrapper_dir/bundle install $wrapper_dir/rake p4ws-versions.sh . p4ws-versions.sh $wrapper_dir/rake package sudo cp pkg/p4ws.tgz $SALT_ROOT sudo salt -G 'node_type:build' cp.get_file salt://p4ws.tgz /var/tmp/p4ws.tgz sudo salt -G 'node_type:build' cmd.run 'tar xvz -C /var/tmp -f /var/tmp/p4ws.tgz' runas=$build_user sudo rm $SALT_ROOT/p4ws.tgz # Setup p4ruby sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd='/var/tmp/p4ws/p4ruby' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/rake gem" runas=$build_user cwd='/var/tmp/p4ws/p4ruby' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/gem install pkg/p4ruby-*.gem" runas=$build_user cwd='/var/tmp/p4ws/p4ruby' # Setup p4_web_services_auth sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd='/var/tmp/p4ws/p4_web_services_auth' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/rake install" runas=$build_user cwd='/var/tmp/p4ws/p4_web_services_auth' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle package" runas=$build_user cwd='/var/tmp/p4ws/p4_web_api/p4_web_api' create_omnibus_package p4_web_api \ '/var/tmp/p4ws/p4_web_api/omnibus-p4_web_api' \ "/var/tmp/p4ws/p4_web_api/omnibus-p4_web_api/pkg/p4-web-api_${P4_WEB_API}-${CHANGELIST}_amd64.deb" \ '/opt/perforce/web-services/p4_web_api' # Setup p4_project_services_data sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd='/var/tmp/p4ws/p4_project_services/p4_project_services_data' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/rake install" runas=$build_user cwd='/var/tmp/p4ws/p4_project_services/p4_project_services_data' # Setup p4_web_api_client sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd='/var/tmp/p4ws/p4_web_api/clients/ruby/p4_web_api_client' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/rake install" runas=$build_user cwd='/var/tmp/p4ws/p4_web_api/clients/ruby/p4_web_api_client' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle package" runas=$build_user cwd='/var/tmp/p4ws/p4_project_services/p4_project_services' create_omnibus_package p4_project_services \ '/var/tmp/p4ws/p4_project_services/p4_project_services/packaging/omnibus-p4_project_services' \ "/var/tmp/p4ws/p4_project_services/p4_project_services/packaging/omnibus-p4_project_services/pkg/p4-project-services_${P4_PROJECT_SERVICES}-${CHANGELIST}_amd64.deb" \ '/opt/perforce/web-services/p4_project_services' # Setup p4_project_services_client sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle install" runas=$build_user cwd='/var/tmp/p4ws/p4_project_services/clients/ruby/p4_project_services_client' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/rake install" runas=$build_user cwd='/var/tmp/p4ws/p4_project_services/clients/ruby/p4_project_services_client' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle package" runas=$build_user cwd='/var/tmp/p4ws/p4_phoenix_services/p4_phoenix_services' create_omnibus_package p4_phoenix_services \ '/var/tmp/p4ws/p4_phoenix_services/p4_phoenix_services/packaging/omnibus-p4_phoenix_services' \ "/var/tmp/p4ws/p4_phoenix_services/p4_phoenix_services/packaging/omnibus-p4_phoenix_services/pkg/p4-phoenix-services_${P4_PHOENIX_SERVICES}-${CHANGELIST}_amd64.deb" \ '/opt/perforce/web-services/p4_phoenix_services' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/bundle package" runas=$build_user cwd='/var/tmp/p4ws/notification_services/notification_services' create_omnibus_package notification_services \ '/var/tmp/p4ws/notification_services/notification_services/packaging/omnibus-notification_services' \ "/var/tmp/p4ws/notification_services/notification_services/packaging/omnibus-notification_services/pkg/notification-services_${NOTIFICATION_SERVICES}-${CHANGELIST}_amd64.deb" \ '/opt/perforce/web-services/notification_services' sudo salt -G 'node_type:build' cmd.run "browserify app.js -t [envify --NODE_ENV production] | uglifyjs -cm > bundle.js" runas=$build_user cwd='/var/tmp/p4ws/ui/static/project_management' create_omnibus_package perforce-helix-web-services-project-management-ui \ '/var/tmp/p4ws/ui/static/project_management/packaging/omnibus-perforce-helix-web-services-project-management-ui' \ "/var/tmp/p4ws/ui/static/project_management/packaging/omnibus-perforce-helix-web-services-project-management-ui/pkg/perforce-helix-web-services-project-management-ui_1.0.0-${CHANGELIST}_amd64.deb" \ '/opt/perforce/web-services/static' # Clean up everything sudo salt -G 'node_type:build' cmd.run 'rm -rf /var/tmp/p4ws*' sudo salt -G 'node_type:build' cmd.run 'rm -rf /var/cache/omnibus/*' sudo salt -G 'node_type:build' cmd.run 'source /usr/local/rvm/scripts/rvm; rvm --force gemset delete build' runas=$build_user shell='/bin/bash' sudo salt -G 'node_type:build' cmd.run 'source /usr/local/rvm/scripts/rvm; rvm gemset create build' runas=$build_user shell='/bin/bash' sudo salt -G 'node_type:build' cmd.run "$build_wrapper_dir/gem install bundler" runas=$build_user shell='/bin/bash' # Transfer packages to the testing machines copy_package /var/cache/salt/master/minions/$build_minion/files/var/tmp/p4ws/p4_web_api/omnibus-p4_web_api/pkg/p4-web-api_${P4_WEB_API}-${CHANGELIST}_amd64.deb copy_package /var/cache/salt/master/minions/$build_minion/files/var/tmp/p4ws/p4_project_services/p4_project_services/packaging/omnibus-p4_project_services/pkg/p4-project-services_${P4_PROJECT_SERVICES}-${CHANGELIST}_amd64.deb copy_package /var/cache/salt/master/minions/$build_minion/files/var/tmp/p4ws/p4_phoenix_services/p4_phoenix_services/packaging/omnibus-p4_phoenix_services/pkg/p4-phoenix-services_${P4_PHOENIX_SERVICES}-${CHANGELIST}_amd64.deb copy_package /var/cache/salt/master/minions/$build_minion/files/var/tmp/p4ws/notification_services/notification_services/packaging/omnibus-notification_services/pkg/notification-services_${NOTIFICATION_SERVICES}-${CHANGELIST}_amd64.deb copy_package /var/cache/salt/master/minions/$build_minion/files/var/tmp/p4ws/ui/static/project_management/packaging/omnibus-perforce-helix-web-services-project-management-ui/pkg/perforce-helix-web-services-project-management-ui_1.0.0-${CHANGELIST}_amd64.deb sudo salt -G 'node_type:eval' state.highstate pillar='{"versions.changelist": "'${CHANGELIST}'", "versions.p4_web_api": "'${P4_WEB_API}'", "versions.p4_project_services": "'${P4_PROJECT_SERVICES}'", "versions.p4_phoenix_services": "'${P4_PHOENIX_SERVICES}'", "versions.notification_services": "'${NOTIFICATION_SERVICES}'"}' $wrapper_dir/gem install p4util # TODO figure out how to install Qt and then let this happen # -> this may need to happen on a specialized machine #$wrapper_dir/bundle exec rake qt_build for test_host in $TEST_HOSTS; do echo initializing data on $test_host $wrapper_dir/p4util init -p $test_host:1666 -a p4init WS_HOST=$test_host $wrapper_dir/bundle exec rake test:p4_web_api WS_HOST=$test_host $wrapper_dir/bundle exec rake test:p4_project_services # TODO run this automatically when this is ready #PHOENIX_SERVICES_URL=http://$test_host/p4_phoenix_services/v1 $wrapper_dir/bundle exec rake test:qt_test done # Clean up the test environment sudo salt -G 'node_type:eval' cmd.run 'service god stop' sudo salt -G 'node_type:eval' cmd.run 'dpkg -r p4-phoenix-services p4-project-services p4-web-api perforce-server perforce-helix-web-services-project-management-ui' sudo salt -G 'node_type:eval' cmd.run 'dpkg -P p4-phoenix-services p4-project-services p4-web-api perforce-server perforce-helix-web-services-project-management-ui' sudo salt -G 'node_type:eval' cmd.run 'rm -rf /var/lib/perforce' # The final output of the build is a tarball that will be consumed by the # publishing step. rm -f pkg/p4ws-all.tgz $wrapper_dir/rake p4ws.deb.manifest mv p4ws.deb.manifest pkg cd pkg && tar czvf p4ws-all.tgz *.deb p4ws.tgz p4ws.deb.manifest && cd .. # TODO rebuild the local 'build' gemset
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#26 | 13972 | tjuricek |
Removing old microservice implementations. The system is now mostly a monolith. Eventually there will be a websocket service. |
||
#25 | 13787 | tjuricek | Add missing 'npm install' step before attempting to create bundle.js | ||
#24 | 13786 | tjuricek | Specify the shell environment for executing the node build task, the default (/bin/sh) seems to not find npm global modules. | ||
#23 | 13707 | tjuricek |
Infrastructure for including a "project management" React application. This attempts to create a fairly simple installer that creates a 'static' folder based on ui/static that gets hosted by the nginx front end. Right now, it's the only app, so the default page is this application. It was called "pws2" during a prototyping phase. Another prototype, "pws" and the related "project" module, is removed since that was a Sinatra-based approach that will be much more difficult to integrate into anything else. I'm running into a couple of issues with notifications setup, it's still not 100%, so I'm disabling this for now from the default 'god' configuration. (The service isn't 100% functional yet, anyway.) |
||
#22 | 13550 | tjuricek |
Agh, running 'rake clean' deletes the pkg directory. The EC environment automatically removes everything here, so this was somewhat superfluous anyway. |
||
#21 | 13548 | tjuricek | Enable debug logging of 'omnibus build' step | ||
#20 | 13547 | tjuricek |
Add /var/cache/omnibus to cleanup routine. It appears that any "file sync" by omnibus first copies files to this cache directory, then syncs to the output folder. We may, however, have issues with file permissions and this temporary folder. |
||
#19 | 13546 | tjuricek |
Adding log messages regarding what omnibus actually output. The test is failing on push of the built packages back to master on the EC cluster. Probably the wrong file name. |
||
#18 | 13544 | tjuricek |
Revise package and gem versioning. Packages will use [gem]-[changelist] as their versions. Gems will use a standard Ruby MAJOR.MINOR.REVISION format. P4WEBAPI-64 |
||
#17 | 13543 | tjuricek | Force charset to be auto for the 'p4util' call to initialize test data | ||
#16 | 13542 | tjuricek | Enable configuration of the build minion via environment variables. | ||
#15 | 13541 | tjuricek | Set 'runas' to $build_user for the create_omnibus_package command, since it executes out on build machines | ||
#14 | 13540 | tjuricek | Allow the 'wrapper directory' configuration to follow from the build user, which is different in the EC setup. | ||
#13 | 13539 | tjuricek | The salt/srv configuration may lie in a different system account on some setups, so allow that to be configured. | ||
#12 | 13538 | tjuricek | Allow the system user on the build minion to be different from the system user on the local master. | ||
#11 | 13537 | tjuricek | Switch to 'stream config' and add some diagnostic commands. | ||
#10 | 13536 | tjuricek |
Remove rvm 'path' initialization. In non-interactive environments, the 'rvm' command likely isn't there, and most local references should be via a wrapping script. This may impact the use of rake, which can be used to launch other tasks via sub-shell commands. I believe the original inclusion of the rvm command here was for calling 'p4util init' on the test environment, which doesn't appear to be needed. |
||
#9 | 13535 | tjuricek |
Add notification_services to deployment, and reconfigure build step to exec bash. The execution bit doesn't seem to stay set on config/bash.sh The notification_services service doesn't have advanced tests just yet. |
||
#8 | 13530 | tjuricek |
Add p4_phoenix_services package and Salt configuration for deployment. This uncovered a couple of issues from the C++ API during it's conversion to C++03. So, in a nutshell, most operations, except for notifications, appear to be working (well, using Vagrant machines). |
||
#7 | 13528 | tjuricek |
Moved rack and app server configuration to be managed via Salt. Also, only using a single value "url" to configure how the p4_project_services instance references the p4_web_api. And, removing the Docker setup, since that won't work for a production system. |
||
#6 | 13527 | tjuricek |
Added a basic p4-project-services .deb package There is some kind of configuration issue with the production config that causes tests to fail. The service is running, however, so this is likely related to having a few things not managed via salt. |
||
#5 | 13524 | tjuricek |
Add a quick message to make it easy to verify just a couple of environment variables when running manually. When monkeying around in Vagrant, it's easy to sudo su [someone else] and watch things fail. |
||
#4 | 13523 | tjuricek | Add the ability for the script to change it's directory location easily, and revise references to RVM wrapper script paths to make it a wee bit more readable. | ||
#3 | 13522 | tjuricek |
Set up build.sh to create p4ws-all.tgz as the only real artifact of the build-test process. This will contain all files that we will "publish" into pkg-ondemand for later consumption. Note: exactly how the salt configuration is distributed is TBD. |
||
#2 | 13521 | tjuricek |
Adjust how rvm is configured on the build runs. This may not work in the non-interactive EC environment. TBD. |
||
#1 | 13520 | tjuricek |
Created a 'cluster' build procedure that creates an installer on build, and executes the install on a test instance. The main change is to package all gem dependencies via 'vendor/cache' (using the 'bundle package' command). Right now, there appears to be an issue with test data initialization, which may need a revised approach. |