diff --git a/.pairs b/.pairs index 83b3a913c..ae74356d5 100644 --- a/.pairs +++ b/.pairs @@ -10,6 +10,7 @@ pairs: dc: Dennis Cahillane; dcahillane df: Daniel Finnie; dfinnie dg: Dave Goddard; dgoddard + dpj: Dirk Janssen; djanssen dl: David Lee; dlee dr: Dimitri Roche; dimroc dt: David Tengdin; dtengdin @@ -32,6 +33,7 @@ pairs: mk: Mike Kenyon; mkenyon ml: Mariana Lenetis; mlenetis mw: Matt Wismer; mwismer + nm: Nick Mahoney; nmahoney pa: Peter Alfvin; palfvin rb: Rachel Brindle; rbrindle rg: Rob Gallagher; rgallagher diff --git a/README.md b/README.md index a4fef722e..1a3dc4b51 100644 --- a/README.md +++ b/README.md @@ -48,25 +48,48 @@ ProjectMonitor is a Rails application. To get the code, execute the following: git clone git://github.com/pivotal/projectmonitor.git cd projectmonitor brew install qt - bundle install + bundle install -### Initial Setup -We have provided an example file for `database.yml`. Run the following to -automatically generate these files for you: +It seems that the `qt` requirement is a relic of the past, we checked +and could not find any reference to it in the code. + +### Initial Setup (local workstation) + +ProjectMonitor depends on a running Postgres database instance. +Connection details for this database should be specified in +`config/database.yml`. An example configuration file has been +supplied in `database.yml.example`, but you may need to edit it +before you can use it. You can copy the file to `database.yml` yourself or use the +following command to do so: rake setup -You likely need to edit the generated files. See below. +If you run a local postgres server that is set up well, you won't have +to change the settings. Do not create the three tables mentioned in +the configuration file yet, the next step will do that for you. + +If you have Postgres installed but not running, you can try this +(confirmed on OS-X Yosemite and Ubuntu 14.04) to run the database from +a subdirectory of the project (useful for development): + + mkdir data + initdb -D data + pg_ctl start -D data -l data/logfile + +You should see several postgres processes if you run `ps -ef | grep +postgres`. -### Set up the database -You'll need a database. Create it with whatever name you want. If you have not -run `rake setup`, copy `database.yml.example` to `database.yml`. Edit the -production environment configuration so it's right for your database: +### Creating the tables on the database + +We assume you have created and updated the `config/database.yml` file. +The following commands will create and populate the databases for the +production environment (first 2 lines) and the development environment +(last 2 lines; development is the default): - cp config/database.yml.example config/database.yml - RAILS_ENV=production rake db:create RAILS_ENV=production rake db:migrate + rake db:create + rake db:migrate ### Authentication support #### IP Whitelist @@ -74,10 +97,10 @@ If you want to use Webhooks, your ProjectMonitor instance will need to be locate publicly accessible server. If you don't want your ProjectMonitor dashboard to also be publicly accessible, you can whitelist access by IP address. -The whitelist is disabled by default, but can be enabled by uncommenting the "ip_whitelist" property +The whitelist is disabled by default, but can be enabled by uncommenting the `ip_whitelist` property in settings.yml and adding a list of IP addresses to whitelist. If you're running ProjectMonitor behind a load balancer (e.g. on a hosted provider such as Heroku), you'll probably want to set -"ip_whitelist_request_proxied" to true. See settings.yml for more documentation. +`ip_whitelist_request_proxied` to true. See settings.yml for more documentation. #### Password authentication Project monitor uses [Devise](https://github.com/plataformatec/devise) to provide both database backed authentication and @@ -97,17 +120,29 @@ and the following configuration options specified: oauth2_secret: 'MY_SECRET' ### Setup Cron with Whenever -We have included a sample whenever gem config in config/schedule.rb. Refer to -the [whenever documentation](https://github.com/javan/whenever) for instructions -on how to integrate it with your deployment. Refer to [Heroku scheduler documentation](https://devcenter.heroku.com/articles/scheduler) for instructions -on how to integrate the rake task with your Heroku deployment. -The default schedule clears log entries and cleans up unused tags daily, and fetches project statuses every 3 minutes. +We have included a sample whenever gem config in +config/schedule.rb. Refer to the +[whenever documentation](https://github.com/javan/whenever) for full +instructions on how to integrate it with your deployment. In a pinch, +you can do at the toplevel of the project: + + whenever --update-crontab + +This will install the `whenever` jobs specified in +`config/schedule.rb` into your personal cron settings. These settings +include a fetch from your CI servers every three minutes. -The fetch project task is what goes out and hits the individual builds. We find -that if you do this too frequently it can swamp the builds. On the other hand, -you don't want ProjectMonitor displaying stale information. At Pivotal we set -it up to run every 3 minutes. +Refer to [Heroku scheduler documentation](https://devcenter.heroku.com/articles/scheduler) for instructions +on how to integrate the rake task with your Heroku deployment. +The default schedule clears log entries and cleans up unused tags +daily, and fetches project statuses every 3 minutes. On CloudFoundry, +scheduling is taken care of by extra workers, see below. + +The `fetch-statuses` project task is what goes out and hits the individual +builds. We find that if you do this too frequently it can swamp the +builds. On the other hand, you don't want ProjectMonitor displaying +stale information. At Pivotal we set it up to run every 3 minutes. ### Start workers The cron job above will add jobs to the queue, which workers will execute. To @@ -136,7 +171,7 @@ Execute: ### Next Steps -Now you need to add a project or two! Keep reading the *Configuration* section for instructions. +Now you need to add a project or two! You can read the *Configuration* section for instructions. ## Configuration @@ -145,7 +180,7 @@ ProjectMonitor. You can log in to set up projects by clicking the "Manage Projec link in the bottom-right corner of the main ProjectMonitor screen. You can either create a user using the console as follows: - rails c production + rails c development User.create!(login: 'john', name: 'John Doe', email: 'jdoe@example.com', password: 'password', password_confirmation: 'password') Or, if you have set up Google OAuth2 as per above, you can simply log in with Google to create a new user account. @@ -189,11 +224,20 @@ like so: ## Deployment ### Cloud Foundry -To deploy to staging: -``` -cf push -f ./config/cf/manifest-staging.yml -``` +Deployment is done using manifest.yml files. There are two files, one +for staging and one for production. The files are identical but for +the endpoints that they use. To use these files, you will have to +modify the line `host: project-monitor-production`: This line +specifies the endpoint or URL, and +project-monitor-production.cfapps.io is already taken by Pivotal. + +Before you can push up your app, you have to create a database to +connect to. The easiest way is to go to the PWS control panel, then +the marketplace and look for the ElephantSQL service. Install a +`panda` (paid version) of the database and call it `db-production`. +Create a second one called `db-staging`. See below for hints on using +the free version. To deploy to production: @@ -201,8 +245,94 @@ To deploy to production: cf push -f ./config/cf/manifest-production.yml ``` +And likewise for staging, using `manifest-staging.yml`. + +After you have pushed the manifest, you have to change the health +check on the apps that do not serve an endpoint (worker, clockwork, +poller), with `cf set-health-check none`. This does not disable +the health check but stops it from checking for a live endpoint. Your +workers may crash repeatedly before this setting takes hold. In +the future, `no-route: true` may set this automatically or a manifest +setting may become available. + + +#### Changes to the manifest files + +The manifest files as supplied will try to use the hostnames that are +in use at Pivotal. You have to supply your own values for the `host` +key (under applications/project-monitor-staging). You should also +remove all `logentries` lines (if still present). Finally, as of the +`Diego` release +CloudFoundry suggest to specify the buildpack with `buildpack: +ruby_buildpack`. + + + +#### Using the ElephantSQL free plan + +If you are using the ElephantSQL free plan, you have only four +connections available. Assuming you use staging for this, edit +`config/cf/manifest-staging.yml` and change the number of rails +instances (project-monitor-production) to one and disable the +`clockwork` task, which does cleanup tasks that are not required for a +staging app. Also make sure you do not have a `psql` or `rails +console` open, and be careful with opening the ElephantSQL dashboard. +Finally, edit the first line of `config/unicorn.rb` to have unicorn +start only one worker process, as each process opens its own database +connection. + + + + +#### Applications started on CloudFoundry + +On cloud deployment platforms like CloudFoundry and Heroku, `cron` is +not available or easy to use. To work around this, the manifest +starts the following independent applications, that all connect to the +same database: + +* Project-monitor: This is the main rails app that will be connected + to your endpoint, `.cfapps.io`. You will get three instances + by default. +* Project-monitor-worker: One instance of this is spun up to do + background tasks, which it reads from the database +* Project-monitor-clockwork: This application will run the rake + commands in `clockwork.rb` periodically. These commands clean up + logs etc. +* Project-monitor-poller: This application will contact the CI + providers periodically. + +On a local machine, this 4 application setup is replaced with the cron +tasks that are controlled by the `config/schedule.rb` file. + +#### Cloudfoundry environment variables + +Once you have set up your CF instance, you will want to modify some +environment variables with `cf set-env`. Specifically, you will +probably want to supply + +* A whitelist of IP addresses allowed to view this project monitor +* Your OAuth setup +* Pivotal uses NewRelic to monitor the monitor, it requries a token + too +* Our internal production setup uses `logentries` and this requires + setup too. Logentries is no longer recommended or supported for new + deployments. + +#### Retired deployment method + +Cloudfoundry hosts apps that serve the endpoints `ci.pivotallabs.com` +(for open source projects) and `pulse.pivotallabs.com` (Pivotal +projects). These were once deployed with `rake +cf_deploy[orgname,spacename,environment]`, where environment is +production or staging. This will tag your current git commit and push +the tag up to github, then do a `cf push`. This mechanism is currently +not used. + ### Heroku -To get running on Heroku, after you have cloned and bundled, run the following commands: + +To get running on Heroku, after you have cloned and bundled, run the +following commands: NB: These instructions are for the basic authentication strategy. @@ -220,7 +350,7 @@ Got a burning idea that just needs to be implemented? Check the CONTRIBUTE.md fi The google group for Project Monitor is [projectmonitor_pivotallabs](http://groups.google.com/group/projectmonitor_pivotallabs) -Copyright (c) 2013 Pivotal Labs. This software is licensed under the MIT License. +Copyright (c) 2013-2015 Pivotal Labs. This software is licensed under the MIT License. [![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/pivotal/projectmonitor/trend.png)](https://bitdeli.com/free "Bitdeli Badge") diff --git a/config/cf/manifest-production.yml b/config/cf/manifest-production.yml index 72dca4fd7..52c0ac815 100644 --- a/config/cf/manifest-production.yml +++ b/config/cf/manifest-production.yml @@ -8,7 +8,6 @@ applications: command: bundle exec rake db:migrate && bundle exec unicorn -p $PORT -c ./config/unicorn.rb services: - db-production - - logentries-production-web - name: project-monitor-production-worker memory: 512M instances: 1 @@ -17,7 +16,6 @@ applications: no-route: true services: - db-production - - logentries-production-worker - name: project-monitor-production-clockwork memory: 512M instances: 1 @@ -26,7 +24,6 @@ applications: no-route: true services: - db-production - - logentries-production-clockwork - name: project-monitor-production-poller memory: 1024M instances: 1 @@ -35,4 +32,3 @@ applications: no-route: true services: - db-production - - logentries-production-poller