Added to documentation and explained migration to CloudFoundry Diego

.pairs

@@ -10,6 +10,7 @@ pairs:
   dc: Dennis Cahillane; dcahillane
   df: Daniel Finnie; dfinnie
   dg: Dave Goddard; dgoddard
+  dpj: Dirk Janssen; djanssen
   dl: David Lee; dlee
   dr: Dimitri Roche; dimroc
   dt: David Tengdin; dtengdin
@@ -32,6 +33,7 @@ pairs:
   mk: Mike Kenyon; mkenyon
   ml: Mariana Lenetis; mlenetis
   mw: Matt Wismer; mwismer
+  nm: Nick Mahoney; nmahoney
   pa: Peter Alfvin; palfvin
   rb: Rachel Brindle; rbrindle
   rg: Rob Gallagher; rgallagher

README.md

@@ -48,36 +48,59 @@ ProjectMonitor is a Rails application. To get the code, execute the following:
     git clone git://github.com/pivotal/projectmonitor.git
     cd projectmonitor
     brew install qt
-    bundle install
+	bundle install
 
-### Initial Setup
-We have provided an example file for `database.yml`. Run the following to 
-automatically generate these files for you:
+It seems that the `qt` requirement is a relic of the past, we checked
+and could not find any reference to it in the code.
+
+### Initial Setup (local workstation)
+
+ProjectMonitor depends on a running Postgres database instance.
+Connection details for this database should be specified in
+`config/database.yml`.  An example configuration file has been
+supplied in `database.yml.example`, but you may need to edit it
+before you can use it.  You can copy the file to `database.yml` yourself or use the
+following command to do so:
 
     rake setup
 
-You likely need to edit the generated files.  See below.
+If you run a local postgres server that is set up well, you won't have
+to change the settings.  Do not create the three tables mentioned in
+the configuration file yet, the next step will do that for you. 
+
+If you have Postgres installed but not running, you can try this
+(confirmed on OS-X Yosemite and Ubuntu 14.04) to run the database from
+a subdirectory of the project (useful for development):
+
+    mkdir data
+    initdb -D data
+	pg_ctl start -D data -l data/logfile
+
+You should see several postgres processes if you run `ps -ef | grep
+postgres`. 
 
-### Set up the database
-You'll need a database. Create it with whatever name you want.  If you have not
-run `rake setup`, copy `database.yml.example` to `database.yml`.  Edit the
-production environment configuration so it's right for your database:
+### Creating the tables on the database
+
+We assume you have created and updated the `config/database.yml` file.
+The following commands will create and populate the databases for the
+production environment (first 2 lines) and the development environment
+(last 2 lines; development is the default):
 
-    cp config/database.yml.example config/database.yml
-    <edit database.yml>
     RAILS_ENV=production rake db:create
     RAILS_ENV=production rake db:migrate
+    rake db:create
+    rake db:migrate
 
 ### Authentication support
 #### IP Whitelist
 If you want to use Webhooks, your ProjectMonitor instance will need to be located on a
 publicly accessible server. If you don't want your ProjectMonitor dashboard to also be
 publicly accessible, you can whitelist access by IP address.
 
-The whitelist is disabled by default, but can be enabled by uncommenting the "ip_whitelist" property
+The whitelist is disabled by default, but can be enabled by uncommenting the `ip_whitelist` property
 in settings.yml and adding a list of IP addresses to whitelist. If you're running ProjectMonitor
 behind a load balancer (e.g. on a hosted provider such as Heroku), you'll probably want to set
-"ip_whitelist_request_proxied" to true. See settings.yml for more documentation.
+`ip_whitelist_request_proxied` to true. See settings.yml for more documentation.
 
 #### Password authentication
 Project monitor uses [Devise](https://github.com/plataformatec/devise) to provide both database backed authentication and
@@ -97,17 +120,29 @@ and the following configuration options specified:
     oauth2_secret: 'MY_SECRET'
 
 ### Setup Cron with Whenever
-We have included a sample whenever gem config in config/schedule.rb. Refer to
-the [whenever documentation](https://github.com/javan/whenever) for instructions
-on how to integrate it with your deployment. Refer to [Heroku scheduler documentation](https://devcenter.heroku.com/articles/scheduler) for instructions
-on how to integrate the rake task with your Heroku deployment.
 
-The default schedule clears log entries and cleans up unused tags daily, and fetches project statuses every 3 minutes.
+We have included a sample whenever gem config in
+config/schedule.rb. Refer to the
+[whenever documentation](https://github.com/javan/whenever) for full
+instructions on how to integrate it with your deployment.  In a pinch,
+you can do at the toplevel of the project:
+
+    whenever --update-crontab
+
+This will install the `whenever` jobs specified in
+`config/schedule.rb` into your personal cron settings.  These settings
+include a fetch from your CI servers every three minutes.
 
-The fetch project task is what goes out and hits the individual builds. We find
-that if you do this too frequently it can swamp the builds. On the other hand,
-you don't want ProjectMonitor displaying stale information. At Pivotal we set
-it up to run every 3 minutes.
+Refer to [Heroku scheduler documentation](https://devcenter.heroku.com/articles/scheduler) for instructions
+on how to integrate the rake task with your Heroku deployment.
+The default schedule clears log entries and cleans up unused tags
+daily, and fetches project statuses every 3 minutes.  On CloudFoundry,
+scheduling is taken care of by extra workers, see below. 
+
+The `fetch-statuses` project task is what goes out and hits the individual
+builds. We find that if you do this too frequently it can swamp the
+builds. On the other hand, you don't want ProjectMonitor displaying
+stale information. At Pivotal we set it up to run every 3 minutes.
 
 ### Start workers
 The cron job above will add jobs to the queue, which workers will execute.  To
@@ -136,7 +171,7 @@ Execute:
 
 ### Next Steps
 
-Now you need to add a project or two! Keep reading the *Configuration* section for instructions.
+Now you need to add a project or two! You can read the *Configuration* section for instructions.
 
 ## Configuration
 
@@ -145,7 +180,7 @@ ProjectMonitor. You can log in to set up projects by clicking the "Manage Projec
 link in the bottom-right corner of the main ProjectMonitor screen. You can either
 create a user using the console as follows:
 
-    rails c production
+    rails c development
     User.create!(login: 'john', name: 'John Doe', email: 'jdoe@example.com', password: 'password', password_confirmation: 'password')
 
 Or, if you have set up Google OAuth2 as per above, you can simply log in with Google to create a new user account.
@@ -189,20 +224,115 @@ like so:
 ## Deployment
 
 ### Cloud Foundry
-To deploy to staging:
 
-```
-cf push -f ./config/cf/manifest-staging.yml
-```
+Deployment is done using manifest.yml files.  There are two files, one
+for staging and one for production.  The files are identical but for
+the endpoints that they use.  To use these files, you will have to
+modify the line `host: project-monitor-production`: This line
+specifies the endpoint or URL, and
+project-monitor-production.cfapps.io is already taken by Pivotal.
+
+Before you can push up your app, you have to create a database to
+connect to.  The easiest way is to go to the PWS control panel, then
+the marketplace and look for the ElephantSQL service.  Install a
+`panda` (paid version) of the database and call it `db-production`.
+Create a second one called `db-staging`.  See below for hints on using
+the free version.
 
 To deploy to production:
 
 ```
 cf push -f ./config/cf/manifest-production.yml
 ```
 
+And likewise for staging, using `manifest-staging.yml`. 
+
+After you have pushed the manifest, you have to change the health
+check on the apps that do not serve an endpoint (worker, clockwork,
+poller), with `cf set-health-check <APP> none`.  This does not disable
+the health check but stops it from checking for a live endpoint.  Your
+workers may crash repeatedly before this setting takes hold.  In
+the future, `no-route: true` may set this automatically or a manifest
+setting may become available. 
+
+
+#### Changes to the manifest files
+
+The manifest files as supplied will try to use the hostnames that are
+in use at Pivotal.  You have to supply your own values for the `host`
+key (under applications/project-monitor-staging). You should also
+remove all `logentries` lines (if still present).  Finally, as of the
+`Diego` release
+CloudFoundry suggest to specify the buildpack with `buildpack:
+ruby_buildpack`.
+
+
+
+#### Using the ElephantSQL free plan
+
+If you are using the ElephantSQL free plan, you have only four
+connections available.  Assuming you use staging for this,  edit
+`config/cf/manifest-staging.yml` and change the number of rails
+instances (project-monitor-production) to one and disable the
+`clockwork` task, which does cleanup tasks that are not required for a
+staging app.  Also make sure you do not have a `psql` or `rails
+console` open, and be careful with opening the ElephantSQL dashboard.
+Finally, edit the first line of `config/unicorn.rb` to have unicorn
+start only one worker process, as each process opens its own database
+connection.  
+
+
+
+
+#### Applications started on CloudFoundry
+
+On cloud deployment platforms like CloudFoundry and Heroku, `cron` is
+not available or easy to use.  To work around this, the manifest
+starts the following independent applications, that all connect to the
+same database:
+
+* Project-monitor:  This is the main rails app that will be connected
+  to your endpoint,  `<HOST>.cfapps.io`.  You will get three instances
+  by default.
+* Project-monitor-worker:  One instance of this is spun up to do
+  background tasks, which it reads from the database
+* Project-monitor-clockwork: This application will run the rake
+  commands in   `clockwork.rb` periodically.  These commands clean up
+  logs etc.
+* Project-monitor-poller:  This application will contact the CI
+  providers periodically. 
+
+On a local machine, this 4 application setup is replaced with the cron
+tasks that are controlled by the `config/schedule.rb` file.
+
+#### Cloudfoundry environment variables
+
+Once you have set up your CF instance, you will want to modify some
+environment variables with `cf set-env`.  Specifically, you will
+probably want to supply
+
+* A whitelist of IP addresses allowed to view this project monitor
+* Your OAuth setup
+* Pivotal uses NewRelic to monitor the monitor, it requries a token
+  too
+* Our internal production setup uses `logentries` and this requires
+  setup too.  Logentries is no longer recommended or supported for new
+  deployments.
+
+#### Retired deployment method
+
+Cloudfoundry hosts apps that serve the endpoints `ci.pivotallabs.com`
+(for open source projects) and `pulse.pivotallabs.com` (Pivotal
+projects).  These were once deployed with `rake
+cf_deploy[orgname,spacename,environment]`, where environment is
+production or staging.  This will tag your current git commit and push
+the tag up to github, then do a `cf push`.  This mechanism is currently
+not used. 
+
 ### Heroku
-To get running on Heroku, after you have cloned and bundled, run the following commands:
+
+To get running on Heroku, after you have cloned and bundled, run the
+following commands:
 
 NB: These instructions are for the basic authentication strategy. 
 
@@ -220,7 +350,7 @@ Got a burning idea that just needs to be implemented? Check the CONTRIBUTE.md fi
 
 The google group for Project Monitor is [projectmonitor_pivotallabs](http://groups.google.com/group/projectmonitor_pivotallabs)
 
-Copyright (c) 2013 Pivotal Labs. This software is licensed under the MIT License.
+Copyright (c) 2013-2015 Pivotal Labs. This software is licensed under the MIT License.
 
 
 [![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/pivotal/projectmonitor/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

config/cf/manifest-production.yml

@@ -8,7 +8,6 @@ applications:
   command: bundle exec rake db:migrate && bundle exec unicorn -p $PORT -c ./config/unicorn.rb
   services:
   - db-production
-  - logentries-production-web
 - name: project-monitor-production-worker
   memory: 512M
   instances: 1
@@ -17,7 +16,6 @@ applications:
   no-route: true
   services:
   - db-production
-  - logentries-production-worker
 - name: project-monitor-production-clockwork
   memory: 512M
   instances: 1
@@ -26,7 +24,6 @@ applications:
   no-route: true
   services:
   - db-production
-  - logentries-production-clockwork
 - name: project-monitor-production-poller
   memory: 1024M
   instances: 1
@@ -35,4 +32,3 @@ applications:
   no-route: true
   services:
   - db-production
-  - logentries-production-poller