Self hosting

README.md

@@ -204,6 +204,150 @@ If you are unable to use Docker, please contact a core team member to get instru
 
 </details>
 
+<details>
+  <summary>Instructions for self-hosting LiteFarm</summary>
+
+### Self-hosting LiteFarm
+
+To self-host LiteFarm, you will need to:
+
+1. Configure domain.
+2. Configure Linux server.
+3. Set up a web server (e.g. Nginx or Apache).
+4. Set up a [Postgres database](#database-setup).
+5. Set up services.
+6. LiteFarm configuration.
+7. Compile and startup LiteFarm.
+
+#### Domain
+
+LiteFarm uses few endpoints and each endpoint needs its own domain:
+
+- main address like `yourdomain.com` where the webapp is hosted,
+- api address like `api.yourdomain.com` where the api is hosted,
+- file service address like `files.yourdomain.com`.
+
+All domain should be pointed to the server IP address on which each service is hosted.
+If all services are hosted on the same server then it will be the same IP address.
+
+#### Linux server
+
+LiteFarm can be hosted on any server that supports Node.js, PostgreSQL or Docker. Recommended is Ubuntu or Debian
+but any Linux distribution should work.
+
+There are few possibilities to self-host LiteFarm:
+
+- from scratch on a server,
+- using Docker.
+
+Node.js should be in version pointed in `.nvmrc` file in the root of the repo.
+Using [NVM](https://github.com/nvm-sh/nvm) is recommended.
+
+Other recommendations:
+
+- install/start LiteFarm `api` service as non-root user,
+- configure firewall to allow only necessary ports,
+- use SSL certificates for secure connections.
+
+When installing from scratch, you will need to install Node.js, PostgreSQL, Redis and other dependencies manually.
+
+#### Web server
+
+LiteFarm is built with Node.js and can be run directly on the server.
+However, it is recommended to use a web server like Nginx or Apache to
+handle incoming requests and route them to the appropriate service.
+
+Using Nginx or Apache will also allow you to use SSL certificates for secure connections.
+
+Nginx or Apache will be also needed if all services are hosted on the same server.
+
+#### Postgres
+
+Database can be used from [Docker container](#postgresql-database)
+or [installed directly](#database---native-installation) on the server.
+
+#### LiteFarm services
+
+LiteFarm uses few services to handle images, documents and certification exports.
+This can be done using Docker containers or installed directly on the server
+as [described above](#services-local-development-dependencies).
+
+#### LiteFarm configuration
+
+When we have all services running, we need to configure packages LiteFarm to use them.
+
+1. Api service `packages/api/.env`:
+
+   ```ini
+   # Set correct NODE_ENV
+   NODE_ENV=production
+   PORT=5001
+   # Public API URL
+   API_PUBLIC_URL=https://api.yourdomain.com
+   # Webapp public URL
+   HOME_PUBLIC_URL=https://youdomain.com
+
+   # S3 storage configuration
+   S3_ENDPOINT=??
+   S3_ENDPOINT_BUCKET=??
+   S3_PUBLIC_BUCKET_NAME=??
+   S3_PRIVATE_BUCKET_NAME=???
+   S3_REGION=???
+   S3_ACCESS_KEY_ID=???
+   S3_SECRET_ACCESS_KEY=???
+   #S3_FORCE_PATH_STYLE=true/false
+
+   # Email configuration (SMTP)
+   #EMAIL_TRANSPORT_HOST=smtp.server.com
+   #EMAIL_TRANSPORT_PORT=465
+   #EMAIL_TRANSPORT_SECURE=true
+   #EMAIL_TRANSPORT_SERVICE=smtp
+   #EMAIL_TRANSPORT_USER=?
+   #EMAIL_TRANSPORT_PASSWORD=?
+   ```
+
+   Other variables should be set as in `.env.default` file.
+
+2. File service `packages/fileservice/.env`. Just use `packages/fileservice/.env.default` file as a template
+   and set the same values for `S3_*` variables as in `packages/api/.env` (you can omit the `PUBLIC` bucket settings)
+   and `JWT_FARM_SECRET` which should be the same as in `packages/api/.env`.
+3. Webapp `packages/webapp/.env`:
+   ```ini
+   VITE_API_URL=https://api.yourdomain.com
+   VITE_S3_SERVICE=https://files.yourdomain.com
+   ```
+   Other variables should be set as in `.env.default` file.
+
+#### Compile and startup LiteFarm
+
+Each endpoint should be compiled and started separately.
+
+1. Webapp:
+   ```bash
+   cd packages/webapp
+   pnpm build
+   ```
+   Now you can copy the `dist` folder to the web server root directory and configure the web server to serve it
+   (static files) or point the web server to the `dist` folder or use `serve` command to serve the files
+   and configure the web server to proxy the requests to the `serve` command/port.
+2. File service:
+   ```bash
+   cd packages/fileservice
+   npm run start
+   ```
+3. Api:
+   ```bash
+   cd packages/api
+   npm run build
+   npm run start:prod
+   ```
+   You need to keep in mind two things:
+
+- all [preliminaries](#preliminaries) should be done before running the commands above,
+- all [services](#services-local-development-dependencies) should be running before starting the api.
+
+</details>
+
 # Testing
 
 ## api

packages/api/.env.self-hosting

@@ -0,0 +1,70 @@
+# Variables for self-hosting configuration
+
+NODE_ENV=production
+TZ=UTC
+PORT=5001
+
+# Public URL of the API
+#API_PUBLIC_URL=https://api.litefarm.my.domain
+
+# Public URL of the webapp
+#HOME_PUBLIC_URL=https://litefarm.my.domain
+
+# Database connection settings
+DATABASE_HOST=localhost
+DATABASE=pg-litefarm
+DATABASE_USER=postgres
+DATABASE_PASSWORD=postgres
+DATABASE_PORT=5432
+
+# S3 Storage configuration
+#S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
+#S3_ENDPOINT_BUCKET=https://${bucket}.nyc3.digitaloceanspaces.com
+#S3_PUBLIC_BUCKET_NAME=litefarmapp
+#S3_PRIVATE_BUCKET_NAME=litefarm-app-secret
+#S3_REGION=us-east-1
+#S3_ACCESS_KEY_ID=?
+#S3_SECRET_ACCESS_KEY=?
+#S3_FORCE_PATH_STYLE=true
+
+# Imaginary URL
+#LOCAL_IMAGINARY=https://image.litefarm.org
+
+
+JWT_SECRET=This_will_(really)_work
+JWT_INVITE_SECRET=Any_arbitrary_string_will_do
+JWT_RESET_SECRET=Production_is_secured_with_a_long_random_string
+JWT_FARM_SECRET=Here_we_can_use_friendly_explanations
+JWT_SCHEDULER_SECRET=Another_token_was_needed_for_the_scheduler
+
+# Email configuration (SMTP)
+#EMAIL_TRANSPORT_HOST=smtp.server.com
+#EMAIL_TRANSPORT_PORT=465
+#EMAIL_TRANSPORT_SECURE=true
+#EMAIL_TRANSPORT_SERVICE=smtp
+#EMAIL_TRANSPORT_USER=?
+#EMAIL_TRANSPORT_PASSWORD=?
+
+# From name
+#EMAIL_SENDER=system@litefarm.org
+# Support email
+#EMAIL_SUPPORT=support@litefarm.org
+
+# Create your own (free in most cases) Google API key at https://console.cloud.google.com/apis/dashboard
+# (Optional) We use google-maps-services-js package so API services on this key can be restricted to APIs listed here:
+#  > https://www.npmjs.com/package/@googlemaps/google-maps-services-js
+# Same as the VITE_GOOGLE_MAPS_API_KEY under webapp
+GOOGLE_API_KEY=?
+
+# Currently a Pro/student account is needed for full functionality (e.g sensor charts)
+# Most functionality is covered by the free OpenWeather API key at https://openweathermap.org/price
+# Note: OpenWeather may take a few hours before the api key is working
+# Same as the VITE_WEATHER_API_KEY under webapp
+OPEN_WEATHER_APP_ID=?
+
+# Create your own Google API key at https://developers.google.com/identity/oauth2/web/guides/get-google-api-clientid
+# (Optional) Authorized javascript origins can be restricted with localhost, and localhost:<port>
+# (Optional) If gmail refresh tokens are used (see below): Authorized redirect URI's could be https://developers.google.com/oauthplayground
+GOOGLE_OAUTH_CLIENT_ID=?
+
+

packages/api/.knex/knexfile.js

@@ -67,7 +67,14 @@ export default {
   production: {
     client: 'postgresql',
     debug: true,
-    connection: process.env.DATABASE_URL,
+    connection: process.env.DATABASE_URL || {
+      host: process.env.DATABASE_HOST,
+      database: process.env.DATABASE,
+      user: process.env.DATABASE_USER,
+      password: process.env.DATABASE_PASSWORD,
+      port: process.env.DATABASE_PORT || 5432,
+      ssl: { rejectUnauthorized: false },
+    },
     migrations,
     seeds,
     ssl: {

packages/api/package.json

@@ -13,7 +13,7 @@
     "test-end2end": "jest endToEnd",
     "jest-w": "jest --watch",
     "start": "set NODE_ENV=development&& node --import @swc-node/register/esm-register src/server.ts",
-    "start:prod": "node dist/app/src/server.js",
+    "start:prod": "node dist/api/src/server.js",
     "build": "rimraf dist && tsc -p src/tsconfig.json",
     "cp-uncompiled-to-dist": "rsync -a --include='*' --exclude='*.cjs' --exclude='*.js' --exclude='*.ts' ./src/templates ./dist/app/src && rsync -a --include='*' --exclude='*.cjs' --exclude='*.js' --exclude='*.ts' ../shared/locales ./dist/shared",
     "dev": "set NODE_ENV=development && node --inspect=0.0.0.0:9230 --watch --import @swc-node/register/esm-register src/server.ts",

packages/api/src/jobs/certification/do_retrieve.js

@@ -14,9 +14,9 @@ export default (nextQueue, emailQueue) => (job, done) => {
     `temp/${exportId}`, // destination
     '--recursive',
     `--endpoint=${
-      process.env.NODE_ENV === 'development'
-        ? process.env.MINIO_ENDPOINT
-        : 'https://nyc3.digitaloceanspaces.com'
+        process.env.S3_ENDPOINT || (process.env.NODE_ENV === 'development'
+            ? process.env.MINIO_ENDPOINT
+            : 'https://nyc3.digitaloceanspaces.com')
     }`,
     '--exclude=*',
   ].concat(files.map(({ url }) => `--include=${url.split('/').pop()}`));

packages/api/src/jobs/certification/upload.js

@@ -14,9 +14,9 @@ export default (emailQueue) => (job, done) => {
     `temp/${exportId}.zip`, // location
     `s3://${fileIdentifier}.zip`, // destination
     `--endpoint=${
-      process.env.NODE_ENV === 'development'
-        ? process.env.MINIO_ENDPOINT
-        : 'https://nyc3.digitaloceanspaces.com'
+        process.env.S3_ENDPOINT || (process.env.NODE_ENV === 'development'
+            ? process.env.MINIO_ENDPOINT
+            : 'https://nyc3.digitaloceanspaces.com')
     }`,
   ];
   const awsCopyProcess = spawn('aws', args, { cwd: process.env.EXPORT_WD });

packages/api/src/jobs/notifications/index.js

@@ -18,7 +18,7 @@ import axios from 'axios';
 import Queue from 'bull';
 import jwt from 'jsonwebtoken';
 const { sign } = jwt;
-const apiUrl = process.env.API_URL || 'http://localhost:5001';
+const apiUrl = process.env.API_URL || ('http://localhost:'+process.env.PORT);
 const mockTimer = !!process.env.MOCK_TIMER;
 
 // UTC day to send weeklies for zones >= 7

packages/api/src/server.ts

@@ -247,6 +247,7 @@ const rejectBodyInGetAndDelete: RequestHandler = (req, res, next) => {
 };
 
 const getAllowedOrigin = () => {
+  if(process.env.HOME_PUBLIC_URL) return process.env.HOME_PUBLIC_URL;
   switch (environment) {
     case 'development':
       return '*';

packages/api/src/templates/sendEmailTemplate.js

@@ -14,6 +14,7 @@
  */
 
 import credentials from '../credentials.js';
+import { getEnvInt, getEnvBool, getEnv } from '../util/env.js'
 
 const dir = path.dirname(fileURLToPath(import.meta.url));
 
@@ -38,7 +39,9 @@ const emails = {
 function homeUrl(defaultUrl = 'http://localhost:3000') {
   const environment = process.env.NODE_ENV || 'development';
   let homeUrl = defaultUrl;
-  if (environment === 'integration') {
+  if(process.env.HOME_PUBLIC_URL) {
+    homeUrl = process.env.HOME_PUBLIC_URL;
+  } else if (environment === 'integration') {
     homeUrl = 'https://beta.litefarm.org';
   } else if (environment === 'production') {
     homeUrl = 'https://app.litefarm.org';
@@ -48,6 +51,13 @@ function homeUrl(defaultUrl = 'http://localhost:3000') {
 
 function gmailAuth() {
   const environment = process.env.NODE_ENV || 'development';
+  if(process.env.EMAIL_TRANSPORT_USER && process.env.EMAIL_TRANSPORT_PASSWORD) {
+    return {
+      user: process.env.EMAIL_TRANSPORT_USER,
+      pass: process.env.EMAIL_TRANSPORT_PASSWORD,
+    }
+  }
+
   if (environment === 'development' && process.env.DEV_GMAIL) {
     return {
       user: process.env.DEV_GMAIL,
@@ -79,10 +89,10 @@ const emailTransporter = new EmailTemplates({
   preview: process.env.DEBUG === 'email-templates' || process.env.DEBUG === 'i18n:*',
   subjectPrefix: process.env.NODE_ENV === 'production' ? false : '[Development] ',
   transport: {
-    host: 'smtp.gmail.com',
-    port: 465,
-    secure: true,
-    service: 'gmail',
+    host: getEnv('EMAIL_TRANSPORT_HOST', 'smtp.gmail.com'),
+    port: getEnvInt('EMAIL_TRANSPORT_PORT', 465),
+    secure: getEnvBool('EMAIL_TRANSPORT_SECURE', true),
+    service: getEnv('EMAIL_TRANSPORT_SERVICE', 'gmail'),
     auth: gmailAuth(),
   },
 });
@@ -91,7 +101,7 @@ function sendEmail(
   template_path,
   replacements,
   email_to,
-  { sender = 'system@litefarm.org', buttonLink = null, attachments = [] },
+  { sender = getEnv('EMAIL_SENDER', 'system@litefarm.org'), buttonLink = null, attachments = [] },
 ) {
   try {
     replacements.url = homeUrl();
@@ -110,7 +120,7 @@ function sendEmail(
       locals: replacements,
     };
     if (template_path.path === emails.HELP_REQUEST_EMAIL.path) {
-      mailOptions.message.cc = 'support@litefarm.org';
+      mailOptions.message.cc = getEnv('EMAIL_SUPPORT', 'support@litefarm.org');
     }
     if (
       attachments.length &&