From 0d852133591d8c73c2c325e2f3830fa7a138a581 Mon Sep 17 00:00:00 2001 From: Colton Shaw Date: Fri, 19 Dec 2025 11:06:22 -0500 Subject: [PATCH 1/2] initial guide on fapolidy and firewalld --- .../configure/calls-deployment.md | 20 ++ .../server/linux/deploy-rhel.rst | 229 ++++++++++++++++++ 2 files changed, 249 insertions(+) diff --git a/source/administration-guide/configure/calls-deployment.md b/source/administration-guide/configure/calls-deployment.md index 6942cf277c3..7c11dab82d6 100644 --- a/source/administration-guide/configure/calls-deployment.md +++ b/source/administration-guide/configure/calls-deployment.md @@ -127,6 +127,26 @@ Mattermost Calls can function in air-gapped environments. Exposing Calls to the - Configuring a STUN server is unnecessary, as all connections occur within the local network. - The [ICE Host Override](https://docs.mattermost.com/configure/plugins-configuration-settings.html#ice-host-override) configuration setting can be optionally set with a local IP address (e.g., 192.168.1.45), depending on the specific network configuration and topology. +### RHEL deployments with firewalld and fapolicyd + +When deploying Mattermost Calls or the rtcd service on RHEL systems with firewalld or fapolicyd enabled, additional firewall and security configuration is required. + +**Required ports:** +- **Calls plugin (integrated)**: Port 8443 UDP/TCP for RTC media traffic +- **rtcd service (standalone)**: Ports 8443 UDP/TCP for RTC media traffic + Port 8045 TCP for API communication with Mattermost + +**Configuration steps:** + +For complete firewalld and fapolicyd configuration instructions, including troubleshooting steps and example rules, see the [RHEL deployment guide](https://docs.mattermost.com/deploy/server/deploy-rhel.html). The guide includes: +- firewalld port configuration for Mattermost, Calls plugin, and rtcd service +- fapolicyd rules for Mattermost server and Calls plugin +- Separate fapolicyd configuration for standalone rtcd service +- Troubleshooting guidance for "operation not permitted" errors + +```{note} +The Calls plugin runs as part of Mattermost and is covered by the standard Mattermost fapolicyd rules. Only the standalone rtcd service requires separate fapolicyd configuration. +``` + ## Limitations - All Mattermost customers can start, join, and participate in 1:1 audio calls with optional screen sharing. diff --git a/source/deployment-guide/server/linux/deploy-rhel.rst b/source/deployment-guide/server/linux/deploy-rhel.rst index 2c9376edef4..57701501ac7 100644 --- a/source/deployment-guide/server/linux/deploy-rhel.rst +++ b/source/deployment-guide/server/linux/deploy-rhel.rst @@ -234,6 +234,235 @@ The final step, depending on your requirements, is to run sudo ``systemctl enabl - `A Sysadmin's Guide to SELinux: 42 Answers to the Big Questions `_ - `Mastering SELinux: A Comprehensive Guide to Linux Security `_ +.. important:: + + **Configure firewalld for government hardened environments**: When deploying Mattermost on RHEL systems with firewalld enabled, you'll need to configure firewall rules to allow access to the Mattermost Server. + + Firewalld is the default firewall management tool on RHEL systems and provides a dynamically managed firewall with support for network zones. By default, it may block incoming connections to Mattermost's port (8065). + + Check if firewalld is running: + + .. code-block:: sh + + sudo systemctl status firewalld + + If firewalld is active, configure it to allow Mattermost traffic: + + **Configure firewalld for Mattermost Server:** + + 1. Open port 8065 for Mattermost Server: + + .. code-block:: sh + + sudo firewall-cmd --permanent --add-port=8065/tcp + + 2. If you're using a reverse proxy (e.g., nginx or Apache) on the same server, also open HTTP and HTTPS ports: + + .. code-block:: sh + + sudo firewall-cmd --permanent --add-service=http + sudo firewall-cmd --permanent --add-service=https + + 3. Reload firewalld to apply the changes: + + .. code-block:: sh + + sudo firewall-cmd --reload + + 4. Verify the configuration: + + .. code-block:: sh + + sudo firewall-cmd --list-all + + You should see port 8065/tcp (and optionally http/https services) in the output. + + .. note:: + + If you're using a custom port for Mattermost, replace ``8065`` with your configured port number. The ``--permanent`` flag ensures the rules persist after system reboots. + + **Configure firewalld for Mattermost Calls (optional):** + + If you're deploying the Mattermost Calls plugin or the standalone rtcd service, additional ports need to be opened. See the :doc:`Calls deployment ` documentation for details on Calls architecture and deployment modes. + + For integrated Calls plugin (running alongside Mattermost): + + .. code-block:: sh + + sudo firewall-cmd --permanent --add-port=8443/udp + sudo firewall-cmd --permanent --add-port=8443/tcp + sudo firewall-cmd --reload + + For standalone rtcd service (requires additional API port): + + .. code-block:: sh + + sudo firewall-cmd --permanent --add-port=8443/udp + sudo firewall-cmd --permanent --add-port=8443/tcp + sudo firewall-cmd --permanent --add-port=8045/tcp + sudo firewall-cmd --reload + + .. note:: + + - Port 8443 (UDP/TCP) is used for RTC media traffic (audio/video/screen sharing) + - Port 8045 (TCP) is used for API communication between Mattermost and rtcd service + - TCP support for RTC requires Calls v0.17+ and rtcd v0.11+ + - If you're using custom ports, replace the default port numbers with your configured values + +.. important:: + + **Configure fapolicyd for government hardened environments**: When deploying Mattermost on RHEL systems with fapolicyd enabled, you may encounter "operation not permitted" errors in the Mattermost logs. + + Fapolicyd (File Access Policy Daemon) is a userspace daemon that determines access rights to files based on trust. It's commonly enabled in secure environments to prevent execution of unauthorized binaries. By default, fapolicyd may block Mattermost binaries and plugins from executing. + + Troubleshoot fapolicyd issues: + + 1. If you're seeing "operation not permitted" errors in your Mattermost logs, first check if fapolicyd is the cause by temporarily stopping it: + + .. code-block:: sh + + sudo systemctl stop fapolicyd + + 2. Test your Mattermost deployment. If it works now, fapolicyd is blocking Mattermost execution and you need to configure it properly. + + 3. Start fapolicyd in debug mode to identify what's being denied: + + .. code-block:: sh + + sudo fapolicyd --debug + + 4. Look for denial messages in the output. They will look similar to: + + .. code-block:: text + + rule=15 dec=deny_audit perm=execute auid=-1 pid=19735 + exe=/opt/mattermost/bin/mattermost : path=/opt/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64 + ftype=application/x-executable trust=0 + + Note the rule number that's denying execution - your rule file must be numbered to come before this rule. + + Configure fapolicyd to allow Mattermost: + + 1. Create a rule file for Mattermost. The naming convention is critical - it must come before the rule that's denying Mattermost. If you're using a stock fapolicyd configuration, ``80`` works fine: + + .. code-block:: sh + + sudo touch /etc/fapolicyd/rules.d/80-mattermost.rules + + 2. Add the allow rules to the file: + + .. code-block:: sh + + sudo vi /etc/fapolicyd/rules.d/80-mattermost.rules + + Add the following rules to allow Mattermost execution: + + .. code-block:: text + + allow perm=execute exe=/usr/bin/sudo trust=1 : dir=/opt/mattermost/ all trust=0 + allow perm=execute exe=/opt/mattermost/bin/mattermost : dir=/opt/mattermost all trust=0 + allow perm=execute exe=/usr/lib/systemd/systemd trust=1 : dir=/opt/mattermost/ all trust=0 + + 3. Check that the rules are valid and will be applied: + + .. code-block:: sh + + sudo fagenrules --check + + You should see "Rules have changed and should be updated" in the output. + + 4. Load the new rules: + + .. code-block:: sh + + sudo fagenrules --load + + 5. Restart the fapolicyd service: + + .. code-block:: sh + + sudo systemctl restart fapolicyd + + 6. Restart Mattermost to verify it works correctly: + + .. code-block:: sh + + sudo systemctl restart mattermost + + 7. Verify that Mattermost is running without errors: + + .. code-block:: sh + + curl http://localhost:8065 + sudo systemctl status mattermost + + .. note:: + + The rule file numbering is critical. Rules are processed in order, and your allow rules must come before any deny rules. If you continue to see denials after following these steps, check your deny rule number (from the debug output) and ensure your rule file number is lower. + + .. note:: + + If you have plugins that execute binaries (like Playbooks or the Calls plugin), the fapolicyd rules above allow execution of any binaries within the ``/opt/mattermost`` directory. If you need more restrictive rules, you can create specific rules for each plugin binary. + + **Configure fapolicyd for rtcd service (optional):** + + If you're deploying the standalone rtcd service for Mattermost Calls, you'll need separate fapolicyd rules for the rtcd binary. The rtcd service is deployed separately from Mattermost and runs its own binary. See the :doc:`Calls deployment ` documentation for details on when and why to use rtcd. + + 1. Create a rule file for rtcd. The naming convention is critical - it must come before the deny rule (typically 80 works): + + .. code-block:: sh + + sudo touch /etc/fapolicyd/rules.d/80-rtcd.rules + + 2. Add the allow rules for rtcd: + + .. code-block:: sh + + sudo vi /etc/fapolicyd/rules.d/80-rtcd.rules + + Add the following rules (adjust paths based on your rtcd installation directory): + + .. code-block:: text + + allow perm=execute exe=/usr/bin/sudo trust=1 : dir=/opt/rtcd/ all trust=0 + allow perm=execute exe=/opt/rtcd/bin/rtcd : dir=/opt/rtcd all trust=0 + allow perm=execute exe=/usr/lib/systemd/systemd trust=1 : dir=/opt/rtcd/ all trust=0 + + 3. Check that the rules are valid and will be applied: + + .. code-block:: sh + + sudo fagenrules --check + + You should see "Rules have changed and should be updated" in the output. + + 4. Load the new rules: + + .. code-block:: sh + + sudo fagenrules --load + + 5. Restart fapolicyd and the rtcd service: + + .. code-block:: sh + + sudo systemctl restart fapolicyd + sudo systemctl restart rtcd + + 6. Verify that rtcd is running without errors: + + .. code-block:: sh + + sudo systemctl status rtcd + + .. note:: + + - The paths shown above assume rtcd is installed in ``/opt/rtcd/``. Adjust these paths based on your actual rtcd installation directory. + - If you continue to see denials after following these steps, run ``sudo fapolicyd --debug`` to identify the specific paths being denied and adjust your rules accordingly. + - See the :doc:`rtcd deployment documentation ` for installation and configuration details. + + See the `Mattermost and fapolicyd `_ support article for additional troubleshooting steps. + Step 6: Update the server ~~~~~~~~~~~~~~~~~~~~~~~~~ From 0f24709b967dc1bbfc9e5f292eb74d08533470e1 Mon Sep 17 00:00:00 2001 From: Colton Shaw Date: Fri, 19 Dec 2025 13:36:05 -0500 Subject: [PATCH 2/2] covert to tabs --- .../server/linux/deploy-rhel.rst | 106 ++++++++++-------- 1 file changed, 62 insertions(+), 44 deletions(-) diff --git a/source/deployment-guide/server/linux/deploy-rhel.rst b/source/deployment-guide/server/linux/deploy-rhel.rst index 57701501ac7..c0e0ad129d4 100644 --- a/source/deployment-guide/server/linux/deploy-rhel.rst +++ b/source/deployment-guide/server/linux/deploy-rhel.rst @@ -176,77 +176,93 @@ Verify that Mattermost is running: curl ``http://localhost:8065``. You should se The final step, depending on your requirements, is to run sudo ``systemctl enable mattermost.service`` so that Mattermost will start on system boot. If you don't receive an error when starting Mattermost after the previous step, you are good to go. If you did receive an error, continue on. -.. important:: +**Hardened RHEL environments** - **Modify SELinux settings**: When deploying Mattermost from RHEL9, which has SELinux running with enforceing mode enabled by default, additional configuration is required. +When deploying Mattermost on hardened RHEL systems, additional security configurations may be required. Select the tab below for your specific security tool: - - SELinux is a security module that provides access control security policies. It's enabled by default on RHEL and CentOS systems. SELinux can block access to files, directories, and ports, which can cause issues when starting Mattermost. To resolve these issues, you'll need to set the appropriate SELinux contexts for the Mattermost binaries and directories, and allow Mattermost to bind to ports. - - Ensure that SELinux is enabled and in enforcing mode by running the ``sestatus`` command. If it's ``enforcing``, you'll need to configure it properly. - - Set bin contexts for ``/opt/mattermost/bin``: SELinux enforces security contexts for binaries. To label the Mattermost binaries as safe, you'll need to set them to the below SELinux context. +.. tab:: SELinux - .. code-block:: sh + When deploying Mattermost from RHEL9, which has SELinux running with enforcing mode enabled by default, additional configuration is required. - sudo semanage fcontext -a -t bin_t "/opt/mattermost/bin(/.*)?" - sudo restorecon -RF /opt/mattermost/bin + SELinux is a security module that provides access control security policies. It's enabled by default on RHEL and CentOS systems. SELinux can block access to files, directories, and ports, which can cause issues when starting Mattermost. To resolve these issues, you'll need to set the appropriate SELinux contexts for the Mattermost binaries and directories, and allow Mattermost to bind to ports. - Now, try starting Mattermost again with + Ensure that SELinux is enabled and in enforcing mode by running the ``sestatus`` command. If it's ``enforcing``, you'll need to configure it properly. - .. code-block:: sh + **Set bin contexts for /opt/mattermost/bin:** + + SELinux enforces security contexts for binaries. To label the Mattermost binaries as safe, you'll need to set them to the below SELinux context. + + .. code-block:: sh + + sudo semanage fcontext -a -t bin_t "/opt/mattermost/bin(/.*)?" + sudo restorecon -RF /opt/mattermost/bin - sudo systemctl start mattermost + Now, try starting Mattermost again with - If you don't receive an error, verify that Mattermost is running: curl ``http://localhost:8065``. You should see the HTML that's returned by the Mattermost Server. You're all set! + .. code-block:: sh - If on starting Mattermost you receive an error, before moving on, check for the existence of a file in ``/opt/mattermost/logs`` - if ``mattermost.log`` exists in that directory, it's more likely you're dealing with a configuration issue in ``config.json``. Double check the previous steps before continuing + sudo systemctl start mattermost - Try different contexts for ``/opt/mattermost``: SELinux enforces security contexts for files and directories. To label your Mattermost directory as safe, you'll need to set an appropriate SELinux context. + If you don't receive an error, verify that Mattermost is running: curl ``http://localhost:8065``. You should see the HTML that's returned by the Mattermost Server. You're all set! - 1. Check current context by running ``ls -Z /opt/mattermost``. When you see something like ``drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 mattermost`` returned, the ``default_t`` indicates that SELinux doesn't know what this directory is for. - 2. Set a safe context by assigning a SELinux type that's compatible with web services or applications by running ``sudo semanage fcontext -a -t httpd_sys_content_t "/opt/mattermost(/.*)?"``. A common one is ``httpd_sys_content_t``, used for serving files. Ensure you match the directory and its contents recursively. Run the ``sudo restorecon -R /opt/mattermost`` to apply the changes. + If on starting Mattermost you receive an error, before moving on, check for the existence of a file in ``/opt/mattermost/logs`` - if ``mattermost.log`` exists in that directory, it's more likely you're dealing with a configuration issue in ``config.json``. Double check the previous steps before continuing. - Allow Mattermost to bind to ports: When Mattermost needs specific ports (e.g., 8065), ensure that SELinux allows it by allowing Mattermost to bind to ports. Run the ``sudo semanage port -l | grep 8065`` command, and if the port's not listed, you'll need to add it by running ``sudo semanage port -a -t http_port_t -p tcp 8065``, replacing the ``8065`` with the required port. + **Try different contexts for /opt/mattermost:** - Handle custom policies: If Mattermost requires actions that SELinux blocks, you'll need to generate a custom policy. + SELinux enforces security contexts for files and directories. To label your Mattermost directory as safe, you'll need to set an appropriate SELinux context. - 1. Check for SELinux denials first in the logs by running ``sudo ausearch -m avc -ts recent``, or by checking the audit log: ``sudo cat /var/log/audit/audit.log | grep denied``. + 1. Check current context by running ``ls -Z /opt/mattermost``. When you see something like ``drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 mattermost`` returned, the ``default_t`` indicates that SELinux doesn't know what this directory is for. + 2. Set a safe context by assigning a SELinux type that's compatible with web services or applications by running ``sudo semanage fcontext -a -t httpd_sys_content_t "/opt/mattermost(/.*)?"``. A common one is ``httpd_sys_content_t``, used for serving files. Ensure you match the directory and its contents recursively. Run the ``sudo restorecon -R /opt/mattermost`` to apply the changes. - 2. If needed, generate a policy module by installing ``audit2allow`` to generate policies automatically. + **Allow Mattermost to bind to ports:** - .. code-block:: sh + When Mattermost needs specific ports (e.g., 8065), ensure that SELinux allows it by allowing Mattermost to bind to ports. Run the ``sudo semanage port -l | grep 8065`` command, and if the port's not listed, you'll need to add it by running ``sudo semanage port -a -t http_port_t -p tcp 8065``, replacing the ``8065`` with the required port. - sudo yum install -y policycoreutils-python-utils - sudo grep mattermost /var/log/audit/audit.log | audit2allow -M mattermost_policy - sudo semodule -i mattermost_policy.pp + **Handle custom policies:** - Test the configuration: Restart Mattermost to confirm the configuation works as expected by running ``sudo systemctl restart mattermost``. In the case of failures, revisit the logs to identify other SELinux-related issues. + If Mattermost requires actions that SELinux blocks, you'll need to generate a custom policy. - Need Mattermost working quickly for testing purposes? + 1. Check for SELinux denials first in the logs by running ``sudo ausearch -m avc -ts recent``, or by checking the audit log: ``sudo cat /var/log/audit/audit.log | grep denied``. - - You can change SELinux to permissive mode by running the ``sudo setenforce 0``. command where policies aren't enforced, only logged. - - This command changes the SELinux mode to "permissive". While in permissive mode, policies aren't enforced, and violations are logged instead of being blocked. This can be helpful for debugging and troubleshooting issues related to SELinux policies. - - Ensure you re-enable enforcing mode once context is working as needed by running the ``sudo setenforce 1`` command. + 2. If needed, generate a policy module by installing ``audit2allow`` to generate policies automatically. - See the following SELinux resources for additional details: + .. code-block:: sh - - `SELinux User's and Administrator's Guide `_ - - `SELinux Project Wiki `_ - - `Introduction to SELinux `_ - - `A Sysadmin's Guide to SELinux: 42 Answers to the Big Questions `_ - - `Mastering SELinux: A Comprehensive Guide to Linux Security `_ + sudo yum install -y policycoreutils-python-utils + sudo grep mattermost /var/log/audit/audit.log | audit2allow -M mattermost_policy + sudo semodule -i mattermost_policy.pp -.. important:: + **Test the configuration:** - **Configure firewalld for government hardened environments**: When deploying Mattermost on RHEL systems with firewalld enabled, you'll need to configure firewall rules to allow access to the Mattermost Server. + Restart Mattermost to confirm the configuation works as expected by running ``sudo systemctl restart mattermost``. In the case of failures, revisit the logs to identify other SELinux-related issues. + + **Need Mattermost working quickly for testing purposes?** + + - You can change SELinux to permissive mode by running the ``sudo setenforce 0``. command where policies aren't enforced, only logged. + - This command changes the SELinux mode to "permissive". While in permissive mode, policies aren't enforced, and violations are logged instead of being blocked. This can be helpful for debugging and troubleshooting issues related to SELinux policies. + - Ensure you re-enable enforcing mode once context is working as needed by running the ``sudo setenforce 1`` command. + + **Additional resources:** + + - `SELinux User's and Administrator's Guide `_ + - `SELinux Project Wiki `_ + - `Introduction to SELinux `_ + - `A Sysadmin's Guide to SELinux: 42 Answers to the Big Questions `_ + - `Mastering SELinux: A Comprehensive Guide to Linux Security `_ + +.. tab:: firewalld + + When deploying Mattermost on RHEL systems with firewalld enabled, you'll need to configure firewall rules to allow access to the Mattermost Server. Firewalld is the default firewall management tool on RHEL systems and provides a dynamically managed firewall with support for network zones. By default, it may block incoming connections to Mattermost's port (8065). - Check if firewalld is running: + **Check if firewalld is running:** .. code-block:: sh sudo systemctl status firewalld - If firewalld is active, configure it to allow Mattermost traffic: + If firewalld is active, configure it to allow Mattermost traffic. **Configure firewalld for Mattermost Server:** @@ -309,13 +325,13 @@ The final step, depending on your requirements, is to run sudo ``systemctl enabl - TCP support for RTC requires Calls v0.17+ and rtcd v0.11+ - If you're using custom ports, replace the default port numbers with your configured values -.. important:: +.. tab:: fapolicyd - **Configure fapolicyd for government hardened environments**: When deploying Mattermost on RHEL systems with fapolicyd enabled, you may encounter "operation not permitted" errors in the Mattermost logs. + When deploying Mattermost on RHEL systems with fapolicyd enabled, you may encounter "operation not permitted" errors in the Mattermost logs. Fapolicyd (File Access Policy Daemon) is a userspace daemon that determines access rights to files based on trust. It's commonly enabled in secure environments to prevent execution of unauthorized binaries. By default, fapolicyd may block Mattermost binaries and plugins from executing. - Troubleshoot fapolicyd issues: + **Troubleshoot fapolicyd issues:** 1. If you're seeing "operation not permitted" errors in your Mattermost logs, first check if fapolicyd is the cause by temporarily stopping it: @@ -341,7 +357,7 @@ The final step, depending on your requirements, is to run sudo ``systemctl enabl Note the rule number that's denying execution - your rule file must be numbered to come before this rule. - Configure fapolicyd to allow Mattermost: + **Configure fapolicyd to allow Mattermost:** 1. Create a rule file for Mattermost. The naming convention is critical - it must come before the rule that's denying Mattermost. If you're using a stock fapolicyd configuration, ``80`` works fine: @@ -461,7 +477,9 @@ The final step, depending on your requirements, is to run sudo ``systemctl enabl - If you continue to see denials after following these steps, run ``sudo fapolicyd --debug`` to identify the specific paths being denied and adjust your rules accordingly. - See the :doc:`rtcd deployment documentation ` for installation and configuration details. - See the `Mattermost and fapolicyd `_ support article for additional troubleshooting steps. + **Additional resources:** + + - `Mattermost and fapolicyd `_ support article Step 6: Update the server ~~~~~~~~~~~~~~~~~~~~~~~~~