Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Changelog

All notable changes to ServiceMaker will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

#### Node.js/Foxx Service Support
- **Node.js Base Image**: Added `Dockerfile.node22base` for Node.js 22 base image with pre-installed ArangoDB packages
- Installs Node.js 22 from NodeSource
- Pre-installs `@arangodb/node-foxx`, `@arangodb/node-foxx-launcher`, and `@arangodb/arangodb` packages
- Creates base `node_modules` with checksums for dependency tracking
- Added to `baseimages/imagelist.txt` as `node22base`

- **Node.js Dockerfile Template**: Created `Dockerfile.nodejs.template` for building Node.js/Foxx service images
- Supports wrapper structure for single service directories
- Configures working directory and user permissions
- Executes `prepareproject-nodejs.sh` for dependency management

- **Dependency Management Script**: Added `scripts/prepareproject-nodejs.sh`
- Copies base `node_modules` from base image
- Installs project-specific dependencies from `package.json`
- Ensures `node-foxx` binary is always available with multiple safety checks
- Tracks new dependencies using SHA256 checksums
- Separates base packages from project packages
- Includes automatic recovery mechanism if base packages are removed during `npm install`
- Handles `package.json` copying to wrapper root for proper dependency installation

- **Project Type Detection**: Extended `detect_project_type()` to support:
- `python`: Projects with `pyproject.toml`
- `foxx`: Multi-service projects with `package.json` and `services.json`
- `foxx-service`: Single service directory with `package.json` (creates wrapper structure)
- `nodejs`: Generic Node.js projects

- **Wrapper Structure Generation**: Automatic wrapper creation for single service directories
- Creates `wrapper/` directory structure
- Copies service directory into `wrapper/{service-name}/`
- Generates `services.json` automatically with mount path configuration
- Copies `package.json` to wrapper root for dependency installation

- **CLI Arguments**:
- `--mount-path`: Required for `foxx-service` type, specifies the mount path for the Foxx service

- **Services JSON Generation**: Added `generate_services_json()` function
- Automatically generates `services.json` for single service directories
- Configures mount path and base path for Foxx services

- **Package.json Support**: Added functions to read Node.js project metadata
- `read_name_from_package_json()`: Extracts project name from `package.json`
- `read_service_info_from_package_json()`: Extracts name and version for Helm charts

- **Entrypoint Enhancement**: Updated `baseimages/scripts/entrypoint.sh` to support Node.js/Foxx services
- Detects service type based on project files
- Automatically runs `node-foxx` for Foxx services
- Falls back to generic Node.js execution for non-Foxx services
- Maintains backward compatibility with Python services

- **Test Service**: Added `itzpapalotl-node` test service in `testprojects/`
- Example Node.js/Foxx service for testing ServiceMaker functionality
- Demonstrates wrapper structure generation and dependency management

### Changed

- **Main Application Logic**: Extended `src/main.rs` to support Node.js projects
- Added project type detection for Node.js/Foxx services
- Updated file copying logic to handle wrapper structure
- Modified Dockerfile generation to use appropriate template based on project type
- Updated Helm chart generation to support Node.js projects
- Added `prepareproject-nodejs.sh` to embedded scripts list

- **Entrypoint Script**: Enhanced `baseimages/scripts/entrypoint.sh` to support Node.js/Foxx services
- Added service type detection based on project files
- Maintains backward compatibility with Python services

- **Base Image List**: Updated `baseimages/imagelist.txt` to include `node22base`

- **File Copying Logic**: Updated `copy_dir_recursive()` to skip `node_modules` directories
- Prevents copying local `node_modules` which should be installed in Docker build

### Fixed

- **Windows Compatibility**: Fixed Windows build issues in `src/main.rs`
- Added conditional compilation for Unix-specific file permissions (`#[cfg(unix)]`)
- Windows builds now skip `set_mode()` calls that are Unix-only

### Technical Details

- **Base Image Structure**:
- Base `node_modules` located at `/home/user/base_node_modules/node_modules`
- Checksums stored at `/home/user/sums_sha256` for dependency tracking
- Base packages: `@arangodb/node-foxx@^0.0.1-alpha.0`, `@arangodb/node-foxx-launcher@^0.0.1-alpha.0`, `@arangodb/arangodb@^0.0.1-alpha.0`

- **Wrapper Structure**:
```
wrapper/
├── package.json # Copied from service for npm install
├── services.json # Auto-generated with mount path
├── node_modules/ # Installed dependencies (base + project)
└── {service-name}/ # Service directory
├── package.json
└── ...
```

- **Dependency Tracking**:
- Uses SHA256 checksums to identify new files vs. base files
- New project dependencies copied to `/project/node_modules/` for tracking
- Base packages remain in base image for efficiency

## [0.9.2] - Previous Release

### Existing Features
- Python service support with `pyproject.toml`
- Base image management for Python 3.13
- Docker image building and pushing
- Helm chart generation
- Project tar.gz creation
- Virtual environment management with `uv`

---

## Version History

- **0.9.2**: Initial release with Python support
- **Unreleased**: Added Node.js/Foxx service support

---

## Notes

- All changes maintain backward compatibility with existing Python projects
- Node.js support is additive and does not affect Python service functionality
- Base images must be built separately using `baseimages/build.sh`
- Windows users should use WSL or Linux environment for building base images

17 changes: 17 additions & 0 deletions Dockerfile.nodejs.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM {BASE_IMAGE}

USER root

COPY ./scripts /scripts
COPY {PROJECT_DIR} /project/{PROJECT_DIR}
RUN chown -R user:user /project/{PROJECT_DIR}

USER user
WORKDIR /project/{PROJECT_DIR}

RUN /scripts/prepareproject-nodejs.sh

EXPOSE {PORT}

CMD ["node_modules/.bin/node-foxx"]

26 changes: 26 additions & 0 deletions baseimages/Dockerfile.node22base
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
FROM debian:trixie-slim

COPY ./scripts /scripts
RUN /scripts/debinstall.sh

# Install Node.js 22
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && \
apt-get install -y nodejs && \
apt-get clean

WORKDIR /home/user
USER user

# Create base node_modules with node-foxx dependencies
RUN mkdir -p base_node_modules && \
cd base_node_modules && \
npm init -y && \
npm install @arangodb/node-foxx@^0.0.1-alpha.0 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to come up with a good selection of "standard" tools and libs to include into the base image, so that many projects can benefit from having a good part of their dependencies in the base image. This crucially depends on us finding a way to avoid copying all dependencies to /project/node_modules.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the following npm libraries to the base image with version pinning for reproducible builds:
Essential utilities: lodash@^4.17.21, dayjs@^1.11.10, uuid@^9.0.1, dotenv@^16.4.5
HTTP clients: axios@^1.7.2
Validation: joi@^17.13.3
Logging: winston@^3.15.0
Async utilities: async@^3.2.5
Security: jsonwebtoken@^9.0.2, bcrypt@^5.1.1

As suggested, avoided copying dependencies to /project/node_modules. npm installs only project-specific dependencies that are missing or incompatible with the base packages in /home/user/node_modules. This keeps the base image immutable while allowing projects to override versions when needed.

@arangodb/node-foxx-launcher@^0.0.1-alpha.0 \
@arangodb/arangodb@^0.0.1-alpha.0

# Create checksums for base node_modules
RUN find base_node_modules/node_modules -type f -print0 | \
xargs -0 sha256sum > sums_sha256

CMD [ "/scripts/entrypoint.sh" ]
1 change: 1 addition & 0 deletions baseimages/imagelist.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
py13base
py13cugraph
py13torch
node22base
38 changes: 31 additions & 7 deletions baseimages/scripts/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,40 @@ if test -e project.tar.gz ; then
tar xzvf project.tar.gz > /dev/null
fi

# Run the entrypoint if configured:
# Detect service type and run accordingly
if test -e entrypoint ; then
ENTRYPOINT=$(cat entrypoint)
echo Running project ...
. /home/user/.local/bin/env
. /home/user/the_venv/bin/activate
for p in /project/the_venv/lib/python*/site-packages ; do
export PYTHONPATH=$p
done
exec python $ENTRYPOINT

# Check if it's a Node.js/Foxx service
if [ -f "package.json" ] && [ -f "services.json" ]; then
# Node.js/Foxx service
echo "Detected Node.js/Foxx service"
if [ -f "node_modules/.bin/node-foxx" ]; then
exec node_modules/.bin/node-foxx
else
echo "Error: node-foxx not found. Make sure node_modules are installed."
exit 1
fi
elif [ -f "package.json" ]; then
# Generic Node.js service
echo "Detected Node.js service"
if [ -f "$ENTRYPOINT" ]; then
exec node "$ENTRYPOINT"
else
echo "Error: Entrypoint file not found: $ENTRYPOINT"
exit 1
fi
else
# Python service (existing logic)
echo "Detected Python service"
. /home/user/.local/bin/env
. /home/user/the_venv/bin/activate
for p in /project/the_venv/lib/python*/site-packages ; do
export PYTHONPATH=$p
done
exec python $ENTRYPOINT
fi
fi

echo No entrypoint found, running bash instead...
Expand Down
126 changes: 126 additions & 0 deletions scripts/prepareproject-nodejs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
#!/bin/bash
# This is to be run in the project directory to install dependencies.
# It copies base node_modules from the base image and installs additional
# project dependencies, tracking changes similar to the Python script.

set -e

# We're already in /project/{PROJECT_DIR} from the Dockerfile WORKDIR
PROJECT_DIR=$(pwd)

# Copy base node_modules if they exist in base image
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do not want this. This is totally against the spirit of "base images". If we do this, we might as well start with an empty base image and put everything. The idea should be this:

Assume a node project needs libraries A, B, C, D, E and F. It declares all of them in its package.json file.
The idea of a "base image" is now to have a Docker image, which is totally immutable (allowing to pre-scan it for security vulnerabilities), which includes a suitable nodejs version (say 22) and might include some of these libraries already. So for example, it could contain A, B and C in versions, which fit would fulfill the requirements of the project.
But libraries D, E and F happen to not be included in the base image.
Now we need a way to use standard tooling to create a runnable environment. So the idea is that the base image contains its stuff under the path /home/user/node_modules (including A, B, and C in the right version).
I think we should now be able to tell npm to create a /project/node_modules, which contains only D, E and F in the right version, so that /project/node_modules/** and /home/user/node_modules together constitute the right environment for the project to run.
This has multiple benefits:

  1. We use automatic tools to install the project.
  2. We only put the additional stuff in /project, so that a potential project.tar.gz is as small as possible.
  3. We keep the base image immutable to allow for pre-scanning.
  4. We only have to scan the stuff in /project for vulnerabilities.
  5. We get a derived docker image, which is self-contained, and we get a project.tar.gz, which can be put on top of the base images, so that with this and the base image (immutable) we can run the project, too.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented dependency checking to avoid duplication and align with the base image approach.
What I implemented:

  • Pre-install dependency check: check-base-dependencies.js analyzes each dependency in package.json against /home/user/node_modules
  • Version compatibility: Uses semver to verify if base package versions satisfy project requirements
  • Selective installation: Only packages missing from base or with incompatible versions are installed to /project/{service-name}/node_modules
  • Result:
    - Base image (/home/user/node_modules) contains A, B, C (immutable, pre-scanned)
    - Project (/project/{service-name}/node_modules) contains only D, E, F (missing packages)
    - NODE_PATH resolves from both locations at runtime

if [ -d "/home/user/base_node_modules/node_modules" ]; then
echo "Copying base node_modules..."
if [ -d "node_modules" ]; then
# Merge with existing node_modules if any
cp -r /home/user/base_node_modules/node_modules/* ./node_modules/ 2>/dev/null || true
else
cp -r /home/user/base_node_modules/node_modules ./node_modules
fi

# Verify node-foxx binary exists after copy
if [ -f "node_modules/.bin/node-foxx" ]; then
echo "✓ Base node-foxx binary copied successfully"
else
echo "Warning: node-foxx binary not found after copying base node_modules"
fi

# Track existing files
cd /home/user
if [ -f "sums_sha256" ]; then
sha256sum -c sums_sha256 || true
fi
cd "$PROJECT_DIR"
else
echo "Warning: Base node_modules not found at /home/user/base_node_modules/node_modules"
fi

# Install additional project dependencies if package.json exists
if [ -f "package.json" ]; then
echo "Installing project dependencies..."

# Always ensure base packages are installed first (they provide node-foxx binary)
# This ensures node-foxx is available even if npm install does a clean install
echo "Ensuring base node-foxx packages are installed..."
npm install --production --no-save \
@arangodb/node-foxx@^0.0.1-alpha.0 \
@arangodb/node-foxx-launcher@^0.0.1-alpha.0 \
@arangodb/arangodb@^0.0.1-alpha.0 || {
echo "Warning: Failed to install base packages, continuing anyway..."
}

# Verify node-foxx exists after installing base packages
if [ ! -f "node_modules/.bin/node-foxx" ]; then
echo "ERROR: node-foxx binary not found after installing base packages!"
echo "Listing node_modules/.bin contents:"
ls -la node_modules/.bin/ 2>/dev/null || echo "node_modules/.bin directory does not exist"
exit 1
fi
echo "✓ Base node-foxx packages installed"

# Install project dependencies (this should preserve base packages)
npm install --production --no-save

# Final verification that node-foxx still exists
if [ -f "node_modules/.bin/node-foxx" ]; then
echo "✓ node-foxx binary exists after installing project dependencies"
else
echo "ERROR: node-foxx binary missing after npm install!"
echo "Reinstalling base packages..."
npm install --production --no-save \
@arangodb/node-foxx@^0.0.1-alpha.0 \
@arangodb/node-foxx-launcher@^0.0.1-alpha.0 \
@arangodb/arangodb@^0.0.1-alpha.0

# Final check
if [ ! -f "node_modules/.bin/node-foxx" ]; then
echo "ERROR: Failed to install node-foxx binary!"
echo "Current directory: $(pwd)"
echo "Listing node_modules/.bin contents:"
ls -la node_modules/.bin/ 2>/dev/null || echo "node_modules/.bin directory does not exist"
echo "Listing node_modules contents:"
ls -la node_modules/ 2>/dev/null | head -20
exit 1
fi
echo "✓ node-foxx binary restored"
fi

# Find all files in node_modules and create checksums
find node_modules -type f -print0 | xargs -0 sha256sum > /tmp/node_modules_sha256_new 2>/dev/null || true

# Find new files (files in current node_modules that weren't in base)
if [ -f "/home/user/sums_sha256" ] && [ -f "/tmp/node_modules_sha256_new" ]; then
cat /tmp/node_modules_sha256_new /home/user/sums_sha256 | sort | uniq -c | grep "^ 1 " | awk '{ print $3 }' > /tmp/newfiles || true
else
# If no base checksums, all files are new
if [ -f "/tmp/node_modules_sha256_new" ]; then
awk '{ print $3 }' /tmp/node_modules_sha256_new > /tmp/newfiles || true
fi
fi

# Move new files to /project/node_modules (similar to Python approach)
if [ -f "/tmp/newfiles" ]; then
mkdir -p /project/node_modules
while IFS= read -r filename; do
# Skip empty lines
[[ -z "$filename" ]] && continue

# Check if source file exists and is relative to current dir
if [[ "$filename" == node_modules/* ]] && [ -f "$filename" ]; then
# Get the directory part of the filename
DIR=$(dirname "$filename")

# Create the destination directory structure
mkdir -p "/project/$DIR"

# Copy the file preserving the directory hierarchy
cp "$filename" "/project/$filename"
fi
done < "/tmp/newfiles"
fi

rm -f /tmp/node_modules_sha256_new /tmp/newfiles
fi

echo "Node.js project prepared successfully"

Loading