Skip to content

Commit 1e13031

Browse files
letonghanrootpre-commit-ci[bot]
authored
[Translation] Support manifests and nginx (#812)
Signed-off-by: letonghan <letong.han@intel.com> Signed-off-by: root <root@a4bf019305c5.jf.intel.com> Co-authored-by: root <root@a4bf019305c5.jf.intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent b205dc7 commit 1e13031

File tree

15 files changed

+1422
-37
lines changed

15 files changed

+1422
-37
lines changed

‎.github/CODEOWNERS‎

100644100755
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,10 @@
33
/ChatQnA/ liang1.lv@intel.com
44
/CodeGen/ liang1.lv@intel.com
55
/CodeTrans/ sihan.chen@intel.com
6-
/DocSum/ sihan.chen@intel.com
6+
/DocSum/ letong.han@intel.com
77
/DocIndexRetriever/ xuhui.ren@intel.com chendi.xue@intel.com
88
/FaqGen/ xinyao.wang@intel.com
9-
/SearchQnA/ letong.han@intel.com
9+
/SearchQnA/ sihan.chen@intel.com
1010
/Translation/ liang1.lv@intel.com
1111
/VisualQnA/ liang1.lv@intel.com
1212
/ProductivitySuite/ hoong.tee.yeoh@intel.com

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Deployment are based on released docker images by default, check [docker image l
4545
| DocSum | [Xeon Instructions](DocSum/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](DocSum/docker_compose/intel/hpu/gaudi/README.md) | [DocSum with Manifests](DocSum/kubernetes/intel/README.md) | [DocSum with Helm Charts](https://github.com/opea-project/GenAIInfra/tree/main/helm-charts/docsum/README.md) | [DocSum with GMC](DocSum/kubernetes/intel/README_gmc.md) |
4646
| SearchQnA | [Xeon Instructions](SearchQnA/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](SearchQnA/docker_compose/intel/hpu/gaudi/README.md) | Not Supported | Not Supported | [SearchQnA with GMC](SearchQnA/kubernetes/intel/README_gmc.md) |
4747
| FaqGen | [Xeon Instructions](FaqGen/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](FaqGen/docker_compose/intel/hpu/gaudi/README.md) | [FaqGen with Manifests](FaqGen/kubernetes/intel/README.md) | Not Supported | [FaqGen with GMC](FaqGen/kubernetes/intel/README_gmc.md) |
48-
| Translation | [Xeon Instructions](Translation/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](Translation/docker_compose/intel/hpu/gaudi/README.md) | Not Supported | Not Supported | [Translation with GMC](Translation/kubernetes/intel/README_gmc.md) |
48+
| Translation | [Xeon Instructions](Translation/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](Translation/docker_compose/intel/hpu/gaudi/README.md) | [Translation with Manifests](Translation/kubernetes/intel/README.md) | Not Supported | [Translation with GMC](Translation/kubernetes/intel/README_gmc.md) |
4949
| AudioQnA | [Xeon Instructions](AudioQnA/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](AudioQnA/docker_compose/intel/hpu/gaudi/README.md) | [AudioQnA with Manifests](AudioQnA/kubernetes/intel/README.md) | Not Supported | [AudioQnA with GMC](AudioQnA/kubernetes/intel/README_gmc.md) |
5050
| VisualQnA | [Xeon Instructions](VisualQnA/docker_compose/intel/cpu/xeon/README.md) | [Gaudi Instructions](VisualQnA/docker_compose/intel/hpu/gaudi/README.md) | [VisualQnA with Manifests](VisualQnA/kubernetes/intel/README.md) | Not Supported | [VisualQnA with GMC](VisualQnA/kubernetes/intel/README_gmc.md) |
5151
| ProductivitySuite | [Xeon Instructions](ProductivitySuite/docker_compose/intel/cpu/xeon/README.md) | Not Supported | [ProductivitySuite with Manifests](ProductivitySuite/kubernetes/intel/README.md) | Not Supported | Not Supported |

‎Translation/docker_compose/intel/cpu/xeon/README.md‎

Lines changed: 49 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -41,30 +41,59 @@ cd GenAIExamples/Translation/ui
4141
docker build -t opea/translation-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f docker/Dockerfile .
4242
```
4343

44+
### 4. Build Nginx Docker Image
45+
46+
```bash
47+
cd GenAIComps
48+
docker build -t opea/translation-nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/nginx/Dockerfile .
49+
```
50+
4451
Then run the command `docker images`, you will have the following Docker Images:
4552

4653
1. `opea/llm-tgi:latest`
4754
2. `opea/translation:latest`
4855
3. `opea/translation-ui:latest`
56+
4. `opea/translation-nginx:latest`
4957

5058
## 🚀 Start Microservices
5159

60+
### Required Models
61+
62+
By default, the LLM model is set to a default value as listed below:
63+
64+
| Service | Model |
65+
| ------- | ----------------- |
66+
| LLM | haoranxu/ALMA-13B |
67+
68+
Change the `LLM_MODEL_ID` below for your needs.
69+
5270
### Setup Environment Variables
5371

54-
Since the `compose.yaml` will consume some environment variables, you need to set up them in advance as below.
72+
1. Set the required environment variables:
5573

56-
```bash
57-
export http_proxy=${your_http_proxy}
58-
export https_proxy=${your_http_proxy}
59-
export LLM_MODEL_ID="haoranxu/ALMA-13B"
60-
export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
61-
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
62-
export MEGA_SERVICE_HOST_IP=${host_ip}
63-
export LLM_SERVICE_HOST_IP=${host_ip}
64-
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/translation"
65-
```
74+
```bash
75+
# Example: host_ip="192.168.1.1"
76+
export host_ip="External_Public_IP"
77+
# Example: no_proxy="localhost, 127.0.0.1, 192.168.1.1"
78+
export no_proxy="Your_No_Proxy"
79+
export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
80+
# Example: NGINX_PORT=80
81+
export NGINX_PORT=${your_nginx_port}
82+
```
83+
84+
2. If you are in a proxy environment, also set the proxy-related environment variables:
85+
86+
```bash
87+
export http_proxy="Your_HTTP_Proxy"
88+
export https_proxy="Your_HTTPs_Proxy"
89+
```
90+
91+
3. Set up other environment variables:
6692

67-
Note: Please replace with `host_ip` with you external IP address, do not use localhost.
93+
```bash
94+
cd ../../../
95+
source set_env.sh
96+
```
6897

6998
### Start Microservice Docker Containers
7099

@@ -99,6 +128,14 @@ docker compose up -d
99128
"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
100129
```
101130

131+
4. Nginx Service
132+
133+
```bash
134+
curl http://${host_ip}:${NGINX_PORT}/v1/translation \
135+
-H "Content-Type: application/json" \
136+
-d '{"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
137+
```
138+
102139
Following the validation of all aforementioned microservices, we are now prepared to construct a mega-service.
103140

104141
## 🚀 Launch the UI

‎Translation/docker_compose/intel/cpu/xeon/compose.yaml‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,12 @@ services:
88
ports:
99
- "8008:80"
1010
environment:
11+
no_proxy: ${no_proxy}
1112
http_proxy: ${http_proxy}
1213
https_proxy: ${https_proxy}
13-
TGI_LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
14-
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
14+
HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
15+
HF_HUB_DISABLE_PROGRESS_BARS: 1
16+
HF_HUB_ENABLE_HF_TRANSFER: 0
1517
volumes:
1618
- "./data:/data"
1719
shm_size: 1g
@@ -25,10 +27,13 @@ services:
2527
- "9000:9000"
2628
ipc: host
2729
environment:
30+
no_proxy: ${no_proxy}
2831
http_proxy: ${http_proxy}
2932
https_proxy: ${https_proxy}
3033
TGI_LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
3134
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
35+
HF_HUB_DISABLE_PROGRESS_BARS: 1
36+
HF_HUB_ENABLE_HF_TRANSFER: 0
3237
restart: unless-stopped
3338
translation-xeon-backend-server:
3439
image: ${REGISTRY:-opea}/translation:${TAG:-latest}
@@ -39,6 +44,7 @@ services:
3944
ports:
4045
- "8888:8888"
4146
environment:
47+
- no_proxy=${no_proxy}
4248
- https_proxy=${https_proxy}
4349
- http_proxy=${http_proxy}
4450
- MEGA_SERVICE_HOST_IP=${MEGA_SERVICE_HOST_IP}
@@ -53,11 +59,31 @@ services:
5359
ports:
5460
- "5173:5173"
5561
environment:
62+
- no_proxy=${no_proxy}
5663
- https_proxy=${https_proxy}
5764
- http_proxy=${http_proxy}
5865
- BASE_URL=${BACKEND_SERVICE_ENDPOINT}
5966
ipc: host
6067
restart: always
68+
translation-xeon-nginx-server:
69+
image: ${REGISTRY:-opea}/translation-nginx:${TAG:-latest}
70+
container_name: translation-xeon-nginx-server
71+
depends_on:
72+
- translation-xeon-backend-server
73+
- translation-xeon-ui-server
74+
ports:
75+
- "${NGINX_PORT:-80}:80"
76+
environment:
77+
- no_proxy=${no_proxy}
78+
- https_proxy=${https_proxy}
79+
- http_proxy=${http_proxy}
80+
- FRONTEND_SERVICE_IP=${FRONTEND_SERVICE_IP}
81+
- FRONTEND_SERVICE_PORT=${FRONTEND_SERVICE_PORT}
82+
- BACKEND_SERVICE_NAME=${BACKEND_SERVICE_NAME}
83+
- BACKEND_SERVICE_IP=${BACKEND_SERVICE_IP}
84+
- BACKEND_SERVICE_PORT=${BACKEND_SERVICE_PORT}
85+
ipc: host
86+
restart: always
6187
networks:
6288
default:
6389
driver: bridge

‎Translation/docker_compose/intel/hpu/gaudi/README.md‎

Lines changed: 50 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -29,34 +29,63 @@ docker build -t opea/translation:latest --build-arg https_proxy=$https_proxy --b
2929
Construct the frontend Docker image using the command below:
3030

3131
```bash
32-
cd GenAIExamples/Translation
32+
cd GenAIExamples/Translation/ui/
3333
docker build -t opea/translation-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
3434
```
3535

36+
### 4. Build Nginx Docker Image
37+
38+
```bash
39+
cd GenAIComps
40+
docker build -t opea/translation-nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/nginx/Dockerfile .
41+
```
42+
3643
Then run the command `docker images`, you will have the following four Docker Images:
3744

3845
1. `opea/llm-tgi:latest`
3946
2. `opea/translation:latest`
4047
3. `opea/translation-ui:latest`
48+
4. `opea/translation-nginx:latest`
4149

4250
## 🚀 Start Microservices
4351

52+
### Required Models
53+
54+
By default, the LLM model is set to a default value as listed below:
55+
56+
| Service | Model |
57+
| ------- | ----------------- |
58+
| LLM | haoranxu/ALMA-13B |
59+
60+
Change the `LLM_MODEL_ID` below for your needs.
61+
4462
### Setup Environment Variables
4563

46-
Since the `compose.yaml` will consume some environment variables, you need to setup them in advance as below.
64+
1. Set the required environment variables:
4765

48-
```bash
49-
export http_proxy=${your_http_proxy}
50-
export https_proxy=${your_http_proxy}
51-
export LLM_MODEL_ID="haoranxu/ALMA-13B"
52-
export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
53-
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
54-
export MEGA_SERVICE_HOST_IP=${host_ip}
55-
export LLM_SERVICE_HOST_IP=${host_ip}
56-
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/translation"
57-
```
66+
```bash
67+
# Example: host_ip="192.168.1.1"
68+
export host_ip="External_Public_IP"
69+
# Example: no_proxy="localhost, 127.0.0.1, 192.168.1.1"
70+
export no_proxy="Your_No_Proxy"
71+
export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
72+
# Example: NGINX_PORT=80
73+
export NGINX_PORT=${your_nginx_port}
74+
```
75+
76+
2. If you are in a proxy environment, also set the proxy-related environment variables:
77+
78+
```bash
79+
export http_proxy="Your_HTTP_Proxy"
80+
export https_proxy="Your_HTTPs_Proxy"
81+
```
82+
83+
3. Set up other environment variables:
5884

59-
Note: Please replace with `host_ip` with you external IP address, do not use localhost.
85+
```bash
86+
cd ../../../
87+
source set_env.sh
88+
```
6089

6190
### Start Microservice Docker Containers
6291

@@ -91,6 +120,14 @@ docker compose up -d
91120
"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
92121
```
93122

123+
4. Nginx Service
124+
125+
```bash
126+
curl http://${host_ip}:${NGINX_PORT}/v1/translation \
127+
-H "Content-Type: application/json" \
128+
-d '{"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
129+
```
130+
94131
Following the validation of all aforementioned microservices, we are now prepared to construct a mega-service.
95132

96133
## 🚀 Launch the UI

‎Translation/docker_compose/intel/hpu/gaudi/compose.yaml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ services:
1010
environment:
1111
http_proxy: ${http_proxy}
1212
https_proxy: ${https_proxy}
13-
TGI_LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
1413
HF_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
1514
HF_HUB_DISABLE_PROGRESS_BARS: 1
1615
HF_HUB_ENABLE_HF_TRANSFER: 0
@@ -36,6 +35,8 @@ services:
3635
https_proxy: ${https_proxy}
3736
TGI_LLM_ENDPOINT: ${TGI_LLM_ENDPOINT}
3837
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
38+
HF_HUB_DISABLE_PROGRESS_BARS: 1
39+
HF_HUB_ENABLE_HF_TRANSFER: 0
3940
restart: unless-stopped
4041
translation-gaudi-backend-server:
4142
image: ${REGISTRY:-opea}/translation:${TAG:-latest}
@@ -65,6 +66,25 @@ services:
6566
- BASE_URL=${BACKEND_SERVICE_ENDPOINT}
6667
ipc: host
6768
restart: always
69+
translation-gaudi-nginx-server:
70+
image: ${REGISTRY:-opea}/translation-nginx:${TAG:-latest}
71+
container_name: translation-gaudi-nginx-server
72+
depends_on:
73+
- translation-gaudi-backend-server
74+
- translation-gaudi-ui-server
75+
ports:
76+
- "${NGINX_PORT:-80}:80"
77+
environment:
78+
- no_proxy=${no_proxy}
79+
- https_proxy=${https_proxy}
80+
- http_proxy=${http_proxy}
81+
- FRONTEND_SERVICE_IP=${FRONTEND_SERVICE_IP}
82+
- FRONTEND_SERVICE_PORT=${FRONTEND_SERVICE_PORT}
83+
- BACKEND_SERVICE_NAME=${BACKEND_SERVICE_NAME}
84+
- BACKEND_SERVICE_IP=${BACKEND_SERVICE_IP}
85+
- BACKEND_SERVICE_PORT=${BACKEND_SERVICE_PORT}
86+
ipc: host
87+
restart: always
6888

6989
networks:
7090
default:

‎Translation/docker_compose/set_env.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (C) 2024 Intel Corporation
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
7+
export LLM_MODEL_ID="haoranxu/ALMA-13B"
8+
export TGI_LLM_ENDPOINT="http://${host_ip}:8008"
9+
export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token}
10+
export MEGA_SERVICE_HOST_IP=${host_ip}
11+
export LLM_SERVICE_HOST_IP=${host_ip}
12+
export BACKEND_SERVICE_ENDPOINT="http://${host_ip}:8888/v1/translation"
13+
export NGINX_PORT=80
14+
export FRONTEND_SERVICE_IP=${host_ip}
15+
export FRONTEND_SERVICE_PORT=5173
16+
export BACKEND_SERVICE_NAME=translation
17+
export BACKEND_SERVICE_IP=${host_ip}
18+
export BACKEND_SERVICE_PORT=8888

‎Translation/docker_image_build/build.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,9 @@ services:
2323
dockerfile: comps/llms/text-generation/tgi/Dockerfile
2424
extends: translation
2525
image: ${REGISTRY:-opea}/llm-tgi:${TAG:-latest}
26+
nginx:
27+
build:
28+
context: GenAIComps
29+
dockerfile: comps/nginx/Dockerfile
30+
extends: translation
31+
image: ${REGISTRY:-opea}/translation-nginx:${TAG:-latest}

‎Translation/kubernetes/intel/README.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Deploy Translation in Kubernetes Cluster
2+
3+
> [NOTE]
4+
> The following values must be set before you can deploy:
5+
> HUGGINGFACEHUB_API_TOKEN
6+
>
7+
> You can also customize the "MODEL_ID" if needed.
8+
>
9+
> You need to make sure you have created the directory `/mnt/opea-models` to save the cached model on the node where the Translation workload is running. Otherwise, you need to modify the `translation.yaml` file to change the `model-volume` to a directory that exists on the node.
10+
11+
## Deploy On Xeon
12+
13+
```
14+
cd GenAIExamples/Translation/kubernetes/intel/cpu/xeon/manifests
15+
export HUGGINGFACEHUB_API_TOKEN="YourOwnToken"
16+
sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" translation.yaml
17+
kubectl apply -f translation.yaml
18+
```
19+
20+
## Deploy On Gaudi
21+
22+
```
23+
cd GenAIExamples/Translation/kubernetes/intel/hpu/gaudi/manifests
24+
export HUGGINGFACEHUB_API_TOKEN="YourOwnToken"
25+
sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" translation.yaml
26+
kubectl apply -f translation.yaml
27+
```
28+
29+
## Verify Services
30+
31+
To verify the installation, run the command `kubectl get pod` to make sure all pods are running.
32+
33+
Then run the command `kubectl port-forward svc/translation 8888:8888` to expose the Translation service for access.
34+
35+
Open another terminal and run the following command to verify the service if working:
36+
37+
```console
38+
curl http://localhost:8888/v1/translation \
39+
-H 'Content-Type: application/json' \
40+
-d '{"language_from": "Chinese","language_to": "English","source_language": "我爱机器翻译。"}'
41+
```

0 commit comments

Comments
 (0)