Implement ExternalService Framework #17009

Wei-hao-Li · 2026-01-12T01:10:27Z

No description provided.

Signed-off-by: Weihao Li <18110526956@163.com>

JackieTien97

partial review, finish CN and antlr parts.

...lational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4

iotdb-core/confignode/src/test/resources/confignode1conf/iotdb-system.properties

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

JackieTien97 · 2026-01-12T08:03:48Z

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

+    crc32.reset();
+    crc32.update(bytes, 0, length);
+
+    long expectedCRC = ReadWriteIOUtils.readLong(inputStream);


Suggested change

long expectedCRC = ReadWriteIOUtils.readLong(inputStream);

int expectedCRC = ReadWriteIOUtils.readInt(inputStream);

JackieTien97 · 2026-01-12T08:04:05Z

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

+
+    ReadWriteIOUtils.write(bytes.length, outputStream);
+    outputStream.write(bytes);
+    ReadWriteIOUtils.write(crc32.getValue(), outputStream);


Suggested change

ReadWriteIOUtils.write(crc32.getValue(), outputStream);

ReadWriteIOUtils.write((int) crc32.getValue(), outputStream);

JackieTien97 · 2026-01-12T08:04:22Z

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

+    crc32.update(bytes, 0, length);
+
+    long expectedCRC = ReadWriteIOUtils.readLong(inputStream);
+    if (crc32.getValue() != expectedCRC) {


Suggested change

if (crc32.getValue() != expectedCRC) {

if (((int) crc32.getValue()) != expectedCRC) {

JackieTien97

Add all ITs you list in the documents and the recovery process is missing

JackieTien97 · 2026-01-12T08:11:07Z

.../node-commons/src/main/java/org/apache/iotdb/commons/schema/column/ColumnHeaderConstant.java

+          new ColumnHeader(SERVICE_NAME, TSDataType.TEXT),
+          new ColumnHeader(DATA_NODE_ID, TSDataType.INT32),
+          new ColumnHeader(STATE, TSDataType.TEXT),
+          new ColumnHeader(CLASS_NAME, TSDataType.TEXT),
+          new ColumnHeader(SERVICE_TYPE, TSDataType.TEXT));


Suggested change

new ColumnHeader(SERVICE_NAME, TSDataType.TEXT),

new ColumnHeader(DATA_NODE_ID, TSDataType.INT32),

new ColumnHeader(STATE, TSDataType.TEXT),

new ColumnHeader(CLASS_NAME, TSDataType.TEXT),

new ColumnHeader(SERVICE_TYPE, TSDataType.TEXT));

new ColumnHeader(SERVICE_NAME, TSDataType.STRING),

new ColumnHeader(DATA_NODE_ID, TSDataType.INT32),

new ColumnHeader(STATE, TSDataType.STRING),

new ColumnHeader(CLASS_NAME, TSDataType.STRING),

new ColumnHeader(SERVICE_TYPE, TSDataType.STRING));

JackieTien97 · 2026-01-12T08:14:49Z

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/service/ServiceType.java

  RPC_SERVICE("RPC ServerService", "RPCService"),
  INFLUX_SERVICE("InfluxDB Protocol Service", "InfluxDB Protocol"),
  MQTT_SERVICE("MQTTService", "MqttService"),
+  EXTERNAL_SERVICE("ExternalService Service", "ExternalService"),


It seems that this enum is never used.

JackieTien97 · 2026-01-12T08:15:32Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNode.java

    if (IoTDBRestServiceDescriptor.getInstance().getConfig().isEnableRestService()) {
      registerManager.register(RestService.getInstance());
    }
+    // registerManager.register(ExternalServiceManagementService.getInstance());


need this? or should be just deleted.

JackieTien97 · 2026-01-12T08:28:05Z

...ode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/parser/AstBuilder.java

+          Optional.of(
+              new ComparisonExpression(
+                  ComparisonExpression.Operator.EQUAL,
+                  new Identifier("datanode_id"),


Suggested change

new Identifier("datanode_id"),

new Identifier(ColumnHeaderConstant.DATA_NODE_ID_TABLE_MODEL),

JackieTien97 · 2026-01-12T08:54:34Z

.../main/java/org/apache/iotdb/db/queryengine/plan/execution/config/TableConfigTaskVisitor.java

+  @Override
+  protected IConfigTask visitCreateExternalService(
+      CreateExternalService node, MPPQueryContext context) {
+    context.setQueryType(QueryType.WRITE);


Suggested change

context.setQueryType(QueryType.WRITE);

context.setQueryType(QueryType.WRITE);

accessControl.checkUserGlobalSysPrivilege(context);

JackieTien97 · 2026-01-12T09:28:47Z

...rg/apache/iotdb/db/queryengine/plan/execution/config/executor/ClusterConfigTaskExecutor.java

+      ExternalServiceManagementService.getInstance().stopService(serviceName);
+      future.set(new ConfigTaskResult(new TSStatus(TSStatusCode.SUCCESS_STATUS.getStatusCode())));
+    } catch (Exception e) {
+      future.setException(new ExternalServiceManagementException(e.getMessage()));


same as above

JackieTien97 · 2026-01-12T09:28:53Z

...rg/apache/iotdb/db/queryengine/plan/execution/config/executor/ClusterConfigTaskExecutor.java

+      ExternalServiceManagementService.getInstance().dropService(serviceName, forcedly);
+      future.set(new ConfigTaskResult(new TSStatus(TSStatusCode.SUCCESS_STATUS.getStatusCode())));
+    } catch (Exception e) {
+      future.setException(new ExternalServiceManagementException(e.getMessage()));


same as above

JackieTien97 · 2026-01-12T09:29:10Z

...rg/apache/iotdb/db/queryengine/plan/execution/config/executor/ClusterConfigTaskExecutor.java

+          new ConfigTaskResult(
+              TSStatusCode.SUCCESS_STATUS, builder.build(), getShowExternalServiceHeader()));
+    } catch (Exception e) {
+      future.setException(e);


same as above

JackieTien97 · 2026-01-12T09:35:59Z

...tanode/src/main/java/org/apache/iotdb/db/service/exernalservice/BuiltinExternalServices.java

+  MQTT(
+      "MQTT",
+      "org.apache.iotdb.externalservice.Mqtt",
+      IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),
+  REST(
+      "REST",
+      "org.apache.iotdb.externalservice.Rest",
+      IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);


Suggested change

MQTT(

"MQTT",

"org.apache.iotdb.externalservice.Mqtt",

IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),

REST(

"REST",

"org.apache.iotdb.externalservice.Rest",

IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);

MQTT(

"MQTT",

"org.apache.iotdb.externalservice.Mqtt",

// IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),

() -> false),

REST(

"REST",

"org.apache.iotdb.externalservice.Rest",

// IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);

() -> false);

better always return false before you really seperate the rest and mqtt service outside. Otherwise, mqtt and rest ITs will fail because org.apache.iotdb.externalservice.Mqtt and org.apache.iotdb.externalservice.Rest is not found

JackieTien97 · 2026-01-12T09:36:10Z

...tanode/src/main/java/org/apache/iotdb/db/service/exernalservice/BuiltinExternalServices.java

+  MQTT(
+      "MQTT",
+      "org.apache.iotdb.externalservice.Mqtt",
+      IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),
+  REST(
+      "REST",
+      "org.apache.iotdb.externalservice.Rest",
+      IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);


Suggested change

MQTT(

"MQTT",

"org.apache.iotdb.externalservice.Mqtt",

IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),

REST(

"REST",

"org.apache.iotdb.externalservice.Rest",

IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);

MQTT(

"MQTT",

"org.apache.iotdb.externalservice.Mqtt",

// IoTDBDescriptor.getInstance().getConfig()::isEnableMQTTService),

() -> false),

REST(

"REST",

"org.apache.iotdb.externalservice.Rest",

// IoTDBRestServiceDescriptor.getInstance().getConfig()::isEnableRestService);

() -> false);

better always return false before you really seperate the rest and mqtt service outside. Otherwise, mqtt and rest ITs will fail because org.apache.iotdb.externalservice.Mqtt and org.apache.iotdb.externalservice.Rest is not found

Wei-hao-Li added 8 commits December 23, 2025 17:27

add api

4d61221

Signed-off-by: Weihao Li <18110526956@163.com>

Merge branch 'master' of github.com:apache/iotdb into exteranlservice

86e3dfa

add process in CN: ExternalServiceInfo

8f93984

Signed-off-by: Weihao Li <18110526956@163.com>

draft

720c7c5

Signed-off-by: Weihao Li <18110526956@163.com>

done

3e4cf34

Signed-off-by: Weihao Li <18110526956@163.com>

Merge branch 'master' of github.com:apache/iotdb into exteranlservice

88568d0

fix Some

cd49eee

Signed-off-by: Weihao Li <18110526956@163.com>

add snapshot test

3ff5dd2

Signed-off-by: Weihao Li <18110526956@163.com>

JackieTien97 requested changes Jan 12, 2026

View reviewed changes

	long expectedCRC = ReadWriteIOUtils.readLong(inputStream);
	int expectedCRC = ReadWriteIOUtils.readInt(inputStream);

	ReadWriteIOUtils.write(crc32.getValue(), outputStream);
	ReadWriteIOUtils.write((int) crc32.getValue(), outputStream);

	if (crc32.getValue() != expectedCRC) {
	if (((int) crc32.getValue()) != expectedCRC) {

	new Identifier("datanode_id"),
	new Identifier(ColumnHeaderConstant.DATA_NODE_ID_TABLE_MODEL),

	context.setQueryType(QueryType.WRITE);
	context.setQueryType(QueryType.WRITE);
	accessControl.checkUserGlobalSysPrivilege(context);

Implement ExternalService Framework #17009

Are you sure you want to change the base?

Implement ExternalService Framework #17009

Uh oh!

Conversation

Wei-hao-Li commented Jan 12, 2026

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants