Implement table & tree disk usage statistics #17169

shuwenwei · 2026-02-05T08:13:49Z

Description

This PR implements disk-usage statistics collection in table level and device level for tree model. It adds the necessary data structures, background tasks, and read APIs to compute and expose disk usage metrics used by monitoring, admission control and operational tooling.

Tree Model (No Cache)

IoTDB> show disk_usage from root.test.**
+---------+----------+--------+-------------+-----------+
| Database|DataNodeId|RegionId|TimePartition|SizeInBytes|
+---------+----------+--------+-------------+-----------+
|root.test|         1|       3|            0|         70|
+---------+----------+--------+-------------+-----------+
Total line number = 1
It costs 0.959s

Implements ShowDiskUsageNode and ShowDiskUsageOperator.
• Disk usage is calculated by scanning relevant TsFiles at query time.
• Supports:
• Path pattern matching
• Time partition filtering
• Existing SQL semantics (SHOW DISK_USAGE)

Table Model (With Cache)

IoTDB:information_schema> select * from table_disk_usage
+--------+----------+-------+---------+--------------+-------------+
|database|table_name|datanode_id|region_id|time_partition|size_in_bytes|
+--------+----------+-------+---------+--------------+-------------+
|   test1|        t1|          1|        5|             0|          142|
|   test1|        t2|          1|        5|             0|            0|
|   test1|        t1|          1|        6|             0|            0|
|   test1|        t2|          1|        6|             0|           82|
+--------+----------+-------+---------+--------------+-------------+
Total line number = 4
It costs 2.821s

Table Model introduces a dedicated disk usage cache:
• TableDiskUsageCache manages all cache operations.
• A single-threaded background worker processes write, read, and maintenance tasks via an operation queue.
• Cache state is persistent across restarts.

Cached Data
• TsFile-level table size statistics
• Object file size deltas, recorded incrementally
• Periodic snapshot + delta compaction

Query Integration
• Exposes statistics via information_schema.table_disk_usage.
• Supports:
• Predicate pushdown (except on aggregated size columns)
• Limit / offset
• Parallel region-level scanning

Copilot

Pull request overview

This PR implements comprehensive disk usage statistics collection for both Tree Model (device-based) and Table Model databases in IoTDB. The implementation provides monitoring capabilities at the table/device level and time partition level.

Changes:

Adds SHOW DISK_USAGE SQL statement for Tree Model with on-demand calculation
Implements table_disk_usage information schema table for Table Model with persistent cache
Introduces background task infrastructure for cache maintenance with periodic compaction
Adds predicate push-down and limit/offset optimization support for information schema tables

Reviewed changes

Copilot reviewed 95 out of 95 changed files in this pull request and generated 33 comments.

Show a summary per file

File	Description
pom.xml	Updates tsfile version to 2.2.1-260205-SNAPSHOT
TsFileID.java	Adds SHALLOW_SIZE constant (contains bug)
InformationSchema.java	Adds table_disk_usage schema and push-down support (contains bug)
TableDiskUsageCache*.java	Core cache implementation with writer/reader classes
ShowDiskUsageNode.java	Plan node for tree model disk usage queries
TableDiskUsageInformationSchemaTableScanNode.java	Plan node for table model information schema scans
ShowDiskUsageOperator.java	Execution operator for tree model
DiskUsageStatisticUtil.java	Base utility class for disk usage calculation
IoTDBDescriptor.java	Configuration support (contains bug)
Integration tests	Comprehensive tests for both tree and table models

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

...b/storageengine/dataregion/utils/tableDiskUsageCache/TimePartitionTableSizeQueryContext.java

...ore/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileID.java

...core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java

...ode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TableDiskUsageTest.java

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

...b/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java

codecov · 2026-02-06T04:54:25Z

Codecov Report

❌ Patch coverage is 53.59168% with 982 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.57%. Comparing base (e4ef4db) to head (a19bd11).
⚠️ Report is 23 commits behind head on master.

Files with missing lines	Patch %	Lines
...ional/InformationSchemaContentSupplierFactory.java	0.00%	185 Missing ⚠️
...ecution/operator/source/ShowDiskUsageOperator.java	0.00%	77 Missing ⚠️
...e/dataregion/utils/TreeDiskUsageStatisticUtil.java	0.00%	64 Missing ⚠️
...utils/tableDiskUsageCache/TableDiskUsageCache.java	76.59%	55 Missing ⚠️
...nner/distribute/TableDistributedPlanGenerator.java	8.51%	43 Missing ⚠️
...b/queryengine/plan/planner/LogicalPlanBuilder.java	0.00%	41 Missing ⚠️
...ngine/dataregion/utils/DiskUsageStatisticUtil.java	63.96%	40 Missing ⚠️
...db/db/queryengine/plan/analyze/AnalyzeVisitor.java	0.00%	38 Missing ⚠️
...kUsageCache/tsfile/TsFileTableSizeCacheReader.java	81.12%	37 Missing ⚠️
...an/planner/plan/node/source/ShowDiskUsageNode.java	47.05%	36 Missing ⚠️
... and 39 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17169      +/-   ##
============================================
+ Coverage     39.47%   39.57%   +0.09%     
  Complexity      282      282              
============================================
  Files          5098     5120      +22     
  Lines        341273   343782    +2509     
  Branches      43471    43795     +324     
============================================
+ Hits         134714   136041    +1327     
- Misses       206559   207741    +1182

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Copilot reviewed 99 out of 99 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-06T06:32:42Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+
+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    tempKeyFile2.createNewFile();


Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:42Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    tempKeyFile2.createNewFile();
+    tempValueFile2.createNewFile();


Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+
+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    keyFile2.createNewFile();


Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    keyFile2.createNewFile();
+    valueFile2.createNewFile();


Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+
+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    keyFile2.createNewFile();


Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+    keyFile1.createNewFile();
+    valueFile1.createNewFile();
+    keyFile2.createNewFile();
+    tempValueFile2.createNewFile();


Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+                + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+    keyFile1.createNewFile();
+    tempKeyFile2.createNewFile();


Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:43Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+
+    keyFile1.createNewFile();
+    tempKeyFile2.createNewFile();
+    tempValueFile2.createNewFile();


Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.

Copilot · 2026-02-06T06:32:44Z

.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java

+        context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+      Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+      for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+        int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));


Potential uncaught 'java.lang.NumberFormatException'.

sonarqubecloud · 2026-02-06T07:42:01Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

shuwenwei added 30 commits October 27, 2025 17:06

add information schema table

f9eee27

fix

9e8b71a

update InformationSchema

b471127

Merge branch 'iotdb_master' into table_disk_usage_statistics

0a8c04f

add pushdown limit offset

687bc24

remove showDiskUsageStatement from table model

1699e44

add it

098392d

Merge branch 'iotdb_master' into table_disk_usage_statistics

ccfb4ec

fix

3fbf6a3

fix

b6787b4

fix

880718f

Merge branch 'iotdb_master' into table_disk_usage_statistics

03b0f2c

fix conflict

640f998

Merge branch 'iotdb_master' into table_disk_usage_statistics

a71f0d7

add cache

524c727

compact

3e3f682

fix

6575e6e

Merge branch 'iotdb_master' into table_disk_usage_statistics_with_cache

8f2de96

record table size for when generating tsfile

991a1b6

use ServiceLoader

dfc7929

add interface for object file table size cache

aafbb1c

refactor

2b87ce7

add abstract cache writer

3c63c68

refactor

79d39aa

refactor

c3945c6

throw exception

d641d10

move constant to parent class

d3ec980

refactor

50bcedd

refactor

3a91c80

refactor

d661431

shuwenwei added 15 commits February 3, 2026 16:47

add statistics

a18f763

fix

48ac272

add config

79b1bb9

remove stale code

56e9bf2

Merge branch 'iotdb_master' into table_disk_usage_statistics_with_cache

325f513

fix bug

eda148d

modify TableDiskUsageSupplier

34efb01

fix

0bbf45d

use final

0076b6b

refactor

eda01fe

fix

d4d641c

fix bug

8ea69e7

fix bug

271bd12

fix

07246a8

update tsfile version

66328b3

JackieTien97 requested a review from Copilot February 5, 2026 08:23

Copilot started reviewing on behalf of JackieTien97 February 5, 2026 08:23 View session

Copilot AI reviewed Feb 5, 2026

View reviewed changes

shuwenwei added 5 commits February 5, 2026 18:40

fix ut

0877c14

fix copilot review comments

398cf2b

fix bug

d7fafe4

fix bug

dd60404

fix it

e7785f3

fix it

4a3bca1

JackieTien97 requested a review from Copilot February 6, 2026 06:23

Copilot started reviewing on behalf of JackieTien97 February 6, 2026 06:23 View session

Copilot AI reviewed Feb 6, 2026

View reviewed changes

fix sort

a19bd11

Implement table & tree disk usage statistics #17169

Are you sure you want to change the base?

Implement table & tree disk usage statistics #17169

Uh oh!

Conversation

shuwenwei commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tree Model (No Cache)

Table Model (With Cache)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 6, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shuwenwei commented Feb 5, 2026 •

edited

Loading

codecov bot commented Feb 6, 2026 •

edited

Loading