ci(tests): replace coverage with sccache for faster sharded tests

[QuantumExplorer]

Issue being fixed or feature implemented

Sharded workspace tests in CI are slow due to cargo-llvm-cov coverage instrumentation overhead. This PR is a competing benchmark against #3237 (which keeps coverage) to measure the speed improvement from removing instrumentation and switching to sccache.

What was done?

Based on the same ci/sharded-rust-tests branch as #3237, with these changes to .github/workflows/tests-rs-workspace.yml:

• Removed cargo-llvm-cov coverage instrumentation — uses cargo nextest run directly instead of cargo llvm-cov nextest
• Replaced Swatinem/rust-cache with sccache (S3-backed compiler cache) for faster incremental builds
• Added pre-built librocksdb static library via .github/actions/librocksdb — eliminates building rocksdb from source
• Removed cdylib stripping step — no longer needed with static rocksdb linking via ROCKSDB_STATIC env var
• Removed coverage upload step — no lcov.info generated in this variant

How Has This Been Tested?

• CI will run the sharded tests on this branch — compare wall-clock times against build(platform): replace per-package Rust tests with sharded workspace runner #3237
• The sccache + librocksdb pattern is already proven in tests-rs-package.yml and tests-rs-nightly-long-running.yml

Breaking Changes

None — CI-only changes.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
• I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Added automated doctest verification and a sharded Rust workspace test run to CI, with partitioned execution and failure artifact collection for troubleshooting.
  • Replaced the legacy package test job with the new sharded workspace workflow and a separate doctests workflow.
  • Updated the main test pipeline to detect Rust workflow and ZK-related changes and to trigger the new Rust test jobs on additional branch events.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds two reusable Rust CI workflows (sharded workspace tests and doctests), removes a legacy package test job, and updates the main tests orchestration to detect Rust/workflow and ZK/shield changes, conditionally triggering the new Rust workflows and additional push triggers.

Changes

Cohort / File(s)	Summary
Doctest Workflow .github/workflows/tests-rs-doctests.yml	Adds a reusable workflow_call workflow that strips cdylib crate-types, sets up Rust (local action), enables rust-cache, and runs workspace doctests (cargo test --doc).
Sharded Workspace Tests Workflow .github/workflows/tests-rs-workspace.yml	Adds a reusable sharded test workflow (6 partitions) that prepares runners (disk cleanup, strip cdylib, Rust + sccache, librocksdb), runs cargo nextest across a large crate set, collects core dumps/crash artifacts per shard, and conditionally runs shielded ZK tests.
Package Tests Workflow (trimmed) .github/workflows/tests-rs-package.yml	Removes the previous comprehensive package test job (replaced by workspace testing); leaves a placeholder comment and retains check_each_feature job.
Main Test Orchestration .github/workflows/tests.yml	Adds push triggers for master and v*-dev; introduces filter-rs-workflows path filter and outputs (rs-workflows-changed, zk-changed); wires new public jobs rs-workspace-tests and rs-doctests, and updates rs-packages/dispatch logic to consider workflow and ZK-related changes.

sequenceDiagram
  participant Dev as Developer
  participant GH as GitHub Actions (tests.yml)
  participant PF as Path-Filter Step
  participant WF as Reusable Workflow (rs-workspace / rs-doctests)
  participant Runner as Runner
  participant Cache as Rust Cache
  participant Art as Artifact Storage

  Dev->>GH: push / PR
  GH->>PF: evaluate paths (rs workflows / RS packages / ZK)
  PF-->>GH: outputs (rs-workflows-changed / zk-changed)
  GH->>WF: dispatch selected reusable workflow(s)
  WF->>Runner: checkout, adjust Cargo.toml (strip cdylib)
  Runner->>Cache: restore/save rust cache / sccache
  Runner->>Runner: install librocksdb, configure core dumps
  Runner->>Runner: run cargo nextest / cargo test --doc (sharded)
  Runner-->Art: upload crash-artifacts on failure

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Poem

🐰 I hopped through YAML with a careful paw,

Nibbled cdylib bits to fix the law,
I split the tests into six tidy parts,
Gathered cores and zipped up their hearts,
Rejoice — doctests and shards hum in CI awe!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly summarizes the main change: replacing coverage instrumentation with sccache for faster sharded tests, which aligns with the core objective and file modifications.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch ci/sharded-rust-tests-no-coverage

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

.github/workflows/tests-rs-workspace.yml (1)

43-69: Avoid a hand-maintained runtime-test allow-list.

Lines 47-66 are now a hand-maintained allow-list for nextest. That makes new workspace crates opt-in for runtime coverage, so they can be skipped silently until this YAML is edited too. Prefer --workspace with explicit --exclude entries, or generate the package list from cargo metadata, so additions are covered by default.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests-rs-workspace.yml around lines 43 - 69, The workflow
currently uses a hand-maintained list of --package entries passed to "cargo
nextest run" which causes new workspace crates to be skipped; replace the
explicit package allow-list with a workspace-driven invocation such as using
"--workspace" and add explicit "--exclude" entries for exceptions (or compute
the package list dynamically from "cargo metadata") so runtime tests opt-out
instead of opt-in; update the "cargo nextest run" invocation (the command
invoking cargo nextest run and the long --package list) to use --workspace plus
the necessary --exclude entries or a generated package list.

.github/workflows/tests-rs-doctests.yml (1)

13-18: Make the manifest rewrites fail loudly.

Lines 15-18 are exact-string sed substitutions. A harmless reordering or spacing change in any of those crate-type arrays turns this into a silent no-op, and the workflow only fails later when the linker issue comes back. Add a match/assert around each rewrite, or move this step to a small TOML-aware script.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests-rs-doctests.yml around lines 13 - 18, The sed-based
crate-type rewrites in the workflow step (the four sed commands that target
packages/rs-sdk/Cargo.toml, packages/wasm-drive-verify/Cargo.toml,
packages/wasm-dpp2/Cargo.toml, packages/wasm-sdk/Cargo.toml,
packages/rs-sdk-ffi/Cargo.toml and packages/rs-platform-wallet-ffi/Cargo.toml)
can silently no-op; change the step so each rewrite fails loudly by asserting
the substitution occurred (e.g., run a post-sed grep/exit-if-not-found or check
sed’s exit status and grep for the expected new string) or replace the step with
a tiny TOML-aware script that loads the Cargo.toml files and programmatically
replaces the crate-type arrays (using a TOML parser) and exits non-zero on any
unexpected shape so the workflow fails immediately on mismatch.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/tests.yml:
- Around line 81-90: The rs-workflows path filter only matches workflow YAMLs
and therefore misses changes to local actions and Rust sources that those
workflows use; update the rs-workflows filter (the dorny/paths-filter entry with
id filter-rs-workflows) to include .github/actions/** and the Rust source paths
those workflows depend on (e.g., rust/, sccache/, librocksdb/ or other
repo-specific Rust dirs) so edits to local actions or Rust code trigger Rust CI,
and either remove .github/workflows/tests-rs-package.yml from this filter or add
a separate workflow-only path pattern so tests-rs-package.yml is exercised
consistently; mirror the same changes in the second occurrence of the
rs-workflows filter elsewhere in the file.

---

Nitpick comments:
In @.github/workflows/tests-rs-doctests.yml:
- Around line 13-18: The sed-based crate-type rewrites in the workflow step (the
four sed commands that target packages/rs-sdk/Cargo.toml,
packages/wasm-drive-verify/Cargo.toml, packages/wasm-dpp2/Cargo.toml,
packages/wasm-sdk/Cargo.toml, packages/rs-sdk-ffi/Cargo.toml and
packages/rs-platform-wallet-ffi/Cargo.toml) can silently no-op; change the step
so each rewrite fails loudly by asserting the substitution occurred (e.g., run a
post-sed grep/exit-if-not-found or check sed’s exit status and grep for the
expected new string) or replace the step with a tiny TOML-aware script that
loads the Cargo.toml files and programmatically replaces the crate-type arrays
(using a TOML parser) and exits non-zero on any unexpected shape so the workflow
fails immediately on mismatch.

In @.github/workflows/tests-rs-workspace.yml:
- Around line 43-69: The workflow currently uses a hand-maintained list of
--package entries passed to "cargo nextest run" which causes new workspace
crates to be skipped; replace the explicit package allow-list with a
workspace-driven invocation such as using "--workspace" and add explicit
"--exclude" entries for exceptions (or compute the package list dynamically from
"cargo metadata") so runtime tests opt-out instead of opt-in; update the "cargo
nextest run" invocation (the command invoking cargo nextest run and the long
--package list) to use --workspace plus the necessary --exclude entries or a
generated package list.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a71b6552-b5b5-45de-92da-48a578a7b8a4

📥 Commits

Reviewing files that changed from the base of the PR and between 18f977e and d010bf0.

📒 Files selected for processing (4)

• .github/workflows/tests-rs-doctests.yml
• .github/workflows/tests-rs-package.yml
• .github/workflows/tests-rs-workspace.yml
• .github/workflows/tests.yml

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Tighten rs-workflows-changed so it matches what Rust CI actually validates.

Lines 85-90 only watch workflow YAMLs, but tests-rs-workspace.yml and tests-rs-doctests.yml also depend on local actions under .github/actions/. Edits to rust, sccache, or librocksdb can therefore skip Rust CI entirely. At the same time, Line 89 includes .github/workflows/tests-rs-package.yml, but Lines 171 and 179 only launch workspace/doctests, so that reusable workflow still is not exercised on workflow-only PRs.

Suggested path-filter expansion

       - uses: dorny/paths-filter@v3
         id: filter-rs-workflows
         if: ${{ github.event_name != 'workflow_dispatch' }}
         with:
           filters: |
             rs-workflows:
               - .github/workflows/tests-rs-workspace.yml
               - .github/workflows/tests-rs-doctests.yml
               - .github/workflows/tests-rs-package.yml
               - .github/workflows/tests.yml
+              - .github/actions/rust/**
+              - .github/actions/sccache/**
+              - .github/actions/librocksdb/**

If .github/workflows/tests-rs-package.yml is meant to stay in this bucket, it also needs a workflow-only execution path; otherwise it should be removed from rs-workflows.

Also applies to: 167-181

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 81 - 90, The rs-workflows path
filter only matches workflow YAMLs and therefore misses changes to local actions
and Rust sources that those workflows use; update the rs-workflows filter (the
dorny/paths-filter entry with id filter-rs-workflows) to include
.github/actions/** and the Rust source paths those workflows depend on (e.g.,
rust/, sccache/, librocksdb/ or other repo-specific Rust dirs) so edits to local
actions or Rust code trigger Rust CI, and either remove
.github/workflows/tests-rs-package.yml from this filter or add a separate
workflow-only path pattern so tests-rs-package.yml is exercised consistently;
mirror the same changes in the second occurrence of the rs-workflows filter
elsewhere in the file.

[coderabbitai]

Actionable comments posted: 2

♻️ Duplicate comments (1)

.github/workflows/tests.yml (1)

82-91: ⚠️ Potential issue | 🟠 Major

Broaden rs-workflows to the local actions these jobs actually depend on.

This filter still only watches workflow YAMLs, so edits under .github/actions/rust/**, .github/actions/sccache/**, or .github/actions/librocksdb/** can change Rust CI behavior without setting rs-workflows-changed. Also, tests-rs-package.yml is listed here, but no downstream job uses this signal to exercise that reusable workflow.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 82 - 91, The rs-workflows
paths-filter is too narrow (the filter id filter-rs-workflows and key
rs-workflows) and only watches workflow YAMLs; update the filter to include the
local action directories that affect Rust CI (add .github/actions/rust/**,
.github/actions/sccache/**, .github/actions/librocksdb/** or any other custom
action dirs) so edits there trigger rs-workflows-changed, and either remove or
correctly wire tests-rs-package.yml from the rs-workflows list if no downstream
job consumes that signal (or add the downstream job to consume it) to keep the
filter consistent with actual dependencies.

🧹 Nitpick comments (1)

.github/workflows/tests-rs-workspace.yml (1)

22-58: Extract the shared Rust CI bootstrap into one reusable step.

Both jobs repeat checkout, manifest rewriting, Rust/sccache/librocksdb setup, and cargo-nextest installation. Only the shard job currently configures core dumps, which is exactly the kind of drift this duplication makes easy. A small composite action or checked script would keep the shard and ZK paths aligned.

Also applies to: 142-172

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/tests.yml:
- Around line 115-123: The current GROVEDB change detector only matches
git-style rev pins and lockfile stanza names; update the grep used after git
diff (the command that currently pipes to grep -qE 'grovedb.*rev\s*=|name =
"grovedb' ) to also detect common dependency forms such as grovedb = "0.20",
grovedb = { version = "..." }, grovedb = { git = "...", branch = "..."} and any
occurrence of grovedb followed by version|git|branch|rev keys; in short, extend
the regex to match patterns like
grovedb\s*=|grovedb.*(version|git|branch|rev)\s*=|name = "grovedb" so
GROVEDB_FILES (and the git diff + grep pipeline) will catch crate version bumps
in Cargo.toml and Cargo.lock.
- Around line 197-205: The zk test gating input currently passes only the
changes.outputs.zk-changed value to the reusable workflow; update the
rs-workspace-tests job so the with: zk-changed input is true if either
needs.changes.outputs.zk-changed OR needs.changes.outputs.rs-workflows-changed
is true (i.e., include the rs-workflows-changed signal in the zk-changed input)
so edits to tests-rs-workspace.yml itself also trigger zk-tests in the called
workflow.

---

Duplicate comments:
In @.github/workflows/tests.yml:
- Around line 82-91: The rs-workflows paths-filter is too narrow (the filter id
filter-rs-workflows and key rs-workflows) and only watches workflow YAMLs;
update the filter to include the local action directories that affect Rust CI
(add .github/actions/rust/**, .github/actions/sccache/**,
.github/actions/librocksdb/** or any other custom action dirs) so edits there
trigger rs-workflows-changed, and either remove or correctly wire
tests-rs-package.yml from the rs-workflows list if no downstream job consumes
that signal (or add the downstream job to consume it) to keep the filter
consistent with actual dependencies.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6fc2c3f5-b108-447c-9b75-deb180791d78

📥 Commits

Reviewing files that changed from the base of the PR and between 72ec2c0 and 575de10.

📒 Files selected for processing (2)

• .github/workflows/tests-rs-workspace.yml
• .github/workflows/tests.yml

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's look at the actual workflow file to understand the context
cat -n .github/workflows/tests.yml | sed -n '110,130p'

Repository: dashpay/platform

Length of output: 1121

---

🏁 Script executed:

# Let's also check how GroveDB is actually declared in the repo's Cargo files
find . -name "Cargo.toml" -o -name "Cargo.lock" | head -20

Repository: dashpay/platform

Length of output: 824

---

🏁 Script executed:

# Let's examine the actual GroveDB dependency declarations in the codebase
rg 'grovedb' -i Cargo.toml | head -20

Repository: dashpay/platform

Length of output: 152

---

🏁 Script executed:

# Test the grep pattern against various GroveDB dependency formats
python3 <<'EOF'
import re

# The exact pattern from the code
pattern = r'grovedb.*rev\s*=|name = "grovedb'

# Test cases for different GroveDB dependency formats
test_cases = [
    'grovedb = "0.20"',
    'grovedb = { version = "0.20", features = ["foo"] }',
    'grovedb = { git = "https://...", rev = "abc123" }',
    'grovedb = { git = "https://..." }',
    'name = "grovedb"',
    'grovedb = { path = "../grovedb" }',
    'grovedb-core = "0.20"',
]

print("Testing grep pattern: " + repr(pattern))
print("-" * 60)
for test in test_cases:
    match = re.search(pattern, test)
    status = "✓ MATCHES" if match else "✗ MISS"
    print(f"{status:12} | {test}")
EOF

Repository: dashpay/platform

Length of output: 508

---

🏁 Script executed:

# Check all Cargo.toml files for grovedb dependency declarations
rg -A 3 'grovedb' Cargo.toml

Repository: dashpay/platform

Length of output: 241

---

🏁 Script executed:

# Check Cargo.lock for grovedb entries to see actual format
rg -A 2 'name = "grovedb' Cargo.lock | head -30

Repository: dashpay/platform

Length of output: 1411

---

The GroveDB bump detector may miss common dependency update formats.

The grep pattern 'grovedb.*rev\s*=|name = "grovedb' only catches git-style rev = ... entries (in Cargo.toml) or stanza headers (in Cargo.lock). It misses version specification formats like grovedb = "0.20" or grovedb = { version = "0.20" }, which would silently skip ZK tests if the dependency format changes to use crates.io versions or branch-based git pins instead of rev-pinned git references.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 115 - 123, The current GROVEDB
change detector only matches git-style rev pins and lockfile stanza names;
update the grep used after git diff (the command that currently pipes to grep
-qE 'grovedb.*rev\s*=|name = "grovedb' ) to also detect common dependency forms
such as grovedb = "0.20", grovedb = { version = "..." }, grovedb = { git =
"...", branch = "..."} and any occurrence of grovedb followed by
version|git|branch|rev keys; in short, extend the regex to match patterns like
grovedb\s*=|grovedb.*(version|git|branch|rev)\s*=|name = "grovedb" so
GROVEDB_FILES (and the git diff + grep pipeline) will catch crate version bumps
in Cargo.toml and Cargo.lock.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

git ls-files .github/workflows/

Repository: dashpay/platform

Length of output: 1123

---

🏁 Script executed:

cat -n .github/workflows/tests.yml | head -250 | tail -100

Repository: dashpay/platform

Length of output: 4404

---

🏁 Script executed:

cat -n .github/workflows/tests-rs-workspace.yml

Repository: dashpay/platform

Length of output: 8186

---

🏁 Script executed:

cat -n .github/workflows/tests.yml | head -150

Repository: dashpay/platform

Length of output: 7677

---

🏁 Script executed:

rg -A 20 "jobs:" .github/workflows/tests.yml | head -80

Repository: dashpay/platform

Length of output: 982

---

Ensure ZK tests run when the workflow definition itself changes.

The zk-changed input to tests-rs-workspace.yml is sourced only from ZK code-path diffing and gates the zk-tests job. When .github/workflows/tests-rs-workspace.yml itself is edited—for example, to modify the zk-tests job definition—the caller still runs (because rs-workflows-changed == 'true'), but zk-tests skips because zk-changed remains false. This prevents validation of workflow-level changes to the ZK test job. Consider including the workflow-path signal (rs-workflows-changed) in the zk-changed input so the job executes whenever its definition or dependencies are modified.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 197 - 205, The zk test gating input
currently passes only the changes.outputs.zk-changed value to the reusable
workflow; update the rs-workspace-tests job so the with: zk-changed input is
true if either needs.changes.outputs.zk-changed OR
needs.changes.outputs.rs-workflows-changed is true (i.e., include the
rs-workflows-changed signal in the zk-changed input) so edits to
tests-rs-workspace.yml itself also trigger zk-tests in the called workflow.

[QuantumExplorer]

Self Reviewed (this removes code coverage for now)