feat: Add bulk sync operations and advanced filtering

[jdrew82]

Summary

Adds 9 new features for bulk sync operations and advanced filtering of synced objects, organized in 3 phases. All changes are fully backwards-compatible — every new parameter defaults to None/False, preserving existing behavior.

Phase 1: Diff-side Filtering

Feature 8 — Diff.filter() / Diff.exclude()

Post-diff, pre-sync manipulation of computed diffs. Returns a new Diff containing only matching (or excluding matching) elements. Recursive through child DiffElements.

diff = dst.diff_from(src)
filtered = diff.filter(actions={"create", "update"}, model_types={"device"})
dst.sync_from(src, diff=filtered)

Files: diffsync/diff.py

Feature 6 — model_types parameter

Restricts diff/sync to specific model type names. Applied at both top_level iteration and child recursion.

dst.diff_from(src, model_types={"site", "device"})

Files: diffsync/__init__.py, diffsync/helpers.py

Feature 7 — sync_attrs / exclude_attrs

Whitelist or blacklist specific attributes per model type during diff calculation.

dst.sync_from(src, sync_attrs={"device": {"role"}})  # only diff role
dst.sync_from(src, exclude_attrs={"device": {"last_seen"}})  # skip last_seen

Files: diffsync/__init__.py, diffsync/helpers.py

Feature 5 — filters query predicates

Per-model-type callable predicates to filter which objects participate in diffs.

dst.diff_from(src, filters={"device": lambda d: d.role == "spine"})

Files: diffsync/__init__.py, diffsync/helpers.py

Phase 2: Sync-side Enhancements

Feature 9 — sync_filter callback

Callback checked before each CRUD operation. Return False to skip the operation.

dst.sync_from(src, sync_filter=lambda action, model_type, ids, attrs: action != "delete")

Files: diffsync/__init__.py, diffsync/helpers.py

Feature 4 — Structured sync_complete operations summary

sync_complete() now receives an operations kwarg with a structured dict of all CRUD operations performed. Backwards-compatible with existing overrides via try/except TypeError fallback.

def sync_complete(self, source, diff, flags=None, logger=None, operations=None):
    # operations = {"device": {"create": [{"ids": ..., "attrs": ..., "model": ...}], ...}}

Files: diffsync/__init__.py, diffsync/helpers.py

Phase 3: Bulk & Parallel Execution

Feature 1 — Bulk CRUD methods

Optional create_bulk(), update_bulk(), delete_bulk() class methods on DiffSyncModel. Default implementations loop over individual calls. Override for batch API/DB operations.

Store-level add_bulk(), update_bulk(), remove_bulk() on BaseStore, LocalStore, and RedisStore (with Redis pipeline optimization).

Files: diffsync/__init__.py, diffsync/store/__init__.py, diffsync/store/local.py, diffsync/store/redis.py

Feature 2 — batch_size parameter

Parameter plumbed through to DiffSyncSyncer for future chunked dispatch of bulk operations.

Files: diffsync/__init__.py, diffsync/helpers.py

Feature 3 — concurrent parallel sync

concurrent=True flag parallelizes sync of independent top-level subtrees using ThreadPoolExecutor. Thread safety achieved via:

• threading.local() for per-element syncer state (model_class, action, logger)
• threading.Lock on LocalStore mutation methods (add, update, remove_item)
• threading.Lock on operations summary dict

dst.sync_from(src, concurrent=True, max_workers=4)

Files: diffsync/__init__.py, diffsync/helpers.py, diffsync/store/local.py

New API Surface

Method	New Parameters
Adapter.diff_from() / diff_to()	model_types, filters, sync_attrs, exclude_attrs
Adapter.sync_from() / sync_to()	model_types, filters, sync_attrs, exclude_attrs, sync_filter, batch_size, concurrent, max_workers
Adapter.sync_complete()	operations
DiffSyncModel	create_bulk(), update_bulk(), delete_bulk()
Diff	filter(), exclude()
BaseStore	add_bulk(), update_bulk(), remove_bulk()

Test plan

• All 133 existing tests pass unchanged
• 36 new tests added in tests/unit/test_new_features.py
• Tests cover: Diff filter/exclude, model_types scoping, sync_attrs/exclude_attrs, query predicates, sync_filter callback, sync_complete operations, bulk CRUD, concurrent sync, and integration combinations
• Review thread safety under high concurrency
• Test with RedisStore backend
• Performance benchmark with large datasets (1000+ models)

🤖 Generated with Claude Code