Replace some uses of `MethodDescCallSite` with `UnmanagedCallersOnly` methods #123832

AaronRobinsonMSFT · 2026-01-31T04:36:55Z

Instead of calling through the MethodDescCallSite machinery, retrieve a pointer to a UnmanagedCallersOnly function and call it like a normal reverse P/Invoke.

- Add Value property to ObjectHandleOnStack and ByteRef - Add UnmanagedCallersOnly overloads for ConvertContentsToNative, ConvertContentsToManaged, ClearNative, and ClearManaged - Add corresponding method entries in corelib.h with _UCO suffix - Add SM_IntPtr_IntPtr_PtrIntPtr_IntPtr_RetVoid signature in metasig.h This enables calling MngdRefCustomMarshaler methods via reverse P/Invoke instead of the more expensive MethodDescCallSite/CallDescrWorker path.

…amCustomMarshaler - Add UnmanagedCallersOnlyCaller template class to callhelpers.h for calling managed methods marked with [UnmanagedCallersOnly] - Add CallerArgConverter templates for automatic argument conversion (OBJECTREF* -> QCall::ObjectHandleOnStack*) - Update DispParamCustomMarshaler to use the new pattern for all four MngdRefCustomMarshaler method calls: - MarshalNativeToManaged (ConvertContentsToManaged) - MarshalManagedToNative (ConvertContentsToNative) - MarshalManagedToNativeRef (ConvertContentsToNative) - CleanUpManaged (ClearManaged) - Add ObjectHandleOnStack constructor with GC frame validation in qcall.h This replaces the expensive CallDescrWorker infrastructure with more efficient reverse P/Invoke calls via UnmanagedCallersOnly methods.

- Update AppContext.Setup to be [UnmanagedCallersOnly] with exception parameter - Update corelib.h APPCONTEXT SETUP signature to include IntPtr for exception - Update corhost.cpp to use UnmanagedCallersOnlyCaller for Setup call - Add SM_PtrPtrChar_PtrPtrChar_Int_IntPtr_RetVoid metasig Since Setup is only called from native code, there's no need to maintain a separate non-UCO overload.

dotnet-policy-service · 2026-01-31T04:37:44Z

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

- Define OBJECT_HANDLE_ON_STACK class in corelib.h for metasig use - Add SM_PtrObjectHandleOnStack_... metasig with typed pointer syntax - Update MngdRefCustomMarshaler UCO methods to use ObjectHandleOnStack* instead of IntPtr in C# signatures - Update AppContext.Setup to use ObjectHandleOnStack* for exception param - Update corelib.h method entries to use the typed signatures This provides better type safety and self-documenting signatures for methods that accept object handles from native code.

Copilot

Pull request overview

This pull request modernizes the calling convention for invoking managed code from native code by replacing MethodDescCallSite machinery with UnmanagedCallersOnly methods, which use the more efficient reverse P/Invoke infrastructure.

Changes:

Introduces UnmanagedCallersOnlyCaller template class in C++ for calling managed methods marked with [UnmanagedCallersOnly]
Adds [UnmanagedCallersOnly] wrapper methods in AppContext and StubHelpers that handle exceptions through an explicit exception parameter
Refactors ObjectHandleOnStack and ByteRef to use property-based access instead of method-based access
Updates method signatures and metadata to support the new calling convention

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/coreclr/vm/callhelpers.h	Adds `UnmanagedCallersOnlyCaller` template class and `CallerArgConverter` for type-safe invocation of UnmanagedCallersOnly methods
src/coreclr/vm/dispparammarshaler.cpp	Replaces `MethodDescCallSite` with `UnmanagedCallersOnlyCaller` for custom marshaler method invocations
src/coreclr/vm/corhost.cpp	Replaces `MethodDescCallSite` with `UnmanagedCallersOnlyCaller` for `AppContext.Setup`
src/coreclr/vm/qcall.h	Adds constructor declaration for `ObjectHandleOnStack`
src/coreclr/vm/qcall.cpp	Implements `ObjectHandleOnStack` constructor with GC frame validation
src/coreclr/vm/metasig.h	Adds method signature metadata for new UnmanagedCallersOnly method signatures
src/coreclr/vm/corelib.h	Updates method definitions to reference new signatures and adds new UCO method definitions
src/libraries/System.Private.CoreLib/src/System/AppContext.cs	Adds `[UnmanagedCallersOnly]` version of `Setup` method with exception parameter
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/QCallHandles.cs	Changes `ObjectHandleOnStack` to use typed pointer and adds `Value` property; changes `ByteRef.Get()` to `ByteRef.Value` property
src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs	Adds `[UnmanagedCallersOnly]` wrapper methods for custom marshaler operations with exception handling; removes pragma warning in favor of discard parameter
src/coreclr/System.Private.CoreLib/src/System/RuntimeHandles.cs	Updates calls from `ByteRef.Get()` to `ByteRef.Value`
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/StaticsHelpers.cs	Updates calls from `ByteRef.Get()` to `ByteRef.Value`

src/libraries/System.Private.CoreLib/src/System/AppContext.cs

src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/QCallHandles.cs

src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

src/coreclr/vm/callhelpers.h

…irectly

src/coreclr/vm/callhelpers.h

src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

huoyaoyuan · 2026-01-31T18:17:45Z

Just curious, why didn't we have "reverse-FCall" that declares managed method in FCall signature and call it directly in C++ code? Why do the arguments have to go through the argument structure and get filled with asm helper?

jkotas · 2026-01-31T18:30:29Z

Just curious, why didn't we have "reverse-FCall" that declares managed method in FCall signature and call it directly in C++ code?

It is not just a simple call. It needs GC and EH transitions. For example, look for CallDescrWorkerInternalReturnAddress.

The idea is to replace this internal transition with UCO, similar how we have replace internal HelperMethodFrames with PInvokes (QCalls).

jkotas · 2026-01-31T18:39:34Z

src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

+                object managedHome = pManagedHome.Value!;
+                ConvertContentsToNative(customMarshaler, in managedHome, pNativeHome);
+            }
+            catch (Exception ex)


We may need something more elaborate like catch (Exception ex) PropagateExceptionToNativeCode(e) to avoid regressing diagnostic experience. It may be needed only in some cases, not sure.

For example, the Main method is called via CallDescWorker today. If we wrap the main method in a UCO that has try/catch iike this, the crash dumps will end up with unwond stack that will make investigation of unhandled exceptions in crash dumps much harder.

cc @janvorli

We may need something more elaborate like catch (Exception ex) PropagateExceptionToNativeCode(e) to avoid regressing diagnostic experience. It may be needed only in some cases, not sure.

Thoughts on what this would look like? Would it be as straight forward as:

static bool PropagateExceptionToNativeCode(Exception e) { // Stack has an assembly's EntryPoint as the last frame in the stack if (...) return false; // Stack has no further managed frames above if (...) return false; return true; }

I suppose the second check is a superset that includes the first.

Yes, something like that. I do not think any of the methods touched in this PR need it. It is only needed for top level entrypoints - Main, finalizer, thread start, etc.

huoyaoyuan · 2026-01-31T19:24:00Z

It is not just a simple call. It needs GC and EH transitions. For example, look for CallDescrWorkerInternalReturnAddress.

Yes I'm aware of these transitions. I'm just unsure why it can't be the form of PROLOG_MACRO; callable(arg1, arg2); EPILOG_MACRO;. Can't the prolog/epilog be represented in C?

…andleOnStack - Updated dispparammarshaler.cpp to pass OBJECTREF* directly - Replaced ObjectHandleOnStack-based metasig entries with typed pointer signatures - Removed OBJECT_HANDLE_ON_STACK class definition from corelib.h - Fixed GetMultiCallableAddrOfCode call in UnmanagedCallersOnlyCaller

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/QCallHandles.cs

src/coreclr/vm/callhelpers.h

jkotas · 2026-02-01T03:22:15Z

Yes I'm aware of these transitions. I'm just unsure why it can't be the form of PROLOG_MACRO; callable(arg1, arg2); EPILOG_MACRO;. Can't the prolog/epilog be represented in C?

The managed calling convention is not exact match of the unmanaged calling convention. A simple callable(arg1, arg2) would not work. We would need to wrap it into platform specific macros. On win-x86, these macros would have to swap argument order. On wasm, these macros would have to setup and push the stack pointer as the first argument. There are issues with return buffer arguments (it always comes after this pointer in managed calling convention) -> more macros. There are issues with zero and sign-extensions of small integer types -> more macros. JIT handles all these arch specific differences as part of UCOs, so the idea is to take advantage of that to reduce amount of arch specific code.

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/QCallHandles.cs

…objects

src/coreclr/vm/callhelpers.h

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

src/coreclr/vm/dispparammarshaler.cpp

AaronRobinsonMSFT · 2026-02-01T07:18:49Z

Copilot uncovered a bug in the existing implementation - #123832 (comment). This PR fixes it, but it does indicate there is a test hole. I'll add tests on Monday.

AaronRobinsonMSFT added 3 commits January 30, 2026 14:50

AaronRobinsonMSFT added this to the 11.0.0 milestone Jan 31, 2026

AaronRobinsonMSFT requested a review from jkotas January 31, 2026 04:36

AaronRobinsonMSFT added the area-VM-coreclr label Jan 31, 2026

Copilot AI review requested due to automatic review settings January 31, 2026 04:36

dotnet-policy-service bot assigned AaronRobinsonMSFT Jan 31, 2026

github-project-automation bot added this to AppModel Jan 31, 2026

Copilot started reviewing on behalf of AaronRobinsonMSFT January 31, 2026 04:37 View session

AaronRobinsonMSFT force-pushed the remove_mdcs_1 branch from 5fd601b to 53b5d65 Compare January 31, 2026 04:44

Copilot AI reviewed Jan 31, 2026

View reviewed changes