Open
Conversation
Parametrize for simulcast Higher resolution to get 3 simulcast layers Use higher tolerance Increase test timeout Clean up Revert timeout increase Clean up Check simulcasted property Format Add HEVC to test matrix Increase timeout Lower resource requirements GitHub Actions runners struggle with the old settings Receive single frame
ladvoc
force-pushed
the
jacobgelman/clt-2627-end-to-end-test-for-video
branch
from
28bff25 to
bba6778
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds end-to-end tests to verify that video streaming works properly, parameterized by codec, resolution, and whether simulcast is enabled.
Methodology
Frames are published with a known, uniform luminance value across the frame, which the subscriber verifies within a certain margin of error. The subscriber also verifies the received frame’s aspect ratio (not exact resolution as it may be smaller due to simulcast). If no frames are received (for example, due to an encoder or decoder not being created in libwebrtc), the test will timeout and fail.
Note: there are probably more robust ways to verify that received frames are correct, but I opted for this method for simplicity. Open to any suggestion here.
Limitations
Currently, only VP8 and VP9 are tested in CI, since I could not get hardware encoding/decoding to work properly in the GitHub Actions runners. In the future, we may consider using self-hosted runners to ensure these paths are covered by automated testing as well.
Other Changes
Updates the CI configuration to build tests for aarch64 on macos-latest. Previously, tests were built for x86_64 even though this runner is ARM-based, which required Rosetta 2 emulation and slowed down test execution.