Community developer's access to GPU / Feedback when creating PR

[yifeif-nv]

Hello all,

I'd like to start a discussion about community developers' access to GPUs and the feedback loop when creating PRs.

It looks like we've recently been hitting a lot of blockers where internal CI results aren't visible to community developers. That seems to make merging PRs a lot harder than it should be compared to the internal flow. I'm thinking of provisioning some GPU runners for community users so you can directly see the pre-merge failure results.

I'd like to know what you think:

1. In your development flow, what issues have you hit when trying to merge a PR?
2. Feel free to highlight any pain points; our team will do our best to eliminate them and make merging as smooth as possible.

I also noticed that some of our developers don't have access to an NVIDIA GPU. I'd suggest looking into GPU leasing services (Something like https://brev.nvidia.com/ or even google colab maybe?) to gain access. You don't necessarily need a fancy GPU: something like an L4, which costs around $0.50 an hour, should be more than sufficient for working on the majority of models. We can see if this fits our needs and whether we can get some shared machines for the community as well.

@kanhaiya-dct @JiaxinD @lukiod @jkzhang7 @Darshan3690 @munnmajithia @roma5087 @AbishekCoder1

[yifeif-nv]

Leave us a comment and we can discuss how to make the PR merging experience better!

[AbishekCoder1]

Thanks for starting this discussion and tagging me, @yifeif-nv
My development flow & pain points:
Coding Blind & Invisible CI: As a developer on Apple Silicon, the biggest blocker is the inability to compile or test the backend locally. We have to rely on CI pipelines to catch basic errors, but when internal CI results aren't visible, debugging turns into a guessing game that slows down the PR process.
Feedback on the proposed solutions:
Community GPU Runners: Provisioning GPU runners so we can directly see pre-merge failure results would be a massive help and solve the "invisible CI" blocker completely.
GPU Leasing (Brev/Colab): This is a fantastic idea and completely solves the local hardware blocker for me. I am ready to spin up an L4 instance to test this workflow!
Since the Music3 model required an 80GB H100 and C++ orchestration, do you have any smaller, Python-based model onboarding tasks available that would fit within the 24GB limits of an L4? I would love to tackle one of those as a test run for this cloud GPU workflow while the team finalizes the "CPU-only" list!

[yifeif-nv]

Cool, thanks for the feedback. I just found another model on Hugging Face https://huggingface.co/superwhisper/s1-mini . This one is a tiny one (0.8 billion parameters), so pretty much any GPU should work.

You can give this one a try. It is essentially derived from Qwen3-ASR. You could also try the original model as well.

A trick I find very useful is using the sliding bar on Hugging Face (https://huggingface.co/models?num_parameters=min:0,max:1B&sort=trending), where you can filter the model parameter. Usually, for an L4, anything smaller than 4B should work.

[yifeif-nv]

https://huggingface.co/fastino/gliner2.5-multi-v1 This one is also pretty small and seems to be a good candidate for the first model family you can try.

[AbishekCoder1]

This is incredibly helpful, thanks @yifeif-nv! That Hugging Face parameter filtering trick is definitely going into my notes for sizing models to an L4.
I will take on the superwhisper/s1-mini model! Since it is derived from Qwen3-ASR, it seems like a perfect entry point. I am going to spin up a Google Colab instance today to start looking at the Python export and see how it hooks into the existing Qwen builder paths. I'll open a draft PR once I get the basic .bundle compiling!

[roma5087]

Hello everyone, Seconding the CI-visibility point. *Registering a model touches five separate inventories, and a contributor **learns them one red CI round at a time:* The docs and the `transform-model` skill describe the manifest plus the family `MODEL.toml`. Any model that counts as "ready" also needs a threshold sidecar, a `tests/validation/model_workloads.yaml` binding, a ` benchmarks/performance/release.yaml` coverage entry or declared exclusion, a `website/data/hf-model-metadata.json` entry, and updates to hardcoded counts in `tests/tools/test_trtmc_validate.py` and ` tests/tools/test_perf_matrix.py`. Each is enforced by a different test, so each one surfaces only after the previous is fixed. Meanwhile `tools/model_ci.py validate` passes cleanly with all of them missing, which is what makes a contributor think they're ready to push. A `--check-registration`mode that reported every missing declaration at once would collapse several CI rounds into one. *Three of the test modules that gate a PR can't be collected from the * *documented contributor setup:*`CONTRIBUTING.md` points at ` requirements/community-ci.txt`; installing exactly that and running `pytest tests/builder/ tests/tools/ --collect-only` collects 3556 tests and errors on four modules — three of them because they import `tensorrt` at collection time: tests/builder/test_repository_contracts.py No module named 'tensorrt' tests/tools/test_model_checks.py No module named 'tensorrt' tests/tools/test_perf_matrix.py No module named 'tensorrt' Those three carry the model-registration contracts described above, and the assertions themselves are pure Python — counts, set membership, a dict comparison. So the checks most likely to fail a first model PR are exactly the ones a contributor cannot run locally, and that stays true with a GPU runner unless the imports are made lazy or the contracts move to CPU-only. *Hardcoded inventory counts make concurrent model additions conflict by * *construction:*Assertions of the form assert len(catalog["models"]) == len(ready_models) == N assert len(bindings) == N "explicitly_excluded_profiles": N, mean every model addition collides with every other one in flight, on lines that carry no semantic information. Any PR that sits for a few days will need a rebase whose only conflicts are numbers, and the contributor has to work out what each number should become. Asserting that the catalog * *covers** the ready set would check the same property without the collisions. *Two patterns worth a look:* - Runtime errors that don't include the value that caused them. A cache-allocation failure that reports only "failed to create" leaves the contributor adding print statements to a C++ runtime to find a bad computed dimension. Including the offending value in the message turns a debugging session into a glance. - Subprocess crashes reported as comparison failures. When the runtime dies before emitting output, the E2E harness reports it as a text-comparison failure with an empty response. That reads as a model-quality problem rather than a hard runtime error, and sends the contributor looking in the wrong place. Surfacing a non-zero exit as an error would prevent that. *Smaller CLI things:* trtmc run --help` prints Error: Unknown flag: --help` ahead of the usage text. An unsupported `--precision` value is rejected only after the full weight load rather than at argument parsing, which costs minutes per attempt on a large checkpoint. `text_trace.step_trace_path` is declared in the global config schema and accepted by `--set` without warning, but only some families' runtimes read it; for the others it produces no file and no message, so a contributor reaching for the built-in per-step trace to debug a divergence gets silence. Thanks, Matt

…

On Wed, Sep 2, 2026 at 1:13 AM yifeif-nv ***@***.***> wrote: Hello all, I'd like to start a discussion about community developers' access to GPUs and the feedback loop when creating PRs. It looks like we've recently been hitting a lot of blockers where internal CI results aren't visible to community developers. That seems to make merging PRs a lot harder than it should be compared to the internal flow. I'm thinking of provisioning some GPU runners for community users so you can directly see the pre-merge failure results. I'd like to know what you think: 1. In your development flow, what issues have you hit when trying to merge a PR? 2. Feel free to highlight any pain points; our team will do our best to eliminate them and make merging as smooth as possible. I also noticed that some of our developers don't have access to an NVIDIA GPU. I'd suggest looking into GPU leasing services (Something like https://brev.nvidia.com/ or even google colab maybe?) to gain access. You don't necessarily need a fancy GPU: something like an L4, which costs around $0.50 an hour, should be more than sufficient for working on the majority of models. We can see if this fits our needs and whether we can get some shared machines for the community as well. @kanhaiya-dct <https://github.com/kanhaiya-dct> @JiaxinD <https://github.com/JiaxinD> @lukiod <https://github.com/lukiod> @jkzhang7 <https://github.com/jkzhang7> @Darshan3690 <https://github.com/Darshan3690> @munnmajithia <https://github.com/munnmajithia> @roma5087 <https://github.com/roma5087> @AbishekCoder1 <https://github.com/AbishekCoder1> — Reply to this email directly, view it on GitHub <#1128?email_source=notifications&email_token=BMZVNHPO3NHLKIFGATAKB2D5M6UBHA5CNFSNUABBM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63RPGEYDOMZSGE4DNJTSMVQXG33OU5WWK3TUNFXW5JLFOZSW45FMMZXW65DFOJPWG3DJMNVQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMZVNHKFHBWCTVOY3D3EUML5M6UBHAVCNFSNUABJKJSXA33TNF2G64TZHMYTEMJWGMZDAMRVHE5UI2LTMN2XG43JN5XDWMJQG4ZTEMJYG2QXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BMZVNHMSMICCPHXQCEQE64L5M6UBHA5CNFSNUABBM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63RPGEYDOMZSGE4DNJTSMVQXG33OU5WWK3TUNFXW5JLFOZSW45FKMZXW65DFOJPWS33T> and Android <https://github.com/notifications/mobile/android/BMZVNHP2KEZ57NWSKIJLXB35M6UBHA5CNFSNUABBM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63RPGEYDOMZSGE4DNJTSMVQXG33OU5WWK3TUNFXW5JLFOZSW45FOMZXW65DFOJPWC3TEOJXWSZA>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[yifeif-nv]

Yes! Thanks matt for pointing this out. And this is indeed the highest priority our team is working on. To decouple things further so everyone just need to modify the code within a single folder. We are working on a huge PR #1093. Which is suppose to make this better.

Currently we are targeting to merging this in over the weekend

[lukiod]

nothing to add from my own two prs here, they went through clean. but the collection-time tensorrt imports on those three test modules matt mentioned seems like the actual root cause more than gpu access itself, a runner wont fix a test that cant even be collected locally. also worth noting the community cpu checks only run on ubuntu-24.04, no mac, no windows, and honestly not even other linux distros. i run arch (omarchy) and ubuntu passing doesnt really tell you it'll work there too, different toolchain and lib versions can break things that look fine on ubuntu. gpu leasing sounds like a good idea regardless.

[yifeif-nv]

Yes agreed on the CPU platform coverage. Ideally the C++ source should be portable (Beside the OS resource access which will break on Windows, Mac is not in the picture here as there is no NVIDIA GPU that can run the SW). But the build system need to be tested for more platform x Arch.

Good feedback and we'll find more resource to harden this

[jkzhang7]

1. Voted for shared machines. As community developers, having shared machines can greatly motivate the production.
2. I would like to have a set of well documented agent skills to inform agent what is the best practice to reduce agent iteration loops during the dev process. Maybe something like TensorRT wiki? So agent can follow the path as much as possible. Described in Docs: repo agent skills are unreachable from every documented entry point #1125 as well.

[yifeif-nv]

Shared machine sounds good. I'm talking with some of our teams internally to see if we can host some shared machine.

[yifeif-nv]

I have mixed feelings about agent skills.

Essentially, anything like a skill will need to be maintained, and that can get outdated quite often. Also, from my perspective, a skill is kind of like injecting our human taste into the AI agents, which essentially steers them towards our human thoughts distribution. If possible, I would try to avoid that.

Instead of having skills, my theory is that, in the long run, this ModelConnect repo should be in the training data for the next-generation models, and they should know things better. Let me know if you think otherwise; I'm really open to discussing the best way forward.

We can also do a pilot study with skills: if it works well, we keep it; if it doesn't, we can remove it.

[JiaxinD]

Hi Yifei, thanks for starting this discussion. The main blocker I hit on #1053 was that the public checks passed while the protected pre-merge gate failed without actionable output, so I couldn't determine whether it was a code, environment, or base-branch issue. Making those results visible would help a lot. Since #1093 is expected to replace the current family layout, should I hold #1053 until it lands and then port the checkpoint-validation fix to the new structure?

[yifeif-nv]

Yes, I think we should hold off on #1053 until #1093 went in. That PR is aiming to address the overlapping hotspot in the codebase to get each contribution more decoupled.

[yifeif-nv]

Hey all, I just want to put an update on this discussion thread.

We are already talking with some internal teams about working on a provisioned GPU for the pre-merge CI. The short-term goal is that all the community contributors and PRs can see all the GPU test results as well.

[kanhaiya-dct]

Hi @yifeif-nv
Here are the best solutions and action items provided by the maintainer (yifeif-nv) and agreed upon by the collaborators:

1. Fixing "Invisible CI" (The Main Priority): The team is actively working with internal teams to provision shared GPU runners for pre-merge CI. This will allow all community contributors to directly see GPU test results and failures on their PRs before merging, eliminating the "guessing game."

2. Decoupling the Codebase: To make merging smoother and prevent merge conflicts, the maintainer mentioned they are landing a massive refactoring PR (refactor: isolate model families end to end #1093). This will restructure the codebase so that developers only need to modify code within a single folder for their contributions. Contributors were asked to hold off on some PRs (like fix(checkpoints): preserve validation under python -O #1053) until this lands.

3. GPU Hardware Solutions for Community Devs: For developers without local GPUs (like Apple Silicon users), the maintainer suggested using cheap GPU leasing services like brev.nvidia.com or Google Colab.

They specifically recommended using an L4 GPU (costing ~$0.50/hour), which is sufficient for smaller models.
They shared a trick to filter Hugging Face models by parameter size (< 4B parameters) and suggested trying out smaller Python-based models first, such as superwhisper/s1-mini (0.8B) or gliner2.5-multi-v1.
4. Agent Skills (Pilot Study): A collaborator (jkzhang7) asked for documented "agent skills" to help AI agents navigate the repo. The maintainer was hesitant, as hardcoded skills require maintenance and restrict the AI's natural distribution. However, they agreed to do a "pilot study" with skills to see if it actually helps, with the long-term goal being that next-gen models will just learn the repo naturally.