Tiers 4 and 5 — multi-term projects that assume a co-supervisor, produce genuinely publishable work, and in several cases connect QHPC directly to research already happening at Queen's. These differ from Volume I in kind, not degree.
Do not route a member into Tier 4 until they have completed at least one Tier 2 and one Tier 3 project. Everything here assumes the measurement discipline of T2.4 (roofline reasoning), the baseline rigour of T3.5 (honest comparison), and fluency with a profiler. A student who arrives without those will produce results that do not survive review — and will have a bad experience doing it.
| Volume I (Tiers 1–3) | Volume II (Tiers 4–5) | |
|---|---|---|
| Duration | One sprint, occasionally two | Two to four sprints; often a full year |
| Supervision | A senior member as sprint lead | A faculty or graduate co-supervisor — required, not optional, for Tier 5 |
| Domain knowledge | None assumed | Tier 5 requires it, supplied by the partner |
| Novelty expectation | Careful replication is a success | A genuine contribution — something not previously known |
| Failure risk | Low; scoped so failure is informative | Real. Some of these will not work out, and that must be acceptable. |
| Output | Sprint report; possible CUCAI submission | Conference paper; possible co-authorship with a research group |
Tier 4 and 5 projects do best with a deliberately mixed team:
A Tier 4 project that produces a negative or partial result is not a failed project — provided the methodology was sound and the work is documented. "We attempted X, it did not yield the expected benefit, and here is the evidence and our explanation" is a legitimate and sometimes valuable contribution.
What is a failure is an undocumented dead end. Insist on the same recording discipline as Volume I: every configuration, every result, logged as it happens. On a multi-term project this matters more, not less, because the people who finish it may not be the people who started it.
Tier 4 topics are HPC research proper — the object of study is the computing system itself. QHPC can execute these independently, with a co-supervisor for guidance rather than domain knowledge.
Tier 5 topics are collaborations — the object of study belongs to someone else's research programme, and QHPC contributes the computational capability. These cannot be started without a partner, and the partner conversation is the first and hardest step. The upside is that they produce the faculty relationships the operating plan's §09 competition applications and §16 consulting strategy both depend on.
Scientific computing has traditionally used 64-bit floating point (FP64) throughout, on the reasonable grounds that accuracy matters. But modern accelerators are far faster at lower precisions — tensor cores designed for machine learning deliver dramatically higher throughput at FP16 or FP8 than at FP64.
Mixed precision exploits this: perform the bulk of the work in low precision, then recover full accuracy through a small number of correction steps in high precision. The canonical instance is HPL-MxP, which performs the LU factorisation in FP16 or FP8 and then applies GMRES-based iterative refinement in FP64 to restore the accuracy of the final solution.
You implement and study this trade-off: how much speed is gained, how accuracy behaves, and where numerical stability breaks down.
HPL-MxP is the third major benchmark alongside HPL and HPCG, and it is increasingly what distinguishes serious teams. It is also effectively unrun at undergraduate level — the entire 72-paper CUCAI corpus contains nothing resembling it.
Frontenac's GPUs have tensor cores built for precisely this. That combination — a current, competition-relevant topic that nobody in the peer network is touching, on hardware QHPC already has — makes this arguably the single highest-value Tier 4 project in the catalogue.
The precision formats and what they cost you:
| Format | Bits | Approx. decimal digits | Approx. dynamic range | Relative throughput |
|---|---|---|---|---|
| FP64 (double) | 64 | ~15–16 | Very wide | 1× (baseline) |
| FP32 (single) | 32 | ~7 | Wide | ~2× on CPU; far more on consumer GPUs |
| TF32 | 19 stored in 32 | ~3–4 | FP32 range | High on tensor cores |
| FP16 (half) | 16 | ~3 | Narrow — overflows above ~65,504 | Very high on tensor cores |
| BF16 | 16 | ~2–3 | FP32 range — fewer mantissa bits, more exponent | Very high; often preferred for stability |
| FP8 | 8 | ~1–2 | Very narrow | Highest; newest, least mature tooling |
The critical distinction is between mantissa bits (precision) and exponent bits (range). FP16 and BF16 both occupy 16 bits but divide them differently: FP16 has more mantissa (more precise) but a narrow exponent (overflows easily); BF16 sacrifices mantissa for FP32's exponent range (less precise but far more numerically robust). For many scientific workloads BF16 is the safer choice, and demonstrating why empirically is a good result.
The iterative refinement idea, stated simply. To solve Ax = b:
1. Factor A in LOW precision → cheap, but the solution x₀ is inaccurate
2. Compute residual r = b − Ax₀ → in HIGH precision; this is cheap (one matvec)
3. Solve A·d = r using the low-precision factors → cheap
4. Correct: x₁ = x₀ + d
5. Repeat until the residual is small enough
The expensive step (factorisation, O(n³)) happens ONCE in low precision.
The correction steps are O(n²) and happen a few times in high precision.
Result: near-FP64 accuracy at close to low-precision cost.
This works provided the matrix is not too ill-conditioned. When it is, refinement fails to converge — and finding that boundary is a central part of this project.
ncu to confirm — it reports tensor-core utilisation directly. A common and invisible failure is writing FP16 code that runs on ordinary FP32 units, giving you the accuracy loss with none of the speed gain.| Quantity | How | Why it matters |
|---|---|---|
| Time to solution | Wall clock, factorisation + all refinement | The headline benefit; must include refinement cost |
| Final residual | ‖b − Ax‖ / ‖b‖ | Did you actually recover accuracy? |
| Forward error | Against a known exact solution | Residual can be small while the answer is wrong for ill-conditioned systems |
| Refinement iterations | Count to convergence | More iterations erodes the speed advantage |
| Convergence failures | Fraction of cases that never converge | The practical limit of the technique |
| Tensor-core utilisation | ncu | Proves you're using the hardware you think you are |
Not actually using tensor cores. The invisible failure. Verify with the profiler, never infer from timing.
Reporting residual instead of forward error. For ill-conditioned systems a tiny residual is entirely compatible with a badly wrong answer. Report both, and understand the difference.
Not controlling condition number. Random matrices are typically well-conditioned, so everything works and you conclude mixed precision is free. The interesting behaviour is at high condition number and you must construct it deliberately.
Ignoring refinement cost. If refinement needs twenty iterations the speed advantage evaporates. Time the whole solve.
Overflow silently producing NaN. FP16 overflows above ~65,504. Check for non-finite values explicitly at every stage rather than discovering them in the final answer.
Comparing against a weak FP64 baseline. The same discipline as T3.5. Use a tuned library implementation, not your own LU.
Extend to FP8, which is the current research frontier and where tooling is least mature. Or apply mixed precision to something other than dense LU — iterative solvers (conjugate gradient with a low-precision preconditioner) are arguably more practically important, since that is what real simulations use.
The most ambitious extension: apply this to a real application from a Tier 5 partner. If the geomechanics or nuclear-materials groups are running large linear solves, demonstrating a mixed-precision speedup on their matrices with their accuracy requirements is a genuinely valuable collaboration and a much stronger paper than a synthetic study.
Modern supercomputers use accelerators from different vendors — NVIDIA, AMD, and Intel all ship GPUs with incompatible native programming models. Writing CUDA locks you to NVIDIA. Rewriting an application for each vendor is prohibitively expensive. A family of performance-portable programming models promises to solve this: write once, compile for any target, retain most of the performance.
The question is whether that promise holds, and what it costs. You take one kernel, implement it in several models, and measure both the performance retained and the development effort required.
This is not a settled question with a textbook answer. It is an active debate in the HPC community with real money behind it — national laboratories have made billion-dollar procurement decisions partly on portability grounds. An honest, careful undergraduate study with reproducible methodology is a legitimate contribution to it.
It is also directly relevant to QHPC's own situation: your Pi cluster is ARM, Frontenac is x86 with NVIDIA GPUs, and the Kingston sovereign facility's architecture is not yet public. Portability is not abstract here.
| Model | Approach | Portability | Typical effort |
|---|---|---|---|
| CUDA | Vendor-native, explicit | NVIDIA only | High — full rewrite |
| HIP | AMD's CUDA-alike; source-portable to NVIDIA | AMD + NVIDIA | Low if coming from CUDA |
| OpenMP target offload | Directives added to existing code | Broad — CPU, NVIDIA, AMD, Intel | Lowest — often a handful of pragmas |
| OpenACC | Directives; historically NVIDIA-focused | Moderate | Low |
| SYCL | C++ single-source, standards-based | Broad | Moderate — idiomatic C++ rewrite |
| Kokkos | C++ abstraction layer over backends | Broad | Moderate; designed for large codebases |
| std::par | Standard C++ parallel algorithms | Growing | Lowest for suitable code |
Realistic scope for QHPC: compare CUDA (as the performance ceiling), OpenMP target offload (as the lowest-effort option), and one of SYCL or Kokkos. Attempting all seven is a way to do none of them properly.
A portability study is far weaker if you only have one vendor's hardware. Frontenac has NVIDIA GPUs. Before scoping this project, establish what else is reachable:
Deciding this in week 1 rather than week 8 is the difference between a coherent project and a salvage operation.
A performance-retention table (portable model as a percentage of native CUDA) across two contrasting kernels, an effort comparison in hours and lines changed, profiler-based explanations for the largest gaps, and a recommendation that acknowledges the trade-off honestly. The finding that a model achieves 85% of CUDA performance for 20% of the effort is exactly the kind of result that changes someone's decision.
An unoptimised CUDA reference. If your CUDA version is naïve, the portable models will look artificially good. The ceiling must be a real ceiling.
Not recording effort. Half the research question, and unrecoverable after the fact. Log hours as you go.
One kernel only. Portability results are kernel-dependent. A single kernel gives an anecdote.
Comparing across different compilers without saying so. Different toolchains have different maturity for each model, and that is a confound worth naming explicitly.
Over-claiming from single-vendor results. If you only tested NVIDIA, you have measured abstraction overhead, not portability. Say so plainly in the title if necessary.
Add maintainability as a third axis: how much does each implementation diverge over time as the kernel evolves? Simulate this by adding a feature to the kernel and measuring the change required in each version. Long-term maintenance cost is the argument portable models are actually sold on, and almost nobody measures it.
The carbon intensity of electricity — grams of CO₂ per kilowatt-hour — varies substantially by hour and by region, depending on what generation is running. A computation performed at one time can therefore have a materially different climate cost than the identical computation performed a few hours later.
Many HPC jobs are delay-tolerant: a batch simulation that a researcher will look at tomorrow does not need to start immediately. Carbon-aware scheduling exploits that flexibility, deferring such jobs into low-carbon windows. Published work in this area reports emissions reductions in the range of roughly 18–57% for bounded delay, depending on grid and workload.
You implement a carbon-aware scheduling policy on the Pi cluster, drive it with real Ontario grid data, and quantify the emissions/delay trade-off.
Ontario's grid is unusually low-carbon — predominantly nuclear and hydro, with comparatively little fossil generation. That means the absolute savings available in Kingston are smaller than in a coal-heavy grid, and the naïve expectation is that carbon-aware scheduling matters less here.
That is itself a finding worth establishing rigorously, and it leads somewhere more interesting: if temporal shifting yields little in Ontario, does spatial shifting — sending work to a different region — yield more? And what does that imply for a sovereign compute facility being built in this province? A study that engages honestly with a null or small result in the local case, then extends it, is considerably more valuable than one that assumes a large effect and finds it.
Three mechanisms are studied in the literature, in increasing order of difficulty:
The central trade-off is emissions reduction versus job delay, and the honest framing is as a constrained optimisation: minimise emissions subject to a bound on delay. A policy that reduces emissions 50% by delaying every job three days is not useful.
The IESO does not publish a real-time carbon-intensity API in gCO₂/kWh. Ontario's system operator publishes raw operational data — hourly generation by fuel type — at reports-public.ieso.ca/public/, but not an emissions figure. You have three options, and choosing between them is part of the project:
Whichever you pick, state the methodology and its uncertainty in the report. Marginal versus average emission factors give materially different answers — average factors describe the grid's current mix, marginal factors describe what actually changes when you add load. For a scheduling study, marginal is arguably the more correct choice and is almost never used. Engaging with that distinction honestly would put this work above most undergraduate treatments of the topic.
A Tier 4 project must situate itself in the literature. These four are the minimum related-work citations for this topic — note that all four post-date most existing undergraduate treatments, which is exactly why this remains "very current" rather than settled.
A previous pass through this section cited "Chen et al., 2025" (a multi-objective deep-RL carbon scheduler reporting 41.3% carbon reduction, attributed to "Journal of Advanced Computing Systems") without independently confirming the paper exists as described. Checked this pass via Scholar Sidekick and direct web search: the paper could not be located anywhere beyond the discovery tool's own result listing, and the named venue's own site shows markers commonly associated with low-quality or predatory publishers (a generic Charleston, WV mailing address; a Gmail contact address behind a registered ISSN). The citation has been removed and replaced above with Liu et al. (2023, Applied Energy) — a real, Crossref-indexed paper doing comparable work, independently verified. This is the exact fabrication-adjacent risk this document's own Material Passport methodology exists to catch; it slipped through in an earlier pass because that pass added citations from a discovery tool's abstract text without running them through verifyCitation first. Treat this as the reminder to run every citation through that check before it goes in a paper, not after a reviewer asks.
Ignoring idle power. The dominant error. If deferring a job means the cluster idles at high power, you may increase total emissions. Measure idle draw first and model it explicitly.
Assuming a large carbon signal. Verify the actual variation in Ontario before designing around it. Design the study so that a small effect is still a reportable finding.
Perfect foresight. Using actual future carbon intensity rather than a forecast inflates results and is not deployable. If you do this for an upper-bound calculation, label it clearly as an oracle.
Reporting only emissions. A policy is only viable if the delay cost is acceptable. Always report both.
Conflating energy and carbon. They are different quantities related by a time-varying factor. Be precise about which you measured and which you computed.
Add heat reuse to the model — data centre waste heat has value in a Kingston winter, and the Bell/Queen's facility's own public description mentions heat recapture. Modelling the combined carbon accounting of compute plus displaced heating is genuinely novel at this scale and directly relevant to a facility being built in this city.
Or go the PCAPS route: if your Pi-cluster workload has any task dependencies (a multi-stage pipeline rather than independent jobs), redesign the policy to be precedence-aware rather than treating every job as independently deferrable. This is a strictly harder and more current version of the same study, and almost nobody outside PCAPS itself has attempted it on a real (non-simulated) cluster.
In Volume I's T2.3 you almost certainly found that scaling degraded at high node counts, and that the culprit was communication. This project examines that culprit directly.
Collective operations — broadcast, reduce, allreduce, alltoall — involve every process in a communicator and are the backbone of parallel scientific code. An allreduce appears in the inner loop of essentially every iterative solver and every distributed training step. MPI implementations provide several algorithms for each collective, choose between them heuristically, and expose tuning parameters that almost nobody adjusts.
You benchmark the algorithms, find where each wins, and quantify what correct tuning is worth on a real application.
Framing worth borrowing verbatim from the ICL paper: HPL-MxP "seeks to highlight the emerging convergence of high-performance computing (HPC) and artificial intelligence (AI) workloads." That sentence is the reason this topic is simultaneously an HPC systems project and a legible CUCAI submission — it sits deliberately on the boundary.
Consider allreduce — every process contributes a value, every process receives the combined result. Several strategies exist:
| Algorithm | How it works | Cost characteristic | Wins when |
|---|---|---|---|
| Linear | All processes send to root; root combines and broadcasts back | O(P) latency in process count | Very small P only |
| Binomial tree | Pairwise combination up a tree, then broadcast down | O(log P) latency, but each step moves the whole message | Small messages, where latency dominates |
| Recursive doubling | Exchange with a partner at increasing distance, doubling each round | O(log P) steps, full message each round | Small-to-medium messages; power-of-two process counts |
| Ring | Data circulates around a logical ring in chunks | O(P) steps but each moves only 1/P of the data — bandwidth-optimal | Large messages, where bandwidth dominates |
| Rabenseifner | Reduce-scatter followed by allgather | Bandwidth-optimal and latency-competitive | Large messages; often the default for good reason |
The crossover between latency-optimal and bandwidth-optimal algorithms is the whole subject. A tree is best for an 8-byte reduction and terrible for an 8-megabyte one; a ring is the reverse. MPI implementations switch between them at a size threshold — and that threshold is a tunable parameter, usually set by a heuristic that was calibrated on hardware unlike yours.
Cluster competitions score on real applications at fixed node counts, where communication is frequently the limiting factor. Collective tuning is free performance requiring no code changes — exactly the category of optimisation a well-prepared team exploits and an unprepared one never considers.
It also connects directly to Volume I's E12: you measured a latency-versus-message-size curve there by hand. This is the same measurement, done properly, across algorithms, with consequences.
osu_allreduce, osu_bcast, osu_alltoall, osu_reduce — across message sizes from 8 bytes to tens of megabytes, at 2, 4, 8, 16, 32, 64 processes.--mca coll_tuned_use_dynamic_rules 1 plus per-collective algorithm selection); other implementations have equivalents. Confirm the parameter actually took effect rather than assuming — a mistyped MCA parameter is silently ignored.MCA parameters silently ignored. Verify with --mca coll_base_verbose or equivalent that the algorithm you requested is the one running. Otherwise you measure the default repeatedly and conclude the algorithms are identical.
Testing only power-of-two process counts. Several algorithms have special cases for powers of two and degrade at other counts. Include 6, 12, 24 to expose that.
Ignoring process placement. Whether ranks are packed onto nodes or spread across them changes communication cost enormously. Fix and report your mapping.
Reporting collective speedup as application speedup. Amdahl applies to optimisations too.
Not accounting for network sharing. On a shared cluster another job's traffic affects yours. Repeat across times of day.
Investigate whether the optimal choice differs between Frontenac's interconnect and the Pi cluster's plain Ethernet. The crossover points should shift substantially — a high-bandwidth low-latency fabric favours different algorithms than commodity gigabit. Demonstrating that the "right" tuning is hardware-dependent, with two concrete data points, is a stronger and more generalisable result than tuning a single machine.
Simulating a quantum computer on a classical one requires tracking the full quantum state, which for n qubits is a vector of 2n complex amplitudes. That exponential is unforgiving: 30 qubits needs about 16 GB in double-complex precision, 34 qubits needs roughly 256 GB, 40 qubits needs about 16 TB. Quantum simulation is therefore a memory-capacity-bound distributed computing problem — which makes it a genuinely good HPC subject regardless of one's views about quantum computing itself.
You benchmark state-vector simulators, find the qubit count at which a single node is insufficient, and characterise what distribution costs.
This is not a quantum computing project. You are not doing quantum algorithms research, and no quantum hardware is involved. It is a memory-bound distributed-computing study that happens to use quantum circuits as the workload — and it should be framed that way in the report, honestly, rather than dressed up as something it isn't.
That said, its recruiting value is real: "we simulate quantum computers on a supercomputer" is an unusually effective club-fair sentence, and it is true.
State vector for n qubits: 2^n complex numbers
Double-complex = 16 bytes each
n = 20 → 16 MB (trivial)
n = 25 → 512 MB (laptop)
n = 30 → 16 GB (single node, comfortably)
n = 33 → 128 GB (large-memory node)
n = 35 → 512 GB (must distribute)
n = 40 → 16 TB (many nodes)
n = 45 → 512 TB (beyond most facilities)
The CAC hardware wiki gives real numbers to plan against:
Aggregating the four 2 TB nodes gets you to 8 TB, or ~39 qubits distributed. That is a concrete, defensible ceiling to state in the report — and it is well beyond what any peer club in the CUCAI network can reach, which is exactly the point of running this project at QHPC rather than anywhere else.
Every additional qubit doubles memory. This produces an exceptionally sharp wall, and locating it precisely on Frontenac is a concrete, useful result.
Distribution introduces a second problem: applying a gate to a qubit whose amplitudes are split across nodes requires communication. Gates on "low-order" qubits are local; gates on "high-order" qubits require an all-to-all exchange. Circuit structure therefore determines communication volume, and reordering qubits can change performance dramatically — which is the most interesting optimisation available here.
Not verifying correctness. A simulator misconfigured to use a different backend (stabiliser, tensor-network) may run impossibly fast because it is not doing state-vector simulation at all. Check against analytically known results.
Circuits too shallow. If your circuit has ten gates, runtime is dominated by setup and you measure nothing.
Ignoring the simulator's own optimisations. Modern simulators fuse gates and may reorder them. That is fine, but it means you are benchmarking the simulator's strategy, not raw state-vector cost. Say which.
Overclaiming relevance. Resist framing this as quantum computing research. Frame it as a memory-bound distributed-computing study. Reviewers respect the honesty and it is a stronger paper.
Compare state-vector simulation against tensor-network methods, which trade generality for the ability to simulate far more qubits when circuits have limited entanglement. The comparison — when does each win, and what circuit property determines it — is a genuinely interesting computational-methods question and considerably more sophisticated than benchmarking one approach.
You build a simulation of the Pi cluster — a model that, given a workload and a hardware configuration, predicts runtime, utilisation, and queue behaviour. You validate it against the real cluster. Then you use it to answer questions you could not otherwise afford to ask: what if we had twenty nodes? what if the network were ten times faster? what if we added GPUs?
The operating plan's §17c establishes that QHPC is compute-rich and capital-poor: Frontenac access is free, hardware money is not. A validated simulator is the direct answer to that constraint — it lets you study hardware configurations you will never be able to buy, at zero cost, and produce defensible results about them.
It is also a strong sponsorship artifact. "Our model predicts that adding four GPU nodes would improve throughput by X for workload Y" is a far more compelling ask than "we would like some GPUs."
A digital twin is only useful if it is accurate enough for the question being asked and no more detailed than that. Deciding the abstraction level is the central design decision:
| Level | Models | Predicts | Effort |
|---|---|---|---|
| Queueing | Jobs, nodes, scheduler policy. Runtime is an input, not a prediction. | Wait times, utilisation, makespan | Low — start here |
| Performance | Adds a runtime model per job type based on cores allocated | Also predicts speedup and efficient core counts | Moderate |
| Communication | Adds a network model with latency and bandwidth | Also predicts scaling limits | High |
| Full system | Memory hierarchy, I/O, power | Nearly everything | Beyond a student project — don't |
Build the queueing model first and validate it thoroughly before adding anything. A simple validated model is worth far more than an elaborate unvalidated one, and most of the questions QHPC actually wants answered — how many nodes should we add, what does the queue look like at twice the membership — are answerable at the queueing level.
Goal: validation. This week determines whether the project has any value at all.
Skipping validation. The fatal error. An unvalidated simulator produces confident numbers that mean nothing, and they are worse than no numbers because people act on them.
Over-modelling. Adding cache and memory models before the queueing layer is validated. Complexity added before accuracy is established cannot be debugged.
Extrapolating far beyond the validated regime. A model validated on 5 nodes making claims about 500 is not evidence. State the range you trust it over.
Circular validation. Fitting model parameters to the same data you then validate against. Hold out a workload the model was not tuned on.
Use the twin to pre-test scheduling policies before deploying them, closing the loop with T3.1 and T4.3. Simulating a carbon-aware policy is far cheaper than implementing it, and a validated model lets you explore the policy space broadly and then implement only the promising configurations. That combination — simulate broadly, validate narrowly, deploy selectively — is how the real field works.
In Tiers 1–4 the computing system is the object of study. In Tier 5, someone else's science is the object of study and QHPC supplies the computational capability. That inversion changes everything about how the project runs.
Tier 5 projects produce the things the operating plan's other sections depend on and cannot manufacture directly: faculty relationships, the documented institutional support that competition applications require, co-authored publications, and the credibility that turns the §16 consulting strategy from a pitch into a track record.
They are also the clearest demonstration of QHPC's actual value proposition. A lab running cluster-class simulations on a workstation does not need a lecture about HPC — it needs someone to move the work onto Frontenac and teach two of its students to do it thereafter. That is a real service, and it is one almost nobody on campus is offering.
Queen's operates the Reactor Materials Testing Laboratory (RMTL) and hosts the Materials at Extremes and Nuclear (MAX-Nuclear) Research Institute. Their work combines experimental ion irradiation — using accelerators to simulate reactor damage in compressed timescales — with multiscale computational modelling spanning density functional theory, ab initio molecular dynamics, classical atomistic simulation, and mesoscale methods.
Research subjects include defect generation under irradiation, radiation-induced damage accumulation, hydride precipitation, and thermomechanical properties of zirconium alloys — the material used in nuclear fuel cladding. Queen's also runs CHEMNUS, an initiative explicitly concerned with training researchers in advanced scientific software engineering for nuclear systems.
Three things line up unusually well. First, the science is inherently compute-bound — DFT and molecular dynamics are among the most demanding workloads in scientific computing. Second, the group already does computational work, so there is no need to persuade anyone that simulation is worthwhile. Third, and most tellingly, Queen's is separately running an initiative about scientific software engineering for nuclear systems — which means someone there has already concluded that software and computing skills are a bottleneck worth investing in.
QHPC's offer writes itself: undergraduate students who know Slurm, MPI, profiling, and reproducible builds, and who want to apply that to a real problem.
Be honest about this in the first conversation. QHPC members are not going to advance nuclear materials theory. What they can plausibly do:
| Contribution | Why the group may value it | Tier 5 difficulty |
|---|---|---|
| Scaling characterisation of an existing simulation code they use (LAMMPS, VASP, Quantum ESPRESSO) | Tells them the efficient core count — most groups guess | Most accessible entry point |
| Build and environment reproducibility — containerising their workflow | Solves a real recurring pain; directly serves CHEMNUS-style goals | Accessible |
| Workflow automation — parameter sweeps via job arrays | Many groups run these manually, one at a time | Accessible; high perceived value |
| Performance profiling and identification of bottlenecks | Rarely done; sometimes reveals large easy wins | Moderate |
| GPU porting or evaluation of GPU-enabled builds | Potentially large speedups they lack time to pursue | Hard |
| Mixed-precision evaluation (linking to T4.1) | Genuinely novel for their workload | Hardest; a second-year project |
Start with scaling characterisation or workflow automation. They are the easiest to deliver, the fastest to demonstrate value, and they build the trust that makes the harder items possible later.
This phase is the project. Budget a full term for it.
Over-promising in the first meeting. The commonest way these die. Promise a scaling study; deliver a scaling study.
Disappearing for three months. Research groups have seen student projects evaporate before. Short, regular contact matters more than a large final deliverable.
Optimising something they don't care about. A 3× speedup on a routine that occupies 2% of their runtime is worthless. Profile first, confirm with them second, optimise third.
Treating their data casually. Unpublished research data is sensitive. Agree handling rules and follow them exactly.
Ambiguous authorship. Settle it in writing before starting. Disputes here damage relationships permanently and are entirely avoidable.
Location: 36 Union Street, Miller Hall. Facilities span Miller Hall and Bruce Wing. Phone +1 (613) 533-2597.
The finding that matters, verbatim from their own facilities page — their Computational Geomechanics Laboratory runs "High Performance Desktop Workstations … configured for geomechanics modelling, 3D geometry modelling, and game engine software packages; Windows and Linux Operating Systems."
Desktop workstations. Not a cluster. And what runs on them is heavyweight: full Rocscience suite; Itasca FLAC, FLAC3D, UDEC, 3DEC, PFC2D, PFC3D, Rhino, Griddle; discrete fracture network tools FRACMAN and Mofrac; Geomechanica Irazu FDEM. Plus Unity and Houdini physics engines for rockfall trajectory work.
Data assets: 10+ years of rock slope monitoring from several large Canadian slopes; 8+ years of geomechanics lab test data across lithologies.
Applications include tunnelling, caverns, mining excavations, and deep geological repositories for nuclear waste storage — which overlaps T5.1's RMTL work and suggests a possible three-way collaboration.
A group with a decade of monitoring data and industry-standard discontinuum/FDEM software, running it on desktops, at a university with 4,500 cores and 88 GPUs available through our CAC partnership. A parameter sweep that takes a week on a workstation is a job array on Frontenac.
Licensing caveat, settle before promising anything: FLAC3D, 3DEC, PFC and Irazu are commercial and typically node-locked or seat-limited. Moving them to a cluster is a licensing question before a technical one. Ask in Phase 0 — the answer may constrain the project to open tools or to their own licence server.
The Queen's Geomechanics and Geohazards Group (QGGG) and the Robert M. Buchan Department of Mining Engineering conduct numerical modelling for deep mining, tunnelling, and waste repositories — brittle fracture mechanics, mine-scale geomechanical models, blast damage, and strainburst mechanisms. Their Computational Geomechanics Laboratory runs FLAC3D, 3DEC, PFC, Irazu (FDEM), and Rocscience tools.
Their published description states that this modelling runs on high-performance workstations — not on a cluster.
That is the entire opportunity in one phrase. A research group doing genuinely cluster-scale simulation, on single machines, because nobody in the group has the systems background to move it. They are not short of scientific expertise or of software; they are short of exactly the thing QHPC has.
This pattern generalises across most experimental and applied-engineering labs, and recognising it is the strategic core of the operating plan's §16. But QGGG is the clearest documented instance.
FLAC3D, 3DEC, PFC, Irazu, and the Rocscience suite are commercial products with per-seat or per-node licensing. This is materially different from the open-source stack the rest of this handbook assumes, and it has consequences you must resolve in Phase 0:
If cluster execution is not permitted, the project is not dead — it pivots. Options: work with an open-source equivalent (there are open FDEM and DEM codes) and demonstrate what cluster execution would buy, or focus on workflow, pre/post-processing, and parameter management, which are often the real bottleneck anyway and are unencumbered.
| Contribution | Value to them | Licence-dependent? |
|---|---|---|
| Parameter sweep automation (job arrays) | Very high — sensitivity analysis is central to geomechanics and is often done by hand | Yes — bounded by concurrent seats |
| Scaling characterisation of their solver | High — tells them whether more cores would help | Yes |
| Pre/post-processing pipeline automation | High | No — safe fallback |
| Data management and result provenance | Moderate–high; often genuinely absent | No |
| Visualisation at scale (ParaView) | Moderate | No |
| Open-source comparison study | Research-interesting in its own right | No |
The licence-independent items are the safe opening move. Automating a post-processing pipeline is unglamorous, immediately useful, entirely unblocked, and builds the trust needed to tackle the licensed work later.
Mining and geomechanics is an area where Queen's has genuine international standing, and the computational demands are growing. A successful engagement here is a strong candidate for the first paid or in-kind consulting relationship under the operating plan's §16, and a credible reference for approaching other engineering departments.
It also opens a specific technical avenue: FDEM and DEM simulations are dominated by contact detection, which is an irregular, memory-bound problem — closely related to the sparse work in Volume I's T3.2 and a legitimate research topic in its own right.
Most Tier 5 projects are blocked on finding a partner. This one has a head start on three fronts:
A researcher is trading away physical accuracy because of a compute budget. QHPC has 4,500 cores and 88 GPUs behind a scheduler we know how to use that almost nobody in this network knows how to use. That is the cleanest possible statement of the operating plan's §16 thesis, with a named person attached.
It is also a cross-institution collaboration, which makes it a stronger CUCAI submission than a solo QHPC paper and builds exactly the peer-network relationship §22's conference ambitions depend on.
| Model | What it solves | Cost | What it loses |
|---|---|---|---|
| 3D | Full Navier-Stokes over the vessel geometry | Hours to days per case | Nothing — this is the reference |
| 1D | Flow reduced to travel along vessel centrelines | Seconds to minutes | Secondary flows, recirculation, wall-shear detail |
| 0D | Lumped-parameter "circuit" analogue of the vasculature | Near-instant | All spatial detail |
Clinicians want 3D fidelity at 0D speed. Two routes are pursued: make 3D cheaper (better solvers, better preconditioners, better hardware use — QHPC's territory), or make reduced models more accurate (ML-augmented, which is Pochapski's route). They are complementary, which is what makes a collaboration natural rather than competitive.
Pick one of these, agreed with the partner. Attempting more than one is how two-term projects fail.
Underestimating the setup burden. Cardiovascular CFD requires meshing, boundary conditions, and physiological parameters. Getting a case to run correctly — not just to completion — takes longer than expected. Budget Phase 1 generously and validate against a published result.
Producing physically meaningless results. A simulation that converges is not necessarily right. You need enough domain guidance to recognise implausible flow. This is exactly what the co-supervisor is for.
Scope sprawl. Four attractive options above; pick one.
Treating clinical framing loosely. This is medically-motivated work. Do not overstate clinical relevance in the write-up — describe it as a computational study of a clinically-motivated problem, which is what it is.
Scale: Queen's Nuclear Materials group, established 2002, now 6 faculty and 25+ students and postdocs. Led by Professor Mark Daymond, Senior Industrial Research Chair since 2012. The Reactor Materials Testing Laboratory opened 2015 at 136 Grant Timmons Drive, Kingston — off main campus, worth knowing before proposing a meeting.
What they do physically: a proton and helium accelerator introduces radiation damage and transmutation products into materials, mimicking reactor conditions. Plus SEM/TEM electron microscopy, nano-indentation, mechanical and corrosion testing.
Funding and industry ties: NSERC/UNENE/Nu-Tech Industrial Research Chair, supported by Ontario Power Generation and CANDU Owners Group. Collaborates with AECL, Bruce Power, Kinectrics, Nuclear Waste Management Organization, Rolls-Royce UK — and with Canadian Nuclear Laboratories Chalk River, Los Alamos, Argonne, and Brookhaven.
Research focus: life management of the existing CANDU fleet, particularly the fuel channel, plus Generation IV materials development.
Why this matters for the approach: this is not a small academic group looking for help. It is a well-funded IRC with national-lab collaborations. Do not pitch it as "we can do your computing." Pitch it as an undergraduate team seeking a co-supervised problem, and be specific about what QHPC brings — parallelisation and scheduler literacy applied to their existing multiscale modelling.
Queen's is a founding partner in the Sudbury Neutrino Observatory Institute, and the university has a substantial particle-astrophysics community. The computational character of this field is distinctive and well suited to QHPC:
Unlike the tightly-coupled simulations in T5.1 and T5.2, this work is dominated by throughput rather than tight parallel coupling. That means job arrays, workflow automation, and I/O optimisation — all Volume I skills — deliver real value without requiring deep MPI expertise or domain physics.
It is also a field where computing infrastructure is visibly the bottleneck, so the value of the offer is easy for the partner to see.
| Contribution | Fit | Prerequisite |
|---|---|---|
| Monte Carlo campaign automation at scale | Excellent — embarrassingly parallel, huge volume | E8 job arrays |
| I/O and data-reduction pipeline optimisation | Excellent — this is usually the bottleneck | T3.3 |
| Workflow reproducibility and containerisation | Strong — these codebases are often long-lived and fragile | T2.1 |
| Storage strategy and data provenance | Strong; frequently under-managed | T3.3 |
| GPU acceleration of a filtering stage | Moderate — depends on the algorithm | T3.5 |
Underestimating data volume. The commonest blocker. Establish it in Phase 0 and involve the CAC.
Collaboration policy surprises. Large physics collaborations have rules about data, authorship, and disclosure. Find out early rather than discovering them at publication.
Optimising a stage that isn't the bottleneck. Profile first — this domain's intuitions about where time goes are often wrong.
Every other Tier 5 project requires persuading a faculty research group. This one requires talking to another student club on the same campus, with an established relationship pathway, a shared conference, and a demonstrated need.
The evidence for the need is unambiguous. Across all 72 CUCAI papers from 2025 and 2026 — roughly 272,000 words — there are zero mentions of a job scheduler or a national compute allocation. Multiple QMIND-affiliated papers describe computationally demanding work; at least one states outright that its scope was reduced due to computational constraints. Their ceiling is the free tier of a hosted notebook.
"Your teams are training models on Colab. We have sponsored access to 4,500 cores and 88 GPUs (25 A100s) on Frontenac, and members who know how to write batch scripts. Bring us the training run you couldn't finish, and we'll co-author the paper."
This is the single clearest instance of the operating plan's §16 thesis, it is on the same campus, and it produces the joint CUCAI submission that §13 wants. It is also the most legible project in this volume to a CUCAI audience, since that corpus is overwhelmingly ML-application work.
Distributed training has a distinct performance character from the simulation workloads elsewhere in this handbook:
| Strategy | How it splits | Communication | When to use |
|---|---|---|---|
| Data parallel | Same model on every GPU, different data batches | Allreduce of gradients each step — this is where T4.4 becomes directly relevant | Default; model fits on one GPU |
| Model parallel | Different layers on different GPUs | Activations passed between stages | Model too large for one GPU |
| Pipeline parallel | Model split into stages, micro-batches flow through | Stage-to-stage | Large models; reduces the idle "bubble" of naïve model parallelism |
| Fully sharded | Parameters, gradients and optimiser state all sharded | Heavy, but memory-efficient | Very large models on limited memory |
For QHPC's purposes the interesting questions are the systems ones: how does training throughput scale with GPU count, where does the allreduce become the bottleneck, and what does the effective batch size do to convergence? That last point is important and frequently mishandled — scaling to more GPUs usually means a larger effective batch, which changes the optimisation problem itself. Reporting a throughput speedup without checking that the model still converges to comparable accuracy is a meaningless result.
Reporting throughput without convergence. The defining error in this genre. More samples per second means nothing if the model no longer reaches the same accuracy.
Not rescaling the learning rate. Larger effective batch usually requires it. Without it, scaling appears to hurt accuracy for reasons that are entirely fixable.
Data loading as the hidden bottleneck. On a fast GPU the input pipeline frequently limits throughput. Profile before blaming communication.
Ambiguous credit. Two clubs, one paper. Settle authorship before starting.
The same offer applies to WAT.ai, UTMIST, Western AI, UVic AI, and Western Cyber Society — all in the CUCAI partner network, all with the same compute ceiling. A single successful QMIND collaboration is the reference that makes those conversations easy, and it converts §16 from a Queen's-only strategy into a national one.
Volume I's E9 asks one student to reproduce one figure. This scales that into a systematic study across the entire published output of Canadian undergraduate computational research — and, more ambitiously, across institutions, by recruiting the peer clubs to reproduce each other's work.
The result is a quantitative picture of reproducibility in this community that does not currently exist anywhere.
Everything else in this handbook applies established methods to new instances. This asks a question nobody has asked: how reproducible is Canadian undergraduate computational research, and what would make it more so?
QHPC is unusually well positioned to answer it. The full CUCAI corpus — 72 papers across two years, both proceedings, all PDFs — is already archived. The peer clubs convene annually in one room. And the analysis requires no programming and no domain expertise, making it accessible to members from any faculty.
This project evaluates work by named students at named institutions, several of whom QHPC wants as collaborators. Handle it carefully:
Descriptive statistics across 72 papers with measured inter-rater reliability; 8–12 documented reproduction attempts from multiple institutions; an analysis of what predicts success; and a proposed checklist. The strongest possible outcome is CUCAI adopting the checklist — at which point QHPC has changed how a national conference operates.
This project produces something rarer than a result: standing. An organisation that publishes a rigorous, generous, useful study of its own community's practices — including its own papers — becomes a natural convener of that community. That is precisely the position the operating plan's §22 conference ladder requires, and it is far cheaper to reach this way than by hosting an event.
Everything in Volumes I and II so far is characterisation research: measure X across conditions Y, explain the mechanism, report honestly. That is real research, it is what cluster competitions reward, and it is the right on-ramp.
It is not what the CUCAI corpus mostly contains. Read the titles again:
Every one of those has the shape "we propose Z; here is the baseline; here is where Z beats it." A characterisation paper has the shape "we measured X; here is what we found." Both are legitimate. Only the first competes well at CUCAI.
| Characterisation (Tiers 1–5) | Method contribution (Tier 6) | |
|---|---|---|
| Claim | "X behaves like this under Y" | "Our approach Z outperforms baseline B on metric M" |
| Needs | Careful measurement, controlled confounds | All of that, plus a baseline you did not write and cannot tilt |
| Fails when | Confounds uncontrolled | Baseline is weak, or improvement is within noise |
| Reviewer asks | "Is this measured correctly?" | "Is this better than what already exists, and why?" |
| Risk | Low — you always get a result | Real — your method may simply not win |
1. The baseline is the paper. A method that beats a weak baseline has proven nothing. Use the strongest existing approach you can actually run, tuned as well as you can tune it, and describe that tuning. This is the same discipline as T3.5's CPU-baseline rule, applied to algorithms instead of hardware.
2. A negative result is publishable; an unfair comparison is not. "We applied method Z and it did not beat the baseline, and here is our analysis of why" is a legitimate paper. "We beat a baseline we crippled" is not, and reviewers detect it.
3. Do not start here. Every Tier 6 project assumes you have already completed the corresponding characterisation project — you cannot propose a better scheduler before you have measured schedulers, or a performance model before you have measured performance. The dependency column below is not advisory.
Amdahl's Law predicts speedup from a single parameter and is wrong in practice, because it ignores communication, contention, and load imbalance. Pure black-box ML predicts runtime accurately but needs a lot of training data and generalises poorly to unseen configurations.
The grey-box approach does both: use the analytical model as the backbone, then train an ML model to predict the overhead coefficient — the gap between theory and reality. You get physical interpretability plus empirical accuracy, and it works with far less training data than a black-box regressor.
This is a published, validated approach — Grey-Box Machine Learning Prediction of Parallel Application Scaling (Alasandagutti et al., HiPC 2025) demonstrates it on two mini-applications and two full applications, reporting that it beats both standard analytical models and black-box regressors while staying robust with limited training data.
Your contribution is not the method — it is applying it to a new application class on new hardware, and reporting where it holds and where it breaks. That is an entirely legitimate undergraduate contribution and a far stronger paper than a scaling study alone.
Random train/test split. Destroys the claim. Hold out configurations.
Untuned black-box baseline. Beating an unconfigured random forest proves nothing.
Too few applications. One application means you have fitted noise. Three minimum, with different communication patterns.
Claiming generalisation you did not test. If all training data is from Frontenac, say the model is Frontenac-specific.
HPL tuning in Volume I was a manual sweep. That does not scale: real tuning spaces span millions of configurations across block size, process grid, compiler flags, thread affinity, and launch parameters. Autotuning searches that space automatically — but which search algorithm to use is unresolved.
The open question is sharp and empirically answerable. Tørring et al. (IPDPSW 2022) compared Random Search, Random Forest, Genetic Algorithms, and two Bayesian Optimization variants over a search space exceeding two million configurations, and found the best algorithm depends on the sample budget: BO variants win at 25–100 evaluations, GA overtakes them at 200+. Their conclusion is explicit — no single algorithm dominates.
Bolet et al. (IPDPSW 2024) extended this to OpenMP tuning across BT, FT, HPCG and Lulesh, comparing BO against Particle Swarm and CMA-ES — and made the pointed observation that most autotuning papers do not report their optimiser's own hyperparameters, leaving results unreproducible.
Your contribution: replicate this comparison on HPL and HPCG on Frontenac — competition-relevant codes on specific hardware, which nobody has published — and report optimiser hyperparameters properly, addressing the reproducibility gap those authors named.
The method is established, the comparison protocol is published, the open question is real, and the outcome is competition-relevant regardless of what you find — you end up with tuned HPL and HPCG configurations for Frontenac either way. Low risk of producing nothing, genuine chance of producing something novel.
No random-search baseline. Sophisticated optimisers frequently fail to beat random search on small budgets. Without it you cannot know.
Single seed per optimiser. Stochastic search needs repetition; report median and spread across seeds.
Ignoring evaluation cost. If one HPL run takes 10 minutes, a 400-evaluation budget is ~3 days per optimiser per seed. Budget the compute before designing the sweep.
Unreported optimiser hyperparameters. The specific flaw you are meant to be fixing.
Deep RL for HPC scheduling is an active field — GAS-MARL (Chen et al., FGCS 2025) applies multi-action deep RL to green-aware scheduling; DeepHCM (Yao et al., 2025) extends it to heterogeneous load-aware clusters and reports slowdown reduced from 10.461 to 8.850.
Training a scheduler from scratch is well beyond a student sprint. But there is a much more tractable formulation in the same literature. Koslovski et al. (FGCS 2023, 34 citations) use Actor-Critic RL not to invent a policy, but to select the best policy from a pool of established algorithms based on current queue state and workload. Their framing, verbatim: "Instead of proposing a new policy, the AC RL is used to select the appropriated scheduling policy from a pool of consolidated algorithms."
That reframing is the whole opportunity. You already measured FIFO, backfill and fair-share in T3.1. The contribution is a meta-scheduler that switches between them based on observed workload state — and you can start with a simple learned classifier before any RL is involved.
Skipping the oracle bound. Without it you cannot tell whether a 3% gain is near-optimal or leaving 30% unclaimed.
Evaluating on training workloads. Generate held-out workloads from different distribution parameters.
Jumping to RL. If a decision tree captures 90% of the oracle gain, the RL adds complexity and no result. Try simple first — and report that you did.
Simulation-only claims. If you never ran it on the real Pi cluster, say so plainly.
It combines three prior projects, engages directly with a current literature, has a real baseline and a principled upper bound, and produces something deployable. Combined with T3.4's walltime-prediction work — better estimates feeding backfill, which T3.1 showed is where backfill's advantage lives — it becomes a coherent multi-year research thread rather than a one-off.
T3.2 ends with a hand-built heuristic: given matrix structure, pick a storage format. Tier 6 turns that heuristic into a learned model and evaluates it properly.
The setup is unusually clean for a method paper: features are cheap to compute from the matrix (dimensions, non-zero count, density, mean and variance of row length, bandwidth, clustering), labels come from your own benchmarking, and the baselines are obvious — always-CSR (what everyone actually does), and your own hand-built heuristic.
The interesting result is rarely raw accuracy. It is the performance the selector achieves relative to the oracle — because misclassifying between two formats that perform nearly identically costs nothing, while one confident mistake on a pathological matrix costs a great deal. Reporting accuracy alone would miss that entirely, and saying so demonstrates you understand the problem.
T4.1 measures what mixed precision costs and buys. This asks the harder question: given a computation, which parts can safely be low precision? Currently that is expert judgement. The contribution is a systematic procedure.
The problem is real and current — the literature includes Ozaki-scheme emulation of FP64 matrix multiply on FP8 tensor cores, and arguments that hardware FP64 may not be the necessary foundation for HPC that it was assumed to be. Precision assignment is an open, active area.
Full automatic precision assignment over an arbitrary program is a research programme, not a sprint. Restrict to one numerical kernel with a small number of clearly separable stages — an iterative solver where you can independently choose precision for the preconditioner, the matrix-vector product, and the residual computation. Three or four binary choices is a search space of 8–16, exhaustively explorable. That is a tractable project. Anything larger is not.
This is the single most CUCAI-shaped project in either volume. It is simultaneously an LLM paper — which is what that corpus overwhelmingly contains — and a genuine HPC systems paper. A team submitting this would be legible to every reviewer in the room while doing work almost nobody else there can do, because almost nobody else has GPUs.
It is also extremely current, and the field moved fast between early and late 2026. What was three papers is now a small ecosystem with distinct schools of thought:
Accepting a kernel that does not verify. The defining error. Every candidate must be checked against a reference output before its timing counts for anything.
No hand-written baseline. Same rule as T3.5 — an LLM kernel that beats naïve code has proven very little. Compare against your best manual attempt.
Ignoring inference cost. If evolving one kernel costs more compute than the kernel will ever save, say so. That is a legitimate and useful result.
Overclaiming novelty. The frameworks exist and are published. Your contribution is the reproduction, the ablation, the cost model, or the transfer study — be precise about which.
Non-determinism. LLM sampling is stochastic. Fix seeds where the API allows, run multiple trials, report variance.
Heterogeneous-GPU LLM serving is a crowded, well-funded systems research area. DynamoLLM (Stojkovic et al., HPCA 2025, pp. 1348–1362) came out of UIUC and Microsoft Azure Research. Helix (Mei et al., ASPLOS 2025) formulates placement and routing jointly as a max-flow problem solved by MILP, evaluated on clusters of 24–42 GPU nodes. Demystifying Cost-Efficiency (Jiang et al., 2025) arrives at a MILP formulation from a price-budget direction. This is not a space where an undergraduate team out-engineers the incumbents, and you should not try.
Every one of them optimizes given a set of GPUs you already hold. Helix takes a known node set with known capacities and network edges; Jiang et al. solve under a price budget and real-time GPU availability. In every case the resource configuration is an input, and obtaining it costs nothing but money.
On a Slurm cluster, heterogeneity has an acquisition cost, and the literature prices it at zero.
Backfill schedulers fill holes. They favour jobs that are small, short, and easy to fit. A request for one A100 plus two A30s plus a V100, all resident simultaneously is close to worst-case for backfill: a gang constraint spanning scarce, differently-populated node classes. The more heterogeneous the request, the longer it waits.
So the very property these papers treat as free upside — GPU diversity matched to request diversity — is, under Slurm, something you pay queue latency to acquire. The two effects have opposite sign, and the crossover between them has never been measured, because no paper in this literature submits to a queue it does not control.
That is a cost term the field omits. Naming it and measuring it is a stronger claim than asking whether someone else's result survives porting.
Helix solves placement over a known node set. Under Slurm you learn your node set when salloc returns. That leaves two options, and neither is Helix-with-different-numbers:
Two further constraints bind. Gang co-residency: pipeline stages must coexist, and KV cache does not survive a job boundary, so you cannot span allocations. Hard walltime: cloud autoscaling degrades gracefully; a Slurm walltime kill does not, and some Frontenac nodes are contributed and carry a 3-hour cap regardless of what you request.
Frontenac's fleet — A100, A30, RTX6000, RTX8000, V100, L4 — is six-way heterogeneous, wider than most of these papers' two- or three-GPU-type testbeds. That width is what makes the acquisition-cost curve measurable here and almost nowhere else.
You are not building a serving system. You are establishing a measurement nobody has: how does the queue-time cost of acquiring a heterogeneous allocation compare against the serving efficiency that allocation buys, and where do the two curves cross?
No published baseline exists to beat, and that is an asset. State it plainly in the paper: no prior work reports heterogeneous-serving performance under a scheduler the optimizer does not control. First-measurement framing is accepted when the setting is clearly motivated, and this one is.
The entire study rests on how Frontenac's scheduler treats mixed requests, and none of that can be assumed. Run scontrol show config and sprio -w and establish: the fair-share weights, the aging policy, whether requeue resets job age, backfill window depth, and any per-partition limits on GPU-type mixing.
If aging does not reset on requeue, part of the T6.8 feedback effect disappears — and you need to know that before designing around it. Report these settings in the paper; they are what make the result reproducible or not.
Measuring from job start instead of submission. The single fatal error. It silently reproduces the cloud papers' assumption that acquisition is free, which erases the entire contribution.
Reporting mean queue time. Allocation waits are heavy-tailed. A mean makes heterogeneous requests look tractable when the 95th percentile is what actually determines whether anyone would run this.
Measuring on an idle cluster. Contention is the independent variable. Results from a quiet Frontenac describe a machine nobody shares.
Reproducing the cloud system exactly. Running Kubernetes on Frontenac repeats an old answer on new hardware. The submit-to-a-queue-you-do-not-control constraint must be real and enforced.
Single-model, single-workload results. Heterogeneous placement gains are workload-shape-dependent in every paper cited. Test at least two request-length distributions before generalising.
Not reporting the scheduler configuration. Without it the result is unreproducible anywhere else, and a reviewer cannot tell whether you measured a property of heterogeneity or a property of one site's fair-share weights.
T4.3 is a characterisation project: implement a simple deadline-based deferral heuristic, measure its emissions-versus-delay trade-off against a theoretical bound, report honestly. That is real research and a legitimate Tier 4 output. This topic is the Tier 6 escalation of the same territory — instead of characterising one fixed heuristic, you design and learn a scheduling policy, and the policy itself is the contribution.
The field has moved fast and is currently thick with exactly this kind of work, none of it aimed at an academic Slurm cluster: an information-and-energy-nexus MDP formulation (Liu et al., Applied Energy 2023, verified matched/high) frames job placement as a two-layer stochastic receding-horizon control problem with real production traces and Nordic balancing-market data; an ensemble reinforcement learning approach (Wang et al., IEEE Transactions on Smart Grid 17:1, 2026, verified matched/high, open-access postprint linked directly) finds a Pareto front between carbon and quality-of-service on real large-scale data-center traces; PCAPS (Lechowicz et al., 2025 preprint — venue unconfirmed, see T4.3) adds precedence-awareness on a 100-node Kubernetes prototype, 32.9% reduction; even Adaptive Carbon-Aware Scheduling Policies for HPC Systems (Benhari & Trystram, JSSPP 2025 / Springer LNCS 2026, verified matched/high) targets HPC specifically but — like all of the above — evaluates in a simulated or cloud-native (Kubernetes) setting, never on a production Slurm cluster with real fair-share and backfill already running.
The previous version of this paragraph cited "Chen et al., 2025" here as well — the same unverifiable "Journal of Advanced Computing Systems" paper flagged and removed from T4.3 above. Replaced with Liu et al. (2023, Applied Energy), which is real, independently confirmed, and does comparable MDP-based scheduling work.
That is the gap. Every published carbon-aware scheduling policy assumes it owns the scheduler outright. QHPC's actual constraint is the opposite: Frontenac already runs Slurm fair-share and backfill for other users, and any carbon-aware layer has to coexist with policies it does not control. Nobody in this literature has asked what happens to a learned carbon-aware policy once it has to share the machine.
The coexistence problem is worse than "there is background load," and this is the part worth building the paper around.
PCAPS defers a task into a low-carbon window and reasons about the downstream blocking that causes. On Kubernetes, where PCAPS was evaluated, that reasoning is sound because the scheduler decides when the downstream task launches. Under Slurm it does not: deferral means resubmission, and a resubmitted job re-enters the queue and competes with everyone else. The carbon saving can be swallowed by queue delay for which the model has no term.
Worse — your deferral decisions change your own future priority. Consuming less now raises your fair-share position later; depending on site configuration, requeue may or may not reset job aging. So the mechanism you are optimising against responds to the choices you make. Every carbon-scheduling paper in the list above treats scheduler behaviour as exogenous. On a fair-share cluster it is a feedback loop.
Nobody has modelled that loop, because nobody in this literature shares a scheduler. Whether the loop is stable, oscillates, or is negligible in practice is an open and genuinely interesting question — and it is answerable with a modest experiment.
Verify before designing around it: run scontrol show config and sprio -w to establish Frontenac's fair-share weights and whether requeue resets aging. If aging does not reset, much of this effect disappears — find out in week 1, not week 8.
You are not going to out-perform a purpose-built MO-PPO system trained on real six-region ISO traces with a summer's worth of undergraduate effort. Frame the contribution as one of these:
Start with the coexistence study; escalate to the feedback loop if it goes well. Coexistence is the safe, bounded version of the same insight — this literature structurally cannot answer it, because no paper in it runs on a cluster it does not fully control. The feedback loop is the ambitious version: it says the scheduler is not just noisy background, it is a system that responds to you.
Testing only on an idle cluster. The entire point is contention. A result with no background load is just T4.3 again with a fancier policy.
An unrealistic background workload. Synthetic competing jobs should resemble real Frontenac usage patterns (job-size and arrival-rate distributions), not uniform noise — check against real sacct history before designing it.
Ignoring the bandit's own learning cost. A policy that needs weeks of exploration to converge may never pay for itself on a workload that changes faster than that. Report convergence time as a cost, not an afterthought.
Comparing against a strawman heuristic. Use T4.3's actual tuned deferral policy as the baseline, not a naive FIFO. The interesting comparison is learned-versus-good-heuristic, not learned-versus-nothing.
Overclaiming from simulation. If the interference sweep is simulated rather than run on real Frontenac/Pi-cluster hardware, say so plainly and treat it as a bound, not a measurement.
| Project | Prerequisite | Risk | Start here if… |
|---|---|---|---|
| T6.2 Autotuning | HPL runbook, T2.2 | Low | Default choice. Useful output regardless of outcome |
| T6.4 Format selector | T3.2 | Low | Team already did the sparse work |
| T6.1 Grey-box prediction | T2.3 | Medium | Team has ML capability and scaling data |
| T6.3 Learned scheduling | T3.1, T3.4 | Medium-high | Multi-term commitment; strongest possible output |
| T6.5 Precision assignment | T4.1, T2.5 | High | Strong numerical background and a co-supervisor |
| T6.6 LLM kernel optimization | T3.5 | Medium | Team wants the most CUCAI-legible topic; needs real CUDA experience first |
| T6.7 Heterogeneous LLM serving | T2.1, T3.4 | Medium | Team wants an AI-audience-legible systems result grounded in Slurm literacy |
| T6.8 Learned carbon-aware scheduling | T4.3, T6.3 | Medium | Team already ran T4.3 and wants to escalate it; sustainability angle matters to sponsors |
Tiers 1–5 could be executed from this handbook alone. Tier 6 cannot. Every project here requires reading the actual papers cited, because your contribution is defined relative to what already exists — and if you do not know what exists, you cannot claim to have added to it.
Budget the first week of any Tier 6 project entirely for reading. Use an academic search tool that covers arXiv, Semantic Scholar and Scopus rather than a general web search; the citation counts and venue quality matter for judging what is actually established versus what is one preprint's claim.
Short, specific, and about them. Route it through an introduction wherever possible.
Subject: Undergraduate HPC team — offer of computational support
Dear Professor [Name],
I'm [Name], [role] with Queen's High-Performance Computing, an
undergraduate team at Queen's that works on cluster computing. Dr Ryan
Grant suggested I get in touch.
We have sponsored access to Frontenac via CAC partnership at the Centre for Advanced
Computing — several thousand cores and GPU nodes — and our members are
trained in Slurm, MPI, profiling, and reproducible builds.
I understand your group runs [specific simulation type]. We'd like to
offer help moving that kind of work onto cluster resources, or
automating parameter studies — at no cost, as a student project.
Would you have 20 minutes for us to hear what you're working on? We'd
rather understand your problem before proposing anything.
[Name] · QHPC · QHPC@engsoc.queensu.ca
Note what this does: names the introducer, states a concrete capability, references their work specifically, makes a bounded ask, and explicitly avoids pitching a solution before understanding the problem.
| Item | Why it matters |
|---|---|
| Scope | One paragraph on exactly what QHPC will deliver. Prevents drift in both directions. |
| Timeline | Including that students disappear during exams. Say so explicitly. |
| Data handling | What QHPC may access, where it may be stored, what happens at the end |
| Publication rights | What QHPC may publish about methodology versus their unpublished science |
| Authorship | Agreed criteria in advance. The single most common source of academic disputes. |
| Software licensing | Especially where commercial tools are involved — see T5.2 |
| Point of contact | One named person each side; meeting cadence |
The operating plan's §29 provides a lightweight engagement template, and §33 proposes a Law-student "Student Counsel" role to maintain it. Have it reviewed before first use — by EngSoc's governance contacts or Queen's Legal Counsel — particularly where money, corporate partners, or third-party data are involved. Nothing in this handbook is legal advice.
Volume I projects fit in a sprint. Everything here does not. Multi-term work fails in characteristic ways, and all of them are preventable.
| Failure | Symptom | Prevention |
|---|---|---|
| Graduation cliff | The one person who understood it leaves | Two people on every project minimum, one junior. This is why the team composition specifies a developing member being trained to lead next year. |
| Summer discontinuity | Four months of nothing; restart from scratch in September | A written handover at the end of term — state of work, what's next, where everything lives. Treat it as a deliverable. |
| Scope drift | Term 2 is doing something unrecognisable from term 1 | A one-page project charter, revisited at each term boundary. Changes are fine — undocumented changes are not. |
| Silent stall | Nobody has touched it in six weeks and nobody noticed | Monthly status to the Director of Research. Two consecutive empty reports triggers a decision: revive or close. |
At the end of every term, every multi-term project produces a handover document containing:
It is the same rule as Volume I, and it matters more here: record as you go. On a four-week project, reconstructing your reasoning costs a day. On a two-year project spanning three cohorts, it is simply impossible — the knowledge is gone.
A multi-term project that is well documented and unfinished can be picked up. One that is undocumented and nearly finished usually cannot.
"Carbon-aware scheduling" is a field. "Performance portability" is a field. A team handed a field will spend the term reading, produce a survey nobody asked for, and submit nothing. The runbooks in this volume deliberately describe fields, because a field can accommodate several projects and several cohorts — but no team should start work until it has converted its topic into one specific question.
Each question below is paired with the result that would refute its hypothesis. This is the single cheapest discipline in the volume and the most frequently skipped. It forces the experiment to be capable of surprising you, it makes the eventual paper honest by construction, and it converts "our approach did not work" from an embarrassment into a finding that was planned for.
These questions are starting points, not assignments. A team that narrows further after week-1 measurements has done the right thing; a team that quietly broadens back out to the subject area has not.
| Topic | The specific question | Hypothesis, and what refutes it |
|---|---|---|
| T4.1 Mixed precision |
At what matrix condition number does FP16 factorisation with FP64 iterative refinement stop converging within a fixed iteration budget, and does BF16 push that threshold higher despite having fewer mantissa bits? | BF16 tolerates roughly two orders of magnitude more ill-conditioning than FP16, because range failures are catastrophic while precision failures are gradual. Refuted if FP16 matches or beats BF16 across the condition-number sweep. |
| T4.2 Portability |
What fraction of tuned native performance does a directive-based portable model retain on one compute-bound and one memory-bound kernel, and how many engineer-hours does each implementation cost? | Retention exceeds 80% on the compute-bound kernel and falls below 60% on the memory-bound one, at under a quarter of the development effort. Refuted if retention is statistically similar across the two kernel types — which would mean kernel character does not drive portability loss. |
| T4.3 Carbon-aware scheduling |
How many hours of deadline slack are required before carbon-aware deferral of a batch workload on the Ontario grid reduces emissions by more than 10%? | Under 12 hours of slack yields under 5% savings, because Ontario's intensity is both low and comparatively flat. Refuted if meaningful savings appear at short slack — which would make the technique far more attractive here than expected. |
| T4.4 Collectives |
On this cluster's interconnect, at what message size does a ring-based allreduce overtake a tree-based one, and does the crossover move with process count? | A crossover exists near the point where bandwidth cost overtakes latency cost, and it shifts upward with process count. Refuted if one algorithm dominates at every size and scale tested. |
| T4.5 Quantum simulation |
How many qubits can be state-vector simulated on one node before memory forces a switch to tensor-network methods, and at what circuit depth does the tensor-network approach become slower than state-vector for circuits that still fit in memory? | The crossover is governed by entanglement growth rather than qubit count, so shallow wide circuits favour tensor networks and deep narrow ones favour state vectors. Refuted if the crossover tracks qubit count and is insensitive to depth. |
| T4.6 Digital twin |
Can a queue simulator calibrated on one term of accounting logs predict the next term's median job wait time within 20%? | Yes for the median, no for the 95th percentile, because the tail is dominated by rare large reservations the simulator cannot anticipate. Refuted if tail prediction is as accurate as median prediction. |
These questions must be co-owned. Bring a draft to the first partner meeting rather than an offer to help: a specific question is what turns a vague offer into a collaboration, and it is what tells the partner within thirty seconds whether you understand their problem.
| Topic | The specific question | Hypothesis, and what refutes it |
|---|---|---|
| T5.1 Nuclear materials |
In the partner's production simulations, what fraction of wall time goes to communication rather than force computation at their current node count, and what is the largest node count at which parallel efficiency stays above 70%? | They are already running past their efficiency knee because node count was chosen by queue availability rather than by measurement. Refuted if efficiency is still above 70% at their working scale. |
| T5.2 Rock mass modelling |
For the partner's characteristic problem sizes, at what mesh size does an iterative solver with a suitable preconditioner overtake the direct solver they currently use? | A crossover exists within the range of sizes they actually run, meaning the current default is wrong for their largest cases. Refuted if the direct solver wins across their whole operating range. |
| T5.3 Cardiovascular CFD |
How far must mesh resolution increase before the computed wall shear stress changes by less than 5%, and what does reaching that level cost in node-hours? | Published results in this area are frequently run below their own convergence threshold, so the converged answer differs materially from the working one. Refuted if the quantity is already converged at the partner's current resolution. |
| T5.4 Particle astrophysics |
What is the single-node throughput ceiling of the partner's data-reduction pipeline, and is it bound by I/O, memory bandwidth, or compute? | It is I/O-bound, and the group has been scaling cores against a bottleneck that cores do not address. Refuted if profiling shows a compute bound. |
| T5.5 Distributed training |
At what combination of model size and batch size does gradient communication overtake computation in data-parallel training on this cluster's interconnect? | The crossover arrives at a smaller model size than practitioners assume, because the interconnect here is not what the published scaling results assumed. Refuted if computation dominates across every configuration that fits. |
| T5.6 Reproducibility |
For one fixed containerised workload, how far do numerical outputs diverge across three institutions' clusters, and is the divergence attributable to compiler, math library, or hardware? | Divergence is dominated by the math library rather than the hardware, which means containers alone do not deliver the reproducibility they are assumed to. Refuted if outputs are bit-identical across sites, or if hardware dominates. |
| Topic | The specific question | Hypothesis, and what refutes it |
|---|---|---|
| T6.1 Grey-box scaling |
Trained only on runs up to 16 nodes, can a grey-box model predict 64-node runtime within 15% for an application it has not seen — and does adding a single 32-node observation halve the error? | Extrapolation error is dominated by the communication term, so one mid-scale observation is worth more than many small-scale ones. Refuted if the extra observation does not materially improve the fit. |
| T6.2 Autotuning under budget |
Given a fixed budget of B evaluations, does model-guided search reach within 5% of the exhaustive-search optimum more often than random sampling — and at what B does the advantage vanish? | The advantage is real at small B and disappears by moderate B, because random sampling catches up once it has covered the space. Locating that B is the contribution. Refuted if random sampling matches guided search at every budget. |
| T6.3 Learned scheduling |
Does a learned backfill policy trained on this site's trace reduce median wait time relative to a properly tuned conventional backfill, without increasing the 95th-percentile wait for large jobs? | Median improves and the large-job tail worsens — the learned policy buys its average by quietly starving the jobs that are hardest to place. Refuted if both median and tail improve together. |
| T6.4 Sparse format selector |
Can a classifier over structural matrix features pick the fastest sparse storage format more often than always choosing the common default — and does the win survive once the cost of format conversion is charged to the classifier? | It beats the default on raw kernel time and loses once conversion cost is included, for any matrix used fewer than some number of times. That break-even count is the paper. Refuted if the advantage survives conversion cost even for single-use matrices. |
| T6.5 Precision assignment |
What fraction of an application's variables can be demoted to lower precision before output error exceeds the application's own stated tolerance, and is that fraction predictable from static properties without running the search? | A large fraction can be demoted, but the demotable set is not predictable statically — meaning the search is necessary and cannot be replaced by a heuristic. Refuted if a simple static rule identifies most of the demotable set. |
| T6.6 LLM kernel optimization |
For a pre-registered, published set of kernels, what fraction of model-generated implementations are both correct and faster than a tuned library baseline — and as kernel complexity rises, does the correctness rate or the speed rate fall away first? | Correctness degrades faster than speed: the generated kernels that remain fast at high complexity are disproportionately the ones that are subtly wrong. Refuted if correctness holds while speed advantage erodes. |
| T6.7 Heterogeneous serving |
Measured from submission rather than from allocation, what additional queue delay does requesting a heterogeneous GPU set incur over a homogeneous set of equal aggregate capability — and at what request rate does that acquisition cost exceed the serving-throughput advantage heterogeneity provides? | A crossover exists and sits at a lower request rate than the serving literature implies, because that literature prices acquisition at zero. Refuted if heterogeneous requests acquire resources as quickly as homogeneous ones under every load regime tested. |
| T6.8 Learned carbon scheduling |
Does deferring jobs for carbon reasons change their owners' subsequent fair-share priority enough to alter later scheduling outcomes — and does that feedback erode the emissions saving the policy was adopted for? | Yes: deferral lowers recent usage, which raises later priority, which pulls subsequent jobs forward into hours the policy would have avoided. No cited paper models this loop. Refuted if the fair-share effect is too small to change placement. |
The Paper Kit covers the craft this volume assumes: literature search and the saturation test, the citation-verification procedure that produced the Material Passport below, benchmarking statistics, section-by-section manuscript construction, authorship, artifact packaging, and a deadline plan worked backwards from submission day.
At the first meeting of any Tier 4–6 project, the team writes its question, its control, its hypothesis, and its refutation condition on one page and gets the Director of Research to sign it. That page goes at the front of the project repository and is the first thing compared against the results at write-up. Where the final paper's question differs from the signed one — which is normal and often good — the team states what changed and why. That paragraph is frequently the most interesting methods content in the paper.
A fair complaint, checked directly against evidence rather than argued from priors: cucai.ca's own homepage highlights an EEG attention-classifier, a CNN/LSTM American Sign Language recognizer with an interactive GUI, and "RL2" — a reinforcement-learning agent that learns Rocket League maneuvers. qmind.ca names Virtual Assistant Attention Detection, Quantum GANs, Android Waste Classification, and Predictive Diabetic Risk Modelling. The pattern across all seven: a named ML technique applied to a named, recognizable task, built into a working artifact, reporting a metric — frequently with a live demo. Neither organization publishes formal proceedings, so this is a small homepage sample, likely biased toward the most demo-friendly work — but the pattern is consistent enough to take seriously.
The full grounding, with both sources quoted and a worked checklist, lives in the Zero-GPU / Single-Node Track's own "Does this look like a CUCAI/QMIND paper?" section — read it there once rather than duplicated here. What follows is the same fix applied to this volume's Tier 4–6 topics: a rewritten, task-first title for every topic, and a named demo for the three Tier 6 flagships most likely to actually go to CUCAI.
The left column is this volume's actual section title. The right column is the same research, retitled so a CUCAI reviewer knows what the system does before reading a word of the abstract. The research question underneath does not change — only the title, and, in three cases below, the addition of a demo.
| Topic | Current title | CUCAI/QMIND-shaped title |
|---|---|---|
| T4.1 | Mixed-precision numerical stability | "When does using less precision break your answer?" — a stability map for FP16/BF16 linear solvers |
| T4.2 | Performance portability | "Write once, run fast everywhere?" — testing a portable GPU-programming model against hand-tuned code |
| T4.3 | Carbon-aware scheduling | "Can a cluster wait for cleaner power?" — measuring what deadline slack actually buys in emissions |
| T4.4 | Collective communication algorithms | "The fastest way to tell every node the same thing" — an algorithm crossover map for this cluster's network |
| T4.5 | Quantum circuit simulation | "How big a quantum computer can you simulate on one node?" — the memory wall for classical quantum simulation |
| T4.6 | Digital twin of the cluster | "Can a model predict how long your job will wait?" — a queue-wait forecaster trained on real logs |
| T5.1 | Nuclear materials partner project | "Where does a nuclear simulation actually spend its time?" — a profiling study for a real research partner |
| T5.2 | Rock mass modelling partner project | "Direct or iterative — which solver actually wins?" — a crossover study for geomechanics simulation |
| T5.3 | Cardiovascular CFD partner project | "Is the simulation converged, or just finished?" — a mesh-convergence audit for blood-flow modelling |
| T5.4 | Particle astrophysics partner project | "What's actually slowing down the telescope's data pipeline?" — a bottleneck hunt for astroparticle data reduction |
| T5.5 | Distributed training partner project | "When does talking to other GPUs cost more than computing?" — a communication-crossover study for model training |
| T5.6 | Reproducibility partner project | "Does the same code give the same answer on a different machine?" — a cross-cluster reproducibility audit |
| T6.1 | Grey-box performance scaling | "Predicting a 64-node run from 16-node data" — a scaling model that knows what it doesn't know |
| T6.2 | Autotuning under a fixed budget | "Can a model out-guess a random search?" — autotuning kernels on a shoestring evaluation budget |
| T6.3 | Learned scheduling policies | "Can an AI scheduler beat the one Slurm ships with?" — a learned backfill policy, trained and tested on real traces |
| T6.4 | Sparse-format selection | "Which storage format makes this matrix fastest?" — a learned format selector for sparse linear algebra |
| T6.5 | Automated mixed-precision assignment | "How much precision can you remove before the answer breaks?" — an automated precision-tuning search |
| T6.6 | LLM-assisted kernel optimization | "Can an AI write a faster GPU kernel than a human?" — grading model-generated code for both speed and correctness |
| T6.7 | Heterogeneous GPU serving | "Is a mismatched pair of GPUs worth the wait?" — pricing the hidden cost of heterogeneous hardware requests |
| T6.8 | Learned carbon-aware scheduling | "Can an AI scheduler cut emissions without breaking its promises to other users?" — a learned carbon policy that has to share the machine |
T6.6 (LLM kernel optimization): named task — "watch an AI try to beat a human-written GPU kernel, live." Demo — generate a kernel on stage for a small, unseen problem size, run it next to the tuned library baseline, show whether it's faster and correct. Headline metric: the correctness-versus-speed split this topic's own research question is built around.
T6.7 (heterogeneous serving): named task — "how long do you wait for a mismatched pair of GPUs versus a matched one?" Demo — submit two jobs live, one requesting matched GPUs and one requesting a heterogeneous set of equal aggregate capability, and show the queue-time gap on a shared dashboard. Headline metric: the acquisition-cost crossover point itself.
T6.8 (learned carbon scheduling): named task — "can the scheduler learn to be greener without learning to be unfair?" Demo — replay a real trace against the learned policy and a heuristic baseline side by side, with a live emissions counter and a live fair-share-priority counter for both, so the coexistence tension this topic's research question is about is visible on screen, not just in a table.
Eight Tier 6 topics exist in this volume. A club running one or two method-contribution projects a year should not choose between them on which sounds most impressive. Choose on four things that are knowable in week 1: whether the required hardware is actually obtainable, whether the baseline you must beat already exists in runnable form, whether a negative result is still publishable, and how much of the work survives if the ambitious half fails.
| Topic | Hardware risk | Baseline availability | Publishable if the headline claim fails? | Verdict |
|---|---|---|---|---|
| T6.7 Heterogeneous serving on Slurm | Moderate — needs a genuinely heterogeneous allocation, which must be confirmed before committing | Strong — the literature it argues with is verified in the passport below and the comparison points are named | Yes. The acquisition-cost measurement stands on its own even if the crossover is not where predicted | Run this first The central claim is a measurement nobody has made, and the measurement is cheap relative to the claim's weight. |
| T6.8 Learned carbon-aware scheduling | Low — simulation-first by design; real allocation only needed for validation | Strong — heuristic schedulers are the baseline and they are trivially available | Yes. "The learned policy did not beat the heuristic, and here is why" is a legitimate and useful result | Strong second The fair-share feedback analysis is novel independent of whether learning wins. |
| T6.6 LLM kernel optimization | Low — the work is generation and evaluation, not large-scale training | Weak point. The baseline must be a properly tuned expert kernel, and constructing that honestly is most of the work | Partly — a negative result here reads as "the model was not prompted well" unless the methodology is airtight | Run only with a strong lead Highest reviewer scrutiny of the three; the crowded literature means weak methodology is caught immediately. |
| T6.1–T6.5 The earlier method topics | Low to moderate | Varies by topic | Generally yes — these are narrower and better bounded | Good training ground Lower ceiling, much lower risk. Correct choice for a team's first Tier 6. |
Prefer the topic whose measurement is the contribution over the topic whose result is the contribution. A project that contributes a measurement produces a paper regardless of which way the number comes out. A project that contributes a result produces a paper only if the result is favourable — which means a year of student effort is riding on an outcome nobody controls.
T6.7 and T6.8 are structured so the measurement is the contribution. That is deliberate, and it is why they are ranked above topics with more exciting headlines.
Write these defences into the methodology while the work is happening. Discovered at review, each of them costs a resubmission cycle; anticipated in week 2, each costs an afternoon.
| Objection | The defence, executed during the work |
|---|---|
| "Your expert baseline is a straw man." The single most likely rejection reason. If the human-written kernel you compare against is unoptimised, any speedup is an artifact of your baseline rather than a property of the method. | Use a tuned vendor-library kernel as the ceiling, profile it, and state how close to the roofline it sits. Record the effort spent tuning it and report that number alongside the model's. |
| "Correctness was assumed." Generated kernels can produce plausible-looking wrong output, and a speedup on incorrect results is meaningless. | Every generated kernel passes a numerical-equivalence check against the reference before it is timed. Report the generation success rate and the correctness failure rate separately — the gap between them is itself a finding. |
| "You measured one model on one day." Model behaviour shifts across versions and sampling settings. | Pin the model version and sampling parameters, report them, and repeat the generation run to quantify variance across attempts rather than reporting a single best case. |
| "Cherry-picked kernels." | Fix the kernel set in advance, publish it, and report every attempt including the failures. A pre-registered kernel list is a cheap and disproportionately persuasive move. |
| Objection | The defence, executed during the work |
|---|---|
| "This is a scheduling artifact of your specific site." Fair — queue behaviour is site-specific and a reviewer will assume the effect does not generalise. | Report the site's queue configuration and fair-share settings explicitly, and reproduce the acquisition-cost curve under at least two different queue-load regimes. Then bound the claim to the class of policy rather than to the site. |
| "You measured from job start, so you measured nothing." The failure mode the runbook already names. If timing begins at allocation, the entire acquisition cost is invisible. | Time from submission, always, and show both curves side by side. The gap between them is the paper's central figure and its clearest rebuttal to this objection. |
| "Why should heterogeneity be priced at all?" | Answer it structurally rather than rhetorically: show the cited optimisation formulations, note that each assumes resources are already held, and demonstrate that the assumption is what fails under a batch scheduler. This is the argument the topic exists to make. |
| "Small scale." | Concede it in one sentence and bound the claim. A carefully measured small-scale effect with a stated mechanism beats an unexplained large-scale one, and reviewers of systems work know it. |
| Objection | The defence, executed during the work |
|---|---|
| "A simple heuristic would have done as well." The standard and often correct objection to any learned scheduler. | Implement the obvious heuristic first, tune it properly, and report it as a first-class baseline rather than a foil. If it wins, that is the paper — and it is a more useful paper than a marginal learned victory. |
| "Your simulator is not a cluster." | Validate the simulator against real traces before using it to draw conclusions, and report the validation error. State plainly which conclusions rest on simulation and which on measurement. |
| "Carbon intensity data is coarse." | Report the data's temporal resolution and provenance, and run a sensitivity analysis showing how much the conclusion moves under plausible data error. This converts a weakness into a demonstration of rigour. |
| "You ignored the effect of deferral on future scheduling priority." The subtlest objection, and the one this topic is uniquely positioned to answer. | Do not ignore it — the fair-share feedback loop is explicitly part of this topic's contribution. Model it, measure it, and make it a named section. No cited paper does, which is why it is worth the effort. |
Every objection above reduces to one of three things: a weak baseline, an unstated scope, or a confound left uncontrolled. Those are also the three things that separate a Tier 6 project from a Tier 3 one. Treat this section as the Tier 6 checklist rather than as advice for the end of the project.
Five papers, chosen because they ground the flagship Tier 6 claims above. Every entry was independently checked with Scholar Sidekick's verifyCitation — the claimed title cross-checked against what the identifier actually resolves to, which catches a real DOI paired with an invented title (the dominant AI citation-fabrication pattern). Identifiers were recovered from the arXiv API and Crossref, never taken from model recall. Scope note: this passport covers only the claims in T6.6–T6.8 that are sourced from these five papers. The rest of Volumes I–II are practitioner runbooks, not a literature review, and most of their content does not derive from this corpus — attaching fabricated page anchors to unrelated sections would itself be a fabrication, so none were added there.
| # | Paper | Identifier | Citation verdict | Full-text status |
|---|---|---|---|---|
| P1 | Astra: A Multi-Agent System for GPU Kernel Performance Optimization | arXiv:2509.07506 | matched / high | Retrieved, parsed, page-anchored below |
| P2 | Carbon- and Precedence-Aware Scheduling for Data Processing Clusters (PCAPS) | arXiv:2502.09717 | matched / high | Retrieved, parsed, page-anchored below |
| P3 | DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency | doi:10.1109/HPCA61900.2025.00102 | matched / high (metadata only) | NOT_FOUND_FULLTEXT — closed access |
| P4 | Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow | arXiv:2406.01566 | matched / high | Retrieved, parsed, page-anchored below |
| P5 | Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs | arXiv:2502.00722 | matched / high | Retrieved, parsed, page-anchored below |
checkOpenAccess on the DOI returned isOa: false, oaStatus: "closed", bestLocation: null — Unpaywall has no legal free copy indexed. The citation itself is verified at the metadata level (title, authors, venue, page range 1348–1362, via Crossref), and it carries a clean retraction check (not retracted, no corrections, no concern). But no full text was obtained, so no quote can be anchored to a page without either fabricating one or paying for IEEE Xplore access. Every DynamoLLM claim used above should be read as abstract/metadata-level, not full-text-verified.
Extracted with pdftotext -layout against the actual PDF, not an LLM's reading of raw PDF bytes — see the caught-fabrication note below for why that distinction mattered here. Page numbers are real, counted from the document's own page breaks.
"we introduce Astra, the first multi-agent system for GPU kernel optimization. Unlike previous approaches, Astra starts from existing CUDA implementations extracted from SGLang...rather than treating PyTorch modules as the specification...Astra achieves an average speedup of 1.32× using zero-shot prompting with OpenAI o4-mini." (Abstract)
"The agents are (line 41)
"The testing agent creates a suite of test cases from the baseline kernel and checks the correctness of candidate kernels...Together, these agents form a feedback loop that supports iterative refinement while preserving correctness." (lines 54–59)
"these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints (i.e., outputs of one task are inputs for another)...Our main contribution is PCAPS, a carbon-aware scheduler that interfaces with modern ML scheduling policies to explicitly consider the precedence-driven importance of each task in addition to carbon...our Spark prototype on a 100-node Kubernetes cluster shows that a moderate configuration of PCAPS reduces carbon footprint by up to 32.9% without significantly impacting the cluster's total efficiency." (Abstract)
"PCAPS interfaces with a probabilistic (PB) scheduler" (Figure 2 caption / Section 4) — a specific instance of the "modern ML scheduling policies" named in the abstract, not a contradiction of it.
"[formulate] inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem"; "heterogeneous clusters ranging from 24 to 42 GPU nodes" (Abstract)
"as a max-flow problem of a directed, weighted graph...Helix then uses a mixed integer linear [programming algorithm]" (Section 3)
"via mixed-integer linear programming, aiming at...under the constraints of price budget and real-time [GPU availability]" (Abstract); "Different GPU capabilities, memory bandwidths, and memory capaci[ties]...so that the system efficiency can be improved." (p.2, line 41–57)
Before extracting real text, It returned: "achieves 1.32× average speedup on kernel benchmarks when using AI 3.5 Sonnet." The real paper — confirmed twice, in the abstract and again on page 6 — says OpenAI o4-mini, never AI. The model name was invented outright.
This was caught only because the true abstract had already been captured earlier via Scholar Sidekick's registry lookup, so there was something to check the fabricated quote against. It is the same lesson as the citation-verification pass two turns ago, one level deeper: a tool that sounds authoritative (reading "the PDF directly") can still confabulate when given content it cannot actually parse. Treat any extracted quote as unverified until it is cross-checked against a second, independent source — which is why every quote above was pulled from pdftotext output, a deterministic text-layer extraction, and not from an LLM's summary of binary PDF bytes.
Wherever a claim in T6.6–T6.8 draws directly on one of these five papers, an HTML comment immediately follows it in the source:
<!--anchor:quote:<identifier>:<page/section>:"<exact quote>" — <verification note>-->
These are source-level provenance markers (view page source to see them) rather than reader-facing footnotes, matching how the rest of this handbook's citations work. Where no full text existed to anchor to — P3 — the comment says so explicitly rather than pointing at a page that was never read.
# HPC Research Starter Pack — Verified APA 7.0 Reference List *17 sources · all verified this session · for student club orientation* ## A. Hardware & Top Systems (1) **1. Atchley, S., et al. (2023). *Frontier: Exploring exascale.* SC '23 (International Conference for High Performance Computing, Networking, Storage and Analysis), Denver, CO.** - Link: https://www.osti.gov/servlets/purl/2224161 - Label: **Peer-reviewed** (conference paper) · Systems - Why it matters: Primary account of the first US exascale machine (AMD MI250X GPUs, HPE Cray EX) — the reference point for today's Top500. ## B. Benchmarks (2) **2. Dongarra, J. J., Luszczek, P., & Petitet, A. (2003). The LINPACK benchmark: Past, present and future. *Concurrency and Computation: Practice and Experience, 15*(9), 803–820.** - Link: https://www.netlib.org/utk/people/JackDongarra/PAPERS/146_2003_the-linpack-benchmark-past-present-and-future.pdf - Label: **Peer-reviewed** (journal) · Benchmark · canonical but dated - Why it matters: The Top500 ranking metric — students must know what it does and doesn't measure. **3. Dongarra, J., Heroux, M. A., & Luszczek, P. (2016). High-performance conjugate-gradient benchmark: A new metric for ranking high-performance computing systems. *International Journal of High Performance Computing Applications, 30*(1), 3–10.** - Link: https://doi.org/10.1177/1094342015593158 - Label: **Peer-reviewed** (journal) · Benchmark - Why it matters: HPCG reflects real-world memory-bound workloads — the corrective to LINPACK. ## C. Programming Models — CPU / Distributed (4) **4. MPI Forum. (2023). *MPI: A message-passing interface standard, version 4.1.*** - Link: https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf - Label: **Grey literature** (standards body) · current - Why it matters: The de facto distributed-memory standard; everything in HPC interoperates with it. **5. OpenMP Architecture Review Board. (2021). *OpenMP application programming interface, version 5.2.*** - Link: https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf - Label: **Grey literature** (standards body) · current - Why it matters: Shared-memory parallelism standard — the first thing to learn on a single node. **6. Kaiser, H., Heller, T., Adelstein-Lelbach, B., Serio, A., & Fey, D. (2014). HPX: A task based programming model in a global address space. *PGAS '14: Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models,* Article 6.** - Link: https://doi.org/10.1145/2676870.2676883 - Label: **Peer-reviewed** (conference) · PGAS - Why it matters: Modern asynchronous, task-based model (C++); shows where post-MPI thinking is going. **7. Chamberlain, B. L., Callahan, D., & Zima, H. P. (2007). Parallel programmability and the Chapel language. *International Journal of High Performance Computing Applications, 21*(3), 291–312.** - Link: https://doi.org/10.1177/1094342007078442 - Label: **Peer-reviewed** (journal) · PGAS - Why it matters: Productivity-oriented parallel language; good contrast case for the programming-model discussion. ## D. GPU & Accelerator Programming (2) **8. Nickolls, J., Buck, I., Garland, M., & Skadron, K. (2008). Scalable parallel programming with CUDA. *ACM Queue, 6*(2), 40–53.** - Link: https://doi.org/10.1145/1365490.1365500 - Label: **Industry magazine** (lightly refereed) · GPU — pair with current CUDA docs - Why it matters: The canonical CUDA introduction; GPUs now dominate TOP500 compute. **9. AMD. (n.d./2024). *Introduction to the HIP programming model.* HIP Documentation (HIP 7.x).** - Link: https://rocm.docs.amd.com/projects/HIP/en/latest/understand/programming_model.html - Label: **Grey literature** (vendor documentation) - Why it matters: HIP is the portable layer behind AMD (Frontier) GPUs — the open alternative to CUDA. ## E. Resource Management & Interconnect (3) **10. Yoo, A. B., Jette, M. A., & Grondona, M. (2003). SLURM: Simple Linux utility for resource management. *Lecture Notes in Computer Science, 2862,* 44–60 (Job Scheduling Strategies for Parallel Processing).** - Link: https://doi.org/10.1007/10968987_3 - Label: **Peer-reviewed** (conference) · historical canonical — current docs at https://slurm.schedmd.com/ - Why it matters: The scheduler virtually every cluster runs; students meet it on day one. **11. MacArthur, P., Russell, R., et al. (2017). An Integrated Tutorial on InfiniBand, Verbs, and MPI. *IEEE Communications Surveys & Tutorials, 19*(4), 2894–2926.** - Link: https://doi.org/10.1109/COMST.2017.2746083 - Label: **Peer-reviewed** (journal) · interconnect - Why it matters: The fabric that ties HPC nodes together; needed to understand performance at scale. **12. Sun Microsystems. (n.d.). *Lustre: A scalable, high-performance file system* [Technical white paper].** - Link: https://cse.buffalo.edu/faculty/tkosar/cse710/papers/lustre-whitepaper.pdf - Label: **Grey literature** (vendor technical report) - Why it matters: Parallel file systems are a core HPC subsystem; Lustre is the dominant one. ## F. Applications & Domain Science (4) **13. Golaz, J.-C., et al. (2019). The DOE E3SM coupled model version 1: Overview and evaluation at standard resolution. *Journal of Advances in Modeling Earth Systems, 11*(7), 2089–2129.** - Link: https://doi.org/10.1029/2018MS001603 - Label: **Peer-reviewed** (journal) · climate - Why it matters: Flagship US exascale-era climate model — a concrete "science on HPC" case study. **14. Economon, T. D., Palacios, F., Copeland, S. R., Lukaczyk, T. W., & Alonso, J. J. (2016). SU2: An open-source suite for multiphysics simulation and design. *AIAA Journal, 54*(3), 828–846.** - Link: https://doi.org/10.2514/1.J053813 - Label: **Peer-reviewed** (journal) · CFD - Why it matters: Open-source CFD the club can actually run; shows aerospace HPC workloads. **15. McKenna, A., Hanna, M., Banks, E., Sivachenko, A., Cibulskis, K., et al. (2010). The Genome Analysis Toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data. *Genome Research, 20*(9), 1297–1303.** - Link: https://doi.org/10.1101/gr.107524.110 - Label: **Peer-reviewed** (journal) · bioinformatics - Why it matters: Genomics pipelines are a major HPC workload; GATK is the canonical example. **16. Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E. H., et al. (2018). Exascale deep learning for climate analytics. *SC '18: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,* Article 51.** - Links: https://doi.org/10.5555/3291656.3291724 · Preprint: https://arxiv.org/abs/1810.01993 - Label: **Peer-reviewed** (conference) · AI/HPC - Why it matters: Deep learning at 999 PF/s on Summit — the AI-on-HPC frontier. ## G. Energy & Sustainability (1) **17. [Preprint — verified Aug 2026]. Energy and sustainability in high performance computing: A survey / roadmap.** - Link: https://arxiv.org/html/2408.01552 - Label: **Grey literature** (arXiv preprint) — not yet peer-reviewed; treat accordingly - Why it matters: Power is the #1 constraint on exascale; gives the club the current energy discussion. --- ## Reading Order (suggested for the club) 1. **Start**: #3 HPCG + #2 LINPACK (what "fast" means) → #1 Frontier (where we are) 2. **Learn to program**: #5 OpenMP → #4 MPI → #8 CUDA → #9 HIP 3. **Run a cluster**: #10 SLURM → #12 Lustre → #11 InfiniBand 4. **See applications**: #13 E3SM, #14 SU2, #15 GATK, #16 Kurth 5. **Think ahead**: #17 energy ## Quality Matrix Summary | Category | Count | Sources | |---|---|---| | Peer-reviewed (journals) | 7 | HPCG, LINPACK, InfiniBand, E3SM, SU2, GATK, Chapel | | Peer-reviewed (conferences) | 4 | Frontier, HPX, Kurth, SLURM | | Standards bodies | 2 | MPI-4.1, OpenMP 5.2 | | Vendor/grey literature | 3 | Lustre, ROCm, energy preprint | | Industry magazine | 1 | CUDA (ACM Queue) | ## Editor / Ethics Self-Check (Phase 5) - **No fabricated sources**: every entry verified against live records, DOIs, or official repositories this session. - **No `[MATERIAL GAP]` flags**: every claim in the pack traces to a cited source. - **Transparency**: vendor docs (ROCm, Lustre) and the energy preprint labeled as grey literature, not peer-reviewed. - **Currency**: standards (MPI-4.1, OpenMP 5.2) current; canonical-but-dated items (LINPACK 2003, SLURM 2003) explicitly flagged with pointers to maintained docs. - **Balance/COI**: mix of standards bodies, journals, and flagged vendor material; no hidden interest. --- All 17 sources delivered with citations + links, quality-labeled, in chat as required. Next available: apply your 2 revision loops if you want adjustments (e.g., add MPI-4.0 vs 4.1 note, swap preprint for a peer-reviewed energy survey, more GPU sources).Volume II complete — Tiers 4–6, 20 runbooks (6 Tier 4, 6 Tier 5, 8 Tier 6) plus 5 reference sections: partner approach, multi-term project management, a specific falsifiable question for every one of the 20 topics, Tier 6 readiness triage, and a reviewer pre-mortem — alongside a verified Material Passport for the Tier 6 flagship corpus. Companion to Volume I (Tiers 1–3) and the QHPC 2026–27 Operating Plan §14.
Compiled 2026-08-04; expanded to full lesson plans 2026-08-08. Tiered curriculum continuing from Volume I; each tier escalates technical depth and research independence. Verified literature base:
verified-bibliography-quality-matrix.md(23 entries) andhpc-reference-corpus.md.Format per session: Objectives → Content → Hands-on → Checkpoint (formative) → Homework. Each tier ends with a summative portfolio artifact. Progression Rule: Tier 4 skills are prerequisites for Tier 5; Tier 5 for Tier 6. Club members may enter at any tier with demonstrated prerequisite competency. Time budget: 6 sessions × 2 hours per tier = 12 hours per tier; total Volume II = 36 hours.
Duration: 6 sessions × 2 hours = 12 hours Focus: Reproducible benchmarking, measurement validity, sample-budget effects Core sources: Husom et al. (2025) [matrix #11]; Tørring & Elster (IPDPSW 2022) [matrix #8]; SOL-ExecBench / µCUTLASS [matrix #17]; Wilhelm et al. (2026) energy-per-token [matrix #20] Key lesson: Autotuning/benchmark outcomes depend on sample budget and search strategy — design experiments accordingly. Deliverable (portfolio): Reproducible benchmark harness + methodology writeup (5–7 pages) covering noise control, sample-size justification, and a published-style results table.
Objectives: After this session, learners can (1) define benchmarking vs. casual timing; (2) enumerate sources of measurement noise (thermal throttling, OS jitter, frequency scaling, cache state, concurrent load); (3) explain why a single run is never a measurement.
Content:
Hands-on: On a shared machine, run the same workload (e.g., llama.cpp -p one short prompt) 5 times back-to-back; record wall time and variance. Then interleave with a background stress process and re-run; observe variance growth. Plot box-and-whisker.
Checkpoint (formative): Each learner writes one paragraph identifying the largest noise source in their runs and one mitigation.
Homework: Read Husom et al. (2025) [11] §2–3 (experimental setup). Bring one question about their noise-control strategy.
Objectives: (1) Justify sample sizes with confidence intervals; (2) run a paired comparison and interpret the outcome; (3) recognize when a difference is measurement noise.
Content:
Hands-on: Given 10 measurements of toolchain A and 10 of B (provided data), compute CI overlap; decide "different or not?" Then bootstrap 100 resamples in Python and report bootstrap CI. Repeat with sample sizes 3, 5, 10 and observe CI width shrinkage.
Checkpoint: Solve a worksheet: "Which n gives you a ±5% CI at 95% confidence for this observed variance?"
Homework: Draft the sample-size subsection of their methodology writeup.
Objectives: (1) Structure a harness (setup → warmup → timed trials → cleanup); (2) automate repetition and result capture to JSON/CSV; (3) separate measurement from analysis code.
Content:
environment-pinning.json.Hands-on: Build a minimal harness for the group's chosen toolchain (e.g., llama.cpp with a fixed prompt) that runs N trials, records wall time + energy via RAPL if available, writes results.json. Verify it is idempotent (re-run → same schema, plausible values).
Checkpoint: Peer review of harness code against a provided rubric (setup/warmup/trials/cleanup/version capture).
Homework: Run the harness overnight on 3 configurations with n=10 each; bring the JSON.
Objectives: (1) Read package and DRAM energy from RAPL; (2) compute energy-per-token / energy-per-inference; (3) explain why idle baseline matters.
Content:
/sys/class/powercap/intel-rapl:0/energy_uj.Hands-on: Script that samples energy_uj at 0.1 s, runs the benchmark, and reports mean power + total energy. Compare with turbostat --show PkgWatt output. Compute energy-per-token for one 128-token generation.
Checkpoint: Learners produce a 3-line summary: total energy, mean power, energy-per-token, with units stated.
Homework: Read Tørring & Elster (2022) [8] on autotuning sample budgets; note two recommendations they will adopt.
Objectives: (1) Reproduce a small autotuning experiment; (2) show that search budget changes the reported "best" configuration; (3) connect to Husom [11] and Tørring [8] findings.
Content:
Hands-on: Run a 2-parameter grid search (e.g., block size × thread count for a tiny kernel or llama.cpp thread count × batch size), each with budget 5 runs/cell and then 30 runs/cell. Compare the argmax cell: did the winner change? Produce a small table.
Checkpoint: Group discussion — "what would you tell a reviewer who asks why your optimum differs from the literature?"
Homework: Finalize methodology writeup sections 1–4 (setup, noise control, sample size, metrics).
Objectives: (1) Assemble a publishable-format methodology section; (2) give and receive structured peer review.
Content:
Hands-on: Present each harness + writeup (5 min each); structured review round; revise one section in-session.
Checkpoint (summative): Portfolio artifact submitted: harness repo + methodology writeup (5–7 pages) with explicit sample-size justification, noise-control plan, and results table.
Exit criteria for Tier 4: Harness runs unattended and produces schema-stable JSON; writeup passes peer rubric; learner can justify n and interpret CI.
Duration: 6 sessions × 2 hours = 12 hours Focus: Strategy selection across optimization approaches: black-box search, generative kernel code, RL-based scheduling Core sources: Bolet et al. (IPDPSW 2024) [matrix #10]; CuTeGen; KernelFoundry; QiMeng-Kernel MTMC; Kernel-Smith; R³; SwizzlePerf; CUDA-LLM FSR; KForge [matrix #16–#19, #22] Key lesson: Optimization strategy and implementation are separable concerns; evolutionary + LLM hybrid approaches show promise. Deliverable (portfolio): Comparative study of ≥2 optimization strategies on a shared workload (design + protocol + results, 8–10 pages).
Objectives: (1) Classify optimization approaches (search-based, generative, learning-based, manual); (2) map real tools onto the taxonomy; (3) articulate the strategy/implementation separation.
Content:
Hands-on: For 5 tools from the corpus (one per category + two hybrid), fill a comparison table: category, input, search mechanism, output, validation step, reported speedups, limitations.
Checkpoint: Each learner defends one tool's classification with a one-paragraph justification.
Homework: Read Bolet et al. [10] abstract + results table; extract BO/PSO/CMA-ES quantitative comparison.
Objectives: (1) Implement or invoke one BO and one evolutionary method on a toy objective; (2) compare their sample efficiency; (3) reproduce Bolet et al. [10]'s main qualitative result.
Content:
Hands-on: In Python (scipy/cma if available, else hand-rolled), tune a toy noisy objective (e.g., 2-D Rastrigin with noise) with each method, budget 200 evals; plot best-so-far curves. Note which finds the best final point and which finds it earliest.
Checkpoint: Learners state, with their own curve as evidence, the budget at which each method's ranking changed.
Homework: Read one of CuTeGen/KernelFoundry/KForge abstract + workflow; prepare a 3-slide summary.
Objectives: (1) Understand the generate-validate-retry loop; (2) identify validity-checking as the bottleneck; (3) compare generative vs. search approaches on effort/quality.
Content:
Hands-on: Prompt an LLM for a simple CUDA kernel (e.g., vector add with tiling); run the compile-verify loop by hand for 3 iterations; count how many of the LLM's versions compile and pass a numerical check vs. a hand-written reference.
Checkpoint: Learners quantify: "of N generated variants, M compiled, P passed checks, Q beat baseline" — and identify where time went.
Homework: Read R³ or SwizzlePerf or CUDA-LLM FSR abstract; one-paragraph "how does it learn?" summary.
Objectives: (1) Explain how RL scheduling learns a policy; (2) contrast with static heuristics; (3) evaluate when RL is worth it.
Content:
green/green_la schedulers of this project are rule-based; RL is the upgrade path the corpus motivates.Hands-on: Given a tiny simulated queue (reuse the T-CPU4 simulator abstraction), hand-simulate FCFS vs. delay-until-green vs. lookahead on 5 jobs; compute grid-energy and delay for each. This mirrors T-CPU4's core experiment at table scale.
Checkpoint: Learners state, for their 5-job trace, which scheduler won on grid energy and which on delay — and why.
Homework: Read GAS-MARL or DRL-MOSHRS abstract; identify one scheduling metric they optimize.
Objectives: (1) Design a fair comparison of ≥2 strategies; (2) fix shared workload + metric + budget; (3) pre-register the protocol (registered-report discipline from T-CPU1/T-CPU2).
Content:
Hands-on: Write a one-page pre-registration for a comparison of their chosen strategy pair (e.g., BO vs. LLM-generative) on a shared toy kernel, with explicit decision rule (e.g., "A beats B if median speedup > 10% with CI non-overlap").
Checkpoint: Peer review of pre-registrations; check for missing workload/metric/budget definitions.
Homework: Execute the pre-registered comparison (can be small: 3 configs × 10 runs).
Objectives: (1) Compile results into a publishable comparison; (2) reflect on strategy/implementation separation; (3) present findings.
Content:
Hands-on: 6-min presentations; structured Q&A; final report assembly.
Checkpoint (summative): Comparative study report (8–10 pages): design, protocol, results table with CI, search-effort comparison, discussion situating findings in the corpus.
Exit criteria for Tier 5: Learner can classify any optimization tool, design a fair comparison, and produce a CI-backed result table; understands when RL/generative approaches are worth their cost.
Duration: 6 sessions × 2 hours = 12 hours Focus: Energy-aware operation, heterogeneous scheduling, multi-objective control Core sources: Grange et al. (2018) [matrix #5]; Thi et al. (2020) Datazero2 [matrix #6]; GAS-MARL; Adaptive MARL; DeepHCM; DRL-MOSHRS; Koslovski et al. (2023) [matrix #14]; Alasandagutti et al. (HiPC 2025) [matrix #21] Key lesson: Green and performance objectives conflict; RL-based schedulers with proactive delay and elastic orchestration manage the trade-off. Deliverable (portfolio): Research proposal OR simulation study (green-aware scheduling) with pre-registered protocol — the capstone of Volume II.
Objectives: (1) Define energy vs. power vs. grid-energy vs. renewable alignment; (2) explain the green/performance conflict; (3) connect to RAPL-measured reality.
Content:
Hands-on: Plot a solar availability curve and a job arrival trace; shade feasible green windows; compute the naive-deferral grid energy by hand for 3 jobs.
Checkpoint: Learners compute grid-energy + delay for naive-deferral and state the trade-off in one sentence.
Homework: Read Grange et al. [5] abstract + model description; summarize the model's inputs/outputs.
Objectives: (1) Explain proactive delay (defer until green) with lookahead; (2) explain elastic orchestration (scale in/out, migrate, checkpoint); (3) implement a lookahead policy on the T-CPU4 simulator.
Content:
green_la lookahead result (no drops, −1.6% to −5.2% grid, delay ≈ FCFS).Hands-on: Extend/run artifact/scripts/tcpu4_simulator.py with a new profile (e.g., a windy "night" profile); run the 3 schedulers; tabulate grid-energy/delay/drops. Compare to the documented sweep results in artifact/results/tcpu4/sweep.json.
Checkpoint: Learners predict scheduler ranking for night-wind profile before running, then check their prediction.
Homework: Read Datazero2 [6] or DRL-MOSHRS abstract; note one elasticity mechanism they could add to the simulator.
Objectives: (1) Formalize multi-objective scheduling (energy × delay × fairness); (2) explain MARL (multi-agent RL) for heterogeneous clusters; (3) identify reward-shaping pitfalls.
Content:
Hands-on: On the T-CPU4 simulator, define a composite objective (grid energy + λ×delay); run a small grid over λ and plot the Pareto front for the LLM workload. Identify the knee point.
Checkpoint: Learners state which λ region dominates and what an SLA constraint would change.
Homework: Read GAS-MARL or Adaptive MARL abstract; describe its state/action/reward mapping in 5 bullet points.
Objectives: (1) Build a validation plan for a simulator against a real system; (2) apply environment pinning and seeded determinism; (3) recognize model limitations (interval-averaging, single-seed).
Content:
Hands-on: Audit the T-CPU4 simulator: run its tests (python -m unittest tests/test_tcpu4_simulator.py), re-run the flat-profile control, and check the sweep file's schema against the documented result format.
Checkpoint: Learners produce a short validation report: what was checked, what passed, one limitation found.
Homework: Draft their capstone pre-registration (research question, hypotheses, decision rules, [MEASURED] slots).
Objectives: (1) Write a complete pre-registration for a green-scheduling research proposal or simulation study; (2) define falsifiable hypotheses and decision rules; (3) prepare for peer review.
Content:
Hands-on: Write the pre-registration (template provided); peer review in pairs against the rejection-criteria checklist; revise once.
Checkpoint: Each learner's pre-registration passes the checklist (metrics concrete, decision rules explicit, result slots marked [MEASURED] or [SIMULATED]).
Homework: Execute the capstone (simulation run, or proposal write-up, or measurement if hardware present).
Objectives: (1) Present capstone results; (2) defend methodological choices; (3) assemble the full Volume II portfolio.
Content:
Hands-on: 8-min defenses with structured Q&A; portfolio checklist review.
Checkpoint (summative): Capstone delivered (research proposal OR simulation study with pre-registered protocol + results); portfolio complete; learner can explain how green and performance objectives conflict and how RL/elastic methods manage the trade-off.
Exit criteria for Tier 6: Learner produces a pre-registered, reproducible green-scheduling study; can situate it in the corpus (Grange, Datazero2, GAS-MARL, DRL-MOSHRS, Alasandagutti); understands the energy-delay Pareto trade-off.
| Tier | Formative | Summative Portfolio Artifact | Exit Criteria |
|---|---|---|---|
| 4 | Session checkpoints + peer reviews | Benchmark harness + methodology writeup (5–7 pp.) | Harness reproducible; n justified; CI interpreted |
| 5 | Strategy classifications + pre-registration reviews | Comparative study of ≥2 strategies (8–10 pp.) | Fair comparison; CI-backed table; strategy/implementation separation articulated |
| 6 | Simulator audits + pre-registration reviews | Capstone: proposal or simulation study with pre-registered protocol | Pre-registered, reproducible, corpus-situated green study |
verified-bibliography-quality-matrix.md (#5, #6, #8, #10, #11, #14, #16–#22), hpc-reference-corpus.md.artifact/scripts/tcpu4_simulator.py (Tier 5/6 simulations), artifact/scripts/tcpu4_sweep.py, artifact/scripts/rag_retriever.py (Tier 4/5 retrieval-side examples), test suites in artifact/tests/.artifact/results/tcpu4/sweep.json, artifact/results/tcpu3/eval.json./sys/class/powercap), optional turbostat.Imported from volume-ii-curriculum.md (2026-08-08). Tier 4–6 lesson plans: 6 sessions x 2h per tier, portfolio artifacts, exit criteria, and assessment summary. Complements the original Volume II tier overview above.