Every topic, written as a runbook a member can actually execute — what it is, why it matters, week by week, with the commands, the measurements, the failure modes, and what "done" looks like. Graduated across five difficulty tiers so nobody's first sprint is a GPU port and nobody's fourth is trivial.
The operating plan lists 65 topics. This document expands them into runnable form. It is written so that a Sprint lead can hand a topic to a team and walk away, and so that a member who has never used a cluster can follow along without asking what a word means.
| Tier | Who | Prerequisite | Sprint | Typical outcome |
|---|---|---|---|---|
| 1 · Foundation | Any student, any faculty, first month | None — can log into a computer | Sprint 1 | A plot, a short report, and confidence |
| 2 · Developing | Completed one sprint | Comfortable in a shell; has submitted a batch job | Sprint 2 | A real measurement with an explanation |
| 3 · Intermediate | Second-year members | Can write or modify a small program; understands parallelism conceptually | Sprints 3–4 | Conference-submittable write-up |
| 4 · Advanced | Senior members, competition team | Profiling, build systems, MPI/OpenMP fluency | Year 2+ | Publishable result; competition-transferable skill |
| 5 · Frontier | With a faculty or grad co-supervisor | Domain knowledge or advanced systems background | Multi-term | Genuine research contribution |
Tier is about prerequisites, not effort. A Tier 1 topic done well is more valuable than a Tier 4 topic done badly. Several Tier 1 topics have produced publishable results elsewhere — E9 (reproducibility) especially. Do not treat the ladder as a hierarchy of worth.
You take one identical computational task and run it in three places — a personal laptop, the QHPC Raspberry Pi cluster, and a node on Frontenac — then measure how long it takes in each, across a range of problem sizes. You produce one chart.
That sounds almost too simple to be a research project. It isn't, because the answer is not what anyone expects. At small problem sizes the laptop usually wins, sometimes by a lot. Understanding why takes you straight into the real concepts: job scheduling latency, data movement cost, startup overhead, and the difference between throughput and latency.
N×N), or estimating π by Monte Carlo (number of samples). Both are a dozen lines of code and both scale cleanly.If you omit resource flags, Slurm on Frontenac gives you 1 CPU, 1024 MB, and a 3-hour limit. A benchmark that silently ran on one core with a gigabyte of memory produces numbers that look like the cluster is slow. Always set -c, --mem, and -t explicitly, and confirm with sacct --format=JobID,AllocCPUS,ReqMem,Elapsed afterwards that you got what you asked for.
Also note some nodes are contributed and carry a hard 3-hour cap regardless of what you request.
| Quantity | How | Why |
|---|---|---|
| Wall-clock runtime | time, or in-program timers around the compute section | The headline number |
| Queue wait (Frontenac) | Slurm accounting: submit time vs start time | A user waiting 40 minutes for a 10-second job cares about this |
| Startup overhead | Time the program at the smallest possible size | Isolates fixed cost from scaling cost |
| Run-to-run variance | Spread across your 3+ repetitions | Tells you whether differences are real |
A single chart with problem size on the x-axis (log scale) and runtime on the y-axis (log scale), three lines, and a clearly marked crossover point where the cluster overtakes the laptop. Accompanied by a paragraph that correctly explains why the laptop wins on the left-hand side.
Comparing different code. If the laptop runs NumPy and the cluster runs pure Python, you've measured the library, not the machine. Same code, same versions, everywhere.
Forgetting queue time. Reporting only compute time makes the cluster look better than it is in practice. Report both.
Too narrow a size range. If everything finishes in under a second you'll see nothing but noise. Push the large end until runs take real time.
Single runs. Machines are noisy — especially shared ones. Three runs minimum, report the median.
Add a fourth platform — a Frontenac GPU node, or multiple Pi nodes rather than one. Or add an energy dimension: estimate energy per computation on each platform and ask which is most efficient, not just fastest. That extension turns E1 into a natural on-ramp to Research topic 02.
Amdahl's Law is the most important single idea in parallel computing. It says: if some fraction of your program can't be parallelised, that fraction sets a hard ceiling on how much faster you can ever go — no matter how many processors you add. If 10% of your program is inherently serial, you can never be more than 10× faster, even with a million cores.
In this project you take a program where you control the serial fraction, run it at increasing core counts, plot the measured speedup against the theoretical prediction, and explain the gap between them. The gap is where the real learning is.
If p is the fraction of the program that can be parallelised and N is the number of processors, the best possible speedup is:
Speedup(N) = 1 / ( (1 - p) + p/N )
As N → ∞, this approaches 1/(1-p).
So with p = 0.90 the ceiling is 10×.
With p = 0.99 the ceiling is 100×.
With p = 0.50 the ceiling is 2× — no matter how many cores.
S units of serial work, then a parallel loop doing P units.T(1) / T(N).A plot with core count on the x-axis and speedup on the y-axis, showing dashed theoretical curves and solid measured curves for several parallel fractions. The measured curves should sit below the theoretical ones and the gap should widen with core count. A correct explanation of that widening gap is the finding.
Too little work per core. At 16 cores with a tiny problem, overhead dominates completely and the curve is meaningless. Make the parallel phase substantial.
Not verifying correctness. A parallel program that races produces wrong answers fast. Check output equivalence before timing.
Hyperthreading confusion. If the machine reports 16 "cores" but has 8 physical ones, your curve will bend oddly at 8. Know your hardware — lscpu tells you.
Timing the whole program including startup. Time the compute region, not interpreter launch.
Investigate Gustafson's Law — the counter-argument to Amdahl, which observes that in practice people grow the problem as they add cores rather than keeping it fixed. Measure weak scaling to demonstrate it, and discuss when each law is the right lens. This is a genuinely sophisticated addition and sets up Research topic 10 directly.
Scientific computing generates enormous files, and moving or storing them is often more expensive than computing them. Compression trades CPU time for space. This project measures that trade-off systematically across several compression tools and several kinds of real data, and produces a practical recommendation.
A ratio-vs-speed scatter plot with the Pareto frontier marked, per data type, plus a one-page practical recommendation. Bonus if you can explain why a given tool wins on a given data type — that requires understanding what the data looks like structurally.
Only testing text. Text compresses beautifully and tells you nothing about scientific binary data. Diversity of corpus is the whole experiment.
Ignoring decompression. Data is written once and read many times. Decompression speed usually matters more.
Caching effects. The second read of a file comes from memory, not disk. Drop caches between runs or use files larger than RAM.
Not verifying round-trips. Always confirm decompressed output matches the original byte for byte.
Test parallel compression variants (pigz, pbzip2, multi-threaded zstd) and measure how the trade-off shifts with core count. Or investigate lossy scientific compressors such as ZFP or SZ, which allow bounded error in exchange for dramatically better ratios — a genuinely current research area.
You implement one identical numerical computation three (or more) ways — pure Python, Python using NumPy, and compiled C — then measure how long each takes across a range of problem sizes. The differences are not subtle. Pure Python is typically 100× to 1000× slower than C for numerical loops, and NumPy usually lands somewhere close to C because it is calling C underneath.
The project is really about answering a deeper question: where does the time actually go? Why is an interpreted language slow? What is NumPy doing differently? When does it stop mattering?
Pick one computation and stick to it. Good choices, in increasing order of interest:
| Kernel | Why it's a good choice | Watch out for |
|---|---|---|
| Vector sum / dot product | Trivial to write correctly in all three; memory-bandwidth-bound | May be too simple — NumPy will crush pure Python immediately |
| Matrix multiply (naïve triple loop) | Compute-bound; the classic; connects directly to HPL later | Don't accidentally compare your naïve C against NumPy's tuned BLAS and call it a language comparison — see failure modes |
| Mandelbrot set | Compute-heavy, no library shortcut, and produces a picture | Iteration counts vary per pixel — that's a feature, it introduces load imbalance |
| 1D heat equation / stencil | Realistic scientific pattern; sets up later stencil work | Requires care with boundary conditions |
Recommendation for a first sprint: Mandelbrot. It is compute-bound, has no library that trivialises it, and the output is an image you can show at a club fair.
for i in range(n) around NumPy operations, you haven't vectorised — you've just made each step slower.-O0, -O1, -O2, -O3. The difference between -O0 and -O3 is itself a finding — often 3–10×.A log-log runtime chart with clearly separated lines, a table of speedup ratios, and — critically — a correct explanation of the mechanism. A report that says "C is faster" is incomplete. A report that says "C is faster because the interpreter performs a type check and a dictionary lookup on every single iteration, whereas the compiler resolves those at compile time and additionally vectorises the inner loop" is a real finding.
Comparing naïve C to tuned BLAS. If your kernel is matrix multiply and you compare hand-written C against numpy.dot, NumPy will win — because it dispatches to a decades-optimised BLAS library. That is a real and interesting result, but it is a comparison of libraries, not languages. Say so explicitly, or pick a kernel with no library equivalent.
Not vectorising the NumPy version. The most common mistake. Looping in Python over NumPy scalars is slower than pure Python.
Forgetting compiler flags. Comparing unoptimised C (-O0) to NumPy is unfair to C and misleading.
Timing including startup. Python takes ~30ms to start. At small problem sizes that dominates everything.
Add more languages — Julia (designed for exactly this trade-off), Fortran (still dominant in HPC and often fastest), Rust, or Python accelerated with Numba or Cython. The Numba result is particularly striking: a single decorator often recovers most of C's performance. Alternatively, profile the pure-Python version with cProfile to show exactly where the time goes, which turns a benchmark into a genuine investigation.
A costing study. You determine what QHPC's compute — the Pi cluster, and a representative Frontenac allocation — would cost if purchased from AWS, Azure, or Google Cloud instead. Then you build a break-even analysis: at what utilisation does owning hardware beat renting it?
This is a business and economics project that happens to be about computers. It requires no programming whatsoever.
A spreadsheet model anyone can re-run with new numbers, a break-even chart (utilisation on the x-axis, cost on the y-axis, two lines crossing), and a headline figure: "QHPC's Frontenac access is equivalent to $X per year of cloud compute." That single sentence is worth a great deal in a sponsorship or funding conversation.
Comparing on-demand cloud pricing to hardware purchase price. On-demand is the most expensive way to buy cloud. A fair comparison uses reserved or spot pricing for sustained workloads.
Ignoring egress. Getting data out of a cloud provider is often the largest line item and is routinely forgotten.
Ignoring idle time. Owned hardware costs the same whether used or not; cloud only bills when running. This is the entire crux of the break-even question.
Stale prices. Cloud pricing changes. Date-stamp every figure and cite the page you took it from.
Add a carbon dimension. Cloud regions publish differing carbon intensities, and Ontario's grid is unusually low-carbon (largely nuclear and hydro). Compare emissions per compute-hour for local versus cloud, and you have turned a cost study into a sustainability study — feeding Research topics 34 and 37 directly.
You document the existing five-node Raspberry Pi cluster so completely that a stranger with no prior knowledge could rebuild it from scratch. Parts list with prices, physical assembly, network configuration, the head node setup, PXE/TFTP network boot, the NFS shared filesystem, SSH keys, Slurm installation and configuration — every step, every command, every mistake to avoid.
Then you test it: hand the guide to someone who wasn't involved and watch them try to follow it. Every place they get stuck is a defect in the guide.
Documentation written by the person who built the system is almost always unusable, because they unconsciously skip the steps that are obvious to them. Putting a newcomer in charge of writing, with an expert available to answer questions, produces a dramatically better guide. The questions the newcomer has to ask are the missing steps.
Someone who has never seen the cluster can follow the guide from an empty table to a working Slurm installation without asking anyone a question. Every command is copy-pasteable, every config file is shown in full, and every stage ends with a check that confirms success.
Assumed knowledge. "Configure DHCP as usual" is not a step. If you cannot copy and paste it, it isn't documented.
No verification steps. A guide that tells you what to type but not how to confirm it worked leaves the reader stranded at the first silent failure.
Skipping the failures. The troubleshooting section is often the most valuable part. Include the things that went wrong, not just the happy path.
Never testing it. An untested guide is a draft. The test is the project.
Turn the manual steps into automation — an Ansible playbook or shell scripts that perform the configuration, so a rebuild takes an hour instead of a weekend. That is the natural bridge from this Tier 1 project into the provisioning work in Tier 3. Alternatively, produce a video walkthrough, which is often more useful for physical assembly than text.
You measure how long file operations take across different storage locations — a laptop SSD, the Pi cluster's local storage, the cluster's NFS shared filesystem, and Frontenac's parallel scratch filesystem. You vary the access pattern: one enormous file versus thousands of tiny ones, sequential versus random, reads versus writes.
The result reliably surprises people. Storage is usually the real bottleneck in scientific computing, and the performance difference between access patterns on the same hardware can be a factor of a hundred.
| Dimension | Levels |
|---|---|
| Location | Laptop SSD · Pi local SD/SSD · Pi NFS mount · Frontenac /global/scratch (GPFS) · Frontenac /global/home · Frontenac node-local $TMPDISK |
| Operation | Sequential write · sequential read · random read |
| File pattern | 1 × 10 GB · 100 × 100 MB · 100,000 × 100 KB |
| Concurrency | 1 process · 4 processes · 16 processes (optional extension) |
dd for simple sequential tests, and understand what each flag does — especially bs (block size) and conv=fsync, without which you are timing the write cache rather than the disk.sync; echo 3 > /proc/sys/vm/drop_caches where you have permission; otherwise use files far larger than RAM).stat, deletion — because on shared filesystems these are frequently the bottleneck rather than bandwidth.Charts showing throughput by location and access pattern, with the many-small-files penalty clearly visible, plus a short practical guide that a researcher in another department could act on immediately.
Caching. The number one error. If your test file fits in RAM, the second read is from memory and you are measuring nothing. Use files larger than RAM, or drop caches between runs.
Not syncing writes. Without fsync, writes may still be in a buffer when you stop the clock. You will report impossible throughput.
Testing a shared filesystem during peak hours. Frontenac scratch is shared with 1,000+ users. Note the time of day, run at multiple times, and report the variance — that variance is itself a legitimate finding.
Filling a shared filesystem. Be careful with 10 GB test files on shared storage. Clean up after every run and check your quota first.
Move from dd to fio, the industry-standard flexible I/O tester, which gives you queue depth, mixed read/write ratios, and proper random-access patterns. Or add concurrency: how does throughput change when four processes write simultaneously? That is the direct on-ramp to Research topic 43.
Take any program with a tunable parameter. Instead of running it a hundred times by hand, you use a Slurm job array to submit all hundred variants as a single command, let the scheduler run them in parallel across the cluster, then collect and visualise the results as a surface or heatmap.
The technique is almost trivially simple — a two-line change to a batch script. Its impact on a researcher's working life is enormous.
#!/bin/bash #SBATCH --array=0-99 #SBATCH --time=00:10:00 #SBATCH --output=results/run_%A_%a.out # SLURM_ARRAY_TASK_ID is 0..99, different in every task. # Use it to select this task's parameter value. PARAM=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" parameters.txt) ./my_program --parameter "$PARAM" > "results/out_${SLURM_ARRAY_TASK_ID}.csv"
That is the entire idea. One sbatch submission creates a hundred independent tasks, each knowing its own index, which the scheduler runs as resources free up.
--time limit.--array=0-4 — five tasks — and confirm it works before scaling up.squeue, sacct, and scancel. You will need all three.--array=17,43,88).A parameter surface or curve derived from ≥100 runs, a clear statement of wall-clock time saved versus running serially, and a reusable template script that another member can adapt in ten minutes.
Every task writing to the same output file. They will overwrite each other and you will lose everything. Always include $SLURM_ARRAY_TASK_ID in output filenames.
Requesting too much time or memory per task. The scheduler will start fewer tasks concurrently, and your sweep takes far longer than it should.
Submitting 10,000 tasks on the first attempt. Test with five. Cluster administrators notice array bombs, and Frontenac has array size limits.
Not checking exit codes. A task that crashed still produces an output file — an empty one. Verify completeness before analysing.
Compare against alternative approaches: GNU Parallel on a single node, or a workflow manager such as Snakemake or Nextflow. When is each the right tool? That comparison is a genuinely useful contribution, because most researchers only ever learn one of them. Or extend into adaptive sweeping — using early results to decide which parameters to explore next, which is the beginning of real optimisation.
You pick one figure from a published undergraduate paper — the CUCAI proceedings archive gives you 72 to choose from — and try to regenerate it using only what the paper tells you. You document precisely where you succeed, where you get stuck, and what information was missing.
This is reproducibility research, and it is a real and currently very active field. You are not required to succeed. A careful, honest account of why a result could not be reproduced is a more valuable contribution than a successful re-run.
It requires no original idea, no novel method, and no advanced skills — yet it produces genuinely publishable work, teaches critical reading better than any seminar, and instills the documentation habits that make the student's own future work reproducible. It is also psychologically safe: there is no way to "fail," because failure is the result.
Good candidates share these properties:
From the archived corpus, strong candidates include the benchmarking-oriented papers, anything using the NASA turbofan dataset, and any scaling study. Avoid papers depending on proprietary data or bespoke hardware.
| Category | What to note |
|---|---|
| Data availability | Was it linked? Did the link work? How long to obtain? |
| Code availability | Repository present? Does it run? Documented? |
| Environment | Were software versions specified? Hardware described? |
| Parameters | Every value you had to guess |
| Ambiguities | Every point where the text supported more than one interpretation |
| Result agreement | Quantitative comparison of your figure to theirs |
| Effort | Hours spent. A genuinely useful and rarely reported number. |
A side-by-side of the original figure and your reproduction, a structured account of every gap encountered, an honest verdict (fully / partially / not reproduced), and a reproducibility checklist other students can use on their own work. A "not reproduced" verdict, well documented, is a complete success for this project.
Choosing an over-ambitious target. A paper needing 10,000 GPU-hours cannot be reproduced in a sprint. Check feasibility in week 1.
Not logging as you go. The log is the paper. Reconstructing it afterwards loses exactly the detail that matters.
Silently filling gaps. When you guess a parameter, that guess is a finding. Record it rather than quietly proceeding.
An accusatory tone. This damages relationships QHPC needs and misrepresents the work. Reproducibility gaps are a systemic norm, not individual failings.
Reproduce a second figure from a different paper and compare reproducibility across them. With three or four, you have the beginnings of a systematic study of reproducibility in Canadian undergraduate computational research — which is genuinely novel and would be a strong CUCAI submission in its own right.
A survey and interview study of Queen's students and researchers: what compute do they have, what do they need, what do they know exists, and where do they get stuck? You produce quantified evidence about a gap that QHPC currently only asserts.
Research involving human participants may require review by the Queen's General Research Ethics Board (GREB). Rules differ for coursework, club activity, and publishable research, and an exemption for one does not extend to another. Ask the faculty advisor before collecting any data, and be explicit that you intend to publish. Getting this wrong retroactively can render the data unusable. Build in time for it.
A defensible sample (aim for 100+ survey responses across at least four departments), clear figures, honest treatment of limitations and sampling bias, and at least one genuinely surprising finding. The killer result, if it appears, is a quantified estimate of how many Queen's students have silently scaled down their work for want of compute.
Skipping ethics clearance. Can invalidate the entire dataset for publication.
Leading questions. "Do you agree that Queen's should provide more computing resources?" measures nothing. Ask about behaviour, not opinions.
Sampling only from computing. Then you've surveyed people who already know. The interesting population is everyone else.
Over-claiming. A 60-response convenience sample does not represent the university. State limits plainly — reviewers respect it.
Extend the same instrument to the peer clubs in the national network — QMIND, UTMIST, WAT.ai, Western AI, UVic AI. You then have a multi-institution study, which is exactly Research topic 35, and a strong CUCAI submission. The interview subjects are all in one room in Toronto in March.
OpenMP lets you parallelise a loop by adding a single line of code above it. In this project you take one such loop, run it with increasing thread counts, and find the point where adding threads stops helping — then work out why it stopped.
There are only a handful of possible reasons, and identifying which one applies is the entire skill: insufficient work per thread, memory bandwidth saturation, false sharing, load imbalance, or hyperthreading.
/* Serial */ for (int i = 0; i < N; i++) result[i] = compute(data[i]); /* Parallel — one line */ #pragma omp parallel for for (int i = 0; i < N; i++) result[i] = compute(data[i]); Compile: gcc -O2 -fopenmp prog.c -o prog Control: export OMP_NUM_THREADS=8
lscpu. How many physical cores, versus logical threads? This single fact explains most confusing results later.OMP_NUM_THREADS from 1 up to twice the physical core count. Three runs each.schedule(static) versus schedule(dynamic). For loops with uneven per-iteration cost — Mandelbrot is exactly this — dynamic scheduling can help substantially.OMP_PROC_BIND=true) and observe the effect on variance.| Cause | Signature | Test |
|---|---|---|
| Not enough work per thread | Knee at low thread count; improves with larger problem | Increase problem size and re-run |
| Memory bandwidth saturation | Plateau regardless of problem size; low arithmetic intensity | Compare against STREAM bandwidth |
| Load imbalance | Helped noticeably by schedule(dynamic) | Switch scheduling policy |
| Hyperthreading | Knee exactly at physical core count | Compare lscpu physical vs logical |
| False sharing | Poor scaling with adjacent per-thread writes | Pad shared arrays to cache-line size |
Race conditions producing wrong answers. Always verify correctness before timing. A fast wrong answer is worthless.
Forgetting -fopenmp. Without it the pragma is silently ignored and you measure serial code at every thread count. If your speedup is exactly 1.0 everywhere, this is why.
Ignoring hyperthreading. Reporting "scaling stops at 8 threads" on an 8-physical-core machine as a mystery, when it is entirely expected.
Sharing the machine. Another user's job on the same node destroys your measurements. Use a dedicated allocation.
Deliberately construct a false-sharing example — have threads write to adjacent array elements — then fix it with padding and measure the difference. It is often dramatic (5–10×) and it is a genuinely memorable demonstration. Alternatively, move to a NUMA machine and investigate first-touch memory placement.
MPI — the Message Passing Interface — is how programs run across multiple machines. Unlike OpenMP, which shares memory within one node, MPI processes have entirely separate memory and must explicitly send each other messages. It is the foundation of essentially all large-scale scientific computing.
You write three programs in sequence, each building on the last, and time each. By the end you understand point-to-point messaging, collective operations, and — most importantly — why communication cost eventually dominates everything.
| # | Program | Concept | What you measure |
|---|---|---|---|
| 1 | Hello world — every process prints its rank and hostname | Ranks, communicators, launching | Startup time vs process count |
| 2 | Ring — pass a message around all processes and back | Point-to-point send/receive; deadlock | Latency per hop; message-size scaling |
| 3 | Reduce — every process computes a partial sum; combine to one total | Collective operations | Manual vs MPI_Reduce performance |
/* Program 1 — the whole thing */ #include <mpi.h> #include <stdio.h> int main(int argc, char **argv) { MPI_Init(&argc, &argv); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); char host[256]; gethostname(host, 256); printf("Rank %d of %d on %s\n", rank, size, host); MPI_Finalize(); return 0; } Compile: mpicc hello.c -o hello Run: mpirun -np 4 ./hello
MPI_Sendrecv) is a core MPI lesson.MPI_Reduce.MPI_Reduce against MPI_Allreduce.Three working programs, a latency-versus-message-size curve with the latency and bandwidth regimes both visible, and a demonstration that the library's collective beats the naïve implementation at scale — with a correct explanation of why (logarithmic tree versus linear gather).
All processes on one node. Then you are measuring shared-memory communication, not the network, and your latency numbers will be unrepresentatively good. Always print hostnames to confirm distribution.
Deadlock treated as failure. It's a learning opportunity — document it.
Timing without a barrier. Processes start at slightly different moments. Use MPI_Barrier before starting the clock.
Message sizes too small to see bandwidth effects. Go up to megabytes, not kilobytes.
Compare your hand-rolled measurements against the OSU Micro-Benchmarks, the standard tool for exactly this. How close did you get? Where do they differ, and why? That comparison is a direct on-ramp to Research topic 47 (collective-communication tuning) and to the standing benchmark suite.
The commands members actually use, in the order they'll need them.
# Submit sbatch script.sh # submit a batch job sbatch --array=0-99 script.sh # submit a job array srun --pty bash # interactive session # Monitor squeue -u $USER # my queued/running jobs squeue -j JOBID # one specific job scontrol show job JOBID # full detail incl. why it's pending sacct -j JOBID --format=JobID,Elapsed,MaxRSS,State # after completion: time and memory used # Control scancel JOBID # cancel one scancel -u $USER # cancel everything of mine # Inspect the system sinfo # partitions and node states sinfo -N -l # per-node detail
#!/bin/bash
#SBATCH --job-name=my_experiment
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --output=logs/%x_%j.out
#SBATCH --error=logs/%x_%j.err
module load gcc openmpi # adjust to the site's module names
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./my_program --size 1000
Always request less time than you think you need, then adjust. Shorter jobs are scheduled sooner via backfill. A job asking for 24 hours when it needs 20 minutes may wait days.
Always check sacct afterwards for MaxRSS. Requesting 64 GB when the job used 2 GB wastes the allocation and delays your own next job.
Almost every flawed result in this handbook's failure-mode sections traces back to bad timing methodology. The rules:
clock_gettime(CLOCK_MONOTONIC, …). In Python, time.perf_counter(). Never time.time(), which can jump backwards.-O3 may delete a computation whose result you never use. Print or accumulate the result so it cannot be optimised away./* C */ struct timespec t0, t1; clock_gettime(CLOCK_MONOTONIC, &t0); /* ... work ... */ clock_gettime(CLOCK_MONOTONIC, &t1); double sec = (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9; # Python import time t0 = time.perf_counter() # ... work ... elapsed = time.perf_counter() - t0
Agree these once so every QHPC paper looks like it came from the same organisation.
Every sprint report uses this structure. It maps onto the IEEE conference format CUCAI expects, so converting a sprint report into a submission is mechanical rather than a rewrite.
| Section | Length | Contains |
|---|---|---|
| Abstract | 150–250 words | The question, the method in one sentence, the headline number, the implication. Write this last. |
| I. Introduction | 400–700 | Motivation, why anyone should care, and an explicit statement of what this report contributes |
| II. Background / Related work | 300–600 | What is already known. Cite real sources — the CUCAI archive is a legitimate corpus to cite. |
| III. Method | 600–1,200 | Hardware, software with versions, parameters, and procedure. Written so a stranger could repeat it. This is where E9 teaches you what to include. |
| IV. Results | 600–1,200 | Figures and tables with prose describing what they show. Description only — no interpretation yet. |
| V. Discussion | 400–900 | What the results mean. Why they came out this way. What surprised you. |
| VI. Limitations | 150–350 | What you did not control, what might be wrong. Include this always — it consistently makes a report read as more rigorous, not less. |
| VII. Conclusion & future work | 200–400 | Restate the finding; name the next question |
| References | — | IEEE numbered style, managed in Zotero |
| Appendix: reproduction | — | Repo link, exact commands, environment details |
Method → Results → Discussion → Background → Introduction → Conclusion → Abstract. Never start with the introduction; you cannot introduce a finding you haven't yet articulated. Figures before prose, always.
Tier 1 answered questions with a single number or curve. Tier 2 introduces controlled comparison — you are no longer measuring one thing, you are isolating the effect of one variable while holding others fixed. That is a genuine methodological step up, and it is where students start producing work that can survive review.
Three new expectations apply from here on:
Containers package an application together with its entire software environment — libraries, compilers, dependencies — so it runs identically anywhere. In HPC this solves a genuinely painful problem: scientific software is notoriously difficult to build, and "it worked on my machine" is the field's most expensive sentence.
But containers add a layer between your program and the hardware. This project quantifies that layer. You run identical workloads three ways — directly on the host ("bare metal"), inside Docker, and inside Apptainer (the HPC-native container runtime, formerly Singularity) — and measure the difference across several classes of workload.
Containers are not virtual machines. A VM emulates an entire computer including its own kernel, which is expensive. A container shares the host kernel and merely isolates the filesystem, process tree, and network view using kernel features (namespaces and cgroups). This is why container overhead is usually small — often under 5% — and why the interesting question is not "is it slower" but "slower at what."
The two runtimes differ in ways that matter for HPC:
| Docker | Apptainer | |
|---|---|---|
| Designed for | Microservices, web deployment | HPC and scientific computing |
| Runs as | Typically requires a privileged daemon | Runs as the calling user, no daemon |
| Why that matters | Most HPC centres will not permit it on compute nodes | Safe on multi-tenant clusters — which is why it exists |
| Image format | Layered (overlay filesystem) | Single SIF file |
| MPI support | Awkward | Designed for it (host MPI can be bound in) |
| Filesystem default | Isolated; mounts must be requested | Home and common paths bound by default |
That last row predicts one of your likely findings in advance: Docker's overlay filesystem is where most measurable overhead lives, so I/O-heavy and metadata-heavy workloads should show the largest gap, while pure compute should show almost none.
You need workloads that stress different subsystems, because container overhead is entirely subsystem-dependent. Use at least four:
| Workload class | Suggested benchmark | Expected overhead | Why |
|---|---|---|---|
| Pure compute | HPL, or your Mandelbrot from E4 | Near zero (<1%) | Instructions execute natively; nothing is intercepted |
| Memory bandwidth | STREAM | Near zero | Same reason |
| Sequential I/O | dd or fio, large files | Small but measurable | Goes through the container filesystem layer |
| Metadata-heavy I/O | Create/stat/delete 100k small files | Largest | Overlay filesystems handle metadata poorly |
| Startup | Time a trivial "hello world" container | Substantial in relative terms | Image mount and namespace setup are fixed costs |
| MPI (extension) | OSU Micro-Benchmarks | Depends entirely on configuration | Host MPI binding vs container MPI is the whole question |
This is the part that decides whether your study is valid.
Identical software versions inside and outside. If the container has GCC 13 and the host has GCC 11, you are measuring compilers. Build the container from the host's versions, or install matching versions on the host.
Identical compiler flags. Same -O level, same architecture flags. A container built with -march=native on a different machine may produce different instructions entirely.
Same physical node. Run all three configurations on the same machine, ideally back to back and interleaved, so thermal state and system load are comparable.
Same storage location. If bare metal writes to local disk and the container writes to a bind-mounted network path, you are measuring the network.
Goal: a working, controlled baseline.
Dockerfile and an Apptainer definition file that install the same software versions. Pin every version explicitly — no apt install gcc without a version.Goal: complete the compute and memory measurements.
Goal: find where the cost actually lives.
Goal: analysis, mechanism, and practical guidance.
A chart showing relative overhead by workload class, with compute near zero and metadata-heavy I/O visibly worse, error bars throughout, and a mechanism explanation that survives scrutiny. Plus a one-page practical guide. If you additionally demonstrate that the Docker overlay penalty disappears with a bind mount, you have a genuinely useful and non-obvious result.
Different software versions inside and out. The single most common invalidating error. Pin everything.
Reporting overhead smaller than your noise floor. If bare-metal runs vary by ±3% and you report 2% container overhead, you have measured nothing. State your noise floor explicitly and only claim differences exceeding it.
Running the three configurations at different times of day. On a shared machine, load changes. Interleave.
Testing only compute. You will conclude "containers are free," which is true for compute and false in general. The I/O workloads are the point.
Cold vs warm image cache. The first container launch pulls and mounts the image. Subsequent ones may not. Decide which you're measuring and be consistent.
Add MPI across containers — the genuinely hard case. Multi-node MPI inside containers requires the container's MPI to be compatible with the host's interconnect libraries, and getting this wrong silently falls back to TCP over Ethernet instead of using the fast fabric, which can cost an order of magnitude. Demonstrating that failure mode and then fixing it is a strong result and directly transferable to competition work.
Alternatively, compare against Spack or EasyBuild environments, which solve the same reproducibility problem without containers. When is each appropriate?
You write one program, then compile it many different ways — different compilers (GCC, Clang, and vendor compilers where available), different optimisation levels, different architecture targeting flags — and measure how much the identical source code varies in speed.
The answer routinely exceeds what people expect. Differences of 2–5× between -O0 and -O3 are ordinary. Differences of 20–40% between compilers on the same code at the same optimisation level are common. On a competition where every percent counts, this is free performance that costs only knowledge.
Optimisation levels are bundles of individual transformations:
| Flag | What it does | Risk |
|---|---|---|
-O0 | Nothing. Straightforward compilation, easy to debug. | None — but slow |
-O1 | Basic: dead code elimination, simple register allocation | None in practice |
-O2 | The standard production level. Inlining, loop optimisations, vectorisation in modern GCC. | Essentially none |
-O3 | Aggressive: more inlining, more loop transformation, more vectorisation | Can occasionally be slower — larger code, worse instruction cache behaviour |
-march=native | Emit instructions specific to this exact CPU (AVX-512, etc.) | Binary may not run on other machines. Critical caveat on heterogeneous clusters. |
-ffast-math | Permit floating-point reassociation and other unsafe algebra | Changes results. Can break numerical correctness. Must be validated, never assumed. |
-flto | Link-time optimisation: optimise across compilation units | Long build times; occasional bugs |
-ffast-math discipline
This flag is the single most tempting and most dangerous item in the list. It permits the compiler to treat floating-point arithmetic as if it were associative, which it is not. It can deliver real speedups — and it can silently change your numerical results.
If you test it, you must validate correctness against the un-optimised build and report the numerical difference. A performance result obtained with -ffast-math and no correctness check is not a result. This is exactly the kind of methodological care that distinguishes a Tier 2 report from a Tier 1 one.
Goal: build matrix defined and correctness harness working.
-ffast-math.Goal: the optimisation-level sweep.
-O3 is slower than -O2 — this happens, and finding one is a genuinely interesting result worth investigating.-O3 regression.Goal: cross-compiler comparison and architecture flags.
-march=native and measure the gain. On modern x86 with AVX-512 this can be substantial for vectorisable loops; on ARM it may be minimal.-march=native on one node, then try to run that binary on a different node type. If it crashes with an illegal instruction, you have demonstrated the risk concretely — and that demonstration is worth including.-ffast-math: measure both the speedup and the numerical deviation from the reference.Goal: mechanism and recommendations.
perf stat to compare instruction counts, cache misses, and branch mispredictions between two builds. Why is the faster one faster?objdump -d, or Compiler Explorer for small examples) to confirm vectorisation actually occurred.-ffast-mathA grouped bar chart of runtime by compiler and optimisation level across benchmarks, at least one mechanistically explained result, an explicit portability caveat demonstrated rather than asserted, and a concrete recommended flag set. The strongest possible version also finds and explains an -O3 regression.
Not verifying correctness. A binary that computes the wrong answer quickly is not faster.
Dead code elimination. At -O2 and above, a compiler may delete a computation whose result is never used — producing an impossibly fast "result." Always consume the output (print it, checksum it, accumulate it).
Comparing compilers at different flag levels. GCC -O2 against Clang -O3 tells you nothing.
Ignoring build time. LTO can multiply compile time. At a competition that matters.
Assuming vectorisation happened. Use -fopt-info-vec (GCC) or -Rpass=loop-vectorize (Clang) to confirm, rather than inferring it from timing.
Add profile-guided optimisation (PGO): build with instrumentation, run a representative workload, rebuild using the collected profile. Gains of 10–20% are common and almost nobody at student level tries it. Alternatively, extend to Fortran — still the dominant language in much of HPC, and Fortran compilers often beat C compilers on numerical loops for reasons worth explaining (pointer aliasing rules).
You take a genuine scientific application — not a benchmark, not a toy — and characterise how its performance changes as you add computational resources. This is exactly what the Sprint Model's applications track asks for, and exactly what a cluster competition judge will probe.
Two distinct questions, often confused:
Real applications almost always strong-scale poorly beyond some point and weak-scale better. Understanding why for your chosen application is the research contribution.
| Application | Domain | Why choose it | Difficulty to build |
|---|---|---|---|
| GROMACS | Biomolecular dynamics | Excellent docs, ships benchmark cases, very widely used, good scaling behaviour to study | Moderate |
| LAMMPS | Molecular dynamics (materials) | Highly modular, ships many example inputs, connects to the Queen's nuclear-materials work | Moderate |
| OpenFOAM | Computational fluid dynamics | Directly relevant to Hyperloop/QADT/Relectric consulting; tutorial cases included | Hard — large, complex build |
| HPCG | Sparse solver benchmark | Easiest build; competition-scored; but it is a benchmark, not an application | Easy |
Recommendation for a first Tier 2 scaling study: LAMMPS. It builds more easily than OpenFOAM, ships with well-documented benchmark inputs designed for exactly this, and its performance characteristics are well understood so you can check your results against published expectations.
Goal: a working build. Budget the whole week — this is genuinely the hard part.
Goal: the strong scaling curve.
Goal: weak scaling, and the beginnings of mechanism.
Loop time summary splits pair, neighbour, communication, and output. Watch the communication fraction grow with core count. That growth is your explanation.Goal: analysis and write-up.
Both scaling curves with the ideal overlaid and the node boundary marked; a time-breakdown stacked bar showing communication share growing; a defended diagnosis; and a concrete recommendation of the efficient core count for this workload. That last item is what a real user wants and what almost no student report provides.
Problem too small. The most common failure. If the single-core run takes 20 seconds, then at 64 cores you are measuring startup and communication only. Make the base case take minutes.
Changing more than one thing. If you change both problem size and core count while intending strong scaling, you have measured nothing interpretable.
Ignoring the decomposition. Many applications partition the domain across processes. A bad partition at a particular core count can produce an anomalous point. Check whether the decomposition is sensible before declaring a scaling anomaly.
Including I/O in the timing. If the application writes a large output file at the end, and that write is serial, it will dominate at high core counts and mask the real compute scaling. Time the phases separately.
Not checking correctness across core counts. A correctly parallelised application should produce equivalent physics regardless of core count. Verify this — if it doesn't, that is a much more interesting finding than the timings.
Repeat on a second application and compare. Two applications with different communication patterns will show visibly different scaling behaviour, and explaining that difference in terms of their algorithms is a genuinely strong undergraduate result — the kind that would stand out in the CUCAI proceedings given how few systems papers appear there.
Alternatively, add the GPU dimension: most of these applications have GPU-accelerated builds. How does the scaling story change when each node has an accelerator? That is the natural bridge into Tier 3.
Modern processors can perform arithmetic far faster than memory can supply data. A CPU core might execute on the order of a hundred billion floating-point operations per second while its share of memory bandwidth delivers only a few billion values per second. For most real scientific code, the processor spends most of its time idle, waiting for data to arrive. This is the memory wall.
In this project you measure your machines' actual memory bandwidth using STREAM — the standard benchmark for exactly this — then use that number to explain performance results you have already collected in other projects. It is the single most explanatory measurement in this handbook.
Every computational kernel has an arithmetic intensity: the number of floating-point operations it performs per byte of data it moves from memory. This single number predicts whether a kernel will be limited by compute or by memory.
| Kernel | Roughly, FLOPs per byte | Limited by |
|---|---|---|
Vector addition (c = a + b) | ~0.08 | Memory, overwhelmingly |
| STREAM Triad | ~0.08 | Memory — by design; that's the point |
| Sparse matrix-vector multiply | ~0.25 | Memory |
| Dense matrix multiply (blocked) | Grows with block size — can reach 10s | Compute, if well implemented |
| N-body (all pairs) | Very high | Compute |
The roofline model plots achievable performance against arithmetic intensity. It has two regions: a sloped section where memory bandwidth is the ceiling, and a flat section where peak compute is the ceiling. The corner where they meet is the machine's balance point. Any kernel to the left of that corner cannot possibly reach peak FLOPs — no matter how well written.
This explains a great deal that is otherwise mysterious: why HPCG scores are typically only a few percent of HPL scores on the same machine (HPCG is memory-bound, HPL is compute-bound), and why adding cores stops helping in E11 for bandwidth-hungry loops.
Goal: a trustworthy bandwidth number.
lscpu), then size accordingly.Goal: bandwidth across platforms and thread counts.
numactl lets you control placement and demonstrate this.Goal: build the roofline and place real kernels on it.
Goal: demonstrate the model's predictive power.
A measured bandwidth figure within a plausible fraction of theoretical peak, a saturation curve showing where extra cores stop helping, a roofline plot with real kernels placed on it, and at least one case where the model correctly predicted an optimisation outcome. The strongest reports use the roofline to explain an earlier project's result that previously seemed arbitrary.
Arrays too small. The classic STREAM error. If your arrays fit in cache you will report bandwidth several times higher than the hardware can actually deliver, and every downstream conclusion will be wrong. Check against theoretical peak — if you exceed it, your arrays are too small.
Quoting Copy instead of Triad. Copy flatters the machine. Triad is the honest number.
Ignoring NUMA. On a two-socket node, threads reaching across sockets see much lower bandwidth. Report placement explicitly.
Confusing bits and bytes. Memory specifications are often quoted in different units than benchmark output. Getting this wrong by a factor of eight is common and embarrassing.
Treating the roofline as exact. It is a first-order model. Real machines have cache hierarchies it doesn't capture. Present it as a bound and a guide, not a prediction.
Build a hierarchical roofline that includes L1, L2, and L3 cache bandwidth as additional ceilings, not just main memory. This shows why blocking works: it moves a kernel's effective intensity up against a faster ceiling. It is a more sophisticated model and a considerably stronger result.
Alternatively, measure bandwidth on a GPU. High-bandwidth memory changes the balance point dramatically, and comparing the two roofline shapes explains why some kernels benefit enormously from GPUs while others don't — setting up Tier 3's GPU work.
You take one program and run it many ways — different compilers, different optimisation levels, different core counts, different machines, different runs of the identical binary — and check whether the numerical results are bitwise identical. Frequently they are not. This project characterises when, why, and by how much.
Floating-point addition is not associative. In exact mathematics, (a + b) + c = a + (b + c). In floating point, these can differ, because each operation rounds. Once you accept that, several consequences follow:
-ffast-math, and sometimes even at -O2 through vectorisation.a*b + c with a single rounding instead of two, giving a more accurate but different answer than the unfused sequence.exp, sin) are accurate to within a bit or so — but not identical.None of these is a bug. All of them mean that "reproducible" needs definition. This project's real contribution is establishing, for a given code, which level of reproducibility is achievable and at what performance cost.
Bitwise: identical bits, every time, everywhere. Achievable, but usually costs performance (deterministic reductions, disabled vectorisation).
Numerical: results agree to within a stated tolerance appropriate to the science. Usually what actually matters.
Scientific: conclusions are unchanged even if numbers differ. The weakest and most important standard.
%.17g for doubles, or dump binary.-ffp-contract=off), with and without -ffast-math.-ffast-math ≫ core count ≈ FMA > compiler > hardware.-ffp-contract=off, disabled fast-math. Then measure the performance cost. That trade-off — reproducibility versus speed, quantified — is the strongest result this project can produce.A matrix showing which factors cause divergence and by how much, an analysis of whether the divergence is scientifically meaningful for the application, and a quantified cost of enforcing bitwise reproducibility. A report that concludes "differences were at the 1e-15 level and do not affect the physical conclusions" is a perfectly good and honest outcome — provided you demonstrated it rather than assumed it.
Insufficient output precision. The commonest error. Six decimal places hides everything.
Uncontrolled randomness. If your program uses a random number generator, seed it explicitly and confirm the generator itself is deterministic across platforms — many are not.
Confusing "different" with "wrong." Bitwise divergence at the 1e-16 level is expected and harmless. Panic here is a misreading. The question is always whether it matters for the science.
Not distinguishing the amplification case. In chaotic or iterative systems, tiny differences grow. Distinguishing "small and stays small" from "small and grows" is the analytically important part.
Investigate compensated summation (Kahan summation) — an algorithm that dramatically reduces accumulated rounding error at modest cost. Measure both its accuracy benefit and its performance cost. It is a genuinely elegant result and it is the kind of thing that impresses a competition judge, because it demonstrates you understand the arithmetic rather than just the tooling.
Tiers 1 and 2 measured systems. Tier 3 modifies them. You are now writing or substantially altering code, configuring services, and producing results that could plausibly appear in conference proceedings. Three further expectations apply:
Tier 3 projects also routinely overrun a single four-week sprint. Plan for two sprints, or scope deliberately to fit one and treat the rest as future work — explicitly, in the report.
A cluster has finite nodes and more jobs than nodes. The scheduler decides what runs next. That decision is a genuine algorithmic problem with real trade-offs between throughput, fairness, and responsiveness — and Slurm exposes enough configuration to study it empirically.
You construct a synthetic but realistic workload, then run it repeatedly under different scheduler configurations on the Pi cluster, measuring makespan, wait times, utilisation, and fairness.
This project requires reconfiguring the scheduler and draining queues. Doing that on a production cluster serving 1,000+ researchers is not acceptable under any circumstances. The Pi cluster exists precisely so that this kind of work is safe — which is exactly the teaching-versus-research split the operating plan's §17c argues for.
| Policy | Rule | Strength | Weakness |
|---|---|---|---|
| FIFO | Strictly first-come, first-served | Perfectly fair by arrival; trivially predictable | A single wide job blocks everything behind it while nodes sit idle |
| Backfill | FIFO, but smaller jobs may jump ahead if they finish before the blocked job's resources are ready | Dramatically better utilisation; no job is delayed relative to FIFO | Depends entirely on accurate walltime estimates from users |
| Fair-share | Priority adjusted by recent usage — heavy users are deprioritised | Prevents monopolisation on multi-user systems | Can starve legitimate heavy users; needs tuning |
| Priority/QoS tiers | Explicit classes with different weights | Supports policy goals (e.g. teaching before research) | Administrative overhead; political |
The backfill insight is the interesting one. Its entire advantage comes from filling gaps around wide jobs — which means it only shows benefit if your synthetic workload actually contains wide jobs and heterogeneous durations. A workload of uniform single-node jobs will show no difference between FIFO and backfill, and a team that constructs such a workload will wrongly conclude backfill doesn't work.
This is the analytically critical step and where most of the intellectual work lies. A realistic HPC workload has:
Generate the workload once, save it as a fixed file, and replay the identical workload under every policy. If the workload differs between runs, the comparison is meaningless.
Goal: reproducible experimental harness.
sacct: submit time, start time, end time, nodes, user.Goal: the policy comparison.
Goal: sensitivity analysis — where the real insight is.
| Metric | Definition | Whose interest it represents |
|---|---|---|
| Makespan | Time from first submission to last completion | The administrator |
| Mean wait time | Average of (start − submit) | The average user |
| 95th-percentile wait | The tail | The unluckiest user — often what people actually complain about |
| Node utilisation | Fraction of node-seconds actually used | The funder |
| Slowdown | (wait + runtime) / runtime | Fairness to short jobs specifically |
| Per-user fairness | Spread of resources delivered across users | The community |
Report several. They conflict — a policy optimising makespan often worsens the wait-time tail — and demonstrating that conflict is more valuable than declaring a winner.
A homogeneous workload. All jobs the same size and duration means all policies perform identically and you have learned nothing. Heterogeneity is the entire experiment.
Not draining between runs. Leftover jobs from the previous configuration contaminate the next.
A workload too short. Scheduling effects need queueing. If jobs never wait, there is nothing to schedule.
Reporting a single metric. Declaring backfill "best" on makespan alone hides its effects on fairness.
Ignoring measurement variance. Establish your noise floor in week 2 and only claim differences that exceed it.
Add a carbon-aware dimension, connecting to §14 topic 37: defer delay-tolerant jobs into low-carbon-intensity windows using public Ontario grid data, and measure the emissions reduction against the makespan cost. Published work in this area reports substantial reductions for bounded delay, and replicating that at small scale with a real scheduler would be a strong and very current contribution.
Alternatively, replay a real workload trace rather than a synthetic one. Several public HPC workload archives exist, and using one substantially strengthens the realism claim.
Most large scientific problems produce matrices that are overwhelmingly zero. A finite-element mesh with a million nodes yields a million-by-million matrix — a trillion entries — of which perhaps ten million are non-zero. Storing it densely is impossible; storing only the non-zeros is routine.
The core operation on such matrices is sparse matrix-vector multiplication (SpMV): y = A·x. It is the innermost loop of virtually every iterative solver, which means it is the innermost loop of most computational science. It is also notoriously inefficient — typically achieving a few percent of a machine's peak FLOPs.
In this project you implement SpMV in several storage formats, measure them across matrices with different structures, and explain the performance differences in terms of memory access patterns.
Three reasons, all of which your measurements should demonstrate:
x, and perform one multiply-add. That is roughly 12–16 bytes moved per 2 FLOPs — an intensity around 0.15. Per T2.4's roofline, such a kernel is hopelessly memory-bound and can never approach peak compute.x. The column indices are arbitrary, so reads from the input vector jump around memory unpredictably. Caches and hardware prefetchers, which assume sequential access, are defeated.Understanding these three explains why HPCG — which is essentially a sparse solver benchmark — scores so much lower than HPL on identical hardware. That connection makes this project directly competition-relevant.
| Format | How it stores | Good for | Bad for |
|---|---|---|---|
| COO Coordinate | Three arrays: row index, column index, value — one entry per non-zero | Simplicity; easy construction; easy to convert from | Wasteful (stores row index redundantly); poor locality |
| CSR Compressed Sparse Row | Values and column indices, plus a row-pointer array marking where each row starts | The default. Compact, good row-wise access, well supported everywhere | Load imbalance when row lengths vary a lot |
| CSC Compressed Sparse Column | CSR transposed — compressed by column | Column operations; transpose products | SpMV requires either atomics or a reduction |
| ELL ELLPACK | Pads every row to the length of the longest, storing a rectangular array | Regular access; excellent on GPUs where uniformity matters | Catastrophic if one row is much longer than the rest — the padding dominates |
| HYB Hybrid | ELL for the bulk, COO for the overflowing long rows | Combines the strengths; common on GPUs | More complex; a tuning parameter to choose |
No format is universally best — the winner depends on the matrix's structure. ELL should beat CSR on matrices with uniform row lengths and lose badly on matrices with a few very dense rows. Demonstrating that empirically, and predicting the winner from a structural statistic before measuring, is the strongest possible outcome for this project.
Goal: correct implementations and a matrix corpus.
Goal: the performance matrix and its explanation.
Goal: parallelisation and load balance.
A performance table across formats and matrices; a demonstration that SpMV saturates memory bandwidth while achieving single-digit percentages of peak FLOPs; a structural statistic that predicts format choice; and a validated selection heuristic. Connecting the result explicitly back to why HPCG behaves as it does elevates the report considerably.
A homogeneous matrix corpus. If all your matrices come from structured meshes, every format performs similarly and the study has no findings. Deliberately include a highly irregular matrix — social graphs are ideal.
Not verifying correctness. ELL padding in particular is easy to get subtly wrong.
Timing the format conversion. Converting COO to CSR is a one-time preprocessing cost. Unless you're studying that specifically, time only the SpMV itself, and say so.
Ignoring the vector. If x fits entirely in cache, the irregular-access penalty vanishes and your results won't generalise. Use matrices large enough that x exceeds cache.
Reporting FLOPs without saying how you counted. The convention is 2 FLOPs per non-zero. State it.
Port CSR SpMV to the GPU and compare against cuSPARSE. GPUs change the format calculus entirely — ELL's regularity becomes far more valuable because GPU threads execute in lockstep. Demonstrating that the best CPU format is not the best GPU format is a strong result and a natural bridge into Tier 4.
Alternatively, connect to the Queen's geomechanics or nuclear-materials work in §14 topics 48–49 — those simulations are full of sparse solvers, and characterising a matrix from a real local research problem turns this into a partnership.
When a thousand processes all want to write results at once, the filesystem becomes the bottleneck — and the difference between a good and bad I/O strategy can be two orders of magnitude. This project characterises parallel filesystem behaviour on Frontenac using the standard tools, then demonstrates the practical consequences of common I/O patterns.
It extends E7 from single-process timing into genuine parallel I/O, and it addresses what is, in practice, the most common performance problem real researchers hit.
| Pattern | What happens | Typical outcome |
|---|---|---|
| File-per-process | Each of N processes writes its own file | Simple and often fast — until N is large, when metadata operations overwhelm the filesystem |
| Single shared file, independent | All processes write to different offsets in one file | Fewer files, but can cause lock contention depending on alignment |
| Single shared file, collective | MPI-IO coordinates processes, aggregating small writes into large contiguous ones | Usually the best for large runs — this is what collective I/O exists for |
| Serialised through rank 0 | All processes send data to one rank, which writes | Simple to reason about, and a severe bottleneck at scale. Common in student code. |
Parallel filesystems (Lustre, GPFS, BeeGFS) distribute a file across multiple storage targets. Stripe count controls how many; stripe size controls the chunk granularity. These are tunable per-file and per-directory, and the defaults are frequently wrong for a given workload — which makes them an easy source of real improvement.
This project generates heavy I/O on infrastructure shared with 1,000+ researchers. Before starting: check quotas, agree a testing window with CAC staff if the load will be significant, clean up test data immediately, and never run large metadata tests during business hours. Being a considerate tenant is also how QHPC builds the CAC relationship that §16 depends on — a team that trashes the scratch filesystem will not be invited to collaborate.
Goal: baseline characterisation.
-a (API: POSIX vs MPIIO), -b (block size), -t (transfer size), -F (file-per-process), and -c (collective).Goal: scaling and pattern comparison.
Goal: tuning, and a real-application demonstration.
Bandwidth-versus-process-count curves showing the plateau; a clear pattern comparison; metadata rates that put the small-files problem in perspective; and — ideally — a real application whose I/O time you measurably reduced. That last item is what makes this a consulting deliverable rather than an exercise.
Page cache contamination. Reading a file just written returns it from memory. Use IOR's reorderTasks option, or files far larger than aggregate node memory.
Testing during peak load. Results will be dominated by other users. Record time of day, repeat, and report variance honestly — or better, negotiate a quiet window.
Transfer size too small. Testing only 4 KB transfers measures per-operation overhead, not the filesystem's capability. Sweep from kilobytes to tens of megabytes.
Filling the filesystem. Multi-node IOR runs generate a great deal of data quickly. Check quota first, clean up immediately, and set a hard size cap in your scripts.
Comparing across filesystems without saying so. Home, scratch, and project directories often have entirely different backing hardware and policies. Label everything.
Investigate higher-level I/O libraries — HDF5 or NetCDF, which most scientific applications actually use. They sit on top of MPI-IO and their tuning parameters interact with the filesystem in ways that are poorly documented and genuinely worth characterising. Comparing raw MPI-IO against HDF5 for the same access pattern is a useful and rarely-done comparison.
Every job a scheduler runs leaves a record: requested resources, actual usage, exit state, queue time, user, partition, walltime requested versus consumed. Across thousands of jobs this is a substantial operational dataset — and a large fraction of jobs fail, often for predictable reasons (out of memory, exceeded walltime, bad path).
You apply machine learning to that data to predict, at submission time, whether a job will fail. Then — more usefully — you interpret the model to identify which submission characteristics predict failure, and turn that into actionable guidance.
It sits exactly at the intersection of ML and systems, which means it is legible to the CUCAI audience — a conference whose corpus is overwhelmingly ML-application work — while being genuinely HPC. It is the single most natural bridge between QHPC and QMIND, and a strong candidate for a joint submission.
It also requires no scientific domain knowledge, which makes it accessible to a broad set of members.
Three options, in descending order of value and ascending order of ease:
Scheduler logs contain usernames, job names, and sometimes file paths — all of which can identify individuals and reveal what research they are doing. If you obtain real data from the CAC:
Getting this right is also how QHPC demonstrates it can be trusted with institutional data — which matters far more to the §16 consulting strategy than any single result.
Goal: data secured and understood.
FAILED, TIMEOUT, OUT_OF_MEMORY, CANCELLED)? These are different phenomena and should probably be modelled separately — a user cancelling a job is not the same event as a job being killed for exceeding memory.Goal: a model that beats the baseline, evaluated honestly.
Goal: interpretation — the part that actually helps anyone.
A model that meaningfully outperforms the base rate on temporally held-out data, evaluated with appropriate metrics, using strictly submission-time features — plus an interpretation section producing concrete guidance. An honest negative result ("failure is largely unpredictable from submission metadata alone, and here is the evidence") is also a perfectly publishable outcome, provided the methodology is sound.
Target leakage. Using post-hoc features. Produces a model with 99% AUC that cannot be deployed. Audit every feature against the question "would I know this at submission time?"
Reporting accuracy on imbalanced data. Meaningless when the base rate is skewed.
Random train/test split on temporal data. Leaks future information; inflates results.
Treating all failures as one class. Timeouts, OOM kills, and user cancellations have different causes and different predictability. Model separately or justify combining them.
Publishing anything traceable to an individual. Beyond being an ethical failure, it would end the CAC relationship permanently.
Extend from failure prediction to runtime prediction — estimating how long a job will actually take. This is directly valuable to the scheduler: T3.1 showed that backfill's effectiveness depends entirely on walltime-estimate accuracy. A model that improves those estimates would measurably improve cluster utilisation, and combining T3.1 and T3.4 into a single study — "can ML-improved walltime estimates improve backfill performance?" — is a genuinely strong piece of work and an excellent joint QHPC/QMIND submission.
You take a CPU kernel you already understand — ideally one you measured in an earlier project — and port it to run on a GPU. You then measure the speedup honestly, find the problem size below which the GPU loses, and explain the result in terms of the hardware.
This is the bridge topic. It is the last runbook in this volume and the prerequisite for most of the Advanced Volume. It is also, by a wide margin, the topic most often done badly — for one specific reason covered below.
Comparing an optimised GPU implementation against an unoptimised CPU baseline.
It is extremely common to see "we achieved 50× speedup" where the CPU version was single-threaded, compiled at -O0, and written without regard for cache behaviour — while the GPU version was carefully tuned. That comparison measures effort, not hardware.
An honest baseline is multi-threaded (OpenMP across all cores), compiled at -O3 -march=native, and vectorised. Against such a baseline, realistic speedups for memory-bound kernels are typically 2–10×, not 50×. Reporting a smaller, honest number with a defensible baseline is worth far more — to a reviewer, to a competition judge, and to anyone who has to make a purchasing decision — than a large dishonest one.
State your baseline explicitly and prominently. Reviewers look for this first.
A CPU has a few dozen sophisticated cores optimised for latency — deep pipelines, large caches, aggressive branch prediction. A GPU has thousands of simple cores optimised for throughput, executing in lockstep groups (warps of 32 on NVIDIA hardware), paired with very high memory bandwidth.
That architecture implies exactly when GPUs win:
| Property of your problem | GPU outcome | Why |
|---|---|---|
| Massive data parallelism (thousands of independent items) | Excellent | Fills the machine; that's what it's for |
| High arithmetic intensity | Excellent | Compute units stay busy |
| Regular, coalesced memory access | Excellent | Adjacent threads reading adjacent addresses is a single wide transaction |
| Heavy branching that diverges within a warp | Poor | Both branch paths execute serially for the whole warp |
| Irregular / random memory access | Poor | Uncoalesced access can cost 32 transactions instead of 1 |
| Small problem size | Loses to CPU | Transfer over PCIe plus launch overhead exceeds the compute saved |
| Sequential dependencies | Poor | No parallelism to exploit |
For a first port, pick something with obvious data parallelism and no tricky dependencies. In rough order of suitability:
Goal: an honest CPU baseline. Do not skip this week.
-O3 -march=native, confirmed vectorisation (-fopt-info-vec).| Node | GPUs | Use for |
|---|---|---|
| frnt107 | 1× A100 40 GB | Start here. Single modern GPU, no multi-GPU complications |
| frnt110 | 1× V100 32 GB | Older architecture — good for a generational comparison |
| frnt140–147 | 2× A30 24 GB | Eight such nodes — best availability, least queue wait |
| frnt156 | 8× RTX8000 48 GB | Largest GPU memory on the system. For anything memory-hungry. |
| frnt154 | 8× A100 40 GB | Multi-GPU work; the distributed-training bridge (Vol II T5.5) |
| frnt190–191 | DGX A100, 8× A100 | Purpose-built DGX systems — the strongest nodes available |
Goal: a correct GPU version. Correct first, fast later.
cudaGetLastError() — CUDA fails silently by default, and a kernel that never launched returns instantly with untouched output, which looks like an impossibly fast result.Goal: optimise, guided by the profiler rather than by guesswork.
nsys (timeline: where does wall time go — transfer or compute?) and ncu (kernel detail: occupancy, memory throughput, warp efficiency).Goal: the crossover, and honest analysis.
| Quantity | Why it must appear |
|---|---|
| CPU baseline configuration | Cores used, flags, vectorisation confirmed. Without this the speedup is uninterpretable. |
| Kernel time vs total time | Separating them shows how much of the win transfers consume |
| Speedup vs problem size | A single speedup number hides the whole story |
| Crossover point | The actionable result |
| Achieved bandwidth / occupancy | Shows whether you used the hardware well or merely used it |
| Optimisation log | What each change bought — the most instructive part for future readers |
| Library comparison | Contextualises your result against what is achievable |
A speedup-versus-problem-size curve with the crossover marked, measured against a genuinely optimised multi-threaded CPU baseline, with transfers included in the headline number and separated in the analysis. Plus a profiler-guided optimisation log showing what each change achieved. A modest, well-defended 4× beats an undefended 40× every time.
The unoptimised baseline. Covered above. The defining error of the genre.
Excluding transfer time. Unless data genuinely stays resident on device for the whole application, transfers are part of the cost.
Not checking CUDA errors. A silently failed launch looks like infinite speed.
Not synchronising before timing. Kernel launches are asynchronous. Without cudaDeviceSynchronize() you time the launch call, not the work — and will report microsecond kernels.
Ignoring warp divergence. If threads in a warp take different branches, both paths execute. Mandelbrot has this property inherently; noticing and quantifying it is a good result.
Reporting only the best run. Same standards as every other project: median of several, with spread.
Port a second kernel with opposite characteristics. If your first was compute-bound and regular, do SpMV from T3.2 — memory-bound and irregular. The contrast between an 8× speedup and a 1.5× speedup, explained in terms of coalescing and arithmetic intensity, is a far stronger result than either alone.
Alternatively compare programming models: CUDA against OpenMP target offload or OpenACC. The directive-based approaches require far less code change — how much performance do they leave on the table? That question opens directly into the Advanced Volume's portability work.
Tiers 1–3 take a member from "has never used a cluster" to "can execute a controlled performance study, modify systems, and write a conference-submittable report." That is the full arc of a first and second year in QHPC, and it covers everything the Sprint Model needs to run.
Tiers 4 and 5 are published separately as the Advanced Volume, because they differ in kind rather than degree: they assume a faculty or graduate co-supervisor, they routinely span multiple terms, and several require domain knowledge from outside computing. That volume covers:
| Tier 4 · Advanced | Tier 5 · Frontier |
|---|---|
| Mixed precision and HPL-MxP | Radiation damage in zirconium alloys (RMTL / MAX-Nuclear) |
| Performance portability across programming models | Rock mass modelling for deep mining (QGGG / Buchan) |
| Collective communication tuning | Cardiovascular CFD and reduced-order models |
| Carbon-aware scheduling | Particle-astrophysics data reduction (SNO Institute) |
| Quantum circuit simulation | Distributed training at scale |
| Digital twin of the cluster | Multi-institution reproducibility studies |
Do not route members into Tier 4 before they have completed at least one Tier 2 and one Tier 3 project. The Advanced Volume assumes the measurement discipline, baseline rigour, and profiling fluency this volume builds. A student who arrives at mixed-precision work without having internalised T2.4's roofline or T3.5's baseline standards will produce results that do not survive review.
The runbooks are written broadly on purpose: a topic has to accommodate several teams, several cohorts, and several levels of ambition. But a team that starts work still holding the broad version will not finish. Breadth is what causes a sprint to end with twelve half-measurements and no figure — the team kept finding new things worth trying because nothing told them what was out of scope.
Narrowing is a skill, it takes about an hour, and it should happen in week 1 before any code is written.
Say the question out loud and ask: could a competent team answer this in four weeks and be done? "How does container overhead affect HPC applications" — no, that is a career. "Does containerising this specific workload cost more runtime than run-to-run noise on this node" — yes, that is four weeks, and it has a real answer that could go either way.
The narrower question is not the lesser one. It is the one that produces a figure, a finding, and a report someone will read.
These are starting points, not assignments. A team that sharpens further after its first measurements has done exactly the right thing.
| Topic | Too broad | The question to actually run |
|---|---|---|
| E1 | "Is the cluster faster than a laptop?" | At what problem size does total time-to-answer on the cluster including queue wait fall below the laptop — and how far does that crossover move between a quiet queue and a busy one? |
| E2 | "Demonstrate Amdahl's Law." | At a fixed parallel fraction of 90%, at what core count does measured speedup fall 20% below Amdahl's prediction, and does that point move when the problem size grows? |
| E3 | "Benchmark compression tools." | For scientific binary array data, which compressor sits on the Pareto frontier at a 1 Gb/s link and at a 10 Gb/s link — and does compress-then-transfer beat raw transfer at either speed? |
| E4 | "Compare Python, NumPy and C." | At what problem size does NumPy's advantage over pure Python stop growing, and how much of the remaining gap to C is interpreter overhead rather than memory layout? |
| E5 | "Analyse cloud costs." | For one representative club workload, at what monthly utilisation does owning the Pi cluster cost less than renting equivalent cloud capacity, once electricity is included? |
| E6 | "Write a build guide." | Can a member who has never used the cluster get a first job running in under 60 minutes from the written guide alone — and at which specific step do most of them stall? |
| E7 | "Time the filesystems." | At what file count does the home-versus-scratch difference for many-small-file workloads exceed 2×, and does the gap depend on file size or only on count? |
| E8 | "Learn job arrays." | For a sweep of N jobs, at what N does a job array beat N individual submissions on total time-to-completion — and is the difference scheduler overhead or queue policy? |
| E9 | "Reproduce a paper." | Working only from the paper and its published artifacts, how close can the headline figure be reproduced — and what specific missing information accounts for the residual? |
| E10 | "Survey campus compute." | Among surveyed research groups, what fraction describe themselves as compute-bound, and of those, what fraction are unaware that a national allocation is available to them? |
| E11 | "Study OpenMP scaling." | At what thread count does this kernel stop scaling, and is the limit memory bandwidth, synchronisation, or load imbalance? Evidence required for whichever you claim. |
| E12 | "Learn MPI." | For a fixed problem size, at what process count does communication time exceed computation time — and does changing the domain decomposition move that point? |
| Topic | Too broad | The question to actually run |
|---|---|---|
| T2.1 | "Measure container overhead." | Does containerising this specific workload cost measurably more runtime than bare metal — that is, more than run-to-run noise on the same node — and does the answer change for I/O-heavy versus compute-heavy phases? |
| T2.2 | "Compare compilers and flags." | How much of the performance difference between two optimisation levels on this kernel survives across three compilers — and does any flag among them change the numerical output? |
| T2.3 | "Study scaling." | At what node count does strong-scaling efficiency for this application drop below 70%, and does weak scaling hold out to the same node count or further? |
| T2.4 | "Investigate the memory wall." | Where does this kernel sit relative to the roofline, and how much of the gap closes with a single blocking transformation? Predict the gain before making the change. |
| T2.5 | "Look at reproducibility." | Does this application produce bit-identical results across repeated runs, across process counts, and across compilers — and which of those three breaks first? |
These are the topics expected to reach a conference submission, so the question matters most here.
| Topic | Too broad | The question to actually run |
|---|---|---|
| T3.1 | "Study scheduling policy." | Under this site's policy, how much does requesting a shorter walltime reduce median queue wait — and is the relationship linear, or is there a threshold where the benefit jumps? |
| T3.2 | "Look at sparse matrices." | Across a chosen matrix collection, does the fastest storage format correlate with any single structural property, or is it predictable only from a combination of them? |
| T3.3 | "Investigate parallel I/O." | At what process count does one-file-per-process I/O become slower than a single shared file with collective I/O — and does the crossover depend on write size? |
| T3.4 | "Predict job failures." | Using only features available at submission time, can failed jobs be identified with precision above 0.5 — and how much does adding the first minute of runtime telemetry improve that? |
| T3.5 | "Port something to the GPU." | What speedup does the GPU port achieve over a properly tuned multithreaded CPU baseline — and how much of the headline speedup people usually quote is actually CPU under-tuning rather than GPU advantage? |
The Paper Kit is the companion for everything after this point — finding the prior work, verifying citations, handling noisy measurements, and building each section of the manuscript. Open it in week 2, not at the end: several of its sections change what you should be recording while the experiment is still running.
Before any code: the team writes the question, what is held fixed, what is varied, the expected answer, and what would refute it. One page. The sprint lead signs it. It goes at the front of the project repository and is the first thing compared against the results at write-up.
Where the final question differs from the signed one — normal, and often a sign the team learned something — the report says what changed and why. That paragraph is regularly the most interesting methods content in the whole write-up, and it is impossible to write if nobody wrote the original question down.
Every runbook in this volume ends with a deliverable, but a deliverable without a standard is graded on whoever happens to read it. That produces inconsistent feedback across teams, and it makes it impossible to tell whether a cohort improved year over year.
Use this rubric for every sprint report, in every tier. It is deliberately weighted so that method quality outranks result quality — a careful negative result should score higher than a lucky positive one, because that is the behaviour the club wants to reinforce.
| Dimension | Weight | What full marks looks like | What loses marks |
|---|---|---|---|
| Method soundness | 30% | Comparisons are like-for-like; confounds are named and controlled; repetitions are sufficient and reported with spread; the baseline is the strongest available rather than the most convenient | Single runs; different code across platforms; an untuned baseline; a varying parameter left unstated |
| Reproducibility | 20% | A reader with access to the same hardware could re-run this from the repository alone: versions pinned, scripts committed, raw results included, environment documented | Results with no accompanying code; "we ran it and got"; an undocumented manual step |
| Interpretation | 20% | The report explains why the curve has the shape it has, and the explanation is supported by evidence rather than plausibility | Describing the plot in prose without explaining it; asserting a cause that was never measured |
| Figures | 15% | One figure carries the finding; axes labelled with units; scales chosen honestly; the reader understands it without the caption | Default styling; unlabelled axes; a truncated axis that exaggerates a difference; six figures where one would do |
| Scope honesty | 10% | An explicit sentence stating what was tested, on what, and what therefore cannot be concluded | Generalising past the evidence; omitting the failures; presenting a single machine's behaviour as a general law |
| Writing | 5% | The finding appears in the first paragraph; jargon is defined on first use; a reader from another faculty can follow it | Burying the result in section four; unexplained acronyms |
A report scoring well on method, reproducibility, and scope honesty is worth submitting externally even if the result is negative or partial. A report scoring well only on figures and writing is not, however impressive the number in it. Sprint leads should be explicit about this when giving feedback, because the instinct in the other direction is strong and it is the single habit most likely to produce work that fails peer review later.
The tier ladder describes prerequisites. This map describes continuity: which Volume I topic sets up which later project, so a sprint lead can route a member deliberately rather than by whatever is unclaimed.
| Start here (Vol. I) | What it teaches that transfers | Leads to |
|---|---|---|
| E1 Cluster vs laptop | Queue time is part of the cost; crossover thinking | Volume II's T6.7, whose entire thesis is that acquisition cost is invisible if you time from the wrong moment |
| E2 Amdahl measured | Measured speedup versus predicted, and explaining the residual | The CPU-only track's T-CPU6, which applies exactly this method to quantized inference |
| E3 Compression benchmark | Pareto reasoning; trade-offs rather than winners | Volume II's T4.1, whose headline figure is an accuracy-versus-time Pareto frontier |
| E4 Python vs NumPy vs C | Serial performance, and why a fair baseline is hard | Volume II's T4.2 and T6.6, both of which live or die on baseline quality |
| E9 Reproduce a figure | What reproducibility actually costs | Volume II's T5.6 multi-institution reproducibility work |
| T2.3 Strong and weak scaling | Scaling methodology proper | T-CPU6, and any competition benchmarking work |
| T2.4 The memory wall | Roofline reasoning; bandwidth versus compute | Prerequisite for all of Volume II's Tier 4 |
| T2.5 Numerical reproducibility | Precision, tolerance, and equivalence checking | Volume II's T4.1 mixed precision, directly |
| T3.1 Scheduling policy | How a batch scheduler actually decides | Volume II's T6.3 and T6.8, both scheduling-method contributions |
| T3.5 First GPU port | Honest cross-platform comparison | Volume II's T4.2 performance portability; gates all Tier 4 GPU work |
Several routes above pass through GPU work. If a term's GPU allocation falls through, the CPU-only track is not a downgrade — it is a parallel path that keeps a member progressing on method rather than stalling on hardware. E2, T2.3, and T2.4 lead into it as naturally as they lead into Volume II, and a member who has done T-CPU6 arrives at Tier 4 with the scaling discipline intact.
Handbook Volume I complete — Tiers 1–3, 22 runbooks, 7 reference sections (Slurm, timing, plotting, report template, topic-narrowing worksheet with a worked question for all 22 topics, sprint rubric, progression map). Volume II (Advanced) covers Tiers 4–6 and the Zero-GPU / Single-Node Track covers CPU-only work; both are published separately.
Last Updated: 2026-08-08 Session: Plotting pipeline (synthetic-first) + grid resolution + cluster pivot + PDF/DOCX compilation + T-CPU1/3/4 full pipeline + Volume II curriculum
phase7-final-manuscript.md — Final Markdown with [MEASURED] slots, abstracts, AI disclosurephase7-manuscript.tex — LaTeX (IEEEtran, IEEE numeric citations, Chinese content removed)phase7-references.bib — 23 entries, all verified (Carry A closed)phase7-cover-letter.md — IGSC submission cover letterphase7-final-quality-checklist.md — PASS on all gatesartifact/)README.md — Updated with Husom model grid, quantization mapping, tier coverage notesLICENSE — CC BY 4.0 legal textzenodo-metadata.json — Pre-reserve DOI, creator placeholdersenvironment-pinning.json — Runtime versions pinned (ollama 0.32.6, llama.cpp 10326, Python 3.11.4)configs/grid.json — Husom 28-model sweep populated + mid/large tiers added (mistral:7b, qwen2.5:7b, llama3.1:8b, granite3.2:8b)configs/prompt.txt — "Explain quantum computing in simple terms."scripts/run_grid.sh — Grid orchestrator (randomized, calibration, cross-check)scripts/measure_cell.py — Per-cell runner (Ollama/llama.cpp, RAPL/SoC/estimate)scripts/analyze.py — Pre-registered analysis (D1-D4, BH-FDR, verdict)t-cpu2-measurement-execution-checklist.md — Phases A-G runbookphase7-final-quality-checklist.md — PASS (format, citations, compliance)external-review-revision-log.md — Carry A closed, all 22 refs verifiedcitation-audit-report.md — Clean (1 cross-artifact fix applied)verified-bibliography-quality-matrix.md — 23/23 verifiedJobSubmitPlugins = lua — Frontenac controller-side Lua submit filter (puppet-managed, invisible from login node) rewrites EVERY job from sg6079000_cpu to hidden partition gpu-rgrant.--partition=cpubase_interac|gpu-L4|teaching|cpubase_6hrs all land gpu-rgrant PENDING; --nodelist=frnt153 → "Requested nodes not in this partition" (server-side rewrite proven); gpu-rgrant hidden from scontrol show partition (sinfo -a only), account-restricted to def-hpcg1971./etc/slurm/slurm.conf controller-only.artifact/scripts/plot_lib.py — pure-function loaders + log-normal ratio SEartifact/scripts/synthetic_data.py — schema-compliant draft data (24 primary cells × 2 legs, 30 runs + 1 warmup, D1 estimate fallback on Pi fp16)artifact/scripts/plots.py — 5 figures + table1: fig1_crossover, fig2_heatmap, fig3_joint, fig4_robustness, fig5_marginartifact/scripts/make_table1_tex.py — booktabs LaTeX table generatorartifact/figures/figure-includes.tex — LaTeX figure blocks for §4.1–4.5 + §6.1 verdict templatesartifact/tests/test_plot_pipeline.py — 15 tests, all passgrid_role field added to grid.json; run_grid.sh/synthetic_data.py filter by role, --include-sensitivity flag.data/synthetic_edge (2 missing-leg cells, 2 boundary-ambiguous cells forced with --calibration-bound-j 5.0). Fixed 2 real bugs: (1) analyze.py crashed when cells.csv mixed tested with non-tested rows (DictWriter fieldnames derived from row 0) — now union_fieldnames() in plot_lib.py; (2) plots.py table1 + make_table1_tex.py dropped reason for non-tested cells (silent 12-char truncation into Ch column) — now a proper Status column, emitted only when non-tested rows exist. fig5_margin renders non-tested cells as grey bars (pixel-verified). Regression tests added (15 total, all pass).pdflatex/mpm/initexmf at C:\Users\[Author]\AppData\Local\Programs\MiKTeX\miktex\bin\x64\. [MPM]AutoInstall=1 set; ieeetran + 17 dependency packages pre-installed via mpm (first pdflatex run had hung on interactive package-install prompt — killed, root-caused, fixed). Note: all manuscripts compile with pdflatex (English-only corpus).phase7-manuscript.pdf — 7 pages, 158,815 bytes, IEEEtran. Pass 1 → bibtex (22 entries, 0 warnings) → pass 2. No LaTeX warnings; only cosmetic underfull hbox/vbox (badness ≤ 10000). 22/22 citations resolved in .aux.phase7-manuscript.docx (26,759 bytes, pandoc from markdown). Verified: affiliation block, all 9 [MEASURED] slots + [CROSSOVER VERDICT] preserved, References rendered (Husom et al., Grange, Koslovski, et al.).All three remaining QHPC topics completed to the same phase1–7 depth as T-CPU2 (11 files each: phase1, phase2, phase3, phase4, phase5a, phase5b, phase6, phase7-final-manuscript.md, phase7-references.bib, phase7-manuscript.tex, phase7-final-quality-checklist.md, repo root, tcpuN-* prefix). Volume II curriculum expanded to full lesson plans.
artifact/scripts/tcpu4_simulator.py — discrete-event simulator (FCFS+backfill / naive threshold / lookahead schedulers, 3 renewable profiles, 4 arrival rates, 2 workload shapes), pure Python.artifact/scripts/tcpu4_sweep.py — full-factorial sweep (48 rows), artifact/results/tcpu4/sweep.json + experiment.json.artifact/scripts/tcpu4_plots.py — 3 figures (artifact/figures/tcpu4/).artifact/tests/test_tcpu4_simulator.py — 14 tests, all pass.green_la lookahead recovers (−1.6% HPC grid, no drops, delay ≈ FCFS). Single seed=42 deterministic.tcpu4-phase7-manuscript.pdf — 4 pages, compiled clean (figure added for fig:edp), 11 verified citations. DOCX via pandoc.artifact/scripts/rag_retriever.py — backend-pluggable retrieval pipeline (MiniLMBackend via sentence-transformers 5.7.0 + torch CPU; HashEmbeddingBackend for tests), persistent store.artifact/data/tcpu3/corpus.jsonl (20 docs) + queries.jsonl (20 queries); artifact/results/tcpu3/eval.json + eval_hash.json.artifact/scripts/rag_eval.py + tcpu3_plots.py (2 figures).artifact/tests/test_rag_retriever.py — 11 tests (10 pass + 1 integration-skipped; MiniLM integration test passed: 384-dim on CPU).tcpu3-phase7-manuscript.pdf — 3 pages, compiled clean, 4 verified citations. DOCX via pandoc.tcpu1-phase7-manuscript.pdf — 3 pages, compiled clean (fixed: torring2022analyzing @article→@inproceedings; qphc2026track citation replaced with prose), 12 verified citations. DOCX via pandoc.volume-ii-curriculum.md — expanded from outline to full lesson plans: Tier 4 (Benchmarking & Empirical Methods), Tier 5 (Optimization Strategy), Tier 6 (Green & Heterogeneous Systems); 6 sessions × 2 h per tier; each session has Objectives / Content / Hands-on / Checkpoint / Homework; tier artifacts + exit criteria + assessment summary table.python -m unittest discover -s tests; 1 skipped integration): 15 plot-pipeline + 14 tcpu4-simulator + 10 rag-retriever (+1 skipped) + 1 MiniLM integration (run separately).Per user request, all stale documents scanned and updated. Summary:
Stamped as historical pipeline artifacts (frozen content + provenance banner, 16 files): phase1-literature-search-report.md, phase2-paper-outline.md, phase3-argument-blueprint.md, phase4-complete-draft.md, phase5a-citation-compliance-report.md, phase5b-abstract.md, phase6-peer-review-report.md, phase6-revision-log.md, phase6-peer-review-round2.md, external-review-eic-report.md, external-review-methodology-report.md, external-review-domain-report.md, external-review-perspective-report.md, external-review-devils-advocate-report.md, external-review-editorial-decision.md, external-review-revision-log.md.
Refreshed shared sources (content updated, 5 files):
verified-bibliography-quality-matrix.md — added 4-topic citation-subset map (T-CPU4 → rows 4–9, 18–23; T-CPU3 → rows 1,3,4,10; T-CPU1 → rows 1,8–17,20); MiniLM prose-only note.hpc-reference-corpus.md — added Volume II lesson-plan grounding (Tiers 4–6 source mapping).qphc-cpu-four-topic-track.md — per-topic status lines (all 4 complete; measurement slots open).citation-audit-report.md — added 2026-08-08 follow-up note (new manuscripts cite only verified matrix; per-manuscript audits clean).cpu-only-curriculum-publication-pathways.md — added per-topic status + gate-status table (gates 1,2,4,5 DONE; gate 3 PARTIAL).Re-verified live deliverable: phase7-final-quality-checklist.md — added 2026-08-08 re-verification note (recompiled after final edits, all gates PASS).
HTML exports stamped (7 files, 2026-08-08): artifact-1.html … artifact-7.html — frozen HTML comment inserted after <title> (inside <head>, no render impact): <!-- STATUS: archived snapshot (saved 2026-08-03/04). Source referenced by citation-audit-report.md; content NOT regenerated 2026-08-08. -->. Content left untouched (original source artifacts; rewriting would falsify provenance). phase7-references.bib left current (used in compilation — not stale).
| Item | File(s) | Action Needed |
|---|---|---|
| Affiliation + ORCID | phase7-final-manuscript.md, phase7-manuscript.tex, phase7-cover-letter.md, artifact/zenodo-metadata.json |
Affiliation DONE (2026-08-08) — "Queen's University, Kingston, ON, Canada" filled in all 4 files. ORCID still pending — optional, author to add (placeholder retained). |
artifact/configs/grid.json |
DONE — mid: mistral:7b, qwen2.5:7b; large: llama3.1:8b, granite3.2:8b. fp16 cells server-only + Pi estimation fallback (pre-registered memory-pressure clause; protocol's "fp16 within 8GB" for >8B is arithmetically impossible, resolved via fallback, documented in grid.json _notes and README). Also DONE (2026-08-08): grid_role field added — 24 primary cells (6 models × 4 quants) + 4 sensitivity (qwen2.5:1.5b; gemma2:2b contributes 0 cells, 3-bit only). |
|
| Environment pinning (post-measurement) | artifact/environment-pinning.json |
Fill OS/kernel/runtimes/model hashes/governors after measurement on each leg |
| Zenodo metadata | artifact/zenodo-metadata.json |
Fill affiliation, ORCID, manuscript DOI after submission |
artifact/environment-pinning.json |
DONE — 7 tokens ("Explain quantum computing in simple terms."), verify with pinned tokenizer at measurement time |
Cluster blocker (resolved as dead-end, not pursued): Account sg6079000_cpu is pinned to hidden gpu-rgrant partition by a controller-side Lua submit filter (root cause, 2026-08-08). frnt201 busy with ~100 MPI jobs ahead; login node has no RAPL. Decision: skip cluster — CAC email template retained below only as fallback.
Local dual-leg plan (active):
/sys/class/powercap/intel-rapl:0/)Fallback template (only if cluster reconsidered):
Subject: Partition access request for account sg6079000_cpu
Hi CAC team,
Account sg6079000_cpu (user sa6079052) is currently restricted to gpu-rgrant partition (single node frnt201).
This forces all jobs to queue behind heavy MPI workloads.
Request: Please grant access to cpubase_interac and/or teaching partitions for CPU-based energy measurement work (RAPL power monitoring on Intel Xeon nodes).
Use case: T-CPU2 protocol server-leg energy measurement — requires RAPL (intel-rapl) on CPU nodes, not GPU.
Thanks,
[Your Name]If cluster access ever granted: Run measurement on idle CPU nodes (frnt129, frnt136-137, frnt150-153) with RAPL.
| Component | Status |
|---|---|
| Synthetic data generator | 24 primary cells × 2 legs ✓ |
| analyze.py (D1–D4, BH-FDR) | 28→24 cells, verdict JSON ✓ |
| plots.py (5 figs + table1) | All rendered, non-blank ✓ |
| Unit tests | 15/15 pass ✓ |
| LaTeX figure includes | Compiled (IEEEtran, 7 pages) ✓ |
| SSH bridge v3 | Working (live OTP prompt) |
| ollama | v0.32.6 ✓ |
| llama.cpp | v10326 (commit 3653e6d6d) ✓ |
| llama3.2:3b model | Pulled (2GB) ✓ |
| RAPL on frnt201 | Verified (package/DRAM/psys) ✓ |
| ollama inference | Working (login node test) ✓ |
| llama.cpp | Needs GGUF model file (download separately) |
| Login node RAPL | Not available (expected) |
High priority (author input):
phase7-final-manuscript.md — ORCID (affiliation filled: Queen's University)phase7-manuscript.tex — ORCID (affiliation filled)phase7-cover-letter.md — ORCID (affiliation filled)artifact/zenodo-metadata.json — ORCID + DOI (affiliation filled)artifact/environment-pinning.json — post-measurement valuesartifact/configs/prompt.txt — token count → environment-pinning.jsonartifact/figures/synthetic/*.png — eyeball figures (label clipping/legend overlap)Local measurement (when Pi + x86 node available):
artifact/scripts/run_grid.sh --leg pi (Pi 5 + INA219)artifact/scripts/run_grid.sh --leg server (x86 Linux + RAPL)artifact/scripts/analyze.py --pi-dir data/pi --server-dir data/server --out results/realartifact/scripts/plots.py --pi-dir ... --server-dir ... --cells results/real/cells.csv --crossover results/real/crossover.csv --verdict results/real/verdict.json --grid configs/grid.json --out figures/realAll pipeline work for T-CPU1/2/3/4 is COMPLETE. Remaining items are author-side / hardware / venue:
Bridge Status: Currently alive in your PowerShell window (CONNECTED). Leave open if continuing soon; close if pausing for days (needs fresh MFA on restart).
Imported from PROJECT_STATUS.md (2026-08-08). Full pipeline status: T-CPU1-4 complete through Phase 7, measurement execution pending cluster access, files to review, and next steps. Complements the Research & Project Handbook above.