Can network position predict which gene pairs kill a cancer cell?
A reproduction and stress test of MAGICAL, a classifier that uses protein-protein interaction (PPI) network features to label gene pairs as synthetic lethal, synthetic viable, or neither. The goal is to check how well the published results hold up, then test how fragile they are.
Try the idea: pick two genes
Click any two nodes in this toy network. The panel shows the kind of network features a classifier can use for a pair. A trained model maps features like these to a class.
No pair selected
Select a first gene, then a second.
Toy graph with made-up gene labels, for explaining features only. It is not MAGICAL's model or data.
The idea in pictures
Synthetic lethality and its opposite, synthetic viability, are both about pairs of genes. The classifier tries to tell them apart from where the genes sit in the interaction network.
How fragile is a network?
Predictions rest on network structure, so damaging the network is a fair stress test. This chart removes genes or interactions at random from the toy network (150 trials per step) and tracks how much stays connected. Drag the slider.
Results at a glance
The model comparison below uses the real results on the paper's split; the schedule is the plan the experiments followed.
What gets implemented
Five stages, in order. Each later stage depends on the one before it.
Rebuild the feature pipeline
Pull the labelled gene pairs and PPI network from the paper's repository and database. Re-extract the network features, checking each against the code's output.
Recreate the published evaluation
Use the same train/test split and metrics as the paper. Report the gap between my numbers and theirs before changing anything.
Compare with simpler models
Logistic regression, random forest and gradient boosting on the same features, plus network-only heuristics such as degree product and shortest-path distance.
Perturb the network
Remove random nodes or edges, rewire while keeping degrees, and re-score. See how fast performance drops.
Shortlist candidate pairs
Rank pairs the model predicts confidently and consistently across perturbations, as leads for biological follow-up.
Results tracker
All five experiments have run on the real data. The numbers come from results/; the figures further down show them in context.
| Experiment | Question | Status | Result |
|---|---|---|---|
| Reproduction | Do I match the paper's reported metrics? | Done | Yes: 80.9% vs the paper's ~81% (seed 0); 73.4% with only 3 clean features |
| Baselines | How much does the full model add over simple classifiers? | Done | RF 80.9%; all-20-column RF 87.5%; gradient boosting 77.2%; logistic 56.6%; degree stump 48.5%; path stump 55.5% |
| Node removal | How does performance fall as nodes are dropped? | Done | Macro-F1 0.71 → 0.52 at 30% removal; 24% → 60% of test pairs lose a network gene |
| Edge rewiring | Do predictions depend on the exact network wiring? | Done | Degree-preserving rewiring collapses macro-F1 to 0.41 (1×) and 0.40 (5×): the model depends on exact wiring |
| Candidate pairs | Which predictions are stable enough to follow up? | Done | 451 stable pairs from 6,535 unlabeled NOT-pool pairs (top confidence 0.97) |
Download the 451 candidate pairs as CSV, annotated with gene symbols and species. Top human predictions include BRCA2–ARRDC3 and HDAC1–BABAM1 (a DNA-repair complex member). Note the official network mixes species: 289 of 451 pairs are human–human.
Setup and schedule
CPU-only: classical machine learning and graph analysis. Start with a small subset of pairs, check the repository's dependencies, then scale to the full benchmark.
magical-repro/ data/ # raw + processed pairs, PPI network src/features.py # network feature extraction src/train.py # model + published split src/baselines.py src/perturb.py # node removal, edge rewiring notebooks/ # figures for each experiment results/
Week 1
Environment, data download, feature extraction on a subset, first checks against the paper.
Week 2
Full reproduction of the published evaluation, then baselines.
Week 3
Perturbation experiments, candidate shortlist, write-up.
Background: why gene pairs matter
Two genes are synthetically lethal when losing either one alone is survivable but losing both kills the cell. This matters for cancer because a tumour often has one gene already broken. A drug that hits its partner can then kill tumour cells while leaving healthy cells, which still have both genes, largely alone.
The best-known example is BRCA-deficient tumours treated with PARP inhibitors. Finding more pairs like this by experiment is slow, because the number of possible gene pairs is enormous. That is the gap computational prediction tries to fill.
Synthetic viability is the opposite surprise: a loss that would normally hurt the cell is rescued when a second gene is also lost. Predicting both, plus the ordinary "no interaction" case, makes this a three-way classification problem rather than a yes/no one.
Network features to extract
The paper uses properties of the protein-protein interaction network. The table lists typical candidates from network science. Which ones MAGICAL actually uses, and how, gets confirmed against the repository before anything is built.
| Feature | What it measures | Why it might help |
|---|---|---|
| Degree | Number of direct interaction partners of a gene | Hubs tend to matter to many processes |
| Betweenness centrality | How often a gene sits on shortest paths between others | Bridges between modules may be hard to compensate for |
| Clustering coefficient | How interconnected a gene's neighbours are | Dense local groups hint at redundancy |
| Shortest path of the pair | Steps between gene A and gene B | Close genes may share function or back each other up |
| Common neighbours | Partners the two genes share | Shared partners suggest a shared pathway |
| Jaccard / Adamic-Adar | Overlap of neighbourhoods, weighted by how rare the shared partners are | Corrects for hubs inflating overlap |
Evaluation plan
With three classes and probably far more "other" pairs than lethal ones, plain accuracy can look good while the model misses most real lethal pairs. So every result gets reported per class.
- Per-class precision, recall and F1 show where the model fails, especially for the rare classes.
- Macro-F1 averages F1 over classes so the large class cannot hide weak ones.
- Matthews correlation coefficient summarises the whole confusion matrix in one number that is hard to inflate.
- Confidence intervals from repeated seeds or bootstrap resamples, so small gaps are not over-read.
Try the calculator below. It works from a confusion matrix (rows are the true class, columns are the predicted class). The starting numbers are an example to show how the metrics behave, not results from this project.
| True \ Predicted | Lethal | Viable | Other |
|---|
| Class | Pairs | Precision | Recall | F1 |
|---|
How the data gets split
How pairs are divided into train and test sets can change the answer a lot. The paper's split is reproduced first, then the stricter ones are added.
| Split | How | What it tests |
|---|---|---|
| Published split | As in the paper and repository | Whether the reported numbers can be matched |
| Random pairs | Shuffle pairs, repeat over several seeds | Spread between runs |
| Gene held out | No gene in the test set appears in training pairs | Whether the model learns real network signal or just memorises genes |
| Source held out | Train on one data source, test on another, if available | Generalisation beyond the original labels |
Robustness protocol
Each perturbation is applied to the network only, then the features are recomputed and the same trained model is scored again. Every setting is repeated over many random seeds and reported as a curve with an interval, not a single number.
| Perturbation | How | What it reveals |
|---|---|---|
| Remove nodes | Delete 5% to 30% of genes at random | Sensitivity to missing proteins |
| Remove edges | Delete 5% to 30% of interactions at random | Sensitivity to an incomplete network |
| Add noise edges | Insert random false interactions | Sensitivity to false positives in the network |
| Degree-preserving rewiring | Swap edge endpoints, keeping every gene's degree | Whether results come from wiring or just from degree |
| Swap the network | Rebuild features from a different interaction database | Dependence on one data source |
The rewiring row is the sharpest test. If performance barely changes when only degrees are kept, the model is mostly learning degree.
From predictions to candidates
- Keep pairs predicted with high confidence and the same class across most perturbations.
- Remove pairs that already appear in the training labels, since those are not new.
- Cross-check against public resources: existing synthetic lethality databases, CRISPR dependency data such as DepMap, and the literature.
- Look for pathway themes among the top pairs instead of trusting single pairs.
- Report them as hypotheses, with the evidence for and against each one.
Risks and how they are handled
| Risk | Mitigation |
|---|---|
| Repository dependencies are old or fail to install | Pin versions in a clean environment; note every fix |
| Published numbers cannot be matched exactly | Report the gap honestly and list what differs |
| Rare classes have few examples | Per-class metrics, repeated seeds, intervals |
| Well-studied genes dominate the network and the labels | Gene-held-out split; check performance by degree bucket |
| Interactions depend on cell type and context | State the limitation; do not claim clinical relevance |
| Information leaks from test to training | Fix splits once, save them, never tune on the test set |
Reproducibility checklist
- Record the repository commit and every package version.
- Fix and log random seeds.
- Save the exact train/test pairs used for every table.
- Keep original-paper numbers and my numbers in separate files and separate figures.
- Write each experiment's question before running it.
Glossary
- PPI network
- A graph where genes or proteins are nodes and physical interactions are edges.
- Synthetic lethal
- A gene pair where losing both is fatal but losing one is not.
- Synthetic viable
- A gene pair where a harmful loss is rescued by losing the second gene.
- Knockout
- Switching a gene off so its protein is no longer made.
- Node and edge
- A gene, and an interaction between two genes.
- Centrality
- A score of how important a node's position in the network is.
- Macro-F1
- F1 averaged equally over all classes.
- Data leakage
- Test information sneaking into training, making results look better than they are.
- Rewiring
- Randomly swapping edges while keeping each node's degree.
Questions
Is this a new discovery?
No. Reproducing a published model confirms or questions the original claims. The new part is the stricter generalisation and robustness testing, reported separately.
Why CPU only?
The method uses classical machine learning and graph statistics, which run on an ordinary laptop. Start with a small subset to check everything works before the full run.
What if I cannot match the paper's numbers?
That is still a result. Document the gap, check data versions and split details, and report what differs rather than tuning until the numbers agree.
Can predictions be used clinically?
No. They are hypotheses for lab follow-up, not evidence about treatment.
Where do I start tomorrow?
Clone the official repository, create a clean environment, read how the features are built, and run the smallest example end to end.
Knockout screen simulator: where synthetic lethality comes from
Genes often back each other up. In this toy cell a nutrient flows through steps to biomass, and the cell lives as long as any route still works. Click genes to knock them out and watch the green path, or click a square in the grid to test a pair. The grid is a full double-knockout screen: every pair, computed instantly.
Synthetic lethal pairs come from redundancy, but a network feature only sees that indirectly. The averages above often overlap between the two groups, which is one reason predicting these pairs from network structure is hard.
An abstract model: a cell lives if a route survives. Real cells add context, timing and rescue effects (synthetic viability is not modelled here). The same screen runs in Python with src/knockout_screen.py in the zip.
Null lab: is a pair closer than its degrees explain?
Features like shared neighbours go up simply because both genes have many partners. To check, this lab compares your pair with 300 networks rewired to keep every gene's number of connections but scramble who connects to whom. If the pair is no closer than that null, a model using the feature may only be reading degree. Paste any interaction list, such as a subset of a real PPI network. It runs in your browser and nothing is uploaded. It starts with the toy network.
These are generic pair features, not MAGICAL's own feature set. The same idea runs on a real network with src/degree_null.py in the zip. The all-pairs scan works for networks of up to 40 genes.
Shortcut lab: can a model pass without learning the biology?
A model can score well by learning which genes tend to be lethal, instead of what makes a particular pair lethal. A random split hides this, because the same genes appear in training and test. A gene-held-out split exposes it. This lab simulates both on invented data. Drag the first slider and watch the memoriser's score fall apart on unseen genes.
Synthetic data: 80 invented genes, all 3,160 pairs, the top quarter labelled lethal, scores averaged over a handful of random runs. The memoriser has no features, so on genes it has never seen it can only guess. This is a simulation to explain the risk, not a result about MAGICAL. In the real project the gene-held-out split and the degree-preserving rewiring play this role.
100 figures
Each figure is labelled by where it comes from. Computed and simulated figures come from the made-up 14-gene toy network, so they show how the methods behave rather than anything about MAGICAL's real data. Concept figures are drawings with illustrative values. MAGICAL results figures are computed on the real network, labels and models.
What this project does and does not claim
Reproducing a published model is not a new discovery. The contribution here is an independent test of how well MAGICAL generalises and how robust it is to changes in the network. Those results will be reported separately from the original paper's numbers, and predicted pairs are hypotheses until tested in the lab.