Assignment 1 · Web crawling and text mining
Rugby report miner
A course server hosted 147 rugby union match reports. My crawler walked the site breadth-first, then used plain string search and one regular expression to work out which team each report was about and what the score was. The server is gone, so the crawl below is replayed on a reconstruction of its link structure, and the extractor runs the original logic, ported line for line, on any text you give it.
- pages crawled
- 147pages crawled
- with team and score
- 64with team and score
- teams tracked
- 7teams tracked
Breadth-first crawl
Starting from index.html, the crawler kept a first-in, first-out queue: pop a page, fetch it, append every link it has not seen or queued yet. Each article links to its previous and next article, so the queue fans out in both directions round the ring and the visit order alternates 001, 147, 002, 146, and so on.
The crawler starts at index.html, which links to one article. Press play to replay the queue.
Queue (1)
- Hodg001
- Visited
- 0
- Team and score found
- 0
- visited, kept in task 2
- visited, dropped
- in the queue
- not yet seen
Paste a match report
The original joins the headline and paragraphs into one string, names the team whose first mention comes earliest, then splits on spaces and keeps tokens that match \d\d?-\d\d?. After stripping quotes, commas, full stops and colons, anything longer than five characters is ignored and the biggest remaining score wins. Edit the text and watch each decision.
Try an example (written for this demo)
First team mentioned and the biggest score agree.
Score rule
Assignment 1.ipynb (produced task2.csv, task3.csv and the report)
Team
Ireland
Score
24-14
Game difference
10
Kept in task2.csv. Both a team and a score were found.
Ireland 24-14 France. Ireland held on in Dublin after France had led 7-3 inside the first quarter. A second-half try from the back row put the hosts in control, and the boot did the rest as Ireland closed out the game. The result leaves Ireland two points clear at the top of the table with one round to play.
| Token | Cleaned | Total | Decision |
|---|---|---|---|
| 24-14 | 24-14 | 38 | chosen |
| 7-3 | 7-3 | 10 | beaten by a later or larger score |
What the miner found
Recomputed from the extracted team and score of every crawled page (no article text needed). The notebook's largest-total rule produced the submitted figures; the final script's rule is one switch away. It changes two scores, lifting New Zealand's average margin from 30.5 to 39.5.
64 of 147 articles kept
Articles per team
Task 4 plotted the five most written-about teams (highlighted).
- England20
- Ireland13
- Wales11
- France8
- Italy6
- New Zealand4
- Scotland2
Average game difference
Task 3: mean absolute winning margin over each team's articles. Under each bar: the number of articles and a 95% interval (added in 2026): a percentile bootstrap from 8 articles, and below that a Student t interval cut off at zero, because a bootstrap of so few values is far too narrow. The wide intervals are the point: they show how little most of these averages rest on.
- New Zealand30.50n = 4 · 95% t interval [13.0, 48.0]
- Ireland17.23n = 13 · 95% bootstrap [7.2, 30.3]
- Italy13.67n = 6 · 95% t interval [5.2, 22.2]
- France12.13n = 8 · 95% bootstrap [4.0, 21.5]
- Wales11.64n = 11 · 95% bootstrap [5.6, 18.2]
- England10.50n = 20 · 95% bootstrap [3.8, 19.5]
- Scotland5.00n = 2 · 95% t interval [0.0, 43.1]
Coverage versus margin
Task 5 asked whether being written about often goes with winning big. The original drew both measures as grouped bars on one axis; here each team is a point. England lead coverage with a modest margin while New Zealand have the biggest margin from four articles, so the report found no clear relationship.
Use the arrow keys to step through points.
View the task 3 to 5 numbers as a table
| Team | Articles | Avg game difference | 95% CI | Home ground |
|---|---|---|---|---|
| England | 20 | 10.5000 | [3.80, 19.45]bootstrap | Twickenham Stadium |
| France | 8 | 12.1250 | [4.00, 21.51]bootstrap | Stade de France |
| Ireland | 13 | 17.2308 | [7.23, 30.31]bootstrap | Aviva Stadium |
| Italy | 6 | 13.6667 | [5.18, 22.16]t, cut at 0 | Stadio Olimpico |
| New Zealand | 4 | 30.5000 | [13.04, 47.96]t, cut at 0 | Unknown |
| Scotland | 2 | 5.0000 | [0.00, 43.12]t, cut at 0 | Murrayfield Stadium |
| Wales | 11 | 11.6364 | [5.64, 18.18]bootstrap | Principality Stadium |
Next: Project 2
Matching products across two shops with four passes.