5 Commits

Author	SHA1	Message	Date
James Mills	26df009e8f	Refactored crawler to use task dispatcher	2021-02-02 13:13:12 +10:00
James Mills	4970b16d61	Add better crawler metrics	2021-02-02 12:07:53 +10:00
James Mills	8a1161cf77	Record and skip urls the crawler has seen before	2021-02-01 23:46:26 +10:00
James Mills	b398a3a709	Add working Crawler, Scraper and Indexer	2021-02-01 22:38:17 +10:00
James Mills	b69b27eeed	Add crawler skeleton and add handler	2021-02-01 21:25:32 +10:00