Commit Graph

5 Commits

Author SHA1 Message Date
James Mills
26df009e8f Refactored crawler to use task dispatcher 2021-02-02 13:13:12 +10:00
James Mills
4970b16d61 Add better crawler metrics 2021-02-02 12:07:53 +10:00
James Mills
8a1161cf77 Record and skip urls the crawler has seen before 2021-02-01 23:46:26 +10:00
James Mills
b398a3a709 Add working Crawler, Scraper and Indexer 2021-02-01 22:38:17 +10:00
James Mills
b69b27eeed Add crawler skeleton and add handler 2021-02-01 21:25:32 +10:00