James Mills
|
93cab4c1bc
|
Add opensearch endpoint and handler
|
2021-02-02 15:12:08 +10:00 |
|
James Mills
|
3923ff91ba
|
Add a /tasks endpoing and extra metdata to crawl tasks
|
2021-02-02 14:58:52 +10:00 |
|
James Mills
|
e1b0dd173f
|
Reduce dispatchers to 2 (by default)
|
2021-02-02 13:36:48 +10:00 |
|
James Mills
|
26df009e8f
|
Refactored crawler to use task dispatcher
|
2021-02-02 13:13:12 +10:00 |
|
James Mills
|
4970b16d61
|
Add better crawler metrics
|
2021-02-02 12:07:53 +10:00 |
|
James Mills
|
8a1161cf77
|
Record and skip urls the crawler has seen before
|
2021-02-01 23:46:26 +10:00 |
|
James Mills
|
b398a3a709
|
Add working Crawler, Scraper and Indexer
|
2021-02-01 22:38:17 +10:00 |
|
James Mills
|
b69b27eeed
|
Add crawler skeleton and add handler
|
2021-02-01 21:25:32 +10:00 |
|
James Mills
|
a9be838b0b
|
Add /cache route
|
2021-01-30 15:58:18 +10:00 |
|
James Mills
|
c4f53a0be9
|
It builds!
|
2021-01-30 15:56:03 +10:00 |
|
James Mills
|
8495ba0319
|
More fixes
|
2021-01-30 15:26:24 +10:00 |
|
James Mills
|
0fb306109c
|
Remove refs to webmention
|
2021-01-30 15:12:41 +10:00 |
|
James Mills
|
0454b9b62b
|
Replaced all other referenced to twt/twtxt
|
2021-01-30 15:06:58 +10:00 |
|
James Mills
|
4529ea3196
|
Initial Codebase (untested)
|
2021-01-30 14:05:04 +10:00 |
|