James Mills
|
d4ab395515
|
Refactor out the use of rice in favor of embed (#2)
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/spyda/pulls/2
|
2022-10-05 00:19:23 +00:00 |
|
James Mills
|
1833d56686
|
Fix import paths
|
2021-07-13 08:33:43 +10:00 |
|
James Mills
|
ce4b180576
|
Add Indexer.Size() and index_size metric
|
2021-02-07 21:12:49 +10:00 |
|
James Mills
|
36cdbd3fc5
|
Tidied up the UX a bit more
|
2021-02-03 21:29:18 +10:00 |
|
James Mills
|
ccfbd18b29
|
Add crawler_duration gauge and server_queries counter
|
2021-02-03 19:09:16 +10:00 |
|
James Mills
|
93cab4c1bc
|
Add opensearch endpoint and handler
|
2021-02-02 15:12:08 +10:00 |
|
James Mills
|
3923ff91ba
|
Add a /tasks endpoing and extra metdata to crawl tasks
|
2021-02-02 14:58:52 +10:00 |
|
James Mills
|
e1b0dd173f
|
Reduce dispatchers to 2 (by default)
|
2021-02-02 13:36:48 +10:00 |
|
James Mills
|
26df009e8f
|
Refactored crawler to use task dispatcher
|
2021-02-02 13:13:12 +10:00 |
|
James Mills
|
4970b16d61
|
Add better crawler metrics
|
2021-02-02 12:07:53 +10:00 |
|
James Mills
|
8a1161cf77
|
Record and skip urls the crawler has seen before
|
2021-02-01 23:46:26 +10:00 |
|
James Mills
|
b398a3a709
|
Add working Crawler, Scraper and Indexer
|
2021-02-01 22:38:17 +10:00 |
|
James Mills
|
b69b27eeed
|
Add crawler skeleton and add handler
|
2021-02-01 21:25:32 +10:00 |
|
James Mills
|
a9be838b0b
|
Add /cache route
|
2021-01-30 15:58:18 +10:00 |
|
James Mills
|
c4f53a0be9
|
It builds!
|
2021-01-30 15:56:03 +10:00 |
|
James Mills
|
8495ba0319
|
More fixes
|
2021-01-30 15:26:24 +10:00 |
|
James Mills
|
0fb306109c
|
Remove refs to webmention
|
2021-01-30 15:12:41 +10:00 |
|
James Mills
|
0454b9b62b
|
Replaced all other referenced to twt/twtxt
|
2021-01-30 15:06:58 +10:00 |
|
James Mills
|
4529ea3196
|
Initial Codebase (untested)
|
2021-01-30 14:05:04 +10:00 |
|