shinyoukai/spyda
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactored crawler to use task dispatcher

Browse Source
This commit is contained in:
James Mills
2021-02-02 13:13:12 +10:00
parent 4970b16d61
commit 26df009e8f
13 changed files with 578 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
go.mod
View File

@@ -46,6 +46,7 @@ require (
github.com/sirupsen/logrus v1.7.0
github.com/spf13/pflag v1.0.5
github.com/steambap/captcha v1.3.1
github.com/stretchr/testify v1.6.1
github.com/temoto/robotstxt v1.1.1 // indirect
github.com/unrolled/logger v0.0.0-20201216141554-31a3694fe979
github.com/vcraescu/go-paginator v1.0.0