Skip to content

Add 7 useragents #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MaxGiting merged 1 commit into from
May 21, 2019
Merged

Add 7 useragents #329

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion raw/Crawlers.json
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions raw/Crawlers.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ APIs-Google
AportWorm\/
AppBeat\/
AppEngine-Google
AppleSyndication
AppStoreScraperZ
Aprc\/[0-9]
Arachmo
Expand All @@ -100,6 +101,7 @@ Astute
asynchttp
Attach
autocite
AutomaticWPTester
Autonomy
axios\/
B-l-i-t-z-B-O-T
Expand Down Expand Up @@ -213,6 +215,7 @@ Crescent
Crowsnest
Criteo
CSHttp
CSSCheck
curb
Curious George
curl
Expand All @@ -228,6 +231,7 @@ DataparkSearch
dataprovider
DataXu
Daum(oa)?[ \/][0-9]
dBpoweramp
ddline
deeris
Demon
Expand Down Expand Up @@ -836,6 +840,7 @@ Recorder
RecurPost\/
redback\/
ReederForMac
Reeder\/
ReGet
RepoMonkey
request\.js
Expand Down Expand Up @@ -1013,6 +1018,7 @@ TheNomad
Thinklab
Thumbshots
ThumbSniper
Thumbor
timewe\.net
TinEye
Tiny Tiny RSS
Expand Down
6 changes: 6 additions & 0 deletions src/Fixtures/Crawlers.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ class Crawlers extends AbstractProvider
'AportWorm\/',
'AppBeat\/',
'AppEngine-Google',
'AppleSyndication',
'AppStoreScraperZ',
'Aprc\/[0-9]',
'Arachmo',
Expand All @@ -121,6 +122,7 @@ class Crawlers extends AbstractProvider
'asynchttp',
'Attach',
'autocite',
'AutomaticWPTester',
'Autonomy',
'axios\/',
'B-l-i-t-z-B-O-T',
Expand Down Expand Up @@ -234,6 +236,7 @@ class Crawlers extends AbstractProvider
'Crowsnest',
'Criteo',
'CSHttp',
'CSSCheck',
'curb',
'Curious George',
'curl',
Expand All @@ -249,6 +252,7 @@ class Crawlers extends AbstractProvider
'dataprovider',
'DataXu',
'Daum(oa)?[ \/][0-9]',
'dBpoweramp',
'ddline',
'deeris',
'Demon',
Expand Down Expand Up @@ -857,6 +861,7 @@ class Crawlers extends AbstractProvider
'RecurPost\/',
'redback\/',
'ReederForMac',
'Reeder\/',
'ReGet',
'RepoMonkey',
'request\.js',
Expand Down Expand Up @@ -1034,6 +1039,7 @@ class Crawlers extends AbstractProvider
'Thinklab',
'Thumbshots',
'ThumbSniper',
'Thumbor',
'timewe\.net',
'TinEye',
'Tiny Tiny RSS',
Expand Down
8 changes: 7 additions & 1 deletion tests/crawlers.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3443,4 +3443,10 @@ deeris/1.0 (+http://www.deeris.com)
Mediametric
TextRazor Downloader (https://www.textrazor.com)
ddline.cn rank history
Web spyder
Web spyder
AppleSyndication/52.8
AutomaticWPTester/1 CFNetwork/975.0.3 Darwin/17.7.0
CSSCheck/1.2.2
dBpoweramp
Thumbor/6.4.1
Reeder/1000.02.01 CFNetwork/454.11.5 Darwin/10.5.0 (i386) (Macmini2)