Skip to content

Add 5 user agents #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MaxGiting merged 1 commit into from
Apr 23, 2019
Merged

Add 5 user agents #325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion raw/Crawlers.json
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions raw/Crawlers.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,8 @@ DataparkSearch
dataprovider
DataXu
Daum(oa)?[ \/][0-9]
ddline
deeris
Demon
DeuSu
developers\.google\.com\/\+\/web\/snippet\/
Expand Down Expand Up @@ -609,6 +611,7 @@ MarkWatch
Mass Downloader
masscan\/
Mata Hari
Mediametric
Mediapartners-Google
mediawords
MegaIndex\.ru
Expand Down Expand Up @@ -1000,6 +1003,7 @@ terrainformatica
Test Certificate Info
testuri
Tetrahedron
TextRazor Downloader
The Drop Reaper
The Expert HTML Source Viewer
The Knowledge AI
Expand Down Expand Up @@ -1098,6 +1102,7 @@ Web Fetch
Web Fuck
Web Pix
Web Sauger
Web spyder
Web Sucker
Webalta
Webauskunft
Expand Down
5 changes: 5 additions & 0 deletions src/Fixtures/Crawlers.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@ class Crawlers extends AbstractProvider
'dataprovider',
'DataXu',
'Daum(oa)?[ \/][0-9]',
'ddline',
'deeris',
'Demon',
'DeuSu',
'developers\.google\.com\/\+\/web\/snippet\/',
Expand Down Expand Up @@ -630,6 +632,7 @@ class Crawlers extends AbstractProvider
'Mass Downloader',
'masscan\/',
'Mata Hari',
'Mediametric',
'Mediapartners-Google',
'mediawords',
'MegaIndex\.ru',
Expand Down Expand Up @@ -1021,6 +1024,7 @@ class Crawlers extends AbstractProvider
'Test Certificate Info',
'testuri',
'Tetrahedron',
'TextRazor Downloader',
'The Drop Reaper',
'The Expert HTML Source Viewer',
'The Knowledge AI',
Expand Down Expand Up @@ -1119,6 +1123,7 @@ class Crawlers extends AbstractProvider
'Web Fuck',
'Web Pix',
'Web Sauger',
'Web spyder',
'Web Sucker',
'Webalta',
'Webauskunft',
Expand Down
5 changes: 5 additions & 0 deletions tests/crawlers.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3439,3 +3439,8 @@ https://www.nominet.uk/privacy-notice
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/61.0.3116.0 Mobile Safari/537.36 Chrome-Lighthouse
Trackuity (+http://trackuity.com)
deeris/1.0 (+http://www.deeris.com)
Mediametric
TextRazor Downloader (https://www.textrazor.com)
ddline.cn rank history
Web spyder
1 change: 0 additions & 1 deletion tests/devices.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30260,7 +30260,6 @@ Mozilla/5.0 (Linux; Android 4.4.4; GT-I9300 Build/KTU84Q) AppleWebKit/537.36 (KH
Mozilla/5.0 (Linux; Android 4.4.4; G7-L01 Build/HuaweiG7-L01) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.89 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 (5298492416)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 6 Build/MMB29V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36 Mediametric
Mozilla/5.0 (Linux; U; Android 5.1; en-US; SLIDE2 Build/LMY47I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 UCBrowser/10.9.0.731 U3/0.8.0 Mobile Safari/534.30
Mozilla/5.0 (Linux; U; Android 4.0.3; en-us; Sony Tablet S Build/TISU0143) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30
Opera/9.80 (SpreadTrum; Opera Mini/4.4.31492/37.8678; U; en) Presto/2.12.423 Version/12.16
Expand Down