npm install google-robotstxt-parser
robots.txt Checker
A JavaScript port of Google's official robots.txt parser — test live in your browser.
Checking…
Use in your project
import { RobotsMatcher } from 'google-robotstxt-parser';
const matcher = new RobotsMatcher();
const allowed = matcher.oneAgentAllowedByRobots(robotsTxt, userAgent, url);
console.log(allowed); // true or false