What is about Robot in web? What is its ROLE ?
Basic and simple definition -
robot is the type of agent who delivers the information from data servers to the Users.
Example - like a postman in the post office. ( who delivers the postcards from post office to the customers ).
Simple right?
Then, why people ignoring these robots ?
don’t they want their postcards (user queries) ?
The thing is, the web developers don’t know - how to right the exact rules for their website.
Now Technically, robot.txt is the text file which is used to instruct the web crawling agents (google bot / bing bot / yahoo bot / yandex bot ect..)that - which pages on your website to be crawled and which should not be crawled in the syntactic format.
If you know the right way to write the instructions to robots file then, your website and sub-pages will be ranked properly.
To know more, follow here

Comments
Post a Comment