Styjun

I understand that robots.txt is a file which is intended for "robots" or should I say "automated crawler". However, does it prevent a human from typing the "forbidden" page and gather the data by hand?

Maybe it's clearer with an example: I cannot crawl this page:

https://www.drivy.com/search?address=Gare+de+Li%C3%A8ge-Guillemins&address_source=&poi_id=&latitude=50.6251&longitude=5.5659&city_display_name=&start_date=2019-04-06&start_time=06%3A00&end_date=2019-04-07&end_time=06%3A00&country_scope=BE

Can I still take "manually" via the my web browser's developers tool the JSON file containing the data?

robots.txt files are guidelines, they do not prevent anyone, human or machine, from accessing any content.

The default settings.py file that is generated for a Scrapy project sets ROBOTSTXT_OBEY to True. You can set it to False if you wish.

Mind that websites may employ anti-scraping measures to prevent you from scraping those pages, nonetheless. But that is a whole other topic.

Based on the original robots.txt specification from 1994, the rules in a robots.txt only target robots (bold emphasis mine):

WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages.

[…]

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed.

So, robots are programs that automatically retrieve documents linked/referenced in other documents.

If a human retrieves a document (using a browser or some other program), or if a human feeds a list of manually collected URLs to some program (and the program doesn’t add/follow references in the retrieved documents), the rules in the robots.txt do not apply.

The FAQ "What is a WWW robot?" confirms this:

Normal Web browsers are not robots, because they are operated by a human, and don't automatically retrieve referenced documents (other than inline images).