Disallowing /lev1/*.html, but allowing /lev1/lev2/*.htmlhow to disallow all dynamic urls robots.txtRobots.txt: Disallow subdirectory but allow directoryDisallow dynamic pages in robots.txtRobots.txt - Disallow all pages but allow all subdirRobots.txt DisallowDisallow dynamic URL in robots.txtAllow /category but disallow /category/subcategoryAllow and Disallow in Robots.txtrobot.txt allow instead of DisallowDisallow /*foo but allow /*bar?foo=foo (i.e. how to disallow an API if query string might contain the same name?)
Manager asking me to eat breakfast from now on
Difference between class and struct in with regards to padding and inheritance
BritRail England Passes compared to return ticket for travel in England
Can error correction and detection be done without adding extra bits?
Which GPUs to get for Mathematical Optimization (if any)?
Is this Android phone Android 9.0 or Android 6.0?
/bin/sh: 0: Can't open sh
🍩🔔🔥Scrambled emoji tale⚛️🎶🛒 #2️⃣
"This used to be my phone number"
How did J. J. Thomson establish the particle nature of the electron?
Does 5e follow the Primary Source rule?
I have found a mistake on someone's code published online: what is the protocol?
Demographic consequences of closed loop reincarnation
Do Australia and New Zealand have a travel ban on Somalis (like Wikipedia says)?
Why is an object not defined as identity morphism?
Wait or be waiting?
Locked-up DOS computer beeped on keypress. What mechanism caused that?
Wildcards not interpreted by shell script when used with output redirection
Why is the Intel 8086 CPU called a 16-bit CPU?
Did Hitler say this quote about homeschooling?
Do medium format lenses have a crop factor?
Why do the digits of a number squared follow a similar quotient?
How can I help our ranger feel special about her beast companion?
Applying for jobs with an obvious scar
Disallowing /lev1/*.html, but allowing /lev1/lev2/*.html
how to disallow all dynamic urls robots.txtRobots.txt: Disallow subdirectory but allow directoryDisallow dynamic pages in robots.txtRobots.txt - Disallow all pages but allow all subdirRobots.txt DisallowDisallow dynamic URL in robots.txtAllow /category but disallow /category/subcategoryAllow and Disallow in Robots.txtrobot.txt allow instead of DisallowDisallow /*foo but allow /*bar?foo=foo (i.e. how to disallow an API if query string might contain the same name?)
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
Must be disallowed:
/lev1/*.html
Must be allowed:
/lev1/lev2/*.html
Is it posible with robots.txt?
robots.txt
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
add a comment |
Must be disallowed:
/lev1/*.html
Must be allowed:
/lev1/lev2/*.html
Is it posible with robots.txt?
robots.txt
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
add a comment |
Must be disallowed:
/lev1/*.html
Must be allowed:
/lev1/lev2/*.html
Is it posible with robots.txt?
robots.txt
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
Must be disallowed:
/lev1/*.html
Must be allowed:
/lev1/lev2/*.html
Is it posible with robots.txt?
robots.txt
robots.txt
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
edited Mar 27 at 9:53
unor
71k18 gold badges151 silver badges260 bronze badges
71k18 gold badges151 silver badges260 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
asked Mar 26 at 10:47
Konstantin LyashenkoKonstantin Lyashenko
262 bronze badges
262 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
http://www.robotstxt.org/robotstxt.html
https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
To exclude all robots from part of the server
User-agent: *
Disallow: /lev1/lev2/*.html
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55355270%2fdisallowing-lev1-html-but-allowing-lev1-lev2-html%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
http://www.robotstxt.org/robotstxt.html
https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
To exclude all robots from part of the server
User-agent: *
Disallow: /lev1/lev2/*.html
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
add a comment |
http://www.robotstxt.org/robotstxt.html
https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
To exclude all robots from part of the server
User-agent: *
Disallow: /lev1/lev2/*.html
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
add a comment |
http://www.robotstxt.org/robotstxt.html
https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
To exclude all robots from part of the server
User-agent: *
Disallow: /lev1/lev2/*.html
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
http://www.robotstxt.org/robotstxt.html
https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
To exclude all robots from part of the server
User-agent: *
Disallow: /lev1/lev2/*.html
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
answered Mar 26 at 10:51
pakopako
3801 silver badge6 bronze badges
3801 silver badge6 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55355270%2fdisallowing-lev1-html-but-allowing-lev1-lev2-html%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
