Can I reference a dynamically generated sitemap in robots.txt?Ethics of robots.txtSitemap for a site with a large number of dynamic subdomainsListing both sitemaps and sitemap index files in robots.txt?XML Sitemap generator in phpCan a relative sitemap url be used in a robots.txt?Url blocked by robots.txt message in Google WebmasterHow to create a whitelist of URLs using a sitemap in robots.txt?sitemaps on cross submitsGoogle Webmaster Tools is not accepting my sitemapInsufficient Sitemap
What is meaning of 4 letter acronyms in Roman names like Titus Flavius T. f. T. n. Sabinus?
Is my background sufficient to start Quantum Computing
gzip compress a local folder and extract it to remote server
Can the word "coexist" be used for more than two things/people/subjects/... etc?
Does a reference have a storage location?
How come having a Deathly Hallow is not a big deal?
Go function to test whether a file exists
Why do the three workers have to go inside the power plant again?
Use real text instead of lipsum in moderncv quote alignment
What is the difference between a historical drama and a period drama?
Show that there are infinitely more problems than we will ever be able to compute
What could a Medieval society do with excess animal blood?
Hiding a solar system in a nebula
Is there any way for a Half-Orc Sorcerer to get proficiency with a heavy weapon?
What do you call the angle of the direction of an airplane?
Contributing to a candidate as a Foreign National US Resident?
Puzzling Knight has a Message for all- Especially Newcomers
Isn't "Dave's protocol" good if only the database, and not the code, is leaked?
What caused the flashes in the video footage of Chernobyl?
Can you move between the attacks of a Twinned Booming Blade
What can a novel do that film and TV cannot?
What is -(-2,3,4)?
SQL Server error 242 with ANSI datetime
Performance of loop vs expansion
Can I reference a dynamically generated sitemap in robots.txt?
Ethics of robots.txtSitemap for a site with a large number of dynamic subdomainsListing both sitemaps and sitemap index files in robots.txt?XML Sitemap generator in phpCan a relative sitemap url be used in a robots.txt?Url blocked by robots.txt message in Google WebmasterHow to create a whitelist of URLs using a sitemap in robots.txt?sitemaps on cross submitsGoogle Webmaster Tools is not accepting my sitemapInsufficient Sitemap
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I'm setting up a sitemap for my website, and what I've done was creating an endpoint that generates the sitemap for it.
If I put the URL for my API on robots.txt, could the crawler bots access that and get the sitemap or will they not send an API request?
dynamic sitemap robots.txt
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
add a comment |
I'm setting up a sitemap for my website, and what I've done was creating an endpoint that generates the sitemap for it.
If I put the URL for my API on robots.txt, could the crawler bots access that and get the sitemap or will they not send an API request?
dynamic sitemap robots.txt
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27
add a comment |
I'm setting up a sitemap for my website, and what I've done was creating an endpoint that generates the sitemap for it.
If I put the URL for my API on robots.txt, could the crawler bots access that and get the sitemap or will they not send an API request?
dynamic sitemap robots.txt
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
I'm setting up a sitemap for my website, and what I've done was creating an endpoint that generates the sitemap for it.
If I put the URL for my API on robots.txt, could the crawler bots access that and get the sitemap or will they not send an API request?
dynamic sitemap robots.txt
dynamic sitemap robots.txt
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
edited Mar 26 at 9:23
unor
70.5k18 gold badges150 silver badges260 bronze badges
70.5k18 gold badges150 silver badges260 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
asked Mar 25 at 18:09
Caio AmbrosioCaio Ambrosio
133 bronze badges
133 bronze badges
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27
add a comment |
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344066%2fcan-i-reference-a-dynamically-generated-sitemap-in-robots-txt%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344066%2fcan-i-reference-a-dynamically-generated-sitemap-in-robots-txt%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There is no difference between an "API request" and an ordinary HTTP request.
– SLaks
Mar 25 at 20:00
Is the sitemap generated on the server-side or on the client-side?
– unor
Mar 26 at 9:24
the sitemap is generated on the server-side
– Caio Ambrosio
Mar 26 at 11:27