DOM not refreshing with Selenium (IE, Chrome and VBA)How to tell if a DOM element is visible in the current viewport?How to find event listeners on a DOM node when debugging or from the JavaScript code?How do I find out which DOM element has the focus?Remove all child elements of a DOM node in JavaScriptHow to Implement DOM Data Binding in JavaScriptHow to return DOM from Selenium after clickHow to initiate click before page is fully loadedCan a website detect when you are using selenium with chromedriver?How to get updated html data without refreshing the page seleniumWeb element not identified by selenium
Should I have one hand on the throttle during engine ignition?
Inscriptio Labyrinthica
Does 5e follow the Primary Source rule?
How did J. J. Thomson establish the particle nature of the electron?
Why aren't there any women super GMs?
Proof that every field is perfect???
I have found a mistake on someone's code published online: what is the protocol?
Which GPUs to get for Mathematical Optimization (if any)?
Connection between SM an CM
Who would use the word "manky"?
Is it legal for a supermarket to refuse to sell an adult beer if an adult with them doesn’t have their ID?
Apex Legends stuck at 60 FPS (G-Sync 144hz monitor)
🍩🔔🔥Scrambled emoji tale⚛️🎶🛒 #2️⃣
Don't individual signal sources affect each other when using a summing amplifier?
How would you say "Sorry, that was a mistake on my part"?
Do higher dimensions have axes?
How important are the Author's mood and feelings for writing a story?
Why are flying carpets banned while flying brooms are not?
Data visualisation: Pie charts with really small values
Probability that the price of stock following a brownian motion goes under a certain value
Is encryption still applied if you ignore the SSL certificate warning for self-signed certs?
Is it possible to have a career in SciComp without contributing to arms research?
Did Hitler say this quote about homeschooling?
Diagram of Methods to Solve Differential Equations
DOM not refreshing with Selenium (IE, Chrome and VBA)
How to tell if a DOM element is visible in the current viewport?How to find event listeners on a DOM node when debugging or from the JavaScript code?How do I find out which DOM element has the focus?Remove all child elements of a DOM node in JavaScriptHow to Implement DOM Data Binding in JavaScriptHow to return DOM from Selenium after clickHow to initiate click before page is fully loadedCan a website detect when you are using selenium with chromedriver?How to get updated html data without refreshing the page seleniumWeb element not identified by selenium
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I'm using Selenium Basic to collect data from a website and store this into a database. The page I'm scraping is dynamic and loads more information as you scroll. I've been able to address most of this by using the implicit/ explicit waits, etc.
I am capturing all the IDs necessary to create the click action, which opens up another javascript popup for me to collect information there. However, even though I've been able to get these new IDs when the page loads by scrolling, when the app uses that new ID to click, I'm getting an error saying the element cannot be found. This is preventing me from opening up the javascript windows for these newly loaded rows.
When I go to collect this new data, the elements don't exist even though I was able to get the IDs for them.
When I look at the DOM in the browser and page source, all of it is there, so I don't believe it's an issue of letting the browser load.
I've tried utilizing the wait methods (implicit/explicit)...I've even put in hard 60 second waits through the routine. No matter what I do, the routine bombs out after the first 10 rows because it can't find the elements to the data it found after scrolling. I've also tried this using Chrome as well.
Unfortunately, the website needs to be private, so I can't provide the full code. The issue that's happening comes here:
driver.FindElementByXPath("//*[contains(text(),'" & DBA!ParseID & "')]").Click
The error I get is "Element not found for XPath("//*[contains(text(),'ID12345"')]
ParseID is the ID found from parsing elements within the body tag. So, I am able to collect all the IDs after loading all the data, but when it goes to click using the above code, it only works for the initial 10 rows. Everything loaded after that will not work (even though they've been loaded in the Browser for quite some time).
What I should be getting is, say 20 IDs which can create 20 clicks to javascript pop-ups to get more information. However, I am getting 20 IDs but the ability to only click on the first 10, even though I've loaded the entire page.
vba selenium dom
asked Mar 26 at 10:10
RobRob
63 bronze badges
|
show 1 more comment
I'm using Selenium Basic to collect data from a website and store this into a database. The page I'm scraping is dynamic and loads more information as you scroll. I've been able to address most of this by using the implicit/ explicit waits, etc.
I am capturing all the IDs necessary to create the click action, which opens up another javascript popup for me to collect information there. However, even though I've been able to get these new IDs when the page loads by scrolling, when the app uses that new ID to click, I'm getting an error saying the element cannot be found. This is preventing me from opening up the javascript windows for these newly loaded rows.
When I go to collect this new data, the elements don't exist even though I was able to get the IDs for them.
When I look at the DOM in the browser and page source, all of it is there, so I don't believe it's an issue of letting the browser load.
I've tried utilizing the wait methods (implicit/explicit)...I've even put in hard 60 second waits through the routine. No matter what I do, the routine bombs out after the first 10 rows because it can't find the elements to the data it found after scrolling. I've also tried this using Chrome as well.
Unfortunately, the website needs to be private, so I can't provide the full code. The issue that's happening comes here:
driver.FindElementByXPath("//*[contains(text(),'" & DBA!ParseID & "')]").Click
The error I get is "Element not found for XPath("//*[contains(text(),'ID12345"')]
ParseID is the ID found from parsing elements within the body tag. So, I am able to collect all the IDs after loading all the data, but when it goes to click using the above code, it only works for the initial 10 rows. Everything loaded after that will not work (even though they've been loaded in the Browser for quite some time).
What I should be getting is, say 20 IDs which can create 20 clicks to javascript pop-ups to get more information. However, I am getting 20 IDs but the ability to only click on the first 10, even though I've loaded the entire page.
vba selenium dom
asked Mar 26 at 10:10
RobRob
63 bronze badges
Perhaps your code trial//*[contains(text(),'" & DBA!ParseID & "')]and the errorElement not found for XPath("//*[contains(text(),'ID12345"')]doesn't match.
– DebanjanB
Mar 26 at 10:33
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17
|
show 1 more comment
I'm using Selenium Basic to collect data from a website and store this into a database. The page I'm scraping is dynamic and loads more information as you scroll. I've been able to address most of this by using the implicit/ explicit waits, etc.
I am capturing all the IDs necessary to create the click action, which opens up another javascript popup for me to collect information there. However, even though I've been able to get these new IDs when the page loads by scrolling, when the app uses that new ID to click, I'm getting an error saying the element cannot be found. This is preventing me from opening up the javascript windows for these newly loaded rows.
When I go to collect this new data, the elements don't exist even though I was able to get the IDs for them.
When I look at the DOM in the browser and page source, all of it is there, so I don't believe it's an issue of letting the browser load.
I've tried utilizing the wait methods (implicit/explicit)...I've even put in hard 60 second waits through the routine. No matter what I do, the routine bombs out after the first 10 rows because it can't find the elements to the data it found after scrolling. I've also tried this using Chrome as well.
Unfortunately, the website needs to be private, so I can't provide the full code. The issue that's happening comes here:
driver.FindElementByXPath("//*[contains(text(),'" & DBA!ParseID & "')]").Click
The error I get is "Element not found for XPath("//*[contains(text(),'ID12345"')]
ParseID is the ID found from parsing elements within the body tag. So, I am able to collect all the IDs after loading all the data, but when it goes to click using the above code, it only works for the initial 10 rows. Everything loaded after that will not work (even though they've been loaded in the Browser for quite some time).
What I should be getting is, say 20 IDs which can create 20 clicks to javascript pop-ups to get more information. However, I am getting 20 IDs but the ability to only click on the first 10, even though I've loaded the entire page.
vba selenium dom
asked Mar 26 at 10:10
RobRob
63 bronze badges
I'm using Selenium Basic to collect data from a website and store this into a database. The page I'm scraping is dynamic and loads more information as you scroll. I've been able to address most of this by using the implicit/ explicit waits, etc.
I am capturing all the IDs necessary to create the click action, which opens up another javascript popup for me to collect information there. However, even though I've been able to get these new IDs when the page loads by scrolling, when the app uses that new ID to click, I'm getting an error saying the element cannot be found. This is preventing me from opening up the javascript windows for these newly loaded rows.
When I go to collect this new data, the elements don't exist even though I was able to get the IDs for them.
When I look at the DOM in the browser and page source, all of it is there, so I don't believe it's an issue of letting the browser load.
I've tried utilizing the wait methods (implicit/explicit)...I've even put in hard 60 second waits through the routine. No matter what I do, the routine bombs out after the first 10 rows because it can't find the elements to the data it found after scrolling. I've also tried this using Chrome as well.
Unfortunately, the website needs to be private, so I can't provide the full code. The issue that's happening comes here:
driver.FindElementByXPath("//*[contains(text(),'" & DBA!ParseID & "')]").Click
The error I get is "Element not found for XPath("//*[contains(text(),'ID12345"')]
ParseID is the ID found from parsing elements within the body tag. So, I am able to collect all the IDs after loading all the data, but when it goes to click using the above code, it only works for the initial 10 rows. Everything loaded after that will not work (even though they've been loaded in the Browser for quite some time).
What I should be getting is, say 20 IDs which can create 20 clicks to javascript pop-ups to get more information. However, I am getting 20 IDs but the ability to only click on the first 10, even though I've loaded the entire page.
vba selenium dom
vba selenium dom
asked Mar 26 at 10:10
RobRob
63 bronze badges
asked Mar 26 at 10:10
RobRob
63 bronze badges
edited Mar 26 at 17:00
Rob
asked Mar 26 at 10:10
RobRob
63 bronze badges
asked Mar 26 at 10:10
RobRob
63 bronze badges
asked Mar 26 at 10:10
RobRob
63 bronze badges
63 bronze badges
Perhaps your code trial//*[contains(text(),'" & DBA!ParseID & "')]and the errorElement not found for XPath("//*[contains(text(),'ID12345"')]doesn't match.
– DebanjanB
Mar 26 at 10:33
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17
|
show 1 more comment
Perhaps your code trial//*[contains(text(),'" & DBA!ParseID & "')]and the errorElement not found for XPath("//*[contains(text(),'ID12345"')]doesn't match.
– DebanjanB
Mar 26 at 10:33
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17
Perhaps your code trial
//*[contains(text(),'" & DBA!ParseID & "')] and the error Element not found for XPath("//*[contains(text(),'ID12345"')] doesn't match.– DebanjanB
Mar 26 at 10:33
Perhaps your code trial
//*[contains(text(),'" & DBA!ParseID & "')] and the error Element not found for XPath("//*[contains(text(),'ID12345"')] doesn't match.– DebanjanB
Mar 26 at 10:33
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17
|
show 1 more comment
1 Answer
1
active
oldest
votes
This issue hasn't been resolved the way I initially expected, but I've accomplished what I needed through a different and more efficient way.
First, when I researched this further by removing certain IDs in my loop, I noticed that this really didn't have much to do with data updating in the DOM or browser, but rather the ID itself not being found by a (still) unknown reason. It actually seems very arbitrary why it's bombing out. The ID matches the ID in the DOM, but when the string is being moved to the XPath, it can't find it in the DOM. I'm not sure why this would occur unless the string is breaking when being passed somehow, but I'll just let that one remain mysterious until someone smarter comes along!
What I did to accomplish what I needed is loop through the actual class N times, and pull the elements I needed within the classes. Rather than use the ID above as a unique identifier, I used the count of class web elements as the identifier. This worked with 90% less code.
Thank you all!
answered Mar 27 at 14:43
RobRob
63 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55354483%2fdom-not-refreshing-with-selenium-ie-chrome-and-vba%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This issue hasn't been resolved the way I initially expected, but I've accomplished what I needed through a different and more efficient way.
First, when I researched this further by removing certain IDs in my loop, I noticed that this really didn't have much to do with data updating in the DOM or browser, but rather the ID itself not being found by a (still) unknown reason. It actually seems very arbitrary why it's bombing out. The ID matches the ID in the DOM, but when the string is being moved to the XPath, it can't find it in the DOM. I'm not sure why this would occur unless the string is breaking when being passed somehow, but I'll just let that one remain mysterious until someone smarter comes along!
What I did to accomplish what I needed is loop through the actual class N times, and pull the elements I needed within the classes. Rather than use the ID above as a unique identifier, I used the count of class web elements as the identifier. This worked with 90% less code.
Thank you all!
answered Mar 27 at 14:43
RobRob
63 bronze badges
add a comment |
This issue hasn't been resolved the way I initially expected, but I've accomplished what I needed through a different and more efficient way.
First, when I researched this further by removing certain IDs in my loop, I noticed that this really didn't have much to do with data updating in the DOM or browser, but rather the ID itself not being found by a (still) unknown reason. It actually seems very arbitrary why it's bombing out. The ID matches the ID in the DOM, but when the string is being moved to the XPath, it can't find it in the DOM. I'm not sure why this would occur unless the string is breaking when being passed somehow, but I'll just let that one remain mysterious until someone smarter comes along!
What I did to accomplish what I needed is loop through the actual class N times, and pull the elements I needed within the classes. Rather than use the ID above as a unique identifier, I used the count of class web elements as the identifier. This worked with 90% less code.
Thank you all!
answered Mar 27 at 14:43
RobRob
63 bronze badges
add a comment |
This issue hasn't been resolved the way I initially expected, but I've accomplished what I needed through a different and more efficient way.
First, when I researched this further by removing certain IDs in my loop, I noticed that this really didn't have much to do with data updating in the DOM or browser, but rather the ID itself not being found by a (still) unknown reason. It actually seems very arbitrary why it's bombing out. The ID matches the ID in the DOM, but when the string is being moved to the XPath, it can't find it in the DOM. I'm not sure why this would occur unless the string is breaking when being passed somehow, but I'll just let that one remain mysterious until someone smarter comes along!
What I did to accomplish what I needed is loop through the actual class N times, and pull the elements I needed within the classes. Rather than use the ID above as a unique identifier, I used the count of class web elements as the identifier. This worked with 90% less code.
Thank you all!
answered Mar 27 at 14:43
RobRob
63 bronze badges
This issue hasn't been resolved the way I initially expected, but I've accomplished what I needed through a different and more efficient way.
First, when I researched this further by removing certain IDs in my loop, I noticed that this really didn't have much to do with data updating in the DOM or browser, but rather the ID itself not being found by a (still) unknown reason. It actually seems very arbitrary why it's bombing out. The ID matches the ID in the DOM, but when the string is being moved to the XPath, it can't find it in the DOM. I'm not sure why this would occur unless the string is breaking when being passed somehow, but I'll just let that one remain mysterious until someone smarter comes along!
What I did to accomplish what I needed is loop through the actual class N times, and pull the elements I needed within the classes. Rather than use the ID above as a unique identifier, I used the count of class web elements as the identifier. This worked with 90% less code.
Thank you all!
answered Mar 27 at 14:43
RobRob
63 bronze badges
answered Mar 27 at 14:43
RobRob
63 bronze badges
answered Mar 27 at 14:43
RobRob
63 bronze badges
answered Mar 27 at 14:43
RobRob
63 bronze badges
63 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55354483%2fdom-not-refreshing-with-selenium-ie-chrome-and-vba%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Perhaps your code trial
//*[contains(text(),'" & DBA!ParseID & "')]and the errorElement not found for XPath("//*[contains(text(),'ID12345"')]doesn't match.– DebanjanB
Mar 26 at 10:33
Can you provide the url and more of the script?
– QHarr
Mar 26 at 11:10
I did think of that. Triple checked; they match. If only it were that easy!
– Rob
Mar 26 at 11:39
Unfortunately I cannot provide the URL because it has personal information on it. I will post more code once I get to the office. Again, the code seems to work up until the page loads more data. Then it's as is if it never updated the DOM.
– Rob
Mar 26 at 11:40
Is there a viewport where items need to be in focus in order to work?
– QHarr
Mar 26 at 12:17