This issue can often show up when you are using an HTML parser like BeautifulSoup or lxml instead of a browser engine via Selenium or Puppeteer. The data you are seeing in the browser might be getting generated via client-side JavaScript after the page load. BeautifulSoup, lxml, and similar HTML parsing libraries do not execute JavaScript.
There are two options to solve this issue:
- Use a browser automation framework like Selenium or Puppeteer and execute the JavaScript before attempting data extraction
- Search for required data in the
<script>tags. Most of the time, the required data is hidden inside<script>tags as JavaScript variables and then rendered on the page after the page load
