Future, you must decide how you wish the crawler to handle several Website technologies. There may be an ongoing debate with regard to the intelligence of search engine crawlers. It is not fully crystal clear When they are full-blown headless browsers or just glorified curl scripts (or some thing https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD