Include links on scraping

[mertdeveci5]

I read that this might be a feature request hence sharing here if someone figured it out.

On using extract, I use include_links=True. However the links in the website are not scraped for some reason. Not sure if I am using this in the wrong way so would appreciate anyone pointing me into the right direction.

Example:

# import the necessary functions
from trafilatura import fetch_url, extract, sitemaps
from rich import print as rprint
# grab a HTML file to extract data from

URL = "https://jam.dev/careers"
downloaded = fetch_url(URL)

sitemap = sitemaps.sitemap_search(URL)

# output main content and comments as plain text
result = extract(downloaded)

# change the output format to XML (allowing for preservation of document structure)
result = extract(downloaded, include_links=True, output_format="xml")

# discard potential comment and change the output to JSON
extract(downloaded, output_format="json", include_comments=False)

rprint(sitemap)
rprint(result)

Here most of the text is scraped EXCEPT the part where job listings are listed. It is critical to get this content though.

[adbar]

This is now addressed in #518, see the issue for further comments.