• TychoOrdo@kbin.social
    link
    fedilink
    arrow-up
    2
    ·
    1 year ago

    Works on a lot of news sites. I think they let web crawlers get the full text in order to improve their search ranking, however it also lets in crawlers with other purposes like archiving.