Title: GitHub - jina-ai/reader: Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/ URL Source: https://github.com/jina-ai/reader Markdown Content: Reader ------ [](#reader) Your LLMs deserve better input. Reader converts any URL to an **LLM-friendly** input with a simple prefix `https://r.jina.ai/`. Get improved output for your agent and RAG systems at no cost. * Live demo: [https://jina.ai/reader](https://jina.ai/reader) * Or just visit these URLs [https://r.jina.ai/https://github.com/jina-ai/reader](https://r.jina.ai/https://github.com/jina-ai/reader), [https://r.jina.ai/https://x.com/elonmusk](https://r.jina.ai/https://x.com/elonmusk) and see yourself. > Feel free to use Reader API in production. It is free, stable and scalable. We are maintaining it actively as one of the core products of Jina AI. [![Image 1: image](https://private-user-images.githubusercontent.com/2041322/324206629-2067c7a2-c12e-4465-b107-9a16ca178d41.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTUwNTk2NjMsIm5iZiI6MTcxNTA1OTM2MywicGF0aCI6Ii8yMDQxMzIyLzMyNDIwNjYyOS0yMDY3YzdhMi1jMTJlLTQ0NjUtYjEwNy05YTE2Y2ExNzhkNDEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDUwNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA1MDdUMDUyMjQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGMwOGVmZjc1YzNkOTIwMWJjYzkxNDlkZWFlZTI1MmIxMzNlNTU3NjZhNGYwMGI1MGFlNDE4Mjk2ZWRjNDIzYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.vRGc3-3qePL4p5spehOl7T0lO_NsieJDAGqVn6_MpUE)](https://private-user-images.githubusercontent.com/2041322/324206629-2067c7a2-c12e-4465-b107-9a16ca178d41.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTUwNTk2NjMsIm5iZiI6MTcxNTA1OTM2MywicGF0aCI6Ii8yMDQxMzIyLzMyNDIwNjYyOS0yMDY3YzdhMi1jMTJlLTQ0NjUtYjEwNy05YTE2Y2ExNzhkNDEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDUwNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA1MDdUMDUyMjQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGMwOGVmZjc1YzNkOTIwMWJjYzkxNDlkZWFlZTI1MmIxMzNlNTU3NjZhNGYwMGI1MGFlNDE4Mjk2ZWRjNDIzYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.vRGc3-3qePL4p5spehOl7T0lO_NsieJDAGqVn6_MpUE) Updates ------- [](#updates) * **2024-05-03**: We finally resolved a DDoS attack since April 29th. Now our API is much more reliable and scalable than ever! * **2024-04-24**: You now have more fine-grained control over Reader API [using headers](#using-request-headers), e.g. forwarding cookies, using HTTP proxy. * **2024-04-15**: Reader now supports image reading! It captions all images at the specified URL and adds `Image [idx]: [caption]` as an alt tag (if they initially lack one). This enables downstream LLMs to interact with the images in reasoning, summarizing etc. [See example here](https://x.com/JinaAI_/status/1780094402071023926). Usage ----- [](#usage) Simply prepend `https://r.jina.ai/` to any URL. For example, to convert the URL `https://en.wikipedia.org/wiki/Artificial_intelligence` to an LLM-friendly input, use the following URL: [https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial\_intelligence](https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence) All images in that page that lack `alt` tag are auto-captioned by a VLM (vision langauge model) and formatted as `!(Image [idx]: [VLM_caption])[img_URL]`. This should give your downstream text-only LLM _just enough_ hints to include those images into reasoning, selecting, and summarization. ### Streaming mode [](#streaming-mode) Streaming mode is useful when you find that the standard mode provides an incomplete result. This is because the Reader will wait a bit longer until the page is _stablely_ rendered. Use the accept-header to toggle the streaming mode: curl -H "Accept: text/event-stream" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main\_Page The data comes in a stream; each subsequent chunk contains more complete information. **The last chunk should provide the most complete and final result.** If you come from LLMs, please note that it is a different behavior than the LLMs' text-generation streaming. For example, compare these two curl commands below. You can see streaming one gives you complete information at last, whereas standard mode does not. This is because the content loading on this particular site is triggered by some js _after_ the page is fully loaded, and standard mode returns the page "too soon". curl -H 'x-no-cache: true' https://access.redhat.com/security/cve/CVE-2023-45853 curl -H "Accept: text/event-stream" -H 'x-no-cache: true' https://r.jina.ai/https://access.redhat.com/security/cve/CVE-2023-45853 > Note: `-H 'x-no-cache: true'` is used only for demonstration purposes to bypass the cache. Streaming mode is also useful if your downstream LLM/agent system requires immediate content delivery or needs to process data in chunks to interleave I/O and LLM processing times. This allows for quicker access and more efficient data handling: Reader API: streamContent1 ----> streamContent2 ----> streamContent3 ---> ... | | | v | | Your LLM: LLM(streamContent1) | | v | LLM(streamContent2) | v LLM(streamContent3) Note that in terms of completeness: `... > streamContent3 > streamContent2 > streamContent1`, each subsequent chunk contains more complete information. ### Using request headers [](#using-request-headers) As you have already seen above, one can control the behavior of the Reader API using request headers. Here is a complete list of supported headers. * You can ask the Reader API to forward cookies settings via the `x-set-cookie` header. * Note that requests with cookies will not be cached. * You can bypass `readability` filtering via the `x-respond-with` header, specifically: * `x-respond-with: markdown` returns markdown _without_ going through `reability` * `x-respond-with: html` returns `documentElement.outerHTML` * `x-respond-with: text` returns `document.body.innerText` * `x-respond-with: screenshot` returns the URL of the webpage's screenshot * You can specify a proxy server via the `x-proxy-url` header. * You can bypass the cached page (lifetime 300s) via the `x-no-cache` header. ### JSON mode (super early beta) [](#json-mode-super-early-beta) This is still very early and the result is not really a "useful" JSON. It contains three fields `url`, `title` and `content` only. Nonetheless, you can use accept-header to control the output format: curl -H "Accept: application/json" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main\_Page Install ------- [](#install) You will need the following tools to run the project: * Node v18 (The build fails for Node version >18) * Firebase CLI (`npm install -g firebase-tools`) For backend, go to the `backend/functions` directory and install the npm dependencies. git clone git@github.com:jina-ai/reader.git cd backend/functions npm install What is `thinapps-shared` submodule? ------------------------------------ [](#what-is-thinapps-shared-submodule) You might notice a reference to `thinapps-shared` submodule, an internal package we use to share code across our products. While it’s not open-sourced and isn't integral to the Reader's functions, it mainly helps with decorators, logging, secrets management, etc. Feel free to ignore it for now. That said, this is _the single codebase_ behind `https://r.jina.ai`, so everytime we commit here, we will deploy the new version to the `https://r.jina.ai`. Having trouble on some websites? -------------------------------- [](#having-trouble-on-some-websites) Please raise an issue with the URL you are having trouble with. We will look into it and try to fix it. License ------- [](#license) Reader is backed by [Jina AI](https://jina.ai/) and licensed under [Apache-2.0](https://github.com/jina-ai/reader/blob/main/LICENSE).