RAG Best Practices - Synthreo Pylon
Configure Knowledge & Retrieval on a Pylon LLM node - choose data structure, tune relevance and chunking, and inspect retrieved context against real test cases.
Purpose
Section titled “Purpose”Configure retrieval for a Pylon DAG’s LLM node and evaluate whether the retrieved material supports the intended answers. Use the test console to inspect retrieval evidence alongside the node’s input and output.
Retrieval supplies relevant source material as context; it does not guarantee a correct answer or retrain the underlying language model.
Knowledge & Retrieval
Section titled “Knowledge & Retrieval”Open Knowledge & Retrieval in the LLM node configuration.
Data structure
Section titled “Data structure”| Control | Use |
|---|---|
| How is your data structured? | Choose Documents (free-form text) or Q&A pairs to match the source |
| Text column | Identify the text column for document data |
| Question column | Identify the question column for Q&A data |
| Answer column | Identify the answer column for Q&A data |
Choose the structure that describes the source you actually supply. A collection of narrative manuals is different from explicit question-and-answer records.
Retrieval and answer controls
Section titled “Retrieval and answer controls”| Control | What to inspect while testing |
|---|---|
| Results to use | Whether the retrieved context is sufficient and relevant |
| Minimum relevance | Under Advanced; compare retained and excluded material against representative questions |
| Re-rank results for accuracy | Compare retrieval quality with and without re-ranking |
| Hybrid keyword + meaning search | Check cases that depend on exact terms as well as semantic similarity |
| Answer only from knowledge | Test answerable questions and questions the supplied knowledge cannot answer |
| Cite sources | Check that citations correspond to the supporting material |
| Sources column name | Identify the configured source output |
| Max sources to cite | Check the number of cited sources against the answer’s needs |
These are the current labels. The editor does not expose Training Settings or Inference Settings tabs, or selectable Full Training, Rebuild Embeddings, Rebuild Index Only and Fetch Data Only modes.
Chunking
Section titled “Chunking”Chunking offers Automatic (recommended) and Fixed size. The section also includes Chunk size and Chunk overlap.
Use representative source passages to evaluate whether chunks preserve enough context. Do not assume a universal chunk size, relevance threshold or result count works for every corpus.
Example
Section titled “Example”For a policy-manual corpus, select Documents (free-form text) and the appropriate Text column. Start with the displayed defaults and record them.
Prepare test questions with expected supporting passages, including a question whose answer is absent from the corpus. Select Run test, open Steps, and inspect the LLM node’s Input, Output and RAG selection items where present.
Compare the retrieved passages with the expected evidence. Change one relevant setting, run the same questions again, and record what improved or regressed. A better final answer alone does not establish that retrieval improved.
Preparing the source
Section titled “Preparing the source”- Keep documents current and identify their owners.
- Remove repetitive material that adds no useful context.
- Use descriptive headings and keep related information together.
- Represent Q&A data as explicit questions and answers.
- Check whether information in tables or images is available in the material the retrieval process receives.
- Include difficult and unanswerable cases in the evaluation set.
If it didn’t work
Section titled “If it didn’t work”- The answer is absent: check the source itself before changing retrieval settings.
- Relevant passages are not retrieved: inspect the selected data structure and columns, then test relevance and retrieval settings.
- Context is irrelevant or incomplete: compare the retrieved material with expected passages and review chunking.
- Responses are slow or context is excessive: measure retrieved context and run behaviour before reducing results or changing the model.
- The answer remains wrong despite useful context: inspect the node’s prompt and response configuration as well as retrieval.