Odoo support RAG: which retrieval should ship?
72 questions over the Odoo 18 Accounting and Inventory docs (1,022 chunks). Same model and judge in every run; only retrieval changes. hybrid-repeat is the hybrid config run a second time to show run-to-run noise.
Runs
| Metric | dense | bm25 | hybrid | dense-rerank | hybrid-repeat |
|---|---|---|---|---|---|
| Accuracy | 75.0% | 87.5% | 87.5% | 88.9% | 88.9% |
| Accuracy on answerable | 70.0% | 85.0% | 85.0% | 86.7% | 86.7% |
| Wrong answers shown | 12.5% | 12.1% | 7.3% | 3.7% | 5.5% |
| Over-escalation | 20.0% | 3.3% | 8.3% | 10.0% | 8.3% |
| Out-of-scope escalated | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Source doc retrieved | 78.3% | 90.0% | 90.0% | 90.0% | 90.0% |
| MRR (source chunk) | 0.537 | 0.657 | 0.674 | 0.613 | 0.674 |
| Cost per question | $0.00029 | $0.00032 | $0.00031 | $0.00032 | $0.00031 |
| Latency p95 | 2.41 s | 2.04 s | 2.54 s | 8.31 s | 2.34 s |
Wrong answers shown = answers delivered to the user that the judge marked partial or incorrect, over all delivered answers. Over-escalation = answerable questions sent to a human.
Configurations
| dense | dense@5 |
| bm25 | bm25@5 |
| hybrid | hybrid@5 |
| dense-rerank | dense+rerank@5 |
| hybrid-repeat | hybrid@5 |
Should it ship?
dense → hybrid SHIP: SIGNIFICANT IMPROVEMENT
accuracy +12.5 pts · 95% CI +2.8 to +22.2 · only dense right: 2 · only hybrid right: 11 · McNemar p = 0.0225
13 questions changed outcome
- When does Odoo automatically recalculate taxes on a quote or invoice?
dense: partial hybrid: correct The answer lists all the trigger points from the reference, including email send, portal views, quote confirmation, invoice validation, subscription invoicing, and eCommerce checkout, with no contradictions. - How can I see extra columns like expiration date or last count date on the inventory adjustments screen that aren't showing up?
dense: escalated hybrid: correct The answer matches the reference by stating columns are hidden by default and instructing to use the column toggle/reveal option and select the checkbox for the desired column. - What counts as an expense versus income in Odoo's accounting setup?
dense: escalated hybrid: correct The answer matches the reference definitions and examples for both expense and income without contradiction. - How can I make Odoo reserve stock a few days ahead of the scheduled delivery date instead of right at it?
dense: incorrect hybrid: correct The answer matches the reference by specifying the Operations Types path, setting Reservation Method to Before scheduled date, and using the Reserve before scheduled date field for days before and starred transfers. - How do I put products into a package during a warehouse transfer like a receipt or delivery order?
dense: escalated hybrid: correct The answer matches the reference steps for detailed operations and Put in Pack, adding only consistent extra detail. - How do I control whether my warehouse staff can create new lot or serial numbers during a transfer, or only pick from ones that already exist?
dense: partial hybrid: correct The answer covers enabling Lots & Serial Numbers in settings, opening the operation type, and using the Create New and Use Existing ones checkboxes, matching the reference. - How can I let my customers pay their invoices instantly by scanning a barcode with their banking app?
dense: partial hybrid: correct The answer matches the reference by explaining EPC QR codes, SEPA credit transfer, instant invoice payment, and the country availability. - How can I set up a product so that Odoo only orders or produces it after a customer actually places an order, instead of keeping stock on hand?
dense: escalated hybrid: correct The answer correctly identifies the Make to Order route, explains that procurement/production triggers only after sales order confirmation, notes the automatic linking of sales order to generated RFQ/manufacturing order, and matches the reference's suitability criteria. - How can I sell a product in individual units if I buy it from my supplier in large packs?
dense: escalated hybrid: correct The answer states the key fact that Odoo supports different units of measure for one product, enabling buying in packs and selling in individual units, and adds consistent detail about UoM conversion. - How do I renumber a batch of vendor bills so they follow a continuous sequence after importing them from my old accounting system?
dense: correct hybrid: escalated escalated an answerable question - How do I make Odoo reserve whatever stock is available for an order instead of only reserving complete packages?
dense: escalated hybrid: correct The answer includes all key steps from the reference, including the path, the Reserve Partial Packagings setting, and the prerequisite of enabling Product Packaging, with no contradictions. - How can I see the value of each lot or serial number in my inventory?
dense: correct hybrid: incorrect The answer describes a different path (Products > Lots/Serial Numbers) instead of the reference's Inventory > Reporting > Valuation with Group By Lot/Serial number, and does not include the key grouping/expand steps. - How do I create a purchase order in Odoo and confirm it?
dense: escalated hybrid: correct The answer matches the reference steps for creating and confirming a purchase order, with only minor extra detail about the Receipt smart button.
dense → bm25 SHIP ALLOWED: IMPROVEMENT NOT PROVEN
accuracy +12.5 pts · 95% CI +1.4 to +23.6 · only dense right: 5 · only bm25 right: 14 · McNemar p = 0.0636
19 questions changed outcome
- How can I make Odoo pick the smallest number of packages when fulfilling an order for a product?
dense: correct bm25: incorrect The answer gives the correct setup steps but describes the strategy's behavior incorrectly, saying it prioritizes an already-opened package, whereas the reference states it pulls from an unopened package rather than a partially used one. - When does Odoo automatically recalculate taxes on a quote or invoice?
dense: partial bm25: correct The answer lists all the trigger points from the reference, including the additional portal-view triggers, with no contradictions. - How does Odoo prove that my posted accounting entries haven't been tampered with?
dense: partial bm25: correct The answer covers SHA-256 hashing of essential fields, determinism, and hash chaining preventing insertion/modification, matching the reference's key facts. - How do I set up a tax so that part of the tax amount goes to a different account instead of the full amount being booked as tax payable?
dense: escalated bm25: correct The answer correctly describes using Distribution for Invoices/Refunds with % of tax lines totaling 100% and setting each line's Account to route portions of the tax, matching the reference's key facts. - How can I see extra columns like expiration date or last count date on the inventory adjustments screen that aren't showing up?
dense: escalated bm25: correct The answer states the columns are hidden by default and instructs clicking to reveal them by selecting the checkbox, matching the reference's key steps. - How do I set up a new product so Odoo knows whether it's a physical item or a service I'm selling?
dense: correct bm25: partial The answer gives the correct navigation and mentions the Product Type field, but it only instructs selecting Service and omits that physical goods are also set up as products with the appropriate type, missing the key fact about distinguishing goods vs. services. - What counts as an expense versus income in Odoo's accounting setup?
dense: escalated bm25: correct The answer matches the reference definitions and examples for both expense and income without contradiction. - How can I make Odoo reserve stock a few days ahead of the scheduled delivery date instead of right at it?
dense: incorrect bm25: correct The answer includes the correct navigation path, the Reservation Method setting, and the Reserve before scheduled date field with days before and starred options, matching the reference. - How do I put products into a package during a warehouse transfer like a receipt or delivery order?
dense: escalated bm25: correct The answer matches the reference by describing the Operations tab detailed operations icon, Add a line with Destination Package, Done quantity, Confirm, and the Put in Pack alternative. - How do I control whether my warehouse staff can create new lot or serial numbers during a transfer, or only pick from ones that already exist?
dense: partial bm25: correct The answer covers enabling Lots & Serial Numbers in settings and the Create New / Use Existing ones options on the operation type, matching the reference steps. - How can I let my customers pay their invoices instantly by scanning a barcode with their banking app?
dense: partial bm25: correct The answer includes the key facts from the reference: enabling EPC QR codes, scanning with mobile banking apps to initiate SEPA credit transfers for instant invoice payment, and the country availability restriction. - How can I set up Odoo so my customers can create vendor bills on my behalf instead of me sending them invoices?
dense: correct bm25: escalated escalated an answerable question - How can I set up a product so that Odoo only orders or produces it after a customer actually places an order, instead of keeping stock on hand?
dense: escalated bm25: correct The answer correctly identifies the Make to Order route and explains that replenishment is triggered on sales order confirmation, matching the reference's key facts. - How can I sell a product in individual units if I buy it from my supplier in large packs?
dense: escalated bm25: correct The answer explains enabling Units of Measure, creating a larger unit in the same category with a ratio, and confirms this supports buying in packs and selling individually, matching the reference. - How can I make sure my invoices meet Argentina's electronic invoicing rules when I bill customers there?
dense: correct bm25: partial The answer mentions AFIP, CAE, and Odoo compatibility but omits the specific supported formats (FE, XML with CAE, digitally signed PDF with QR codes), ten-year tamper-proof archiving, and automatic VAT calculation that the reference requires. - How can I make sure the oldest stock gets picked first when fulfilling orders, so older items don't sit around too long?
dense: escalated bm25: correct The answer correctly identifies FIFO as the removal strategy that reserves the oldest lots first, matching the reference's key fact and example. - How do I make Odoo reserve whatever stock is available for an order instead of only reserving complete packages?
dense: escalated bm25: correct The answer includes the correct navigation path, the Reserve Partial Packagings setting, and the prerequisite of enabling Product Packaging, matching the reference. - How can I see the value of each lot or serial number in my inventory?
dense: correct bm25: partial The answer gives a different navigation path (Products > Lots/Serial Numbers) instead of the reference's Inventory > Reporting > Valuation with Group By Lot/Serial number, though it does mention the valuation report and Valuation smart button. - How do I create a purchase order in Odoo and confirm it?
dense: escalated bm25: correct The answer matches the reference steps: New RfQ, add vendor, add products, Confirm Order, and the Receipt smart button appears.
hybrid → dense-rerank SHIP ALLOWED: IMPROVEMENT NOT PROVEN
accuracy +1.4 pts · 95% CI -5.6 to +8.3 · only hybrid right: 3 · only dense-rerank right: 4 · McNemar p = 1.0000
7 questions changed outcome
- How do I purchase more credits for digitizing documents in Odoo?
hybrid: partial dense-rerank: correct The answer includes the correct navigation path and the Buy credits action, matching the reference; it omits the XML/OCR note but that is supplementary detail. - When does Odoo automatically recalculate taxes on a quote or invoice?
hybrid: correct dense-rerank: partial The answer only mentions confirming documents and manual recalculation, missing the specific automatic trigger points (email send, portal view, sales order confirmation, invoice validation, subscription invoicing, eCommerce checkout) from the reference. - What counts as an expense versus income in Odoo's accounting setup?
hybrid: correct dense-rerank: escalated escalated an answerable question - How can I make Odoo reserve stock a few days ahead of the scheduled delivery date instead of right at it?
hybrid: correct dense-rerank: partial The answer mentions the 'Before scheduled date' reservation method but omits the key configuration path (Inventory > Configuration > Operations Types > General tab) and the field to set the number of days before the scheduled date. - How do I set up my products so that they get picked based on how close they are to expiring, and where do I configure the days for that?
hybrid: escalated dense-rerank: correct The answer covers enabling the features, setting tracking and expiration date, and explains the expiration/removal date day calculations, matching the reference. - How can I make sure the oldest stock gets picked first when fulfilling orders, so older items don't sit around too long?
hybrid: incorrect dense-rerank: correct The answer correctly identifies FIFO as the strategy that picks the oldest stock first, matching the reference's key fact. - How can I see the value of each lot or serial number in my inventory?
hybrid: incorrect dense-rerank: correct The answer includes the reference's exact steps (Inventory > Reporting > Valuation, search bar, Group By Lot/Serial number) plus additional valid detail, with no contradictions.
hybrid → hybrid-repeat SHIP ALLOWED: IMPROVEMENT NOT PROVEN
accuracy +1.4 pts · 95% CI +0.0 to +4.2 · only hybrid right: 0 · only hybrid-repeat right: 1 · McNemar p = 1.0000
1 questions changed outcome
- How do I purchase more credits for digitizing documents in Odoo?
hybrid: partial hybrid-repeat: correct The answer includes the correct navigation path and the Buy credits action, matching the reference's key steps, though it omits the note about XML files not consuming credits.