The Price Was in the PDF

We were debugging a catalogue question that should have been easy.

Customer question

“What is the price for this size per box?”

The PDF had the answer. The requested size was visible in the table. The price was sitting a few columns away. I could find it in seconds.

The AI answer looked equally straightforward. It mentioned the right size, used the right unit, and returned a neatly formatted price.

It was the wrong price.

The system had found the requested size in one part of the extracted text and a nearby price in another. The language model did a perfectly reasonable job with the evidence it received. Unfortunately, the evidence had already lost the table.

This is the sort of bug that becomes obvious after you see it once. A person reads a table by moving across a row and looking up at the header. Most basic retrieval systems read a stream of text.

250 is not enough information

Suppose an extracted cell contains 250. I cannot tell whether it is a price, quantity, part number, page number, or stock count. I need to know what sits on the left, what appears above it, and which table contains it.

The row path might contain a product family, SKU, and dimensions. The column path might say “Price” and “per 100 pieces.” The value is250.

Keeping this context together helps twice. Retrieval has a precise record to search, and an engineer can inspect the source when an answer looks suspicious.

This second benefit matters a lot. Debugging “the AI gave a wrong answer” is painful. Debugging “this value came from page 14, table 3, row 8, under this header” is ordinary engineering work.

Extraction decides what retrieval can know

It is tempting to treat PDF extraction as plumbing. Get the text out, split it into chunks, and move on to the interesting AI work. Tables make that shortcut expensive.

During ingestion, we need to retain rows, columns, merged cells, headers, spans, and page position. HTML tables bring their own versions of the same problem through rowspan and colspan. Once both sources enter a common table model, we can build different representations for different questions:

I like this approach because it avoids asking one representation to do every job. A vector chunk is useful when someone asks, “What kinds of products are in this catalogue?” A structured record is better when the question contains a part number, two dimensions, and a pricing basis.

Real PDFs refuse to stay neat

The clean example fits on one page. Customer documents rarely cooperate.

A table continues on the next page and drops its title. A PDF parser splits one visual table into three fragments. A small heading sits above the rows it governs. Several product variants share one physical row. Two product families happen to use the same dimensions.

Now a query for M10 × 25 can find several plausible candidates. If a table fragment has lost its parent heading, the nearest price may belong to another family.

Customer questions contain different kinds of words

Consider:

• M10 and 25 identify the required row.
• price tells us which field to return.
• per 100 pieces identifies the pricing basis.

Treating all of them as generic search terms creates strange rankings. A row can score well because it repeats “price” and “100” while missing the requested dimensions.

The query plan needs to separate required constraints, softer descriptions, requested fields, units, and exact identifiers. Required dimensions and identifiers should be enforced before the database trims the candidate list.

That ordering caused one of the more frustrating classes of failure. The application ranking code looked correct, but it never received the correct row. A fast query had already discarded it.

I do not expect one search method to win

Catalogue questions vary too much. An exact lookup is excellent when a customer sends a SKU. Structured row search handles dimensions, variants, quantities, and prices. Keyword search catches literal catalogue language. Vector search helps when the customer describes a product without using the document’s wording.

We run these routes together and let the most specific evidence lead. Source scope is part of the question. If a workflow asks for a particular catalogue and that source is missing, I would rather return a clear gap than quietly search every uploaded file. A polished answer from the wrong catalogue is difficult to detect and even harder to trust.

The first patch is usually too specific

When one catalogue fails, the quickest fix is obvious: add its product names to the prompt or prefer the page that contains the screenshot. I understand the temptation. It makes the demo work again.

It also teaches shared retrieval infrastructure the vocabulary of one customer. The next catalogue has different names, abbreviations, and layouts.

• Required keys occur in the same logical record.
• A heading precedes and governs a set of rows.
• Fields align across table fragments.
• Every value retains its source.
• The requested dimensions match before ranking.

Customer vocabulary still matters. It belongs in that customer’s knowledge and agent instructions. The common retrieval layer should preserve relationships without memorising every product family it encounters.

Rollouts need awkward questions

Retrieval changes are risky because a bad result can still sound sensible. We can build a new logical-record index beside the existing path, run it in shadow mode, and compare the evidence returned by both systems.

Representative tests need to include the annoying cases:

I care more about the retrieved record and its provenance than the final sentence in these tests. Fluent writing can hide bad retrieval. A visible row mismatch gives us something we can fix.

What I took away from this

Querygen answers questions inside the conversations where a business already works. A buyer may ask for a price, pack quantity, specification, or available size and expect a direct reply.

That makes the boring parts of document processing important. The answer begins when we preserve the table. It improves when the query understands the difference between a product constraint and a requested field. It becomes easier to trust when a person can follow the result back to its row, header, unit, and source.