Enterprise File Ingestion Pipeline

The challenge

Retrieval systems typically work on day one and fail by day two hundred, when file volume, format variety, and silently failed jobs break the index.

What we built

We rebuilt file ingestion so the portal could take any file type at all, lift the single-file ceiling to 200 MB, and process uploads in the background so a large dataset no longer blocks anyone from using the portal while it lands. A reworked chunking and embedding strategy improved retrieval quality and cut the wait between upload and a file being usable. It now absorbs roughly 2,000 uploads a week for about 3,000 users, holds around 65,000 files, and processes up to 1,000,000 tokens a minute.