Why I chose GPUs for a faster, cheaper ingestion pipeline

The small machine was becoming an expensive choice

A small CPU instance looks like a sensible place to save money. The hourly rate is low. The workload runs in the background. Nobody needs an immediate response. Let it take a little longer.

That reasoning feels comfortable until “a little longer” becomes the largest contributor to the bill.

While working on a document ingestion pipeline, I found that generating embeddings was consuming much of the processing time. I evaluated the CPU options, tested the GPU path, and chose GPUs for the embedding stage so the ingestion could progress faster.

The decision was about the cost of completing the work. GPU inference was substantially faster in the tests, and the estimates made it the more economical option for this batch. The hardware cost more per hour; the pipeline needed far fewer hours.

Find the bottleneck before adding workers

An ingestion pipeline does several different kinds of work: download data, extract and parse content, clean it, split it into chunks, generate embeddings, and write the results into storage. Embeddings turn those chunks into numerical representations that support tasks such as semantic search.

In this pipeline, workers could prepare data but then spent a significant amount of time waiting for the CPU embedding service. That made the cost larger than inference alone. The surrounding workers also stayed alive while they waited.

Splitting the dataset into shards gave workers independent portions to process. But every shard still depended on the embedding service. Adding clients would help only if that service had capacity to handle them.

I also checked the scaling behavior. Having autoscaling configured does not guarantee that the requests you send will trigger it. A few clients making sequential, slow requests can behave very differently from the burst of concurrent requests a scaling rule expects. Measuring where work queued was more useful than simply increasing the worker count.

Compare the cost to finish the batch

The comparison below uses deliberately altered figures to illustrate the evaluation. It preserves the important distinction: modest CPU concurrency, higher CPU concurrency, and GPU-accelerated embedding. The GPU result is shown as a range because the exact final bill is not established here.

These are estimated scenarios, not an audited invoice or a controlled benchmark proving that more shards are cheaper. If four times as many identical workers finish in one-quarter of the time, their combined worker-hours remain approximately unchanged. Lower total cost requires another benefit, such as better utilization or less time keeping shared resources running.

Why I went ahead with the GPU path

I kept the existing embedding model and moved its inference onto GPU-backed compute. That let me address the stage limiting progress without turning the ingestion into a wider pipeline rewrite.

The tests showed the embedding stage returning results much faster. My decision was to use that improvement to accelerate the ingestion, then evaluate additional capacity against actual throughput. Faster responses also meant the surrounding workers could spend less time waiting.

For rented compute, the starting calculation is simple: add up each resource’s billing rate multiplied by its billed duration. A faster resource earns its premium when the time reduction outweighs the higher rate. For a pipeline, include the supporting resources whose runtime changes too.

This is the engineering contribution I care about: identifying the limiting stage, comparing workable alternatives, and choosing a configuration that moves the whole job forward at a better estimated cost.

Per-second billing still charges for time

GPU billing needs some precision. Per-second billing does not mean a provider counts completed embeddings or charges only when the GPU is doing useful arithmetic. You are paying for allocated resources over time.

Azure Container Apps, for example, meters allocated GPU-seconds alongside CPU and memory resources. Its serverless GPU apps do not receive the reduced idle rate available to some CPU apps. The opportunity is to complete more useful work during each billed second and then release the capacity.

Shutdown belongs in the cost model. A fast batch followed by hours of unnecessarily running infrastructure can give back the saving. Completion checks and resource cleanup should be part of the job’s lifecycle.

• Microsoft: billing in Azure Container Apps

Other benchmarks show why this is worth testing

Databricks explored the same question in its 2021 article “Are GPUs Really Expensive?” For its small-dataset sentiment-analysis test, a high-end GPU configuration completed inference roughly six times faster than the CPU configurations while its calculated inference cost was lower than either CPU option. Batching and utilization also affected the results.

A separate Databricks Community technical article reported a 37-fold inference speedup and a 16-fold cost reduction for text classification, using batches of 500 rows over a 500,000-row dataset.

These are published results for other workloads, not measurements of my pipeline or current pricing guidance. I would not paste their multipliers into an estimate. They are useful evidence that the higher hourly price alone is a poor reason to dismiss GPUs.

• Databricks: Are GPUs Really Expensive? (2021)
• Databricks Community: zero-shot and few-shot text classification

Embedding is only one place ingestion gets heavy

The other stages deserve their own measurements. These are candidates to investigate, rather than bottlenecks I would assume every pipeline has. A GPU will not automatically speed up an ordinary parser or remove database contention.

Downloading and decompression can repeat the same work in every shard. Parsing can expand compact files into large in-memory objects; document extraction and OCR can add substantial processing. Cleaning, chunking and tokenization can repeat work or generate unnecessary input through excessive overlap.

Writing and indexing can become expensive through small writes, network round trips, contention and index maintenance. Retries can multiply any of those costs, particularly when a restart repeats already completed work.

Once embedding becomes faster, another stage may become the limit. Databricks’ inference guidance discusses separating preprocessing from inference, including using CPUs for ETL and GPUs for inference. Prepared data can also be reused across later runs. Give each stage the resources it can use effectively.

• Databricks: best practices for inference and preprocessing

A second GPU needs to earn its place

After moving inference to GPUs, the next question was how much GPU capacity the pipeline could use. I considered additional capacity against measured progress and the cost of keeping the ingestion workers running.

Under ideal scaling, doubling the GPUs and halving runtime leaves total GPU-hours unchanged. If the same supporting workers finish sooner, their compute cost can fall. But parsing, request concurrency and database writes must keep up. Otherwise, extra GPU capacity may sit underused.

The figure is a hypothetical example of that arithmetic. It is not a claim that our run achieved perfect scaling or that another GPU is free.

Count successful output, then stop the clock

A configuration that processes easy documents quickly but repeatedly fails on long ones has not completed the same job. A useful comparison includes representative long inputs, retries, startup time, output validation and shutdown.

For the next ingestion workload, I would measure preparation time, time waiting for inference, embedding throughput, and write time. Then I would compare cost per successfully ingested document or chunk alongside total completion time, keeping the model and quality requirements consistent.

A lightweight machine remains a good choice for small batches, occasional requests and workloads without effective GPU support. The choice depends on the model, the inputs and the work around it.

For this ingestion, I chose GPUs. Measuring the bottleneck and doing the cost analysis gave me a clear reason to make that call: faster progress and a lower estimated compute bill for the batch. The number I want to optimize is what it costs to get the dataset ready—and how soon we can start using it.