Two test runs can send the same requests at the same arrival rate and still ask different amounts of work from the system.
One searches a small dataset. The other searches years of accumulated records. The traffic chart looks familiar. The data behind each request has changed.
That is the problem behind my observation that data growth is load. More rows, wider payloads, different cardinality, cache state and index size belong in the workload description.
This is a proposed comparison method, not a report of a completed experiment. Its purpose is to test a specific explanation for a slowdown—and accept the result when that explanation fails.
Choose one data dimension
“Production has more data” is too broad to investigate cleanly. Start with the part of the workload you think changes the work:
- Row count: how much data exists, and how much the selected operation visits.
- Payload width: how much information each request or result carries.
- Cardinality: how many distinct values exist in a field used by the operation.
- Cache state: which data is already available in the caches that matter to the workflow.
- Index size and configuration: how large the indexes are and how they are defined.
These dimensions can move together. Growing a dataset may also grow its indexes. Changing the value distribution may change which records a query matches. Record those relationships rather than claiming an isolation you did not achieve.
A larger dataset does not guarantee a slower workflow. That is a reason to measure the operation, not a reason to assume the outcome.
Keep the comparison interpretable
Use the same application version, environment, request mix, arrival schedule and observation window. Record achieved arrivals, completed requests, errors and timeouts in both runs. A configured request rate alone does not establish that the two runs delivered equivalent work.
Change one chosen data dimension between a baseline dataset and a comparison dataset. Keep the other dimensions as consistent as the question allows. Save how each dataset was built so that the comparison can be repeated.
Choose a cache treatment before running: a documented warm-up procedure, or a deliberately cold comparison with a documented reset. Apply the same procedure to both datasets. It may not produce identical cache occupancy; that difference can be part of the mechanism you are investigating. Record it.
Keep index definitions and database settings fixed for a data-size comparison. Record statistics maintenance and any query-plan changes. If you also add an index or change cache configuration, treat that as another experiment.
Repeat the comparison and vary run order where practical. Otherwise, a result that follows “first run versus later run” can be mistaken for a result that follows dataset size.
Write the prediction before the run
My prediction checklist asks what changes, by how much, under what load, and what might worsen. Apply the same discipline here.
State the expected workflow effect and the resource behavior that would support your explanation. Choose a threshold that matters to the workflow and a measurement window that can test it. Then write what would count against the hypothesis.
For a PostgreSQL investigation, query plans and buffer activity can help examine the database portion of that explanation. The official EXPLAIN guide describes buffer hits and reads. Those observations still need to be connected to the user’s complete workflow; a database observation alone does not explain all of its elapsed time.
Blank comparison worksheet
Fill this out before collecting results. Leave observations blank until the runs finish.
- Workflow and timing boundary: [user action; measured start and end]
- Hypothesis and mechanism: [chosen data dimension; expected resource effect]
- Prediction and disproof: [expected change; meaningful threshold; evidence against it]
- Baseline and comparison: [dataset identifiers; changed dimension; linked changes]
- Controls: [version; environment; arrivals; request mix; duration; cache procedure; index definitions; statistics maintenance]
- Run record: [order; repetitions; achieved arrivals; completions; errors; timeouts]
- Observations: [workflow latency distribution; suspected resource; query plans where relevant; confounds]
- Decision: [supported; not supported; inconclusive; next action and owner]
Let the result change the next step
If the predicted workflow and resource effects appear repeatedly, investigate the mechanism further. The comparison supports a next step; it does not establish that every future dataset will behave the same way.
If you see no meaningful difference, record that null result. If latency changes but the resource evidence contradicts the explanation, revise the explanation. If uncontrolled changes dominate, call the result inconclusive and improve the comparison.
Before raising request rate again, make the data assumptions visible. They belong in the test record alongside traffic.