Throughput measures the amount processed per unit of time, for example tokens generated per second across all requests. Latency measures the time of a single request. Batching requests increases overall throughput but can lengthen the latency of each one.
Example
A server produces 400 tokens/s in total, shared among 8 simultaneous requests.