Table of Contents

Maximizing Request Throughput to Third-Party APIs: A Practical Testing Approach
#

A community member recently asked an interesting question: “How can I send requests to third-party APIs as quickly as possible, without worrying about overwhelming their servers? What’s the best way to develop and validate this approach?”

Here’s a comprehensive strategy that I’ve found works really well in practice:

The Game Plan
#

1. Go Async or Go Home You’ll definitely want to use WebClient for its asynchronous, non-blocking I/O capabilities. Alternatively, frameworks like Vert.x are excellent choices. I’d hold off on virtual threads for now - while they’re exciting, they’re not quite production-ready yet.

2. Isolate Your Testing Environment Here’s the key insight: if you only care about your code’s performance and want to maximize pressure on the target API, never test directly against the third-party interface during development. You need to isolate your code first and ensure it performs as expected.

Why? The response time has way too many variables and instabilities. Think about it - network bandwidth between you and the API, your network card performance, potential rate limiting on their end, and if you don’t limit connection counts, CDNs might just block you entirely. Plus, even with non-blocking requests, if you fire off 10,000 simultaneous requests, many will just queue up in the network layer anyway.

3. Test Locally with Realistic Simulations For testing your own code while simulating realistic delays, I typically run tests locally. My go-to approach uses TestContainers with an httpbin image (kennethreitz/httpbin:latest). For your specific scenario, you can add timing to each request and call the /anything endpoint to collect responses - this endpoint simply echoes back all the parameters you send.

Want to simulate bandwidth constraints? Add a toxicproxy image and route your httpbin calls through it for bandwidth throttling. Need to simulate API delays? Use the /delay/0.1 endpoint (for 100ms delays).

Code Example
#

Here’s a practical example (quick test setup, not fine-tuned, just to demonstrate the testing approach). First, let’s create a TestContainer base class for reusability:

import eu.rekawek.toxiproxy.Proxy;
import eu.rekawek.toxiproxy.ToxiproxyClient;
import eu.rekawek.toxiproxy.model.ToxicDirection;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.Network;
import org.testcontainers.containers.ToxiproxyContainer;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.io.IOException;

@Testcontainers
public class CommonMicroServiceTest {

    private static final Network network = Network.newNetwork();

    private static final String HTTPBIN = "httpbin";
    public static final int HTTPBIN_PORT = 80;
    public static final GenericContainer<?> HTTPBIN_CONTAINER
            = new GenericContainer<>("kennethreitz/httpbin:latest")
            .withExposedPorts(HTTPBIN_PORT)
            .withNetwork(network)
            .withNetworkAliases(HTTPBIN);
    /**
     * <a href="https://java.testcontainers.org/modules/toxiproxy/">toxiproxy</a>
     * Using toxiproxy to wrap httpbin
     * Can simulate network failures and other conditions with toxiproxy
     * Available port range: 8666～8697
     */
    private static final ToxiproxyContainer TOXIPROXY_CONTAINER = new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.5.0")
            .withNetwork(network);

    private static final int GOOD_HTTPBIN_PROXY_PORT = 8666;
    private static final int READ_TIMEOUT_HTTPBIN_PROXY_PORT = 8667;
    private static final int RESET_PEER_HTTPBIN_PROXY_PORT = 8668;

    public static final String GOOD_HOST;
    public static final int GOOD_PORT;
    /**
     * Represents scenarios where requests reach the server but timeout or can't respond (e.g., server restart)
     */
    public static final String READ_TIMEOUT_HOST;
    public static final int READ_TIMEOUT_PORT;
    public static final String RESET_PEER_HOST;
    public static final int RESET_PEER_PORT;

    /**
     * Represents scenarios where requests never leave, TCP connections can't be established
     */
    public static final String CONNECT_TIMEOUT_HOST = "localhost";
    /**
     * Port 1 is guaranteed to be unreachable
     */
    public static final int CONNECT_TIMEOUT_PORT = 1;


    static {
        //Not using @Container annotation for lifecycle management since we need to generate proxies in static block
        //No worries about container cleanup - testcontainers starts a ryuk container to monitor and close all containers
        HTTPBIN_CONTAINER.start();
        TOXIPROXY_CONTAINER.start();
        final ToxiproxyClient toxiproxyClient = new ToxiproxyClient(TOXIPROXY_CONTAINER.getHost(), TOXIPROXY_CONTAINER.getControlPort());
        try {
            Proxy proxy = toxiproxyClient.createProxy("good", "0.0.0.0:" + GOOD_HTTPBIN_PROXY_PORT, HTTPBIN + ":" + HTTPBIN_PORT);
            //Disable traffic flow, will cause READ TIMEOUT
            proxy = toxiproxyClient.createProxy("read_timeout", "0.0.0.0:" + READ_TIMEOUT_HTTPBIN_PROXY_PORT, HTTPBIN + ":" + HTTPBIN_PORT);
            proxy.toxics().bandwidth("UP_DISABLE", ToxicDirection.UPSTREAM, 0);
            proxy.toxics().bandwidth("DOWN_DISABLE", ToxicDirection.DOWNSTREAM, 0);
            proxy = toxiproxyClient.createProxy("connect_timeout", "0.0.0.0:" + RESET_PEER_HTTPBIN_PROXY_PORT, HTTPBIN + ":" + HTTPBIN_PORT);
            proxy.toxics().resetPeer("UP_SLOW_CLOSE", ToxicDirection.UPSTREAM, 1);
            proxy.toxics().resetPeer("DOWN_SLOW_CLOSE", ToxicDirection.DOWNSTREAM, 1);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        GOOD_HOST = TOXIPROXY_CONTAINER.getHost();
        GOOD_PORT = TOXIPROXY_CONTAINER.getMappedPort(GOOD_HTTPBIN_PROXY_PORT);
        READ_TIMEOUT_HOST = TOXIPROXY_CONTAINER.getHost();
        READ_TIMEOUT_PORT = TOXIPROXY_CONTAINER.getMappedPort(READ_TIMEOUT_HTTPBIN_PROXY_PORT);
        RESET_PEER_HOST = TOXIPROXY_CONTAINER.getHost();
        RESET_PEER_PORT = TOXIPROXY_CONTAINER.getMappedPort(RESET_PEER_HTTPBIN_PROXY_PORT);
    }
}

And here’s the actual test code:

@Test
public void test() {
    // Create a custom connection provider
    ConnectionProvider provider = ConnectionProvider.builder("customConnectionProvider")
            .maxConnections(100) // Increase max connections, but not too high to avoid CDN DDoS detection
            .pendingAcquireMaxCount(10000) // Increase waiting queue size
            .build();

    HttpClient httpClient = HttpClient.create(provider)
            .responseTimeout(Duration.ofMillis(100000)); // Response timeout
    WebClient build = WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).build();
    List<Mono<String>> monos = Lists.newArrayList();
    for (int i = 0; i < 10000; i++) {
        //Using TestContainer port to simulate 0.1s delay
        Mono<String> stringMono = build.get().uri("http://localhost:" +
                CommonMicroServiceTest.HTTPBIN_CONTAINER.getMappedPort(HTTPBIN_PORT) + "/delay/0.1")
                .retrieve().bodyToMono(String.class);
        monos.add(stringMono);
    }
    long start = System.currentTimeMillis();
    String block = Mono.zip(monos, objects -> {
        log.info("{}", objects);
        return "ok";
    }).block();
    log.info("block: {} in {}ms", block, System.currentTimeMillis() - start);
}

Test Results: block: ok in 10362ms

This aligns perfectly with our expectations:

10,000 requests, each with 0.1s delay, connection pool of 100
Total time ≈ 0.1 × 10000/100 = 10s

Pro Tips
#

I regularly use toxicproxy to simulate various failure scenarios: server disconnections, requests reaching the server but getting no response, requests that never reach the server, partial request transmission failures, and more. This toolkit is incredibly valuable when building robust microservice infrastructure!

The beauty of this approach is that you get complete control over your testing environment while still maintaining realistic conditions. You can push your code to its limits without worrying about external factors, then confidently deploy knowing exactly how your system will behave under pressure.