C++ Under the Hood: Money, Latency, and Safety

Taylor Built Solutions Logo

BLUF – Writing an automated trading tool in C++23 isn’t about chasing tricks to save every nanosecond possible. At retail exchange speeds, the things that actually protect your account and your data are unglamorous systems fundamentals. Order sizes, prices parsed from JSON, and P&L all stay in exact fixed-point math, where a double never gets the chance to round them. A timed-out order placement never gets blindly retried. Warm HTTPS connections keep a round trip to Tokyo from costing half a second. And tick-by-tick quotes get batched into TimescaleDB without blocking the WebSocket thread or losing sub-millisecond arrival order.

In this post:

  • Money: Fixed-Point Decimal Math (and Keeping double Out Of JSON)
  • Safety: std::expected, Never Blindly Retrying Orders, and Failing Closed
  • Latency: Keeping Connections Warm (429 ms Down To 129 ms) and Measuring Everything
  • Throughput: Batching Quotes Into TimescaleDB Without Losing Order
  • Conclusion

In the first post in this series, I introduced Kestrel, a C++23 library and CLI tool for pulling stock and crypto market data, backtesting strategies, paper trading, and executing live orders. The full source is on GitHub if you want to follow along in the code. In the second post, I walked through how I used multiple AI coding agents and mechanical C++ guardrails to build it without ending up with a plate of spaghetti.

This time I want to pop the hood and look at the engineering choices that actually move money, talk to exchanges, and write market data to disk. None of these are exotic high-frequency trading tricks—Kestrel talks to retail REST and WebSocket APIs over the public internet—but every single one of them came out of either a real measurement or a failure mode that can quietly corrupt P&L or place an order twice.

As before, nothing here is financial advice. This post is about keeping a trading bot’s bookkeeping honest, not about making it profitable.


Money: Fixed-Point Decimal Math (and Keeping double Out Of JSON)

Anyone who has spent time maintaining payroll systems learns early that binary floating point has no business anywhere near accounting. The IEEE 754 formats behind float and double store numbers in base 2, so most decimal fractions—even a humble 0.1—can only be approximated. 0.1 + 0.2 != 0.3 is a punchline in introductory CS classes, right up until a trading bot sends an order quantity of 0.30000000000000004 to Binance.US and gets bounced with -1111 (too much precision), or tiny rounding errors accumulate across thousands of maker fills and fees until your local position disagrees with the exchange.

In Kestrel, market data ingestion, order sizing and tick/lot rounding, balances, fees, P&L (PnlTracker), and database storage never convert through float or double. Everything in that path uses kestrel::types::Decimal (include/kestrel/types/decimal.hpp and src/types/decimal.cpp).

Side Note: There are two intentional exceptions. MarketMaker‘s volatility-widened spread and its nonlinear skew_power/skew_age adjustments convert to double briefly for fractional powers, then round back with std::llround. Unquoted TOML floats in config files are converted through their shortest round-trip text with std::to_chars before Decimal::Parse sees them.

An int64_t Mantissa With __int128 Scratch Space

Decimal stores two fields: a signed 64-bit integer mantissa (mMantissa) and a base-10 scale (mScale) between 0 and 18, representing mMantissa / 10^mScale. That gives exact representation for crypto quantities down to 18 decimal places as well as US equity prices. Comparisons work on numeric value rather than on the raw fields, so 1.50 == 1.5; the class gets all six comparison operators from == and the three-way <=> operator that arrived in C++20.

The tricky part of fixed-point math in C++ is intermediate overflow. Multiplying a price by a quantity carried at 18 decimal places, or aligning two values to a common scale before adding them, can easily blow past int64_t before you ever get a chance to round the result back down. Because Kestrel targets GCC and Clang on Linux, decimal.cpp leans on __int128 for intermediate scale alignment, multiplication, and division:

__extension__ using i128 = __int128;

constexpr std::array<i128, 39> MakePow10() {
    std::array<i128, 39> t{};
    t[0] = 1;
    for (std::size_t i = 1; i < t.size(); ++i) {
        t[i] = t[i - 1] * 10;
    }
    return t;
}
constexpr auto kPow10 = MakePow10();Code language: HTML, XML (xml)

With that constexpr powers-of-10 table (up to 10^38), every operation computes its exact result in i128 first. What happens next depends on the operation:

  • +, -, and * narrow back to int64_t through one helper, FromWide. When a result needs more than 18 fractional digits or a mantissa wider than int64_t, FromWide drops fractional digits once (rounding to nearest, avoiding double-rounding) until it fits. Only an integer part that won’t fit int64_t throws std::overflow_error.
  • Decimal::Parse goes through the same FromWide narrowing, but catches the overflow and returns it as a kParse error (“value out of range”), since bad input from an exchange is a runtime problem, not a bug.
  • <=> compares the widened i128 values directly without narrowing at all.
  • Operations with a fixed target scale (RoundToIncrement, Divide, widening Rescale, and negating INT64_MIN) check FitsI64 directly and throw std::overflow_error if the scaled mantissa overflows. Kestrel treats that as a programming bug rather than a recoverable runtime error.

Two small details in decimal.cpp are easy to get wrong on a first pass:

  • Floor division on negative numbers: C++’s / and % operators truncate toward zero, not toward −∞. DivRound explicitly normalizes negative remainders (if (r < 0) { r += d; --q; }) before applying Rounding::kDown (toward −∞), Rounding::kUp (toward +∞), or Rounding::kNearest (ties away from zero).
  • Parsing scientific notation and long decimals: Exchanges occasionally return tiny numbers in scientific notation (like 1e-8) or long fractional strings. Decimal::Parse parses directly from std::string_view into i128, tracking a sticky flag if non-zero digits appear past its 2^120 (~36-digit) accumulation limit so rounding stays honest instead of silently truncating.

Side Note: __int128 is a GCC/Clang extension, not standard C++. MSVC doesn’t provide it, so decimal.cpp carries a TODO for a small wide-integer fallback if I ever decide to support native Windows builds outside WSL.

Keeping double Out Of JSON—In Both Directions

Building a Decimal type doesn’t help much if your JSON library rounds every number to a double the moment it parses it—and that is exactly what nlohmann::json::parse() does by default. The JSON spec doesn’t actually require that. It lets parsers limit precision and points out that double precision is what most of them support, so that’s what most of them do. Crypto exchanges usually quote prices as strings ("64210.50"), but Schwab and some metadata endpoints send raw JSON numbers (182.35).

On the inbound side, util::ParseJson (src/util/json.cpp) uses a tiny nlohmann::json_sax<json> subclass, ExactNumberSax, that intercepts number_float and stores the parser’s raw literal text as a JSON string instead of accepting the converted double:

class ExactNumberSax : public nlohmann::json_sax<json> {
   public:
    // ...
    bool number_float(number_float_t, const string_t& literal) override {
        return Add(json(literal));
    }
    // ...
};Code language: HTML, XML (xml)

From there, util::JsonDecimal parses strings via Decimal::Parse and integers directly into Decimal(value, 0).

The same trap exists on the outbound side for Schwab, whose order API expects unquoted JSON numbers for price, stopPrice, and quantity. Putting a fractional value into an nlohmann::json object would pass it through double, so BuildSingleOrderJson (src/trading/schwab_trading_venue.cpp) hand-formats the order payload directly from Decimal::Normalized().ToString() and uses nlohmann::json(schwab_symbol).dump() only to escape the string symbol. A price moves from the exchange’s response text into Decimal, out to order payloads, and into PostgreSQL’s exact NUMERIC columns without ever becoming a double.

Rounding In the Safe Direction Before an Order Hits the Wire

Before any venue sends an order (paper or live), it runs through ApplySymbolRules (src/trading/order_rules.cpp) using the exchange’s tick_size, lot_size, min_order_size, and min_notional:

  • Limit buys round kDown to tick_size (never bid more than the strategy asked).
  • Limit sells round kUp to tick_size (never offer for less than the strategy asked).
  • Stops round kNearest to tick_size.
  • Quantities always round kDown to lot_size (never order more than sized or funded). Orders below min_order_size—and limit orders whose limit_price * quantity falls below min_notional—are rejected locally before spending a network call.

On Binance.US, there is one last wrinkle: the exchange rejects a price or quantity with -1111 if it carries more decimal places than the tick or lot allows, even if the extra digits are trailing zeros. AtScaleOf in binanceus_trading_venue.cpp rescales every formatted number to increment.Normalized().Scale() before putting it in the query string.


Safety: std::expected, Never Blindly Retrying Orders, and Failing Closed

Once your math is exact, the next way a trading system hurts you is ambiguity over the network.

Recoverable Errors As Values (std::expected)

Anywhere I/O or exchange rules can fail, Kestrel returns a types::Result<T>, which is an alias for std::expected<T, types::Error>, the value-or-error type that C++23 added to the standard library. A types::Error carries a structured ErrorCategory (kNetwork, kTimeout, kRateLimit, kAuth, kRejected, kWouldTake, kInsufficientFunds, kNotFound, kParse, kServer, and a few others), along with optional http_status, provider_code, and retry_after fields.

Keeping recoverable errors in the return type makes failure paths explicit at every call site, and lets the HTTP and retry layers distinguish between errors worth backing off from and errors that demand stopping immediately.

A good example is Binance.US rate limits versus IP bans (src/net/cpr_http_client.cpp and src/trading/binanceus_api.cpp). A 429 Too Many Requests maps to kRateLimit, which is transient, so it gets retried after the server’s Retry-After delay. For an IP ban, Binance.US borrows 418—the joke “I’m a teapot” code from RFC 2324—and retrying into a ban only makes it longer. When BinanceUSApi sees a 418, it records the ban window from Retry-After, and BinanceUSApi::CheckBan() then short-circuits every request locally until that window expires. Nothing reaches the exchange until the ban is over.

Why Kestrel Never Blindly Retries a Timed-Out Order

For read-only requests—pulling historical bars, reading open orders, checking balances—retrying on transient failures is a no-brainer. util::Retry (include/kestrel/util/retry.hpp) wraps idempotent calls with exponential backoff (up to 5 attempts, 500 ms initial, 2x multiplier, 30 s cap, ±20% random jitter) whenever error.IsTransient() is true (kNetwork, kTimeout, kRateLimit, or kServer), honoring any server-supplied Retry-After header as a floor (capped at 1 hour).

For POST /api/v3/order, blind retries are a footgun.

Suppose you send a market buy to an exchange and libcurl returns a timeout 15 seconds later—or a proxy drops the connection, or the server returns HTTP 502. You are now in an unknown state. Maybe your packet never reached the matching engine. Or maybe the matching engine filled your order 120 ms after you sent it, and the response got lost on the way back. If your HTTP wrapper blindly retries that POST, you can easily end up buying twice.

Kestrel splits this explicitly at the API layer. BinanceUSApi::Send takes a bool idempotent parameter:

if (!idempotent) {
    return attempt();
}
return util::Retry(mOptions.retry, attempt, mOptions.sleeper);Code language: PHP (php)

Order placements, OCO pairs, and cancels pass idempotent = false, so util::Retry never touches them. (The single automatic resend inside Send is when Binance.US returns error -1021—timestamp outside recvWindow—because the exchange explicitly rejected the request before executing it; Kestrel resyncs its clock offset against /api/v3/time and resends.)

Flowchart of how the Kestrel trading bot handles an order whose outcome is unknown: it sends the order once, looks it up by client order ID after a timeout or server error, and never sends it again.

Instead of retrying a timed-out order, Kestrel reconciles it:

  1. Tag every order up front. BinanceUSTradingVenue attaches a unique newClientOrderId (kst-<8-char random tag>-<monotonic counter>) to every order before sending it.
  2. Distinguish refusals from unknowns. If POST /api/v3/order fails with a definite refusal (insufficient funds, bad price filter, auth error), nothing was placed—return the error directly. If the error says the order may have gone through (Error::MayHaveTakenEffect(): kNetwork, kTimeout, kServer, or a 2xx body that won’t parse, kParse), do not send it again.
  3. Look up the client order ID. Kestrel calls LookupClientOrder (GET /api/v3/order?origClientOrderId=...)—which is an idempotent read—up to twice with a 1-second pause (kLookupPause).
  4. Recover or surface the ambiguity loudly. If the order shows up, Kestrel returns a normal OrderAcknowledgement built from the exchange’s state and life goes on. If it is still not found (or the lookup itself fails), Kestrel returns the original error with the client_order_id and a warning that the order may still exist.

The same rule holds across the other venues, adapted to what each API allows:

  • Kraken (KrakenTradingVenue) assigns a cl_ord_id and checks both OpenOrders and ClosedOrders on an unknown outcome. Kraken only enforces cl_ord_id uniqueness among open orders, so when the caller supplied a custom ID, the ClosedOrders lookup is bounded by time.
  • Charles Schwab (SchwabTradingVenue) doesn’t accept a client order ID on equity orders, so Kestrel has to find the order another way:
    • Before sending the POST, it takes an in-memory snapshot of the order IDs it has already seen (KnownOrdersSnapshot()). That costs no extra HTTP round trip.
    • If the POST hits an unknown outcome—or returns 2xx without an order ID—RecoverPlacedOrder polls recent account orders up to twice, 200 ms apart.
    • It ignores IDs that were already known or claimed by concurrent polls, and looks for a single new order matching the symbol, side, type, time in force, quantity, price, and creation window.
    • If more than one order matches, it logs a warning and refuses to guess.
  • OCO pairs on Binance.US (PlaceOco) guard against half-tracked exits too: if a 2xx response doesn’t yield both legs in orderReports, Kestrel first looks up both legs by their client order IDs (limit_id and stop_id). If either lookup fails—or if an unknown-outcome lookup finds only one leg—it cancels the whole pair via DELETE /api/v3/orderList rather than leaving an untracked leg resting on the book.

Side Note: That list didn’t start out consistent. A cross-model review caught that each venue kept its own idea of what “may have taken effect” meant. Kraken and Schwab already treated an unparseable 2xx as an unknown outcome, but Binance.US returned ParseOrder‘s kParse error as if it were a refusal. Worse, Breakout::PlaceStop (src/strategy/breakout.cpp) treated any OCO error outside its own short list as a refusal and immediately placed a standalone STOP_LOSS sell, which could put a second sell beside an already-resting OCO if a 2xx body came back garbled. The fix (commit 8e227a5) added one predicate, Error::MayHaveTakenEffect(), which all three venues and breakout now share. A garbled 2xx on Binance.US now gets looked up by client order ID like a timeout, and breakout checks GetOpenOrders before placing an unanswered stop, pair, or exit again. Sharing the lookup code itself across venues is still on the list in TODO.md.

A Few More C++ Safety Touches Around Live Orders

Three smaller details in the networking, risk, and shutdown paths turned out to be worth far more than the lines of code they took.

Never follow redirects with API keys. In CprHttpClient::Send, session->SetRedirect(cpr::Redirect{false}) is set unconditionally. Exchange REST endpoints don’t redirect, and libcurl will otherwise forward custom headers (X-MBX-APIKEY, API-Key, API-Sign) across a 3xx redirect to another host.

Closing the Halt() vs. PlaceOrder() race. When the kill switch in RiskGatedVenue::Halt() trips, flipping mHalted (std::atomic<bool>) and calling CancelAll() once isn’t quite enough. An order that checked mHalted a microsecond earlier could still be in flight inside PlaceOrder() or PlaceOco(), landing on the exchange after CancelAll() returns.

Both methods hold mMutex from admission through the venue call. So Halt() cancels all open orders, briefly acquires and releases mMutex as a barrier to drain any in-flight placement, and calls CancelAll() a second time:

mHalted = true;
spdlog::warn("risk: trading halted: {}", reason);
auto cancelled = mVenue->CancelAll();

// An order already inside PlaceOrder read the flag before it was set and
// can still land after the cancel above. PlaceOrder holds the order lock
// from its checks to the venue call, so once the lock is ours that order
// has landed, and any later one sees the halt. Cancel again to catch it.
{ std::lock_guard lock(mMutex); }
auto recancelled = mVenue->CancelAll();Code language: PHP (php)

Even after a halt, Admit() still lets position-reducing exits through.

Signal-safe shutdown without racing open entries. In the first post I mentioned that kestrel_app run cancels open entry orders on Ctrl-C / SIGTERM while leaving protective exits resting on the book. Under the hood, that takes two pieces:

  • Signals: BlockStopSignals() (app/commands/run.cpp) masks SIGINT and SIGTERM with pthread_sigmask before any worker threads spawn. A dedicated StopWatcher thread then collects them synchronously with sigtimedwait, so the mutex-taking Runner::Stop() never executes inside an async signal handler. A second signal calls std::_Exit(130) if you want out immediately.
  • Shutdown order: Runner::End() stops the fill poller, then cancels open entries via EntriesToCancel() (sorting stops first and capping kept exits to actual holdings). It runs one final PollFills() before OnStop(), so an entry that filled mid-cancel still reaches OnFill to place its protective stop.

Latency: Keeping Connections Warm (429 ms Down To 129 ms) and Measuring Everything

Early on I had a hunch that calling Binance.US from my machine in California might be slow enough to matter for faster strategies like market_maker. Rather than guess, I added instrumentation first.

Measuring the Latency of Every HTTP Call (TimedHttpClient)

Every HttpClient built by Kestrel’s factory is wrapped in TimedHttpClient (src/net/timed_http_client.cpp). Each Send() times one individual attempt (so retries show up as separate samples) and records it into a thread-safe LatencyStats keyed by "METHOD url" (or a redacted label).

Two design choices make the metrics practical to keep on all the time:

  • No secrets in endpoint labels: HttpRequest separates url from raw_query (signed parameters) and lets callers supply an optional label for paths containing account hashes or order IDs (as Schwab’s endpoints do). TimedHttpClient only ever logs MethodName(method) + " " + request.Label(), so API signatures and account IDs never leak into logs or database tables.
  • Two views, bounded memory: LatencyStats keeps a whole-run map (Summarize()) logged when a command exits, and an independent window map (TakeWindow()) drained every 60 seconds into PostgreSQL’s venue_latency table by RunRecorder. Each endpoint tracks exact call counts (EndpointLatency::count, stored in SQL column calls), errors, rate_limited (HTTP 429/418), min_ms, and max_ms across every request, plus a 4,096-sample ring buffer for nearest-rank p50, p95, and p99. Any call taking at least [http] slow_request_ms (default 2,000 ms) logs a warning immediately.

What the Numbers Showed—and Why Connection Pooling Cut 300 ms

Once venue_latency started filling up, runs 7 through 11 on Binance.US showed the same thing on every single endpoint: a median round trip around 415 ms.

Digging into where those milliseconds were going turned up two compounding causes:

  1. Binance.US serves its REST API out of AWS Tokyo (ap-northeast-1) behind a CloudFront edge (SFO53, about 15 ms from me in California).
  2. My original CprHttpClient constructed a fresh cpr::Session (and underlying libcurl easy handle) on every single Send(). That meant every REST call tore down and re-established DNS, TCP, and TLS from scratch.

The fix in CprHttpClient (src/net/cpr_http_client.cpp) is a small, thread-safe pool of up to kMaxIdleSessions = 4 idle cpr::Session handles (TakeSession() / ReturnSession()), shared between a provider’s DataSource and TradingVenue by factory.cpp. Each Send() checks out its own session—so the Runner thread and the background fill poller never touch the same libcurl handle concurrently—and returns it when the call finishes.

Kraken is the one venue that still serializes signed calls on top of that pool. Kraken enforces strictly monotonic nonces per API key, so KrakenTradingVenue::SendOnce holds mSendMutex across both nonce assignment and mHttp->Send().

Reusing a cpr::Session has two subtle sharp edges that are easy to miss:

  • Sticky state between calls: A reused cpr::Session remembers options from the previous request. If you send a POST with a JSON body and later reuse that handle for a GET or DELETE without explicitly resetting everything, stale query parameters, headers, or bodies leak into the next call. CprHttpClient::Send overwrites the URL, parameters (explicitly clearing cpr::Parameters{} when raw_query is used), headers, timeout, redirect policy, and calls session->RemoveContent() whenever request.body is empty.
  • Dropping half-dead sockets (while keeping healthy HTTP errors): Normal HTTP 4xx and 5xx responses still return the cpr::Session to mIdle, because the underlying TCP/TLS connection is healthy. Only when r.error fires (a timeout or libcurl transport failure) does Send() return early without calling ReturnSession(), destroying the unique_ptr<cpr::Session> on the spot so a wedged socket is never reused.

To prove the pool actually worked without touching the internet, tests/cpr_http_client_test.cpp spins up LoopbackServer, a tiny in-process server that keeps connections open the way an HTTP/1.1 server does, and verifies that:

  • five sequential requests open exactly one TCP connection;
  • a GET and DELETE after a POST carry none of its body or query;
  • three concurrent threads open at most three connections;
  • a transport failure is reported cleanly without poisoning subsequent calls;
  • redirects are refused rather than forwarding signed headers.

The pooling tests passed 20 repeats under ThreadSanitizer when the pool landed.

Then I tested the old and new clients back-to-back against the live Binance.US API from California on October 29, 2026 using the exact same read-only account --days 30 command (34 signed requests):

MetricFresh cpr::Session per callReused session pool
Median (p50)429 ms129 ms
Minimum214 ms121 ms

Shaving 300 ms off the median was a bigger win than eliminating a local TCP + TLS handshake to SFO53 (~15 ms away) could explain on its own. Keeping my HTTPS connection to CloudFront open appears to keep CloudFront’s upstream connection to Tokyo warm as well—turning a multi-round-trip cold start across the Pacific into a single warm round trip.


Throughput: Batching Quotes Into TimescaleDB Without Losing Order

Recording historical bars is easy—even 1-minute bars across dozens of symbols only trickle in once a minute. Recording live tick-by-tick quotes (kestrel_app stream 3600 --store) is a different animal. Binance.US’s bookTicker feed and Schwab’s level-1 quote streamer fire constantly, often delivering multiple updates for the same symbol in the same millisecond.

Trying to run one synchronous PostgreSQL INSERT per quote inside the WebSocket callback breaks down immediately: the network and disk latency stalls the WebSocket thread, which delays quotes to the strategy or causes the socket buffer to back up.

Decoupling the WebSocket Thread From Database Writes (QuoteWriter)

QuoteWriter (include/kestrel/db/quote_store.hpp and src/db/quote_store.cpp) puts a bounded in-memory buffer and a dedicated writer thread between the WebSocket callbacks and PostgreSQL:

  1. Number and buffer fast on the WebSocket thread: QuoteWriter::Push increments a per-(venue, symbol) 64-bit sequence counter (seq), appends the quote to mBuffer, and returns immediately. If mBuffer hits kMaxBatch (1,000 quotes), it wakes the writer thread early; otherwise the writer thread wakes every 500 ms.
  2. Swap and flush off-thread: QuoteWriter::Flush acquires mMutex just long enough to batch.swap(mBuffer), then releases the lock so incoming quotes never wait on SQL I/O. QuoteStore::Save packs up to 1,000 rows (kRowsPerInsert = 1000 × 12 columns = 12,000 bind parameters, comfortably under PostgreSQL’s 65,535-parameter limit) into a single multi-row parameterized INSERT.
  3. Retry without OOM or duplicate-key death spirals: An early version of QuoteStore::Save had no ON CONFLICT clause. When a flush held more than 1,000 quotes, it split into multiple INSERT statements; if the second statement failed, Flush put the entire batch back in the buffer to retry. On the next flush, the first 1,000 rows—already committed—tripped the unique constraint every single time, wedging the buffer until it overflowed and dropped quotes. Commit 1dce515 added ON CONFLICT (venue, symbol, received_time, seq) DO NOTHING, which made retries idempotent. Failed batches are prepended back in front of newer quotes (preserving chronological order), and total buffered quotes are capped at kMaxBuffered = 100,000 so a long database outage drops the oldest quotes instead of running the machine out of RAM.

Keeping Sub-Millisecond Arrival Order Intact

Kestrel timestamps quotes in UTC milliseconds (std::chrono::sys_time<milliseconds>). That is plenty for bars and strategy timers, but on a busy crypto pair or liquid ETF, several quotes routinely land with the exact same millisecond timestamp.

Migration 3 added a seq BIGINT NOT NULL column to quotes with UNIQUE (venue, symbol, received_time, seq). Inside QuoteWriter::Push, mSeqMutex is held across both incrementing seq and pushing into mBuffer, guaranteeing that sequence order strictly matches arrival order.

Prime() lets a caller (like kestrel_app stream) preload the highest stored seq per symbol before the WebSocket opens, so numbering continues smoothly across restarts without hitting PostgreSQL on the first tick. If an unprimed symbol arrives, Push() falls back to a lookup. When backtests call QuoteStore::Load, the query orders by seq, replaying every tick in the exact order Kestrel originally saw it.

Database Timeouts, Index Trimming, and TimescaleDB Compression

Two final lessons showed up once Kestrel had been recording live sessions for a while.

Bound every database wait. As I mentioned in the last post, Database::Query (src/db/database.cpp) never automatically re-runs a failed SQL statement after a dropped connection. It accepts arbitrary SQL, including non-idempotent inserts like TradeStore::StartRun‘s INSERT INTO runs ... RETURNING run_id, so EnsureConnected() reconnects on the next call instead.

What I hadn’t originally wired up were connection and statement timeouts, which meant a stalled database could hang a live stream thread indefinitely. Commit 1dce515 added explicit libpq bounds:

  • connect_timeout=10
  • TCP keepalives every 10 s after 30 s idle, with 3 probes
  • tcp_user_timeout=60000
  • a configurable statement_timeout, defaulting to 5 minutes—long enough for large backtest quote loads, short enough to break a hung call

Quotes grow fast—trim indexes and compress old chunks. Recording 33 Schwab symbols eats roughly 320 MB per session uncompressed (~10 MB per symbol per session). Two changes in commit 4f478b4 kept that manageable:

  1. Migration 10 dropped two redundant indexes (quotes_venue_symbol_time and TimescaleDB’s default quotes_received_time_idx). Because every quote query in Kestrel filters on (venue, symbol), the remaining quotes_seq_unique (venue, symbol, received_time, seq) and quotes_venue_symbol_seq indexes already served every read. That alone cut quote index size from 1.8 GB down to 1.1 GB.
  2. Automatic TimescaleDB compression: When Kestrel connects to a TimescaleDB community build (timescaledb.license = 'timescale', version 2.11+ so compressed chunks still accept ON CONFLICT upserts), Database::Migrate() enables native columnar compression. For quotes, that means segmenting by venue, symbol, ordering by received_time, seq, and compressing 1-day chunks after 2 days; market_trades (after 14 days) and bars (after 7 days) get the same treatment. Setup runs with SET LOCAL lock_timeout = '5s', so startup waits at most a few seconds behind a long backtest read instead of blocking indefinitely. On Debian/Ubuntu’s Apache-only package, it gracefully skips compression and keeps working.

Side Note: Turning on compression exposed a sneaky startup trap during code review. If QuoteStore::LastSeq() ran an unbounded MAX(seq) for a symbol, TimescaleDB could end up scanning and decompressing weeks of old chunks just to find the last sequence number at startup. LastSeq() now does a two-step lookup. First it grabs the newest received_time via ORDER BY received_time DESC, seq DESC LIMIT 1, which stops at the newest chunk thanks to TimescaleDB’s ordered ChunkAppend. Then it takes MAX(seq) over just the trailing 1-hour window [received_time - 1 hour, received_time]. That window catches small backward clock steps (like an NTP adjustment where a higher seq landed at an earlier millisecond) while still letting TimescaleDB prune every older compressed chunk.


Conclusion

None of what I covered here is flashy. There is no custom kernel networking, no lock-free ring buffer written in inline assembly, and no secret indicator. It is mostly boring systems engineering:

  • Keep double out of your accounting types and out of your JSON boundary.
  • Treat unknown network outcomes as unknown—never blindly retry a non-idempotent write.
  • Measure latency per endpoint before guessing, and don’t throw away warm TLS connections on every call.
  • Decouple hot streaming threads from disk I/O with bounded, idempotent batches and indexes that match your actual queries.

Next up in the series is Integrating Crypto Exchanges and Charles Schwab—what it took to put Binance.US, Kraken, and Charles Schwab behind one DataSource and TradingVenue interface, including Schwab’s 7-day OAuth refresh cycle, US equity market-hours gating, and the places where live exchange behavior disagreed with the official docs.

If you’ve built financial or market-data systems in C++ (or another language), what subtle bug bit you first—floating-point drift, retry semantics, connection pooling, or database growth? Let me know in the comments!

Leave a Reply

Your email address will not be published. Required fields are marked *