BLUF – Writing an automated trading tool in C++23 isn’t about chasing tricks to save every nanosecond possible. At retail exchange speeds, the things that actually protect your account and your data are unglamorous systems fundamentals. Order sizes, prices parsed from JSON, and P&L all stay in exact fixed-point math, where a double never gets the chance to round them. A timed-out order placement never gets blindly retried. Warm HTTPS connections keep a round trip to Tokyo from costing half a second. And tick-by-tick quotes get batched into TimescaleDB without blocking the WebSocket thread or losing sub-millisecond arrival order.
In this post:
- Money: Fixed-Point
DecimalMath (and KeepingdoubleOut Of JSON) - Safety:
std::expected, Never Blindly Retrying Orders, and Failing Closed - Latency: Keeping Connections Warm (429 ms Down To 129 ms) and Measuring Everything
- Throughput: Batching Quotes Into TimescaleDB Without Losing Order
- Conclusion
In the first post in this series, I introduced Kestrel, a C++23 library and CLI tool for pulling stock and crypto market data, backtesting strategies, paper trading, and executing live orders. The full source is on GitHub if you want to follow along in the code. In the second post, I walked through how I used multiple AI coding agents and mechanical C++ guardrails to build it without ending up with a plate of spaghetti.
This time I want to pop the hood and look at the engineering choices that actually move money, talk to exchanges, and write market data to disk. None of these are exotic high-frequency trading tricks—Kestrel talks to retail REST and WebSocket APIs over the public internet—but every single one of them came out of either a real measurement or a failure mode that can quietly corrupt P&L or place an order twice.
As before, nothing here is financial advice. This post is about keeping a trading bot’s bookkeeping honest, not about making it profitable.
Money: Fixed-Point Decimal Math (and Keeping double Out Of JSON)
Anyone who has spent time maintaining payroll systems learns early that binary floating point has no business anywhere near accounting. The IEEE 754 formats behind float and double store numbers in base 2, so most decimal fractions—even a humble 0.1—can only be approximated. 0.1 + 0.2 != 0.3 is a punchline in introductory CS classes, right up until a trading bot sends an order quantity of 0.30000000000000004 to Binance.US and gets bounced with -1111 (too much precision), or tiny rounding errors accumulate across thousands of maker fills and fees until your local position disagrees with the exchange.
In Kestrel, market data ingestion, order sizing and tick/lot rounding, balances, fees, P&L (PnlTracker), and database storage never convert through float or double. Everything in that path uses kestrel::types::Decimal (include/kestrel/types/decimal.hpp and src/types/decimal.cpp).
Side Note: There are two intentional exceptions. MarketMaker‘s volatility-widened spread and its nonlinear skew_power/skew_age adjustments convert to double briefly for fractional powers, then round back with std::llround. Unquoted TOML floats in config files are converted through their shortest round-trip text with std::to_chars before Decimal::Parse sees them.
An int64_t Mantissa With __int128 Scratch Space
Decimal stores two fields: a signed 64-bit integer mantissa (mMantissa) and a base-10 scale (mScale) between 0 and 18, representing mMantissa / 10^mScale. That gives exact representation for crypto quantities down to 18 decimal places as well as US equity prices. Comparisons work on numeric value rather than on the raw fields, so 1.50 == 1.5; the class gets all six comparison operators from == and the three-way <=> operator that arrived in C++20.
The tricky part of fixed-point math in C++ is intermediate overflow. Multiplying a price by a quantity carried at 18 decimal places, or aligning two values to a common scale before adding them, can easily blow past int64_t before you ever get a chance to round the result back down. Because Kestrel targets GCC and Clang on Linux, decimal.cpp leans on __int128 for intermediate scale alignment, multiplication, and division:
__extension__ using i128 = __int128;
constexpr std::array<i128, 39> MakePow10() {
std::array<i128, 39> t{};
t[0] = 1;
for (std::size_t i = 1; i < t.size(); ++i) {
t[i] = t[i - 1] * 10;
}
return t;
}
constexpr auto kPow10 = MakePow10();Code language: HTML, XML (xml)
With that constexpr powers-of-10 table (up to 10^38), every operation computes its exact result in i128 first. What happens next depends on the operation:
+,-, and*narrow back toint64_tthrough one helper,FromWide. When a result needs more than 18 fractional digits or a mantissa wider thanint64_t,FromWidedrops fractional digits once (rounding to nearest, avoiding double-rounding) until it fits. Only an integer part that won’t fitint64_tthrowsstd::overflow_error.Decimal::Parsegoes through the sameFromWidenarrowing, but catches the overflow and returns it as akParseerror (“value out of range”), since bad input from an exchange is a runtime problem, not a bug.<=>compares the widenedi128values directly without narrowing at all.- Operations with a fixed target scale (
RoundToIncrement,Divide, wideningRescale, and negatingINT64_MIN) checkFitsI64directly and throwstd::overflow_errorif the scaled mantissa overflows. Kestrel treats that as a programming bug rather than a recoverable runtime error.
Two small details in decimal.cpp are easy to get wrong on a first pass:
- Floor division on negative numbers: C++’s
/and%operators truncate toward zero, not toward −∞.DivRoundexplicitly normalizes negative remainders (if (r < 0) { r += d; --q; }) before applyingRounding::kDown(toward −∞),Rounding::kUp(toward +∞), orRounding::kNearest(ties away from zero). - Parsing scientific notation and long decimals: Exchanges occasionally return tiny numbers in scientific notation (like
1e-8) or long fractional strings.Decimal::Parseparses directly fromstd::string_viewintoi128, tracking astickyflag if non-zero digits appear past its 2^120 (~36-digit) accumulation limit so rounding stays honest instead of silently truncating.
Side Note: __int128 is a GCC/Clang extension, not standard C++. MSVC doesn’t provide it, so decimal.cpp carries a TODO for a small wide-integer fallback if I ever decide to support native Windows builds outside WSL.
Keeping double Out Of JSON—In Both Directions
Building a Decimal type doesn’t help much if your JSON library rounds every number to a double the moment it parses it—and that is exactly what nlohmann::json::parse() does by default. The JSON spec doesn’t actually require that. It lets parsers limit precision and points out that double precision is what most of them support, so that’s what most of them do. Crypto exchanges usually quote prices as strings ("64210.50"), but Schwab and some metadata endpoints send raw JSON numbers (182.35).
On the inbound side, util::ParseJson (src/util/json.cpp) uses a tiny nlohmann::json_sax<json> subclass, ExactNumberSax, that intercepts number_float and stores the parser’s raw literal text as a JSON string instead of accepting the converted double:
class ExactNumberSax : public nlohmann::json_sax<json> {
public:
// ...
bool number_float(number_float_t, const string_t& literal) override {
return Add(json(literal));
}
// ...
};Code language: HTML, XML (xml)
From there, util::JsonDecimal parses strings via Decimal::Parse and integers directly into Decimal(value, 0).
The same trap exists on the outbound side for Schwab, whose order API expects unquoted JSON numbers for price, stopPrice, and quantity. Putting a fractional value into an nlohmann::json object would pass it through double, so BuildSingleOrderJson (src/trading/schwab_trading_venue.cpp) hand-formats the order payload directly from Decimal::Normalized().ToString() and uses nlohmann::json(schwab_symbol).dump() only to escape the string symbol. A price moves from the exchange’s response text into Decimal, out to order payloads, and into PostgreSQL’s exact NUMERIC columns without ever becoming a double.
Rounding In the Safe Direction Before an Order Hits the Wire
Before any venue sends an order (paper or live), it runs through ApplySymbolRules (src/trading/order_rules.cpp) using the exchange’s tick_size, lot_size, min_order_size, and min_notional:
- Limit buys round
kDowntotick_size(never bid more than the strategy asked). - Limit sells round
kUptotick_size(never offer for less than the strategy asked). - Stops round
kNearesttotick_size. - Quantities always round
kDowntolot_size(never order more than sized or funded). Orders belowmin_order_size—and limit orders whoselimit_price * quantityfalls belowmin_notional—are rejected locally before spending a network call.
On Binance.US, there is one last wrinkle: the exchange rejects a price or quantity with -1111 if it carries more decimal places than the tick or lot allows, even if the extra digits are trailing zeros. AtScaleOf in binanceus_trading_venue.cpp rescales every formatted number to increment.Normalized().Scale() before putting it in the query string.
Safety: std::expected, Never Blindly Retrying Orders, and Failing Closed
Once your math is exact, the next way a trading system hurts you is ambiguity over the network.
Recoverable Errors As Values (std::expected)
Anywhere I/O or exchange rules can fail, Kestrel returns a types::Result<T>, which is an alias for std::expected<T, types::Error>, the value-or-error type that C++23 added to the standard library. A types::Error carries a structured ErrorCategory (kNetwork, kTimeout, kRateLimit, kAuth, kRejected, kWouldTake, kInsufficientFunds, kNotFound, kParse, kServer, and a few others), along with optional http_status, provider_code, and retry_after fields.
Keeping recoverable errors in the return type makes failure paths explicit at every call site, and lets the HTTP and retry layers distinguish between errors worth backing off from and errors that demand stopping immediately.
A good example is Binance.US rate limits versus IP bans (src/net/cpr_http_client.cpp and src/trading/binanceus_api.cpp). A 429 Too Many Requests maps to kRateLimit, which is transient, so it gets retried after the server’s Retry-After delay. For an IP ban, Binance.US borrows 418—the joke “I’m a teapot” code from RFC 2324—and retrying into a ban only makes it longer. When BinanceUSApi sees a 418, it records the ban window from Retry-After, and BinanceUSApi::CheckBan() then short-circuits every request locally until that window expires. Nothing reaches the exchange until the ban is over.
Why Kestrel Never Blindly Retries a Timed-Out Order
For read-only requests—pulling historical bars, reading open orders, checking balances—retrying on transient failures is a no-brainer. util::Retry (include/kestrel/util/retry.hpp) wraps idempotent calls with exponential backoff (up to 5 attempts, 500 ms initial, 2x multiplier, 30 s cap, ±20% random jitter) whenever error.IsTransient() is true (kNetwork, kTimeout, kRateLimit, or kServer), honoring any server-supplied Retry-After header as a floor (capped at 1 hour).
For POST /api/v3/order, blind retries are a footgun.
Suppose you send a market buy to an exchange and libcurl returns a timeout 15 seconds later—or a proxy drops the connection, or the server returns HTTP 502. You are now in an unknown state. Maybe your packet never reached the matching engine. Or maybe the matching engine filled your order 120 ms after you sent it, and the response got lost on the way back. If your HTTP wrapper blindly retries that POST, you can easily end up buying twice.
Kestrel splits this explicitly at the API layer. BinanceUSApi::Send takes a bool idempotent parameter:
if (!idempotent) {
return attempt();
}
return util::Retry(mOptions.retry, attempt, mOptions.sleeper);Code language: PHP (php)
Order placements, OCO pairs, and cancels pass idempotent = false, so util::Retry never touches them. (The single automatic resend inside Send is when Binance.US returns error -1021—timestamp outside recvWindow—because the exchange explicitly rejected the request before executing it; Kestrel resyncs its clock offset against /api/v3/time and resends.)

Instead of retrying a timed-out order, Kestrel reconciles it:
- Tag every order up front.
BinanceUSTradingVenueattaches a uniquenewClientOrderId(kst-<8-char random tag>-<monotonic counter>) to every order before sending it. - Distinguish refusals from unknowns. If
POST /api/v3/orderfails with a definite refusal (insufficient funds, bad price filter, auth error), nothing was placed—return the error directly. If the error says the order may have gone through (Error::MayHaveTakenEffect():kNetwork,kTimeout,kServer, or a 2xx body that won’t parse,kParse), do not send it again. - Look up the client order ID. Kestrel calls
LookupClientOrder(GET /api/v3/order?origClientOrderId=...)—which is an idempotent read—up to twice with a 1-second pause (kLookupPause). - Recover or surface the ambiguity loudly. If the order shows up, Kestrel returns a normal
OrderAcknowledgementbuilt from the exchange’s state and life goes on. If it is still not found (or the lookup itself fails), Kestrel returns the original error with theclient_order_idand a warning that the order may still exist.
The same rule holds across the other venues, adapted to what each API allows:
- Kraken (
KrakenTradingVenue) assigns acl_ord_idand checks bothOpenOrdersandClosedOrderson an unknown outcome. Kraken only enforcescl_ord_iduniqueness among open orders, so when the caller supplied a custom ID, theClosedOrderslookup is bounded by time. - Charles Schwab (
SchwabTradingVenue) doesn’t accept a client order ID on equity orders, so Kestrel has to find the order another way:- Before sending the
POST, it takes an in-memory snapshot of the order IDs it has already seen (KnownOrdersSnapshot()). That costs no extra HTTP round trip. - If the
POSThits an unknown outcome—or returns 2xx without an order ID—RecoverPlacedOrderpolls recent account orders up to twice,200 msapart. - It ignores IDs that were already known or claimed by concurrent polls, and looks for a single new order matching the symbol, side, type, time in force, quantity, price, and creation window.
- If more than one order matches, it logs a warning and refuses to guess.
- Before sending the
- OCO pairs on Binance.US (
PlaceOco) guard against half-tracked exits too: if a 2xx response doesn’t yield both legs inorderReports, Kestrel first looks up both legs by their client order IDs (limit_idandstop_id). If either lookup fails—or if an unknown-outcome lookup finds only one leg—it cancels the whole pair viaDELETE /api/v3/orderListrather than leaving an untracked leg resting on the book.
Side Note: That list didn’t start out consistent. A cross-model review caught that each venue kept its own idea of what “may have taken effect” meant. Kraken and Schwab already treated an unparseable 2xx as an unknown outcome, but Binance.US returned ParseOrder‘s kParse error as if it were a refusal. Worse, Breakout::PlaceStop (src/strategy/breakout.cpp) treated any OCO error outside its own short list as a refusal and immediately placed a standalone STOP_LOSS sell, which could put a second sell beside an already-resting OCO if a 2xx body came back garbled. The fix (commit 8e227a5) added one predicate, Error::MayHaveTakenEffect(), which all three venues and breakout now share. A garbled 2xx on Binance.US now gets looked up by client order ID like a timeout, and breakout checks GetOpenOrders before placing an unanswered stop, pair, or exit again. Sharing the lookup code itself across venues is still on the list in TODO.md.
A Few More C++ Safety Touches Around Live Orders
Three smaller details in the networking, risk, and shutdown paths turned out to be worth far more than the lines of code they took.
Never follow redirects with API keys. In CprHttpClient::Send, session->SetRedirect(cpr::Redirect{false}) is set unconditionally. Exchange REST endpoints don’t redirect, and libcurl will otherwise forward custom headers (X-MBX-APIKEY, API-Key, API-Sign) across a 3xx redirect to another host.
Closing the Halt() vs. PlaceOrder() race. When the kill switch in RiskGatedVenue::Halt() trips, flipping mHalted (std::atomic<bool>) and calling CancelAll() once isn’t quite enough. An order that checked mHalted a microsecond earlier could still be in flight inside PlaceOrder() or PlaceOco(), landing on the exchange after CancelAll() returns.
Both methods hold mMutex from admission through the venue call. So Halt() cancels all open orders, briefly acquires and releases mMutex as a barrier to drain any in-flight placement, and calls CancelAll() a second time:
mHalted = true;
spdlog::warn("risk: trading halted: {}", reason);
auto cancelled = mVenue->CancelAll();
// An order already inside PlaceOrder read the flag before it was set and
// can still land after the cancel above. PlaceOrder holds the order lock
// from its checks to the venue call, so once the lock is ours that order
// has landed, and any later one sees the halt. Cancel again to catch it.
{ std::lock_guard lock(mMutex); }
auto recancelled = mVenue->CancelAll();Code language: PHP (php)
Even after a halt, Admit() still lets position-reducing exits through.
Signal-safe shutdown without racing open entries. In the first post I mentioned that kestrel_app run cancels open entry orders on Ctrl-C / SIGTERM while leaving protective exits resting on the book. Under the hood, that takes two pieces:
- Signals:
BlockStopSignals()(app/commands/run.cpp) masksSIGINTandSIGTERMwithpthread_sigmaskbefore any worker threads spawn. A dedicatedStopWatcherthread then collects them synchronously withsigtimedwait, so the mutex-takingRunner::Stop()never executes inside an async signal handler. A second signal callsstd::_Exit(130)if you want out immediately. - Shutdown order:
Runner::End()stops the fill poller, then cancels open entries viaEntriesToCancel()(sorting stops first and capping kept exits to actual holdings). It runs one finalPollFills()beforeOnStop(), so an entry that filled mid-cancel still reachesOnFillto place its protective stop.
Latency: Keeping Connections Warm (429 ms Down To 129 ms) and Measuring Everything
Early on I had a hunch that calling Binance.US from my machine in California might be slow enough to matter for faster strategies like market_maker. Rather than guess, I added instrumentation first.
Measuring the Latency of Every HTTP Call (TimedHttpClient)
Every HttpClient built by Kestrel’s factory is wrapped in TimedHttpClient (src/net/timed_http_client.cpp). Each Send() times one individual attempt (so retries show up as separate samples) and records it into a thread-safe LatencyStats keyed by "METHOD url" (or a redacted label).
Two design choices make the metrics practical to keep on all the time:
- No secrets in endpoint labels:
HttpRequestseparatesurlfromraw_query(signed parameters) and lets callers supply an optionallabelfor paths containing account hashes or order IDs (as Schwab’s endpoints do).TimedHttpClientonly ever logsMethodName(method) + " " + request.Label(), so API signatures and account IDs never leak into logs or database tables. - Two views, bounded memory:
LatencyStatskeeps a whole-run map (Summarize()) logged when a command exits, and an independent window map (TakeWindow()) drained every 60 seconds into PostgreSQL’svenue_latencytable byRunRecorder. Each endpoint tracks exact call counts (EndpointLatency::count, stored in SQL columncalls),errors,rate_limited(HTTP 429/418),min_ms, andmax_msacross every request, plus a 4,096-sample ring buffer for nearest-rankp50,p95, andp99. Any call taking at least[http] slow_request_ms(default 2,000 ms) logs a warning immediately.
What the Numbers Showed—and Why Connection Pooling Cut 300 ms
Once venue_latency started filling up, runs 7 through 11 on Binance.US showed the same thing on every single endpoint: a median round trip around 415 ms.
Digging into where those milliseconds were going turned up two compounding causes:
- Binance.US serves its REST API out of AWS Tokyo (
ap-northeast-1) behind a CloudFront edge (SFO53, about 15 ms from me in California). - My original
CprHttpClientconstructed a freshcpr::Session(and underlying libcurl easy handle) on every singleSend(). That meant every REST call tore down and re-established DNS, TCP, and TLS from scratch.
The fix in CprHttpClient (src/net/cpr_http_client.cpp) is a small, thread-safe pool of up to kMaxIdleSessions = 4 idle cpr::Session handles (TakeSession() / ReturnSession()), shared between a provider’s DataSource and TradingVenue by factory.cpp. Each Send() checks out its own session—so the Runner thread and the background fill poller never touch the same libcurl handle concurrently—and returns it when the call finishes.
Kraken is the one venue that still serializes signed calls on top of that pool. Kraken enforces strictly monotonic nonces per API key, so KrakenTradingVenue::SendOnce holds mSendMutex across both nonce assignment and mHttp->Send().
Reusing a cpr::Session has two subtle sharp edges that are easy to miss:
- Sticky state between calls: A reused
cpr::Sessionremembers options from the previous request. If you send aPOSTwith a JSON body and later reuse that handle for aGETorDELETEwithout explicitly resetting everything, stale query parameters, headers, or bodies leak into the next call.CprHttpClient::Sendoverwrites the URL, parameters (explicitly clearingcpr::Parameters{}whenraw_queryis used), headers, timeout, redirect policy, and callssession->RemoveContent()wheneverrequest.bodyis empty. - Dropping half-dead sockets (while keeping healthy HTTP errors): Normal HTTP
4xxand5xxresponses still return thecpr::SessiontomIdle, because the underlying TCP/TLS connection is healthy. Only whenr.errorfires (a timeout or libcurl transport failure) doesSend()return early without callingReturnSession(), destroying theunique_ptr<cpr::Session>on the spot so a wedged socket is never reused.
To prove the pool actually worked without touching the internet, tests/cpr_http_client_test.cpp spins up LoopbackServer, a tiny in-process server that keeps connections open the way an HTTP/1.1 server does, and verifies that:
- five sequential requests open exactly one TCP connection;
- a
GETandDELETEafter aPOSTcarry none of its body or query; - three concurrent threads open at most three connections;
- a transport failure is reported cleanly without poisoning subsequent calls;
- redirects are refused rather than forwarding signed headers.
The pooling tests passed 20 repeats under ThreadSanitizer when the pool landed.
Then I tested the old and new clients back-to-back against the live Binance.US API from California on October 29, 2026 using the exact same read-only account --days 30 command (34 signed requests):
| Metric | Fresh cpr::Session per call | Reused session pool |
|---|---|---|
Median (p50) | 429 ms | 129 ms |
| Minimum | 214 ms | 121 ms |
Shaving 300 ms off the median was a bigger win than eliminating a local TCP + TLS handshake to SFO53 (~15 ms away) could explain on its own. Keeping my HTTPS connection to CloudFront open appears to keep CloudFront’s upstream connection to Tokyo warm as well—turning a multi-round-trip cold start across the Pacific into a single warm round trip.
Throughput: Batching Quotes Into TimescaleDB Without Losing Order
Recording historical bars is easy—even 1-minute bars across dozens of symbols only trickle in once a minute. Recording live tick-by-tick quotes (kestrel_app stream 3600 --store) is a different animal. Binance.US’s bookTicker feed and Schwab’s level-1 quote streamer fire constantly, often delivering multiple updates for the same symbol in the same millisecond.
Trying to run one synchronous PostgreSQL INSERT per quote inside the WebSocket callback breaks down immediately: the network and disk latency stalls the WebSocket thread, which delays quotes to the strategy or causes the socket buffer to back up.
Decoupling the WebSocket Thread From Database Writes (QuoteWriter)
QuoteWriter (include/kestrel/db/quote_store.hpp and src/db/quote_store.cpp) puts a bounded in-memory buffer and a dedicated writer thread between the WebSocket callbacks and PostgreSQL:
- Number and buffer fast on the WebSocket thread:
QuoteWriter::Pushincrements a per-(venue, symbol)64-bit sequence counter (seq), appends the quote tomBuffer, and returns immediately. IfmBufferhitskMaxBatch(1,000 quotes), it wakes the writer thread early; otherwise the writer thread wakes every500 ms. - Swap and flush off-thread:
QuoteWriter::FlushacquiresmMutexjust long enough tobatch.swap(mBuffer), then releases the lock so incoming quotes never wait on SQL I/O.QuoteStore::Savepacks up to 1,000 rows (kRowsPerInsert = 1000× 12 columns = 12,000 bind parameters, comfortably under PostgreSQL’s 65,535-parameter limit) into a single multi-row parameterizedINSERT. - Retry without OOM or duplicate-key death spirals: An early version of
QuoteStore::Savehad noON CONFLICTclause. When a flush held more than 1,000 quotes, it split into multipleINSERTstatements; if the second statement failed,Flushput the entire batch back in the buffer to retry. On the next flush, the first 1,000 rows—already committed—tripped the unique constraint every single time, wedging the buffer until it overflowed and dropped quotes. Commit1dce515addedON CONFLICT (venue, symbol, received_time, seq) DO NOTHING, which made retries idempotent. Failed batches are prepended back in front of newer quotes (preserving chronological order), and total buffered quotes are capped atkMaxBuffered = 100,000so a long database outage drops the oldest quotes instead of running the machine out of RAM.
Keeping Sub-Millisecond Arrival Order Intact
Kestrel timestamps quotes in UTC milliseconds (std::chrono::sys_time<milliseconds>). That is plenty for bars and strategy timers, but on a busy crypto pair or liquid ETF, several quotes routinely land with the exact same millisecond timestamp.
Migration 3 added a seq BIGINT NOT NULL column to quotes with UNIQUE (venue, symbol, received_time, seq). Inside QuoteWriter::Push, mSeqMutex is held across both incrementing seq and pushing into mBuffer, guaranteeing that sequence order strictly matches arrival order.
Prime() lets a caller (like kestrel_app stream) preload the highest stored seq per symbol before the WebSocket opens, so numbering continues smoothly across restarts without hitting PostgreSQL on the first tick. If an unprimed symbol arrives, Push() falls back to a lookup. When backtests call QuoteStore::Load, the query orders by seq, replaying every tick in the exact order Kestrel originally saw it.
Database Timeouts, Index Trimming, and TimescaleDB Compression
Two final lessons showed up once Kestrel had been recording live sessions for a while.
Bound every database wait. As I mentioned in the last post, Database::Query (src/db/database.cpp) never automatically re-runs a failed SQL statement after a dropped connection. It accepts arbitrary SQL, including non-idempotent inserts like TradeStore::StartRun‘s INSERT INTO runs ... RETURNING run_id, so EnsureConnected() reconnects on the next call instead.
What I hadn’t originally wired up were connection and statement timeouts, which meant a stalled database could hang a live stream thread indefinitely. Commit 1dce515 added explicit libpq bounds:
connect_timeout=10- TCP keepalives every 10 s after 30 s idle, with 3 probes
tcp_user_timeout=60000- a configurable
statement_timeout, defaulting to 5 minutes—long enough for large backtest quote loads, short enough to break a hung call
Quotes grow fast—trim indexes and compress old chunks. Recording 33 Schwab symbols eats roughly 320 MB per session uncompressed (~10 MB per symbol per session). Two changes in commit 4f478b4 kept that manageable:
- Migration 10 dropped two redundant indexes (
quotes_venue_symbol_timeand TimescaleDB’s defaultquotes_received_time_idx). Because every quote query in Kestrel filters on(venue, symbol), the remainingquotes_seq_unique (venue, symbol, received_time, seq)andquotes_venue_symbol_seqindexes already served every read. That alone cut quote index size from 1.8 GB down to 1.1 GB. - Automatic TimescaleDB compression: When Kestrel connects to a TimescaleDB community build (
timescaledb.license = 'timescale', version 2.11+ so compressed chunks still acceptON CONFLICTupserts),Database::Migrate()enables native columnar compression. Forquotes, that means segmenting byvenue, symbol, ordering byreceived_time, seq, and compressing 1-day chunks after 2 days;market_trades(after 14 days) andbars(after 7 days) get the same treatment. Setup runs withSET LOCAL lock_timeout = '5s', so startup waits at most a few seconds behind a long backtest read instead of blocking indefinitely. On Debian/Ubuntu’s Apache-only package, it gracefully skips compression and keeps working.
Side Note: Turning on compression exposed a sneaky startup trap during code review. If QuoteStore::LastSeq() ran an unbounded MAX(seq) for a symbol, TimescaleDB could end up scanning and decompressing weeks of old chunks just to find the last sequence number at startup. LastSeq() now does a two-step lookup. First it grabs the newest received_time via ORDER BY received_time DESC, seq DESC LIMIT 1, which stops at the newest chunk thanks to TimescaleDB’s ordered ChunkAppend. Then it takes MAX(seq) over just the trailing 1-hour window [received_time - 1 hour, received_time]. That window catches small backward clock steps (like an NTP adjustment where a higher seq landed at an earlier millisecond) while still letting TimescaleDB prune every older compressed chunk.
Conclusion
None of what I covered here is flashy. There is no custom kernel networking, no lock-free ring buffer written in inline assembly, and no secret indicator. It is mostly boring systems engineering:
- Keep
doubleout of your accounting types and out of your JSON boundary. - Treat unknown network outcomes as unknown—never blindly retry a non-idempotent write.
- Measure latency per endpoint before guessing, and don’t throw away warm TLS connections on every call.
- Decouple hot streaming threads from disk I/O with bounded, idempotent batches and indexes that match your actual queries.
Next up in the series is Integrating Crypto Exchanges and Charles Schwab—what it took to put Binance.US, Kraken, and Charles Schwab behind one DataSource and TradingVenue interface, including Schwab’s 7-day OAuth refresh cycle, US equity market-hours gating, and the places where live exchange behavior disagreed with the official docs.
If you’ve built financial or market-data systems in C++ (or another language), what subtle bug bit you first—floating-point drift, retry semantics, connection pooling, or database growth? Let me know in the comments!
