Notes on distributed systems, AI agent architecture, and engineering leadership.https://sagarthakkar.com/https://sagarthakkar.com/blog/knowledge-distillation-black-box-llms/https://sagarthakkar.com/blog/knowledge-distillation-black-box-llms/Proprietary LLM costs explode at scale. Here's how to compress frontier models into deployable, cost-effective alternatives you actually control.Mon, 29 Jun 2026 00:00:00 GMT# Why Architects Can't Ignore Knowledge Distillation of Black-Box LLMs Your enterprise just licensed access to a state-of-the-art proprietary LLM. The model is brilliant—frontier-class reasoning, nuanced outputs, enterprise-grade. The problem? Cost explodes at scale. API pricing is per-token. Production traffic multiplies the bill. And you have zero visibility into the model's internals—no weights, no gradients, no fine-tuning hooks. You're trapped. You need the quality. You can't afford the cost. You can't own the model. Enter **knowledge distillation for black-box LLMs**—a technique that lets you transfer the reasoning capability of proprietary models into smaller, cheaper, deployable student models you actually control. ## The Problem Black-Box LLMs Create Proprietary language models (whether from closed-source vendors or API-only providers) dominate the frontier: - State-of-the-art reasoning and instruction-following - Managed infrastructure (no self-hosting headache) - But: closed weights, vendor lock-in, cost-per-token bleeding The classic distillation path—fine-tuning a student model on teacher outputs—works. But it assumes you can: 1. Label data extensively 2. Run inference on both teacher and student 3. Absorb the cost of teacher inference at massive scale For black-box APIs, #3 kills the business case. Distillation becomes too expensive.  *Figure 1: The problem (API costs, no ownership) vs the solution (distilled student models, full control).* ## Knowledge Distillation Without Teacher Access: Proxy-KD The core insight: **You don't need internal weights to extract knowledge.**  *Figure 2: Three-stage Proxy-KD flow — black-box teacher knowledge transfers through open proxy to deployable student model.* Proxy-KD sidesteps the closed-box problem by: ### 1. Using a Proxy Teacher Instead of directly distilling from the proprietary model, distill from an open proxy that mimics its behavior. The proxy is either: - Another open LLM fine-tuned to mimic the closed model's outputs - A smaller model trained to produce similar logits - A synthesized model built from observed input-output pairs The proxy costs orders of magnitude less to run than the proprietary model. ### 2. Distilling from the Proxy Use standard distillation: generate synthetic data from the proxy, train a student model to match the proxy's outputs and reasoning patterns. ### 3. Optional: Refinement from the Real Teacher If budget allows, do a final refinement pass on a small high-value dataset using the actual proprietary model. This polishes the student without the full cost. ## Why This Matters for Architects ### Cost Control - Proprietary models: $0.01–0.10 per 1K tokens (varies). At 1B tokens/day, that's **$10K–100K monthly**. - Distilled student: Deploy open weights (Llama, Mixtral, etc.) on your infrastructure. Marginal inference cost → near-zero per-token. - **ROI window: 2–6 months of API savings covers the distillation cost.** ### Operational Ownership - Closed APIs: You're dependent on the vendor's uptime, rate limits, pricing changes. - Distilled student: You own the weights. Deploy anywhere—on-prem, containerized, edge devices, air-gapped. ### Latency & Control - API models: Network round-trip + vendor queue. Latency is unpredictable. - Local models: Sub-second inference, deterministic performance, no external dependency. ### Quality Preservation - The student captures 75–90% of the teacher's quality on most tasks. - For many enterprise use cases (document classification, customer support, data extraction), this is more than sufficient.  *Figure 3: Student model accuracy across proxy model sizes — 75–90% of teacher quality achieved with smaller, cheaper proxies.* ### Empirical Results **Table 1: Overall Benchmark Performance** | Dataset | Teacher Accuracy | Proxy-KD Student | Quality Ratio | |---------|------------------|------------------|---------------| | MMLU (reasoning) | 88.2% | 81.5% | 92% | | GSM8K (math) | 92.1% | 78.4% | 85% | | HumanEval (code) | 84.6% | 74.2% | 88% | | SQuAD (QA) | 95.3% | 89.7% | 94% | | Classification tasks | 91.4% | 87.3% | 96% | | **Average** | **90.3%** | **82.2%** | **91%** | **Table 2: Ablation Study — Component Impact** | Component | Effect on Accuracy | Compute Cost | Necessity | |-----------|-------------------|----------------|-----------| | Proxy model (any size) | baseline | ~1/10th teacher | Essential | | Proxy fine-tuning (LoRA) | +2-3% | ~5% overhead | Important | | Token-level distillation loss | +1-2% | <1% overhead | Helpful | | Teacher refinement pass (small dataset) | +3-5% | ~10-20% teacher cost | Optional (RoI-dependent) | | Data augmentation (synthetic) | +1-2% | ~5% overhead | Helpful for domains | ## When to Distill | Scenario | Distill? | Why | |----------|----------|-----| | One-off API calls, low volume | ✗ | Cost doesn't justify overhead | | High-volume production workload (1M+ calls/month) | ✓ | API costs become painful; distillation ROI is fast | | Latency-sensitive application | ✓ | Local inference beats API round-trip | | Compliance/data-residency requirement | ✓ | Can't send data to external API; self-hosted is mandatory | | Cost optimization initiative | ✓ | If you're already paying proprietary bills, distillation is the lever | | Prototype/research phase | ✗ | Keep the powerful model; optimize later | ## The Implementation Path 1. **Baseline:** Measure quality on your domain tasks (few-shot eval set, ~100 test cases). 2. **Proxy creation:** Fine-tune an open model (Llama 2, Mistral, etc.) on data/outputs from the proprietary model. 3. **Distillation:** Generate synthetic data from the proxy, train a smaller student model (e.g., 7B parameters vs. 70B). 4. **Evaluation:** Test on your baseline. Aim for 80%+ of teacher quality. 5. **Deployment:** Ship the student model to prod. Monitor drift; refresh every 6–12 months. ## Cost-Benefit Example **Scenario:** Customer support chatbot, 500K conversations/month. | Method | Monthly Cost | Ownership | Latency | |--------|--------------|-----------|---------| | Proprietary API @ $0.03 per 1K tokens | ~$15K | None | ~300ms | | Distilled student (open model, on-prem) | ~$500 (infra) | Full | ~50ms | | **Monthly savings** | **$14.5K** | | **6× faster** | | **Distillation cost (amortized)** | – | – | Paid back in ~1 month | ## The Catch Distillation isn't free: - **Upfront cost:** Engineering time + compute for proxy creation and student training (~2–8 weeks, $10K–50K in cloud cost). - **Quality trade-off:** The student won't match the teacher on frontier reasoning tasks. It works best for structured domains (classification, extraction, summarization). - **Maintenance:** Refresh the student periodically as the world shifts; re-label if the task distribution changes. ## The Bigger Picture This is architecture-level thinking: **When you're locked into proprietary models, distillation is your path to independence.** As LLM costs become a line-item expense in enterprise budgets, the engineers who know how to compress frontier models into deployable, cost-effective alternatives will own the competitive advantage. The proprietary vendors want you to think distillation is impossible without their weights. It isn't. The technique is proven. The math is sound. The question is whether you're willing to invest the engineering effort to reclaim control. --- ## References **Original Research:** - Chen et al. (2024). "Knowledge Distillation of Black-Box Large Language Models" ([arXiv:2401.07013](https://arxiv.org/abs/2401.07013)) - Introduces Proxy-KD: distilling proprietary LLM knowledge via an open-source proxy model - Empirical validation across reasoning, classification, and generation tasks **Related Work:** - Hinton et al. (2015). "Distilling the Knowledge in a Neural Network" — foundational distillation framework - Hsieh et al. (2023). "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data" — task-specific distillation for reasoning **Open Model Baselines:** - Meta's Llama 2 & Llama 3 — strong open-source baselines for student models - Mistral 7B — efficient, production-ready alternative **Enterprise Implementation Resources:** - vLLM — optimized LLM serving (inference speed & cost) - Ollama — local LLM deployment & management - Ray Tune — distributed training for distillation workflows --- **Next Step:** If you're running on proprietary LLM APIs at scale, audit your usage logs. If monthly costs exceed $5K, distillation becomes a strategic project, not a nice-to-have. The clock is ticking. Every month you delay is money left on the table.AIarchitecturecost-optimizationLLMsenterprisetechnicalhttps://sagarthakkar.com/blog/51-agent-orchestration/https://sagarthakkar.com/blog/51-agent-orchestration/The model was never the hard part. Orchestration, routing, and failure modes are. Here's the architecture.Sat, 20 Jun 2026 00:00:00 GMTI run a personal AI system of 51 agents. Not a demo I spun up for a blog post - a system I operate every day, that has failed in every way a distributed system can fail, and that I've had to debug at 11pm like any other production service. The single most expensive mistake I see teams make is this: they design agent systems like chatbots. Free-form conversation, one model trying to do everything, a prompt that grows until nobody understands it. It demos beautifully. It collapses the moment real traffic and real edge cases arrive. > Most production AI agent systems fail because people design them like chatbots. Agents need clear routing - orchestrator, leads, specialists - not free-form conversation. The same pattern as any well-run org. ## Agents are an org chart, not a conversation The mental model that actually scales is the one every functioning company already uses. An orchestrator at the top that owns intent and routing. Leads beneath it that own a domain. Specialists beneath them that do one thing well and hand the result back up. Nobody talks to everybody. Authority and scope are explicit. This isn't an aesthetic preference. It's the same reason you don't run a 200-person company as one giant group chat. Routing is what keeps the system legible as it grows. The moment any agent can call any other agent, you've built a graph with no failure boundaries - and you will not be able to reason about it. ## Failure modes are the actual design Tutorials assume the model returns what you asked for. Production assumes it won't. A specialist times out. A tool returns malformed JSON. A lead loops because two sub-agents disagree. The interesting engineering is entirely in how the system behaves when a step fails - retries with backoff, circuit-breakers around flaky tools, a budget on tokens and wall-clock per request, and a clear answer to "what does the orchestrator return when a branch never comes back?" If you can't draw the failure path, you don't have an architecture. You have a prompt that happens to work today. ## You cannot operate what you cannot see Observability is not optional and it is not a dashboard you add later. Every agent hop, every tool call, every token spent is a span. I trace the full tree of a request the way you'd trace a distributed transaction - because that's exactly what it is. Without it, a regression is invisible until your bill or your latency tells you, and by then you're debugging blind. > The model is a commodity. The architecture around it - routing, failure handling, observability - is the entire job. Swap the model and a good system keeps working. Swap the model on a bad one and you find out how much you were leaning on luck. ## What changed my mind I used to think better models would absorb the architecture problem - that a smart enough model would route itself. Two years of operating this system convinced me of the opposite. LLMs amplify the cost of bad architecture. The more capable the model, the more confidently it executes a badly-routed plan, and the more brutal the design debt becomes when it breaks. Start with the org chart. Draw the failure paths. Instrument everything. The model is the last thing to worry about, not the first.AIAgentsArchitectureOrchestrationhttps://sagarthakkar.com/blog/why-numbers-are-not-enough/https://sagarthakkar.com/blog/why-numbers-are-not-enough/Most people believe mathematics starts with numbers. That belief works-until it doesn't.Wed, 14 Jan 2026 00:00:00 GMTMost people believe mathematics starts with numbers. That belief works-until it doesn't. Imagine you're looking at a student's preparation for an exam. You ask a simple question: *How much did you study?* The answer comes back: **10 hours**. That number feels informative. It is precise. It is measurable. It is also incomplete. Ten hours *doing what*? Reading passively? Solving problems? Watching videos? Being coached? The moment you ask that follow-up question, pure numbers stop being sufficient. Something deeper is required. That is where linear algebra actually begins. --- ## Scalars: Quantities Without Direction A **scalar** is a single number that represents magnitude-*how much* of something exists. Time, temperature, total money spent, total hours studied. All scalars. Scalars are powerful when direction does not matter. If you are measuring how long something took or how heavy something is, a scalar is exactly what you want. But scalars collapse information. When you say "10 hours studied," you flatten every possible way of studying into one number. The number is accurate, but the description is dishonest. The world rarely behaves like a scalar system. --- ## When Scalar Thinking Breaks Down Now consider two students: * Student A studied **8 hours alone** and **2 hours with a coach** * Student B studied **2 hours alone** and **8 hours with a coach** Scalar thinking says they are identical: 10 hours each. Reality disagrees. Their effort is distributed differently. Their learning experience is different. Their outcomes are likely different. The problem is not that the number is wrong. The problem is that the number is **too simple**. When multiple influences act at once, scalar thinking hides structure instead of revealing it. --- ## Vectors: Quantities With Direction To describe the students honestly, we need more than one number. We need something like this: [ \begin{bmatrix} \text{self-study hours} \ \text{coaching hours} \end{bmatrix} ] This object is called a **vector**. But calling it an "ordered list of numbers" misses the point. A vector represents **magnitude and direction together**. In this case: * One direction corresponds to self-study * Another direction corresponds to coaching * The vector tells us *how much effort was applied in each direction* Each student is no longer a number. Each student is a **state**.   --- ## Why Direction Changes Everything Two vectors can have the same length and still mean very different things. Student A's effort points mostly toward self-study. Student B's effort points mostly toward coaching. Same magnitude. Different direction. Linear algebra treats these as fundamentally different objects, because **direction encodes behavior**. This is the first major mental shift: > Numbers tell you *how much*. > Vectors tell you *how*. Once you accept this, many real-world problems suddenly become representable instead of awkward. --- ## From Values to States A scalar answers: > "How big is it?" A vector answers: > "Where is it in relation to everything else?" That difference is subtle but profound. In data science, machine learning, economics, physics, and engineering, we rarely care about isolated quantities. We care about **configurations**. A user is not "30 years old." A user is a point in a space of age, income, behavior, preferences, and history. A system is not "under load." A system is operating across CPU, memory, network, and latency dimensions simultaneously. These are not scalar problems. They never were. --- ## Why Linear Algebra Starts Here Linear algebra is often introduced with symbols, rules, and matrices. That ordering is backwards. The real starting point is this realization: > Many phenomena cannot be described honestly with a single number. Vectors exist because reality has direction. Once you begin modeling systems as vectors instead of scalars, everything else-dot products, matrices, projections-becomes a natural extension instead of an abstraction. --- ## The Takeaway Scalars are not wrong. They are incomplete. Vectors do not complicate reality. They **respect it**. If you are still thinking in scalars, linear algebra will feel artificial. Once you start thinking in vectors, it will feel inevitable.linear-algebramathsvectorshttps://sagarthakkar.com/blog/token-vs-session-security/https://sagarthakkar.com/blog/token-vs-session-security/A practical breakdown of session vs token security, when to use which, and modern best practices.Sat, 13 Dec 2025 00:00:00 GMT Below is the simplest, most practical breakdown of **Token-Based** vs **Session-Based** security. ## ✅ Session-Based Security (Traditional Web Login) ### How Session-Based Security Works 1. **User logs in**. 2. **Server creates a session object** and keeps it in memory, DB, or cache (e.g., Redis). 3. **A session ID is stored in a browser cookie**. 4. **Client** sends the cookie with every request.  ### Session-Based Characteristics - **Server maintains state**. - Sessions expire or logout triggers invalidation. - Good for classic web apps (monoliths, server-rendered). ### Session-Based Pros - **Simple to implement**. - **Easy to revoke sessions**. - Well supported by frameworks. ### Session-Based Cons - **Doesn’t scale easily** across multiple servers (needs sticky sessions or shared store like Redis). - **Mobile apps & APIs don’t fit well** (cookies are tricky on mobile). - Requires server memory/storage for each user session. --- ## ✅ Token-Based Security (Modern APIs & Mobile/Web) ### How Token-Based Security Works 1. **User logs in**. 2. **Server generates a token** (JWT, opaque token, PASETO) and signs it. 3. **Token is sent to client** and included in every request (usually in `Authorization` header). 4. **Server does not store session state** - it validates the signature to trust the token.  ### Token-Based Characteristics - **Stateless**. - Scales horizontally easily. - Works across mobile, web, APIs, and microservices. ### Token-Based Pros - **Perfect for distributed systems**. - **No session storage required** on server. - Easy to integrate with third parties. - Works across domains (CORS friendly). ### Token-Based Cons - **Harder to revoke** (token is valid until expiry unless you maintain a blacklist/allowlist). - **Larger attack surface** if token is stolen (especially if stored in `localStorage`). --- ## 🆚 Simple Comparison in One Line  | Feature | Session-Based | Token-Based | | :--- | :--- | :--- | | **Server stores session?** | Yes | No | | **Scales easily?** | No | Yes | | **Best for** | Web apps | APIs, mobile, microservices | | **Revocation** | Easy | Harder | | **Implementation** | Simple | Requires careful design | --- ## 🧠 Architect-Level Guidance - What Else You MUST Know in 2025 ### 1️⃣ JWT is NOT always the right choice People overuse JWTs. Use JWTs when: - You need cross-service trust. - You have distributed microservices. - You avoid session storage by design. **Avoid JWTs when:** - You need quick revocation options. - You don’t trust clients fully. - Sessions are short-lived and centralized. ### 2️⃣ PASETO is the modern, safer alternative to JWT - JWT has known pitfalls (alg confusion, signature issues). - **PASETO** (Platform-Agnostic Security Tokens) fixes those by design. ### 3️⃣ OAuth2 / OpenID Connect ≠ Token-Based Auth These are protocols, not just "authentication methods". They use tokens, but solve: - Authorization delegation. - Identity federation. - SSO (Single Sign-On). ### 4️⃣ Zero-Trust Security Models Prefer Tokens In microservices, you need: - Mutual TLS (mTLS). - Service-to-service tokens. - No server-stored sessions. ### 5️⃣ For Internal Enterprise Apps → Hybrid Approach - Use **Session cookies for browser UX**. - Use **Token exchange for APIs**. Both can co-exist and are common in modern systems. ### 6️⃣ For Mobile Apps → ALWAYS Token-Based - Sessions break easily on mobile networks. - Tokens are the standard for resilience.SecurityArchitectureAuthhttps://sagarthakkar.com/blog/linear-algebra-insight-for-engineers/https://sagarthakkar.com/blog/linear-algebra-insight-for-engineers/Discover why matrices are not just tables of numbers, but functions in disguise. A paradigm shift that transforms how engineers think about systems, transformations, and architecture.Tue, 10 Dec 2024 00:00:00 GMTMost of us were taught matrices as **tables of numbers**. Rows. Columns. Formulas. Memorize and move on. And that's exactly why so many engineers quietly develop a fear of linear algebra. But here's the insight that recently clicked for me - and I'm still processing it layer by layer: **A matrix is not data. It's a function-disguised as data.** I'm still comprehending this idea more deeply, but even at this stage, it has completely shifted how I look at systems, architecture, and transformations. And I'll keep sharing more as I understand it better. ## Why This Matters to Engineers and Architects When you multiply a matrix by a vector, you're not calculating-you're **transforming**. - **Stretch** dimensions - **Rotate** coordinate systems - **Project** into different spaces - **Compress** information - **Shift** perspectives - **Re-orient** reference frames - **Change** viewpoints This isn't math trivia-this is exactly how: - Neural networks transform data through layers - Graphics engines move 3D objects in real-time - Robotics systems adjust direction and orientation - Simulations evolve states over time - Embeddings encode semantic meaning Matrices are everywhere around us-quietly doing the work of functions. ## So Why Do They Look Like Tables? Because storing the actual transformation like "Rotate 30° and stretch the Y-axis by 3x" would be messy to write and unstructured for computers. So mathematics found a clean solution: **Store how the function acts on basis vectors.** That grid of numbers isn't recording data-it's recording the *behavior* of a transformation. Each column tells you where a basis vector lands after the transformation is applied. ## A Matrix is a Controller, Not a Container Think of a video game controller: 🎮 The controller is not the movement-it's the *device that causes movement.* Similarly: - **Matrix** → the controller - **Vector** → the character's position - **Multiplication** → pressing buttons - **Output** → new position Once you see matrices as actions-not numbers-the whole subject becomes intuitive. (At least it's starting to for me, and I'll keep you posted as I go deeper.) ## Why This Matters for Engineering Leadership As architects and tech leaders, we constantly deal with: - High-dimensional problems - State changes and transitions - Transformations of data - Modelling system behavior - Abstraction design - Complex system interactions Understanding the "matrix = function" mental model strengthens how we think about **systems, change, operations, and behavior**. Linear algebra becomes more than a subject-it becomes a **thinking framework** for understanding how systems transform and evolve. ## Key Takeaway > **A matrix is a linear transformation stored in a data-friendly format.** I'm still refining my mental model, but this insight alone unlocked 50% of the subject for me. If you're an engineer who's struggled with linear algebra, try shifting your perspective: stop seeing matrices as static tables and start seeing them as dynamic transformations. The moment you make that shift, everything from machine learning to computer graphics to system design starts making a lot more sense. --- *What insights have transformed your understanding of technical concepts? I'd love to hear your "aha" moments in the comments.*Linear AlgebraEngineeringArchitectureLearning