# k3ld Services — Six capabilities, one tenant

## The six capabilities

1. **Data catalog & governance** — Catalog, lineage, glossary, and data-quality checks — so models are built on data you trust and can defend.
2. **Connect & pipelines** — Managed ingestion and ELT with run history and live cost tracking on every job.
3. **Custom model training** — Docs become instruction data automatically, then we fine-tune (LoRA) or align (DPO/GRPO) a model for your behavior.
4. **Evaluation** — Built-in benchmarks and eval runs that rank your checkpoints with evidence before you ship.
5. **Ship & serve** — One click from a completed run to a real-time endpoint, batch, serverless, agent, or model-catalog bake.
6. **Monitor & govern** — Production is where models decay: latency, drift, usage, and alerts — with retraining one click away.

## The Pipeline editor (capability tour)

Three areas: **Inputs, Processes, Outputs**.

- **Inputs** — catalog tables; file sources (S3/GCS/Azure Blob with glob patterns); managed S3; external databases queried with SQL (Snowflake, BigQuery, Postgres, Databricks).
- **Processes** — roughly **80 nodes**: transforms (filter, join, aggregate, window, JSON parse…), ML (profile, anomaly, correlation, pattern mining, PII/PHI, train/forecast/classify/cluster), flow control (branch, merge, fork, loop, gate, quality gates), and NN building blocks.
- **Outputs** — write to catalog tables, S3, managed S3, or views.

**Compute is auto-routed.** SQL-compilable mining (profile, anomaly, encode, scale, PII/PHI) runs serverless; ML training nodes run on EMR Spark; NN training runs on SageMaker. You can override the engine, and a visual map shows exactly which nodes can connect to which.

**Pipelines can branch, merge, fork, loop, and gate** — branch data by a condition, fork identical copies into parallel model runs, loop in batches, and gate on a schedule, webhook, or another pipeline completing. Quality gates can warn, fail, or block on a SQL rule.

## The catalog & Explore area

Every tenant gets a data catalog: registered tables with schemas, plus connections to object storage (S3/GCS/Azure) and external warehouses (Snowflake, BigQuery, Postgres, Databricks). Search the catalog, preview tables, check row counts before building anything. Query directly via notebooks or the External DB node, build dashboards from results, and use **Ask AI** to draft a pipeline from plain English. Governance: row-level security policies and masking rules; every read/write lands in the audit trail.

## Learning from data without a model (data mining & analytics)

- **Profile node** — per-column statistics (counts, nulls, distinct, min/max/mean).
- **Anomaly Detection** — flags IQR outliers.
- **Correlation Matrix** — Pearson or Spearman.
- **Pattern Mining** — FP-Growth or SVM for association rules (market-basket insights).

All SQL-compiled and serverless. **Reporter** produces richer output — HTML dashboards, JSON-LD for machine consumption, Jupyter cells, or PDF — with schema, lineage, and charts, landing in S3 or a catalog table.

## Turning documents into training data (LLM Format)

The **LLM Format node** reads raw text, CSV, or JSON from a file source and uses an LLM to transform each document into training-ready JSONL. Pick the target format — instruction (prompt/completion), chat (multi-turn messages), or text (pretraining) — with auto-filled default system prompts. Map columns with dotted paths (e.g. `title → prompt`, `body → completion`). It shards large corpora and checkpoints after each shard, so interrupted runs resume instead of restarting. Output feeds a Data Source node for pretraining or the Fine-Tune (LoRA) path.

## Shipping a model (deploy, scale, serve)

model.k3ld's Ship area creates a SageMaker endpoint with autoscaling, data capture for auditing predictions, and automatic idle-sleep so you're not billed when nobody's calling it. Choose the instance or let the platform recommend the best fit. Canary-style deploys shift traffic gradually; an inference recommender benchmarks your artifact first. LoRA adapters attach to existing endpoints without a full redeploy. Everything stays private to your tenant until you expose it.