From AI Coding to AI Engineering
A reproducible research project on how semantic models, constrained APIs, compile-time checks, and runtime verification affect the reliability and efficiency of LLM-based software development.
Large Language Models can generate code at extraordinary speed. However, generating code is not the same as engineering reliable software.
As LLMs move from autocomplete and isolated code generation toward autonomous software development, a central question becomes increasingly important:
How can we make AI-generated software reliable, reproducible, auditable, and economically efficient?
This project studies a specific hypothesis:
AI reliability can be improved not only by making the model smarter, but also by making the valid solution space smaller.
Instead of relying only on prompts, instructions, skills, or increasingly capable models, we investigate whether software engineering constraints themselves can provide a stronger harness for AI:
- domain models and metadata;
- domain-specific languages and semantic APIs;
- generated APIs across multiple programming languages;
- type systems and compile-time constraints;
- runtime validation;
- audit and execution traces;
- tests and acceptance criteria;
- commit and quality gates.
TeaQL is used as one concrete implementation and experimental platform for this research, but the goal of the project is broader than evaluating a single framework.
The project aims to produce reusable evidence about a more general software engineering question:
What kinds of AI software engineering failures should be prevented by prompts, skills, APIs, compilers, runtimes, tests, or acceptance gates?
The first generation of AI coding has focused primarily on improving the model:
Human Intent
↓
Prompt / Specification
↓
LLM
↓
General-Purpose Code
This approach gives the LLM a very large solution space.
For the same business requirement, an AI agent may choose different data access patterns, SQL implementations, transaction boundaries, security mechanisms, tenant isolation strategies, audit mechanisms, or error-handling paths.
Many of these implementations may compile.
Some may even pass basic tests.
But only a subset may satisfy the actual business, security, operational, and compliance intent.
This project explores another architecture:
Human Intent
↓
LLM / Agent
↓
Semantic API / DSL
↓
Type System / Compiler
↓
Constrained Runtime
↓
Verification
↓
Evidence
The central idea is simple:
Do not ask the model to repeatedly rediscover engineering decisions that can be encoded once into the system.
We study the transition from:
AI Generation
to:
AI Engineering
The distinction is important.
AI Generation asks:
How can an LLM generate more correct code?
AI Engineering asks:
How should software systems be designed so that AI-generated work is easier to constrain, verify, audit, and accept?
Our working model is:
Prompts guide.
Skills teach.
APIs constrain.
Compilers reject.
Runtime verifies.
Evidence supports acceptance.
Not every rule belongs in the same layer.
A natural-language instruction may be sufficient for some tasks.
Other requirements should be encoded as APIs, type constraints, runtime rules, tests, or explicit acceptance criteria.
This project attempts to identify those boundaries empirically.
The project begins with the following research questions.
Does reducing the valid solution space through semantic APIs and engineering constraints improve the success rate of LLM-based software development?
Do constrained development environments reduce:
- token consumption;
- completion time;
- repair iterations;
- failed compilations;
- failed tests;
- manual intervention?
Can stronger engineering constraints allow smaller or less expensive models to complete tasks that otherwise require stronger models?
In other words:
Can part of repeated model intelligence be replaced by reusable engineering intelligence?
Which classes of failure are best addressed at which layer?
Prompt-catchable
Skill-catchable
API-preventable
Compile-time detectable
Runtime-detectable
Test-detectable
Acceptance-only detectable
Can a shared domain model serve as a durable source of truth from which consistent semantic APIs are generated across multiple technology stacks?
Can AI-generated software be evaluated through reproducible execution evidence rather than relying primarily on the agent's own claim that a task is complete?
TeaQL provides one implementation of the constrained engineering approach studied in this project.
A domain model can act as a shared source of truth and generate consistent semantic APIs for multiple programming languages.
For example:
Q::tasks()
.comment("Find overdue tasks for follow-up")
.with_task_status_is_open()
.with_name_sounding_like("sharding")
.purpose("load tasks for dashboard")
.execute_for_list(ctx)Q.tasks()
.comment("Find overdue tasks for follow-up")
.with_task_status_is_open()
.with_name_sounding_like("sharding")
.purpose("load tasks for dashboard")
.execute_for_list(ctx);The API is intentionally close to natural language, but it is not unconstrained natural language.
The same business semantics can be generated consistently across Java and Rust from a single domain model.
The working architecture is:
Domain Model
│
Single Source of Truth
│
┌──────────┴──────────┐
│ │
Java API Rust API
│ │
└──────────┬──────────┘
│
LLM / Agent
│
Business Logic
│
Runtime
│
Audit / Trace / Evidence
This creates an opportunity to study a broader principle:
Natural-language-like interfaces for probabilistic agents, backed by deterministic semantics and verification.
The initial experiments will compare different levels of engineering constraint under otherwise comparable conditions.
General-Purpose Framework
+ Prompt
+ LLM / Agent
General-Purpose Framework
+ Prompt
+ AGENTS.md / Skills / Rules
+ LLM / Agent
Semantic Domain Model
+ Generated API / DSL
+ Compile-Time Constraints
+ Runtime Verification
+ LLM / Agent
Additional experiments may compare:
Strong Model + Weak Constraints
with:
Smaller Model + Strong Constraints
The objective is not merely to determine which system produces code faster.
The objective is to measure whether the resulting software satisfies the intended requirements with less uncertainty and less repair effort.
Each experiment should collect evidence across several dimensions.
- task completion time;
- total token usage;
- number of agent iterations;
- number of tool calls;
- number of generated or modified files.
- compilation success;
- unit test results;
- integration test results;
- acceptance criteria pass rate;
- semantic correctness.
- number of failed attempts;
- number of repair cycles;
- manual intervention required;
- regression rate.
Where applicable:
- tenant isolation violations;
- unauthorized access paths;
- audit coverage;
- sensitive-data handling;
- security gate failures.
Possible measures include:
- duplicated logic;
- bypassed abstractions;
- API consistency;
- implementation variance;
- complexity introduced by the agent.
- completeness of runtime trace;
- auditability;
- reproducibility;
- ability to explain why a result was accepted.
Reproducibility is a first-class requirement of this project.
Every experiment should produce an evidence package similar to:
experiments/
└── <experiment-id>/
├── manifest.json
├── README.md
├── task.md
├── acceptance-criteria.md
├── environment.json
├── model.json
├── prompts/
├── skills/
├── agent-traces/
├── generated-code/
├── patches/
├── compile-logs/
├── test-results/
├── runtime-traces/
├── sql-traces/
├── audit-logs/
├── token-usage.json
├── timing.json
├── evaluator-results/
└── final-evaluation.json
Where model providers expose sufficient information, experiment metadata should record:
- model name and version;
- inference configuration;
- date and time;
- agent version;
- framework version;
- operating system;
- compiler version;
- dependency versions;
- repository commit hashes.
Exact reproducibility may not always be possible with nondeterministic hosted models.
When exact reproduction is impossible, the project should aim for:
procedural reproducibility
That means another researcher should be able to rerun the same task, configuration, evaluation process, and evidence collection pipeline.
This project follows a simple rule:
No performance or reliability claim should exist without traceable evidence.
Summary charts and aggregate metrics are useful, but the underlying raw evidence should be preserved whenever licensing, privacy, and security constraints allow.
The preferred chain is:
Claim
↓
Metric
↓
Evaluation Result
↓
Execution Evidence
↓
Raw Artifact
The project should make it possible to move in both directions:
from experiment to conclusion,
and from conclusion back to the original evidence.
Successful examples alone are not sufficient.
The project will maintain a structured collection of AI software engineering failures.
Potential failure classes include:
- misunderstood business semantics;
- hallucinated APIs;
- bypassed abstractions;
- incorrect query logic;
- missing tenant isolation;
- missing audit information;
- insecure implementation;
- compilation failures;
- type-system violations;
- runtime validation failures;
- test failures;
- acceptance criteria failures;
- agent-declared completion without supporting evidence.
Each failure should be classified by the earliest layer that could reasonably prevent or detect it.
Example:
Failure:
Missing tenant isolation
Possible classifications:
- Prompt-catchable
- Skill-catchable
- API-preventable
- Runtime-detectable
- Test-detectable
A key goal of the project is to distinguish between:
instructions that should remain soft
and:
requirements that should become executable constraints
TeaQL may be both an experimental subject and part of the evidence infrastructure.
This creates a potential source of bias.
Therefore, evaluation should not rely exclusively on TeaQL's own mechanisms.
Where possible, experiments should include:
- independent test suites;
- external evaluators;
- static analysis;
- manually reviewed acceptance criteria;
- blinded comparisons;
- repeated trials.
LLM-based judges may be used as supplementary evaluators, but should not be treated as the sole source of truth for important correctness claims.
The project begins with several hypotheses.
Semantic APIs reduce implementation variance for routine business logic.
Compile-time and runtime constraints reduce the number of failures that survive into final acceptance.
A constrained solution space reduces repair iterations and token usage.
Smaller models operating within strong constraints can successfully complete some tasks that require stronger models in weakly constrained environments.
A single domain model can improve semantic consistency across Java and Rust implementations.
Runtime evidence improves the reliability of task acceptance compared with agent self-reporting alone.
These are hypotheses, not conclusions.
The purpose of the project is to test them.
This project is not intended to prove that:
- TeaQL is universally superior to all frameworks;
- DSLs are appropriate for every software domain;
- prompts and skills are unnecessary;
- stronger models are unnecessary;
- all software engineering can be reduced to generated APIs;
- autonomous agents can replace software engineers.
The current working assumption is more modest:
Different forms of knowledge and constraints belong at different layers of the engineering system.
The project studies where those layers should be.
The project aims to produce several public artifacts.
A collection of representative software engineering tasks with explicit acceptance criteria.
Scripts, configurations, prompts, traces, generated code, and evaluation tooling.
Raw and normalized experimental data suitable for independent analysis.
A structured taxonomy and collection of real agent failure modes.
Periodic reports documenting observations before formal publication.
Potential publication topics include:
- constrained LLM-based software engineering;
- semantic APIs for AI agents;
- compile-time constraints as AI guardrails;
- evidence-driven acceptance of AI-generated software;
- model capability versus engineering constraints;
- cross-language semantic consistency from shared domain models.
The project currently starts from three practical ideas.
Do not rely only on better prompting when a recurring engineering decision can be encoded into an API, type, model, or runtime rule.
Domain semantics should live in durable, executable structures rather than depending entirely on conversational context.
An AI agent saying that a task is complete is not sufficient.
Completion should be supported by compilation, tests, traces, audits, or other independently verifiable evidence.
We believe the first generation of AI coding has been primarily about teaching AI how to write software.
The next generation may be about redesigning software engineering so that AI can build software reliably.
That requires more than a better model.
It requires an engineering environment that provides:
Semantics
Boundaries
Verification
Evidence
The long-term question behind this project is therefore:
What should software engineering become when the primary consumer of its abstractions is no longer only the human programmer, but also the AI agent?
This repository is an attempt to answer that question through reproducible engineering practice.
Status: Experimental / Research in Progress
The project is at an early stage.
Initial work will focus on:
- defining benchmark tasks;
- standardizing experiment manifests;
- establishing evidence collection;
- identifying baseline frameworks;
- designing controlled comparisons;
- publishing the first reproducible experiment.
Results, terminology, and hypotheses are expected to evolve as evidence accumulates.
Contributions are welcome in areas such as:
- benchmark design;
- experiment reproduction;
- evaluation methodology;
- statistical analysis;
- agent failure classification;
- alternative frameworks;
- Java and Rust experiments;
- additional programming languages;
- security and correctness evaluation.
A contribution guide will be added as the project matures.
The model provides intelligence.
The engineering system provides semantics, boundaries, verification, and evidence.
And our current working summary is:
Prompts guide. Skills teach. APIs constrain. Runtime verifies.
To be determined.
For experiments that depend on external models, datasets, frameworks, or APIs, their respective licenses and terms apply.