SeqGPT: a small laboratory for visible patterns

Large language models make it difficult to see the relationship between a training example and a learned behavior. Natural language contains too many overlapping patterns, and training runs are expensive enough that curiosity can become a budget decision.

SeqGPT shrinks the laboratory. It generates numerical sequence families, trains compact transformer variants to continue them, and measures how performance changes as patterns become more complex. GPT-style, Llama-like, and mixture-of-experts models can be swapped across the same workbench.

Make the hidden thing visible

Synthetic patterns are not a substitute for language. They are an instrument. A generated sequence has a known rule, so failure and generalization can be discussed against something concrete. Cross-pattern evaluation asks whether a model learned a reusable structure or merely became fluent in the generator it saw.

The mixture-of-experts experiments add another observable surface: which expert receives which kind of pattern, and whether specialization emerges over training. Visualizing that routing turns an architectural claim into something that can be inspected.

A framework awaiting findings

SeqGPT provides training, generation, configurable complexity tests, checkpointing, Docker workflows, and multi-GPU support. It is best understood as a public experiment rather than a finished research result. The next valuable layer is not more architecture names; it is reproducible findings, baselines, and clear records of which hypotheses failed.

That unfinished state is worth showing in a private cabinet. Bricolage includes building the jig that makes later learning possible. The discipline is to label a jig as a jig. SeqGPT is a place to ask smaller questions of smaller models and to keep enough of the experiment visible that an answer can be trusted.

The project is public at shawnrancatore/seqGPT.