Learning When to Think While Listening in Large Audio-Language Models
ICLR 2027Under reviewInternational Conference on Learning Representations

I'm Zhiyuan Song, and you can also call me Allen. I'm an M.S.E. student in Computer and Information Science (AI Track) at the University of Pennsylvania, where I work as a Deep Learning Researcher advised by Prof. Jiatao Gu on multimodal foundation models and LLM reasoning. I'm also an AI Engineer Intern at Millennium Management in New York.
My primary research interests lie in building intelligent, real-time interactive systems. My current focus spans LLM post-training, Omni & multi-model systems, real-time interaction models, agent harnesses, continual learning, and long-horizon tasks. I am also passionate about robust AI infrastructure and optimizing distributed training pipelines.
Outside research, I build AI products and the infrastructure behind them. ThunderTalk is an open-source, on-device voice toolkit for macOS, and Wemute is a local-first file converter. I am also building agent systems that carry real work: digital employees, a research and data-analysis pipeline, and a shared skill and memory layer for coding agents. At Millennium, an internal AI tool I built grew from my team to about 190 colleagues.
I completed my undergraduate studies at the University of Virginia, where I double majored in Computer Science and Applied Statistics and graduated with High Distinction, and was fortunate to conduct research advised by Prof. Madhav Marathe, Prof. S.S. Ravi, and Prof. N. Rich Nguyen.
I'm always glad to talk with founders and researchers about early-stage ideas. Here is how I got here and what I would bring to a founding team. Feel free to reach out.
Master of Science in Engineering (M.S.E.), Computer and Information Science (AI Track).
Bachelor of Arts with High Distinction, Computer Science & Applied Statistics.
Honors: Dean's List; UVA Grad App Fund Scholarship; NSF Fund.
AI Engineer Intern — Cloud & AI/ML Infrastructure
Deep Learning Researcher with Prof. Jiatao Gu — LLM post-training (SFT → RL) on 4×B200
Founder — my own brand and studio; built Legal Manus, a RAG legal assistant
Completed my summer internship at Millennium Management and got an extension — continuing as an AI Engineer Intern.
Started as an AI Engineer Intern at Millennium Management, working on cloud & AI/ML infrastructure.
New preprint: "Learning When to Think While Listening in Large Audio-Language Models" is on arXiv, under review at ICLR 2027.
I started my M.S.E. in CIS at UPenn and joined Prof. Jiatao Gu's group as a Deep Learning Researcher.
Led the team as Tech Lead & MLE at AdventureX Hackathon, building a full-duplex real-time audio AI assistant.
Graduated from University of Virginia with High Distinction.
Paper "SL4HOI", with Prof. N. Rich Nguyen, accepted at KDD-UC 2024 in Barcelona.
Founded Entropinia — it began as a clothing brand and grew into my own studio, where I built AI products such as Legal Manus.
Joined the UVA Biocomplexity Institute as a Research Assistant under Prof. Madhav Marathe and Prof. S.S. Ravi.
The summer after my first year of high school, I flew alone to Washington, D.C. for a summer program at Georgetown, my first time in the United States. I filmed three vlogs there, and that is where I taught myself to edit: choosing music and sound effects, finding footage, cutting every shot by hand to the beat, and taking apart other people's videos to see how they were made. The laptop was slow, but the videos got made, and they reached 34K views on Bilibili. Back at school I became president of the film club and led a promo video for the coffee club. I also ran the broadcasting club, and later headed the student union's publicity department, which the broadcasting club sat under.
I wanted to go to the University of Virginia, so I suggested to our school counselor that we invite UVA to give a talk. The counselor made it happen, and an admissions officer came to our school. I spoke with the officer during the Q&A, sent a thank-you letter afterwards, and we kept in touch by email. I applied that year and was admitted.
At UVA I double majored in Computer Science and Applied Statistics and graduated in three and a half years. In my second year I was a teaching assistant for discrete mathematics. Through that course I met Zirou Qiu, a PhD student working with Prof. Madhav Marathe, who brought me into the lab. He has been a close friend ever since. On a later summer project, the supervising professor remarked that the undergraduate was the one leading the graduate student. In 2024 I presented a first-author paper at the KDD Undergraduate Consortium in Barcelona, where my assigned mentor was Prof. Jie Tang, a professor at Tsinghua University and chief scientist of Zhipu AI. We talked at length about large language models and GLM.
I have sold things online since high school: sneakers during the resale boom, then second-hand goods on Xianyu, where as a student I have sold close to ¥80,000 in total, with 189 reviews, all of them positive. In college I noticed that the TC-Helicon VoiceTone C1, a pitch-correction pedal popular with musicians in China, cost about half as much in the US. I brought a first batch home as a test. Every unit sold at about twice my cost, and my listing rose to the top of the search results. By the time I came back with a second batch, others had found the same gap: US prices were climbing, stock was running out, and I sold that batch at cost. What the first batch really earned was a head start.
One of my closest friends from high school is a designer, and together we started a clothing label, Entropinia. He led the design and I ran the business side: I worked out and completed the registration, licence and tax filings myself, negotiated with a factory owner, and went through design drafts, the logo and round after round of samples until the pieces were ready for production. When the direction changed, I did not close the business. It became my consulting studio in China, where I advise students and build AI products, and everything I had learned about licences, contracts and taxes carried over.
I handled most of my own graduate applications and was admitted to Penn. By the end I knew the process well enough to teach it. In the months before starting, I taught university courses for several tutoring companies and led a team as tech lead at AdventureX, China's largest hackathon, with about 870 builders from 67 countries. I then published a guide to CS graduate applications on Bilibili, and students from Fudan, Peking University and UC Berkeley asked me to advise them. My model is semi-DIY, and it is hands-on. I meet each student one-on-one for one to two hours every week. I provide the structure and references for each essay, the student writes the first draft, and we revise it together, version after version, until every statement for every program is final. I propose the school list and lay out the trade-offs, and the student makes the final call, because it is their application and their future. Advising one student is close to applying all over again myself. I drafted the contracts too, and all three were admitted to strong programs. I have advised new students every cycle since.
My first Millennium process went to a fourth and final round, and ended without an offer. I thought about it from the recruiter's side: their job is to fill the headcount, and a candidate who has already reached the final round is easier to place than someone starting from zero. The New York posting had been taken down, but the same AI engineer role was still open in Miami on the firm's website, so I asked the recruiter whether I could be considered for it. The recruiter asked around and came back with a match: a different team, in New York after all. Two more interviews, six rounds in total, and I had the offer. The internship has since been extended.
At Millennium I built an internal AI tool, starting from an early framework. The hard part was not writing it but getting people to use it. My early pushes brought modest results, so I kept improving it on colleagues' feedback. Once it was good enough, people began recommending it to each other. It started with my own team, about 190 colleagues use it today, and the number is still growing.
LLM post-training: supervised fine-tuning, reinforcement learning and evaluation design. I wrote our DAPO trainer from scratch and train on 4×B200 GPUs with DeepSpeed ZeRO-3. Two first-author papers, one at KDD-UC 2024 and one under review at ICLR 2027.
I can take a product from idea to release on the platform it needs: full-stack web, macOS, iPhone and iPad apps, WeChat mini programs, browser games, agent systems, and audio and speech pipelines. ThunderTalk and Wemute are open-source examples that are live today.
Shooting and editing video, audio production and music. I make the videos for my own products.
Registering and running a business, drafting contracts, negotiating with factories, selling, and getting a product adopted inside a large firm.
From school clubs and student organizations to research projects and hackathons, I have usually been the one leading the team. Most of the people I have built things with are still close friends. I have also taught and advised students for years.
My rule is simple: I win or I learn. If you are working on something hard and worth doing, I would like to hear about it.
University of Virginia
Instructor: Rich Nguyen
Instructor: Kevin Sullivan
I actively mentor undergraduate students (sometimes high school students), guiding them through curriculum planning, research directions, and graduate school applications in Computer Science, Data Science, and related fields. Below are the recent admission results of students I have closely advised:
UIUC (MSCS), Georgia Tech (CSE), UT Austin (ECE), UW-Madison (MSCS PMP), UMD (MSCS), UNC Chapel Hill (MSCS), Brown University (ScM CS), UCSD (CS75), NYU Courant (MSCS).
UChicago (MPCS), National University of Singapore (DSML), UW Seattle (EEPMP), NYU Courant (MSIS), NYU Tandon (MSCS), USC (CS32), Columbia (MSCE).
UPenn (Systems Engineering), Columbia University (Applied Math), UMich (Data Science), Northwestern University (Stats & DS), JHU (Applied Math & Stats), NYU Courant (Mathematics), NYU Tandon (Mathematical Science).
Prospective students: if you are interested in working with me or seeking advising, please reach out via this dedicated email — song@entropinia.com — for all mentoring-related inquiries.
A small, opinionated list of papers and posts I've found unusually useful — not a survey, just the ones I keep returning to.
Learning new skills from demonstrations without a reward function: the model, conditioned on a demonstration, acts as its own teacher and produces on-policy training signal. It forgets far less than plain fine-tuning. Read the limitations section too.
An agent writes its own tasks, each with a verification function, and is then trained on them with RL. A concrete answer to where tasks, verifiers and training signal come from when nobody labels them.
Trains a code agent to alternate between writing code and verifying it with tool feedback, so its self-verification becomes reliable enough to keep improving at test time. Worth noting which reference information exists in training but not at inference.
RL post-training is only as good as its verifier. This pipeline synthesizes tests hard enough to catch well-disguised wrong solutions, and shows how much verifier quality changes what the model learns downstream.
Shows that repeatedly distilling an agent's experience into its weights can make capability collapse over iterations instead of compounding, and looks at why. A useful warning before building any multi-round self-improvement loop.
The first empirical look at loop engineering — agents prompted by schedules and repo events instead of a human. 217 real loops, and a clear picture of where their state actually lives.
A source-code study of eleven production coding agents. Seven canonical subsystems, 29 design patterns, and the finding that everyone hand-rolls the async loop.
How to keep an agent productive across many context windows: an initializer agent, feature lists, git commits and tests as the hand-off artifacts. Practical and directly reusable.
A clean replacement for standard residual connections in attention — the kind of small architectural change that's easy to ablate and surprisingly effective.
A useful conceptual frame for how we should be designing the interaction layer of AI systems, not just the model layer. Short, opinionated, worth the ten minutes.
Excellent practitioner's overview of how 16 RL training libraries handle async rollout, weight sync, and staleness. Saves you weeks of reading source code.
Convincing demonstration that automated, reflection-based prompt search can beat RL on the same tasks. Worth knowing before you spin up another GRPO run.
The clearest survey I've found of what actually scales (and what doesn't) when you push RL compute on LLMs. Good companion to the DeepSeekMath paper.
Direct neighbor to my own work on think-while-listening. Solid framing of the latency-vs-quality tradeoff in streaming audio reasoning.
The 'simpler than you think' argument lands. A surprisingly minimal architecture that holds its own against far more elaborate folding systems.
A different take on multimodal representation learning — they actually instrument the model to trace how information flows across modalities, rather than just reporting benchmarks.
The most cogent treatment so far of full-duplex speech-LLM dialogue. Required reading if you care about real-time interactive audio systems.
MLA (Multi-head Latent Attention) is the part to focus on. The cleanest KV-cache compression trick I've seen that actually pays off at scale.
Discrete flow-matching done right. Bridges flow models and discrete-space generation, with concrete results on protein design.
The original GRPO paper. If you only read one source for understanding the algorithm everyone is currently building on, read this one.
RFdiffusion. Showed that fine-tuning a structure-prediction model with a diffusion objective unlocks de novo protein design at unprecedented quality.
Diffusion applied to molecule-to-protein docking. The geometric noise space — translation + rotation + torsion — is the part I keep coming back to.
The original Latent Diffusion paper. Foundation for nearly every modern image generation system; the latent-space framing alone is worth the read.
DDIM. The non-Markovian formulation that made diffusion sampling fast enough to be practical. Still the cleanest derivation of accelerated diffusion sampling.

Free, open-source voice toolkit for macOS, 100% on-device via MLX. Press a key in any app and speak: the text appears at the cursor, with a live preview as you talk. It also transcribes recordings with speaker labels, reads text aloud in a cloned voice, and can proofread dictation with the AI tools you already use.

A calendar and to-do list for macOS that coding agents can use. Tell Claude Code or Codex what your day looks like and it shows up in the app: tasks, routines, important dates and a Markdown page for each day, through a CLI and an MCP server. No account, no cloud.

A local-first file converter for Mac and the browser — images, video, audio, Office, PDF and e-books, plus OCR, background removal and subtitles, with nothing uploaded. Its CLI turns any file into text an LLM can read.

An AI-powered desktop pet companion that lives on your screen — built with PySide6 and real-time emotion recognition.

A RAG-powered legal chatbot for U.S. & Chinese law, with 600K+ legal chunks and 98% retrieval accuracy.

Official website for Entropinia — a contemporary clothing brand blending streetwear aesthetics with minimalist design.
I spent a summer building a multi-agent code-review tool at Millennium, then stayed for the fall to run it. The second half taught me more than the first: how software actually reaches production, what users complain about and in what order, and why adoption is part of the engineering.
I moved my coding agents from a flat list of skills to routers that load skills on demand. Startup context fell by 92%, routing accuracy held, and the failures taught me more than the wins.
A short tour of the tools I work on outside research: ThunderTalk, Wemute, a skill hub for coding agents, a planner, a finance app and an early agent-team project, and the three ideas they share.
Why I built an open-source voice-to-text app that runs entirely on-device, what MLX changed about the math, what shipping a 1.0 actually costs — and what 1.5 added: Studio, local text-to-speech and voice cloning.
I have trained in the gym since middle school and kept it up for more than ten years.
I write lyrics and compose music. Search for "欧拉Song" on NetEase Cloud Music.
I publish videos on Bilibili. Search for the same username to find my channel.
Welcome to collaborate on music production and video production!