🎓 Class: how a version reaches production — Cross CI/CD

Welcome to class. Today we follow one single thing: a version number. It starts on your laptop and ends running in production. Along the way it passes through GitHub, two TeamCity servers, a registry, a bot, and a security gate. The big idea is simple. You only do three things: you bump the version, you review a pull request, and you click merge. Everything else happens by itself. And every automatic step is checked by another step that does not trust it. Let's meet the pieces one by one.

Here is the map. Think of it as four neighborhoods. First, GitHub, where the code lives. There are the service repositories, like user-api or bank-conciliation, the production-deployment repository, which describes what runs in production, and a fork of it owned by a bot organization. Second, Cross TeamCity on cross-server. This is the factory. It builds images, and now it also runs the release bot. Third, the registry at registry dot cross-otc dot com. It is the warehouse where finished images are stored. Fourth, the production cluster. Inside it lives a second, separate TeamCity project called CrossProduction, with its own private deployer agent, and of course your running applications. The most important rule of this map: the factory never holds a key to production.

Step one is yours. You change a version number and merge to master in the service repository. Where is the version? For Go APIs it is the at-version comment in cmd server main dot go. For the websocket gateway it is info dot version in swagger dot yaml. For web apps and helpers like bank-conciliation, it is the version field in package dot json. Why a version number and not just the commit? Because a version is a promise: version 0.14.11 is published exactly once and never overwritten. A human readable number makes the pull request, the logs and the rollback obvious.

Step two. Cross TeamCity watches the master branch of every service. When you push, the build for that service starts by itself. First it reads the version and asks the registry: does this tag already exist? If yes, it says skipping, finishes green, and publishes nothing. That is why pushing a commit without a version bump is harmless. If the version is new, it builds the image for both A M D 64 and A R M 64, pushes it, and calculates the digest. A digest is a fingerprint, a long sha256 code that changes if even one byte of the image changes. Finally it writes a small file called image dot json. Think of image dot json as a receipt: it says which version, which digest, and which commit were published by this exact build. Remember that receipt, it comes back twice.

A quick but important side lesson. Why do we care about digests? A tag, like colon 0.14.11, is just a label stuck on a box. Someone could move that label to a different box. A digest is computed from the content of the box itself, so it cannot lie. If anyone swaps the image, the digest no longer matches. That is why production pins both: the repository, then colon version, then at sha256 digest. The version is for you to read. The digest is for the machine to verify. Kubernetes pulls by the digest, so the version text cannot trick it.

Step three is the new robot. In Cross TeamCity there is a build called Release PR. It runs right after any image build succeeds, and also every hour at minute seven. It does one comparison. On one side, the newest version each build published, read from those image dot json receipts. On the other side, what production master currently pins, read from the images dot lock file of each namespace. If TeamCity has something newer, the bot writes a new images dot lock, commits it on its own fork, and opens one pull request per service on production-deployment. If a pull request is already open, it refreshes it on top of the latest master. If master already has that version, it closes the old pull request. And it never proposes a downgrade.

What does that pull request change? Only images dot lock files. There is one per namespace: cross, cross minatech, and cross helpers. Each line has a name and the full pinned reference, version and digest. The Kubernetes yaml files never contain real image names. They contain placeholders like at IMAGE colon cross-exchange slash user-api at. At deploy time, the placeholder is replaced with the line from the lock. So the whole release is a one line diff that you can read in two seconds. Binance adapter api runs in two namespaces, cross and cross minatech, so its single pull request edits both locks together. One service, one pull request, always.

Now, why a fork? Why not let the bot push a branch directly to production-deployment? Because any token that can push branches to that repository is one step away from pushing to master. GitHub's free plan does not let us block that with branch protection. So we took the write power away completely. The bot owns its own copy of the repository, in a separate organization called cross release bot. It can write there freely, because nothing deploys from there. Then it opens a pull request from the fork into the real repository. A pull request from a fork is only a suggestion. The real repository does not change until you merge. And because the fork lives in a different organization, the bot's write permission does not exist anywhere inside cross exchange.

The bot uses two GitHub Apps, and splitting them is on purpose. The first is cross release push. It belongs to the bot organization and it can write file contents, but it is installed on exactly one repository: the fork. The second is cross release pr. It belongs to cross exchange and it can read contents and write pull requests, installed only on production-deployment. Write pull requests means it can open, edit and close pull requests. It does not mean it can push code or merge. So if someone stole the push app, they could only scribble on the fork. If someone stole the pr app, they could only open pull requests that you still have to merge. Neither app alone, and not even both together, can change production. Both are private, have webhooks turned off, and each sees one single repository.

Let's list every token, because tokens are where security lives. Our GitHub tokens are build scoped tokens. TeamCity asks the GitHub App for a fresh token when a build starts, and the token is revoked when the build ends. Nothing long lived sits on disk. The Release PR build gets two of them: the push token for the fork, and the pr token for production-deployment. To read the image dot json receipts, it uses the temporary login that TeamCity gives every running build, so there is no stored TeamCity password either. And here is the key decision: the image builds, the ones that run your code and npm and go modules, get no GitHub token at all. Code that you did not write, like a dependency, runs during image builds. So we keep the bot in a completely separate build, where only our own tool runs.

Where do the app secrets live? In Cross TeamCity, in the Root project, under Connections. There are two connections, cross release push with id project ext 5, and cross release pr with id project ext 6. Each one stores the app id, client id, client secret and private key, encrypted inside TeamCity. The downloaded pem key files were deleted from your laptop. Why the Root project and not the Cross project? Because the Cross project's settings come from git, so its connection page is read only, and we did not want secrets in git. The Cross settings in git only contain two harmless parameters that point to those ids. Each app also has TeamCity's unique callback address registered, which is required for the connection test to pass.

Step four is you again. The pull request shows a small table: the namespace, the image, the old version, the new version, the digest, a link to the TeamCity build that produced it, and the source commit. If you like it, you merge. Merging is the approval. If you do not want it, you close it, and the bot will not reopen it for the same versions. Only a newer version brings it back. One more rule: it must be merged by you, kevynsax. GitHub's free plan cannot enforce that with branch protection, so the production gate enforces it instead, as you will see in a second.

Step five. The merge lands on production-deployment master. The CrossProduction TeamCity project watches it. Each namespace has its own deploy build, and its trigger only fires when files under apps slash that namespace change. So merging a bank-conciliation release only deploys cross helpers, nothing else. This build only runs on one special agent, the cross production deployer, which lives inside the production cluster and is the only thing with permission to change it. To read the merge history it uses a third GitHub App, cross otc production teamcity, with read access only. Again, a build scoped token that disappears when the build ends.

Step six is the guard at the door, called the release gate. Before deploying, it trusts nobody, not even the bot. Check A: it walks back through every commit on master since the last deploy, and each one must be a pull request merged by kevynsax. A direct push, or a merge by someone else, stops everything. Check B: for every image line whose digest changed, it goes to Cross TeamCity, finds the build that published that version, and compares the digest with the image dot json receipt. Remember the receipt? This is where it comes back. If the bot was hacked and wrote a fake digest, the receipt would not match, and the deploy is refused. To read those receipts it uses a special user, production record reader, who can only view the Cross project, and its token is stored as a password parameter in CrossProduction. If both checks pass, the gate fills in the placeholders, applies the manifests, waits for the rollout, and records this revision as the new last deployed point.

Let's step back and see the walls. Wall one: the builders, the machines that run untrusted dependency code, hold no production credentials at all. Wall two: the cross teamcity GitHub app, used by the factory, was removed from production-deployment, so the factory cannot even read the production repository. Wall three: only one agent inside the cluster can deploy, and only after the gate passes. Wall four: two independent proofs are needed for any image to reach production. Your merge, checked through GitHub, and TeamCity's receipt, checked through a read only user. An attacker would need to break you and break TeamCity at the same time.

Let's practice with some what ifs. What if the bot's tokens are stolen? The thief can open pull requests and write to the fork. You don't merge, so nothing deploys. What if a pull request contains a fake digest pointing to an evil image? The gate compares it with the TeamCity receipt, it does not match, deploy refused. What if someone pushes straight to master? The gate sees a commit that is not a pull request merged by you, deploy refused. What if a malicious package sneaks into an image build? That build has no GitHub token and no production access, so there is nothing to steal. Each layer assumes the one before it might fail.

Finally, why these choices. We run the bot as a separate build, so its tokens never sit next to third party code. We used a fork and two apps instead of one app with write access, so nothing in the bot can write to the production repository. The gate checks the TeamCity receipt instead of just asking the registry, because anyone with a registry password could push an image, but only the real build writes the receipt. The bot refreshes on an hourly schedule instead of watching production master with git, so Cross TeamCity did not need any new access to that repository. And the app secrets live in the Root project connections, so they stay out of git. Every choice follows one principle: give each piece the smallest power it needs, and make a second piece check it.

Recap. Your whole job is three clicks. One: bump the version and merge in the service repository. Two: review the pull request the bot opens on production-deployment. Three: merge it. The image build, the bot, the deploy, and the gate do the rest, and they check each other the whole way. One practical note: the first time the bot runs, it will propose kyc api and kyc web app 4.0.1, because production intentionally stays on 2.0.0. Just close those two pull requests, and the bot will leave them alone until a newer KYC version appears. That's the class. Thanks for listening.