Episode: How Zalando Engineers Software and AI at Scale - Bartosz Ocytko, Executive Principal Engineer at Zalando SE

So it was yesterday evening. I was working on our LiteLLM proxy, and we had an issue that was a degradation of functionality that was introduced two hours before. I noticed and it was was an admin functionality. And this admin functionality had a error where there was a mandatory argument, and this mandatory argument basically broke the API contract that existed before.

And I have to find a way of how to basically get rid of this regression.

So even with your executive role, you still are hands deep in the code.

That's the only way how you stay current in terms of the tools, because I of course used some of the AI tools to analyze what the actual issue was, since I do not follow the pull requests on this code base very thoroughly on a day-to-day basis.

So hands on is the theme here. I love it. Speaking about executive principal engineer, can you explain us a little bit what that roles comes with it? What responsibility do you have with that?

Sure, the way we define principal engineering in the company is basically that as soon as you exit the team scope, you inherit the scope of the manager that you report to. So a principal engineer, which comes in at Zalando after a senior, is inheriting the scope of the person that they report to, the head of engineering. And they have three to five teams that they take care of.

principal engineers take care of the architecture and the systems and the system landscape that those teams are responsible for. And they have to find a way of how to cope with the complexity and the scale and the operational health of those systems, supporting their manager. Similarly,

in my role, I'm reporting to the CTO and I'm supporting our executive team that is responsible for the engineering systems that we are delivering at Zalando.

And my focus is mostly on the B2C systems and the builder infrastructure and builder platform that we have for our engineers and applied scientists.

Fantastic. So you got a really bright overview across the whole company. That will be interesting discussions and I'm lucky for you. You don't have to switch to management roles and lead the organization. You can stay in your engineering lane.

so I actually have an updated role since the beginning of the year when we are trying a little bit of a new setup. So I'm blessed to have a team of experts. there are three experts that are part of my team, and we're seeing how we can help coordinate between ourselves, and I have a formal leadership role for this team now.

That's fantastic. And I have reserved a spot for later that we dive deep into that engineering team. Fantastic. We have another format in the podcasts where I asked the previous guests on a podcast, question, was Ralf Müller from DB Systel. is an AI evangelist and a chief architect. And he gave me the question for you. Do you review your code reviews yourself still? And if so,

What would it take you to trust the AI and leave it all to the AI?

I think it depends on how critical this code is. Right, the code that I mentioned for the proxy, I reviewed the few lines of code that were necessary to fix the bug because I wanted to understand like what is going to really change, and because the test coverage for this endpoint didn't exist, otherwise we would have found the regression during deployment. So together with the code, I also wrote the test. And the more you have a code base where you can quickly verify.

Whether the code base and the changes are working, the more you're able to trust AI if you are familiar with the tools, if you have a feel for the constraints, for the limitations of them. Because then when you make the changes, when you make the change, you understand what the result can be. Right? So we have, for example, our Tech Radar documentation. And in the TechRader documentation, we have a build process that is.

compiling static websites and generating markdown for the MkDocs site. And recently I was building the AI Radar where there is an additional view with four more quadrants. And basically I needed to have this radar built in a similar way, like code that was existing already. So the coding agent was basically able to understand the structure of the existing code.

And when with a well-defined output that it was easy to verify in terms of the files and folder structure that I wanted to achieve, the agent loop was working in multiple steps and multiple iterations, and it verified itself whether the output is correct. And there I did not read the full transformation of the code because I saw that the result is okay, and I saw that there was no regression in the layout breaking, right? So that's

My take that it highly depends on what your setup of the code base is and what is the risk of the code that you

Yeah, one thing is preparation, of course. And the other thing is, trust in the criticality. So if it's high risk, then you better look twice. And if it's like documentation or something, that's a lower risk.

It's basically

what you can rely on in terms of the setup. If you have high test coverage, if you have restrictions that you put in the code base where you enforce this test coverage, then you can trust the system because you know that it's built for accepting random changes to the code. And you can trust that those changes will be verifiable and easy to understand where when they break.

Do you see anytime soon where we don't look at code at all?

So code review was typically a handoff between team members, where a team member was understanding why there is a change made to the system and what kind of change is made and what you can do in order to revert it. Right. So you described if there is any risk and reverting the change, if there is a flag that you can flip, if this is something that is working in production.

it highly depends on the setup that you're able to achieve. Right? Because if you're able to explain code changes in a different way to engineers who are then going to support and do the on-call support after hours, and if you feel that the automation that you created con creates release nodes or creates the necessary flags that are required to operate the system.

Or if you can trust an automatic rollback that is triggered as part of the deployment process, then you're like more eager to let go of some of those controls. Right. And then you have a compliance angle. Sometimes compliance will stop you, and you're always have to have the human decision in order to roll out the change. but it also is highly dependent on what code you do for yourself. I think on social media you read a lot about individuals.

Who are building for themselves and they're the only user. The more you have a team, you actually have a responsibility about the code that you assume, not as an individual, but as a team. And so far, the code review was a place where you would challenge some of the direction that is taken with the code. You would discuss whether a potentially different approach is going to work better because it will make the code more extensible.

So it was a highly teaching moment for all the engineers who were involved.

Yeah.

I'm in generel afraid that if you take it away completely and not replace it with something else that has similar conversations, then you will lose all the good value of code review. Code review is not about naming a specific variable, pointing out typos, but it's all about finding out what the constraints are that are put into the system.

What are the edge cases that are being missed out? And trying to anticipate secondary order effects of the changes if they accumulate in a similar pattern or in a similar fashion over a longer period of time. And some of

those things you cannot replace with just an agent that will look at your code, at least not today. But you have to have those conversations front-loaded somewhere else. What I see frequently happening is that people are.

just throwing a pull request and saying, hey, I did a PR, and then you the burden onto the reviewer. Whereas in the past, you would be constrained with your capacity more. So you would have the conversation earlier: hey, is this something that we need to build? Is this

Hmm.

something that if I build it, will you accept this PR? And I think the open source projects that are stopping contribute contributions because they are overloaded with the incoming PRs.

They are experiencing exactly that, that the conversation about what to build is moving away to the moment where you get a PR, which is way too late.

That's interesting. I got at least two takes on that. The first thing is once you have that, speed with AI, you think you can build everything and you have to clearly choose what is, really makes sense to build. And on the other hand side is what you said. some of the experts in the internet that really, top notch in front of everything, they really claiming that it's important to look at the coach to not lose touch.

So if you don't understand what's going on, if you have like for a week, a day, I crumbling something together and you need to adjust it to the left or to the right, but you can't be precise in your formulation because you lose, you were losing your connection with the code. Do you see something?

Yeah, it's

like it's the same as if you would be an ops team that is responsible for operating applications that are built by another team, and all you get is release notes. Right? There will be setups where this is sufficient and where this is enough because of the pace of change, because of the scope of the product that can be very well understood. But as soon as you go beyond the original scope, as soon as you go beyond the conventions that were

present in the system before.

Then you will lose the ability to tell: is this something that's easy to develop? Is this something that is easy to add? Because you will not understand even what configuration knobs you have or what mechanisms you have during operations to stop a batch job, to intervene if something is not working. Right. So I think this is what you highlight as the importance of understanding and control of the code base that is ever evolving. And

Yeah.

like

There are teams who work with pull requests, and this is like the main way of how in the past teams have agreed on having changes or agreed on up level up-leveling quality, but there are also other practices, right? If you do trunk-based development, you have changes that accrue and accumulate over time, and as long as the build is green, you continue. While you have other means of how you understand what is changing in the code base, while you have other means to assess like what is the end product that we are building.

And are there any conventions that we want to adjust and change? Right. So I think we see that many of the engineering practices that existed before but were weird to apply, many of the lessons from them apply to the world where a lot of change is done in a short amount of time.

Yeah, that's finally makes sense why we do this and why teams with good engineering practices are way ahead instead of the others. You also mentioned something responsibility. If you push something in production at Zalando, you have some responsibility. You have a life system with a huge scale. Can you share a little bit at what scale you guys operate, what your architecture is and just a little bit where the...

more than 2,000 engineers work on every day.

So scope-wise, we cover B2C and B2B for e-commerce. For the B2C part, we have a marketplace. So we have a team responsible for the marketplace, onboarding of merchants, dealing with the products and the sales metrics and sales integrations with the merchant systems. Then we have a team that is responsible for the e-commerce experience.

Right, so it would be the main Zalando website and app, where you have domains like product, domains like customer, transaction processing, and content. And those teams are responsible for the e-commerce experience and for all the integration of the different systems that need to be in place. Then you have the fulfillment that is responsible for integration of warehouses. We have our own warehouses that we operate with our own software, where you have

also a lot of AI products that are deciding what is the

order and how you split the order into individual warehouse orders or shipment orders as we call them and you basically need to understand like do I ship one parcel, do I ship multiple parcels, from which warehouse do I do this? Then when you're processing the individual orders in the warehouse, you need to understand what is the optimal path in the warehouse itself.

And we have teams that build those algorithms that verify hi how we can optimize the route that a person takes in the warehouse in order to pick the items that they were assigned to. And you have also teams that are responsible for the logistics integrations with the different carriers that are picking up the orders in the warehouses. We have a team that's responsible for the builder infrastructure, so they operate our 250 Kubernetes clusters. they are responsible for.

making sure that we have a way of deployment with the CI C D system, with controls around deployments, with controls around reliability, and this team is providing the necessary building blocks to operate the thousands of microservices that we have deployed all across the business.

Yeah, fantastic. When I think about e-commerce, I always have to think about Black Friday and Cyber Week and things like this. What are peak moments for you and how you guys prepare for that?

I mean the peak moment is always the cyber week, as it's called, with the culminating moment of Black Friday. this is where a lot of customers expect that there will be deals and you have this direct traffic and organic traffic that is coming to the site. but you also have a lot of marketing campaigns because this is the moment in time where customers are there, so you better serve them with some great deals and some great expectations. It's also a moment in time where everything runs at peak load.

Right. It's all the marketing activities. it's all the integrations with partners, it's all the warehouses that run at peak load, where they're limited by physical limits, of course, in terms of the machines and the stations that you have for the workers. but still like any delay that you would have in the processing is going to cost you a lot in terms of the throughput that you have in the warehouse. And of course, on the website, in order to serve the users and serve campaigns that are driven by push notifications.

You have in a short moment of time a lot of users that, of course, over hours accumulate. And then all your queues that you have somewhere in the system where usually you can wait a few hours and they clean themselves and you do not have impact and shipping delays. On days like Black Friday, this is the moment in time where everything has to work the right way. And we prepare over the year to observe basically how our system scalability is changing. We do monthly load tests.

in production where we are scaling the system up, where we are pushing price changes, orders and traffic through the system in order to observe if there is any degradation and if there are any scaling limits that we were not aware of before.

Yeah, that's very interesting. Was there one moment or one special occurrence that you remember? I mean, you're at Solando for a longer time where you have like, where the whole infrastructure surprised you about the scale that you saw.

I mean there were moments earlier in my Zalando journey where there was a thundering herd problem, where one system was basically under load and it started losing capacity because of a configuration change, where a cache expired and then bringing up this fleet with all the retries and all the delayed scale that is hitting such a service, basically required an order of magnitude more

instances in order to serve the traffic and recover from the incident. I think the most difficult incident for me to coordinate as an incident commander. we wrote about this on our blog. It was a typo in a configuration that resulted in DNS hosted zones to be removed. And what this meant is that the whole platform went offline, including some of the admin tools, right? And

What we had to do is we had to restore the whole platform in a sequence. Right? So you had to take a call. Do you restore the users browsing the site? Or do you restore the order process or the warehouses and basically manage expectations? And we had like a bunch of people who are then restore restoring the DNS records in parallel. And they asked me, okay, so what service do I restore next? And you know, we have hundreds of services, so

first we started with some of the services that we knew are the critical points because they process some of the web traffic or the API aggregators. And then slowly by the journey part of product data display, catalog, cart, and checkout, we worked our way through this mesh of services in order to restore the entries. and the system's reliability helped us here because of circuit breakers, because of

our tiering of applications where we say which are the most important, which are the least important, we were able to prioritize in which order do we restore the effort. Right. And there you really see how many people need to work in parallel in order to restore a service and restore the experience for the customers.

I mean, people were asking you for the sequences that you have in lists. Did you prepare for that? Or was it like trial

and error, guesswork from experience? How did you come up with the right order of?

It was

experience from knowing the architecture of the system and also from knowing past incidents. but it was also like good guesswork. Right. And I asked people to report which systems still have problems. And there was one configuration service that was used somewhere in the warehouses. And I did not recall that we have this service because it's small because of its small footprint of use. So I needed tips, of course, from people, but the task was more to organize the communication flow.

Right, because you had a lot of people who were saying, Hey, this is not working, this is not working, they used the global incident chat. And then we had a call ongoing in parallel, and in the call, I was like serving the next person who was asking, Hey, what is the next system to restore? After we did it for 10 and we validated, we used a spreadsheet with the names where people had the cluster name or they had the three core service names that they have to look at, and then the restoration process accelerated over time like a snowball.

And the meme is still valid. It's always DNS, right?

Yeah, of course.

You mentioned before you have also AI products for your end customers already in place. Can you share a little bit of the experience how to run AI in production and how is it working for you?

So, you know, like we built AI products in the very early days, as every e-commerce system, the first one was recommendations with some collaborative filtering behind. And in the early beginning it was like CPUs and the algorithms ran on CPU. There was XGBoost as a framework or others that were used.

And teams learned to operate some of those systems. They learned how to define reliability targets for themselves. And later we built a framework for them. the internal framework is now about to be deprecated finally after five or eight years of operations. We did one mistake. We didn't care about the endpoint on serving the model. It was more about all the preparation and the training processes that were organized for a team in multiple stages. But serving is something that is super important.

Right, so serving started for us at with SageMaker. And because we didn't provide a platform building block for that, teams had to duplicate the setups, come up with their own, and some were more sophisticated, supporting A/B testing, faced traffic rollout, and some were like more I just switched to the new endpoint on the client side and I just deploy two models in parallel. There are some challenges that we have with like specific technologies. We are on AWS, so it's SageMaker.

Where it has limitations when it comes to the scaling capabilities and the cluster sizes that you can have with the model served. And we have, for our example, for our EdTech use cases migrated to Nvidia Triton in order to serve models through this framework. And right now we're running a project where we are modernizing basically our ML model serving.

in order to also include more structured way of managing GPU capacity. because the typical problem that you have at scale is how do you get GPU reservations? How do you make sure that teams who need capacity get the capacity and teams who have lower utilization of GPUs basically are able to donate those GPUs for the time being where they do not have their training routes.

So do you have your internal GPU brokerage system or is it more like you look about the demands and then based on the demands you assign GPUs to different services?

So we're building this up, right? So it's not in production

yet. so it's early stage, but we are moving into a direction where we basically do not have team level clusters and team level GPU capacity, but where we can have more capacity

shared across teams.

That's machine learning part. Do you guys have also GenAI components for your end customers?

we do have Gen AI components, like one of them is like very prominent in content generation. right. So you take, for example, sneakers and you want to have a video that is going to show the sneaker in a rotated fashion in like a 3D and then you also want a nice background to be shown. we have imagery generation.

where you generate teasers for campaigns also using genai so you put the products and you describe a little bit like what your anchor is for in terms of the background and this image is generated. We have a customer facing assistant. We were collect quite early and we built it back then on the ChatGPT API endpoints. We are continuously upgrading both the capabilities of this assistant that is available on the website and in the app.

Where can basically use it as an alternative interface to search. We've integrated image upload where you can search by image, where you can also ask about your customer care cases, about like where is your order and things like this. And those capabilities are just using the APIs that we have built for our customer care agents or the other features that are rendering static pages for you. But now you have this conversational interface.

face that a lot of customers are trying out and we see growth in usage every month.

That's fantastic. Do you guys host your own model or are you getting model data from hyperscalers or somewhere else?

so for the AIs that we build, we have like fine-tuned versions of open source models, for example for the image processing, where we have extraction of product attributes based on images. For the ML that we do and warehouses and for recommendations, we basically have our own models built from scratch.

that are proprietary and we basically host them on SageMaker or on or on gpus and do the inference there. And for genai we are using APIs in order to access models. Zalando assistant is using the open AI model.

And before we move into AI engineering, question about open source. Zalando has on his repositories, thousands of stars, they were famous for your open source contribution. How did you build that culture in your company and how do you live open source at Zalando?

So the culture was coming from the top. we had back then Eric Bowman as VP of engineering, and the mandate for the teams was basically that hey, if you can do something in the open, start early and start it as an open source project. Right. And we had a lot of teams that were migrating to the cloud back then. this was like the early days of us using AWS. So

Very favorable timing because you need to build many things. And we had, for example, Skipper, which is a Go request router. We built it in the beginning in order to route traffic between AWS and data centers. Right? So we needed a way of how to express that some routes are already running on the cloud and some routes are still running in the data center, and we needed the host names to be dynamic. so

This system evolved over time and it was always developed in the open. And over time it attracted contributors. the maintainers were also very open to receive like reports of missing functionality or reports of missing features that some of them came as PRs. And basically this is an example of a successful project because of strong maintainership.

but also because of its simplicity and usefulness in terms of other companies needing it and requiring it for similar use cases and over time it became a Kubernetes ingress controller and because of this powerfulness of the route language it's very easy to find use cases on your own where you can write a proxy and mock

Some requests where you can rewrite the requests that are incoming and fix some weird third-party systems that are sending malformed requests that you can replace on the fly in the proxy itself. And it isn't, of course, all that shiny in all the other projects. So we had some projects where we decided, okay, there is no uptick in terms of contributors from the outside. It's nice that we have it in the open.

But there is no bigger benefit from it. And we decided to in-source those projects in order to relieve the team a little bit from the burden of everything being public. And in some cases those teams moved quicker then. in other cases, like in the case of Skipper, it helped to have this as an open source project because of the design. You

Mm-hmm.

think twice about how are you going to encode your internals that you have.

within the company and you think about the abstractions that would be needed for other use cases. So it's more challenging to design systems like that.

and a little bit more

effort as well, right?

it is more effort and it requires like more thought, but the end result is a better product that's more maintainable, that's more extensible because you think about those extension mechanisms and how to build them. And we also had have projects that were like very successful internally, and we have like hundreds of applications that depend on them, where some of those Java modules and Java libraries became successful, but then there was a burden of maintainership.

We had colleagues who left a company that were very eager and building open source. And it turned out that they were kind of the sole maintainers, and that open source is not included in our backstage registry. So for management, those systems do not exist. They are a dependency the same as open sources. But there is like the sole maintainer, like in the meme, that is helping merge some of those PRs. And the way we solve this is we do reach out to the community of engineers and say, hey, we are having

projects that were that are looking for maintainers. And we try to fill the maintainer maintainer spots there. It's not easy, right? Because people are very eager in the call out and to get the visibility. But when it comes to real work, it depends. Right. And

Yeah, sure.

now I think that Gen AI is actually making it easier because with LLMs and coding agents you can more easily contribute to code that you do not know. So the onboarding time is quicker.

And it's easier for the team members to get started and also support those open source projects themselves.

And also, the AI helps to explain the code. So if you have a huge code base and where do you start, let it explain by the AI can help you to get a jumpstart at least.

Exactly. It can explain

to you some of the aspects. You also asked like how do we get buy-in? once a year, we build an overview of the top ten most rated projects and we try to capture whether any press mentions. do we have any blog posts that people wrote about the libraries? How many contributors do we have internally, externally? And we try to give the maintainers like a way.

how they can put a reference to this deck to their performance review to also show, hey, this is something

that I was working on, it had impact. And I also see this as like as part of my job to make sure that those external contributions are valued and seen within the company, even though they might not be on the day to day in the forefront.

So open source is not just writing code for you, it's more, it's like scanning the market, the attraction, how to, and also decide if it's still something that should be in the outside or we in-source it again. How much time budget do engineers get? Is there like, you have only 20 % time budget or is it, they can work as they want on open source? So.

So the

team members who work on Skipper, because it's part of our core infrastructure in the request router, they just work on the product and part of their commits is just going to be always open source. For team members who find a bug in an open source repository and they need it for their own work, it's also part of the work.

Because they can ship a patch and they can basically get this internally patched version of a library upstream from the contribution point of view. The biggest difficulty is usually the CLAs that you need to sign. So we have like an internal documentation page where we explain which CLAs we've already signed as corporate CLAs, which one of them do we need to still sign later. And this is like typically the most detracting.

aspect for for the colleagues who want to contribute because they need to wait until you reviewed some of the licensees. If this is the first time where we contribute to a project with this specific CLA. For the other colleagues, they have an option to basically discuss with their managers that they are maintaining a library, and we also help explain the value of this library to our community by basically saying, Hey, you have 400.

Applications that depend on this open source library. So it has value and we cannot leave it alone, and you will not be able to patch make security patches to your application footprint because you are one of those dependencies. Right. And we're going to see if we can split the load more evenly when it comes to those contributions. but what works best is as part of your like focus areas.

that are later evaluated as part of the performance, as part of your goals for the year. You basically put in that you have a goal to support Project X, and this is an open source project.

And then it's basically your daily work, right?

it's probably not daily, but like once a month you

Yeah.

do something. It depends on how active the library is itself.

Excellent. I like to switch topics and ask you a little bit about how you at Zalando approach AI engineering. So what are your approach? Are you have a platform for it? Just explain me how you do. Do you have minions or do you have like other famous engineering companies, something running already? I'm sure you do.

So I think it's good to start like how we give access to LLMs. So we have a LiteLLM proxy that we deployed two and a half years ago. And this proxy is providing access to models from different providers. Right? We use the cloud providers and of course AWS with the bedrock service for that. but we also have connections with OpenAI and with G C P and teams can basically choose.

tool that they use as long as it can connect to this API. Right? The typical hurdle for tools is how do you configure an API token? A API token is retrieved through a command line tool that we have, and then it needs to be injected in the authorization header in order for the API call to work. before coding agents existed, we wrote a small CLI tool. the CLI tool is

Allowing you to access models, allowing you to list what models are available. And it's kind of like a first discovery point for people if they are starting their journey, right? Because it's very easy to write ZLM with the command that you want to have, which is going to be the prompt. It's easy to redirect files to it. We built an interactive mode where you can like just chat along.

And before coding agents existed, this was like the main way of how people interfaced with the API. Right. And as soon as you have an API, there is an open source project, it'll give you a chat, so you do not have to use your command line, of course. but we expanded the command line to help with configuration of coding agents that team people are using, right? So the default configuration you can easily apply with like a config command, and then you pick what tool you're using.

And it's going to like put the right file in the right place for the user. We also have an easy way to have MCP server access in this tool because we ship as part of the command line a configuration file that has the host names or the repository names of the MCP servers that we built internally. And this way a person who has never interacted with an MCP can get like a command line command from their colleague.

They put it there and they have used an MCP. We use this also in trainings, which makes it for for an easy experience for people. And then from this point they can dive deeper and deeper and deeper. And what else do we have?

the command line tool also has some security features, right? So if you have hosted MCP servers that are internal, while the standards were still adjusting on what MCP authorization method to pick, we basically had MCP servers that were deployed that were just demanding our platform OAuth token. And we made it easy in the command line tool to basically inject this token for trusted host links, right? So you can

easily convert a hosted MCP to an MCP that is standard input-output because you say ZLLM proxy and then you put the URL and then you have a standard input-output server running. Similarly, if you have hosted MCPs that are used by the CLI, they will also auto-inject the token when it's needed. This allowed us to like bridge the gap until we have an authorization server, until we have an MCP registry.

And until we have figured out how to do the all in a corporate way, in a secure way, because we basically had the small tool that we built. was also

built by the community, not by like a team. The team built

Okay.

the proxy, but the CLI was like a hack week project that then escalated to a community project with like six people working on it.

And you basically solve the situation that they have local credentials on the file system and service accounts that you don't want. get the credentials injected.

Exactly, like it basically eliminates the need to configure an authorization header somewhere in your MCP configuration, or that you need to change the configuration every time the token expires. We basically wanted to avoid that people have all those bad practices that you read somewhere on the internet. Put your token here, right? We wanted to say, hey, do not configure any tokens because we will do it for you.

And how do you decide which MCPs are allowed and which are not? And is it completely free or do you have like a set of MCPs that you provide?

So we do have some community build MCP servers that allow you to access like our logs, the backstage system with all of its features, including the ones that we added on top with CI CD, where you can read logs, where you can read status, where you can understand what was the last build and things like that. we have some of the third-party systems that we are using, like linear, where the provider is providing the MCP.

And right now we're not having a solution where we enforce the MCP servers with like a registry or an allow list. We will get there eventually. but it's also a little bit tricky to basically block everyone from experimentation. We have a very active chat channel where people are asking, hey, has anyone used X? Is this safe to use? And we also have in our Tech Radar a page for MCP where you can copy the configuration.

if you need it, and where we keep a record of the MCP servers that are trusted and that we recommend people to use. Right? Typically the power users, they do not need guidance. they will just do whatever is possible and not restrict it. Whereas if you take a look at the people who need guidance who are more earlier on their journey, they will go to those registry, they will they will rely on the central guidance. Where we are moving towards is to having

part of our platform, the MCP registry, where it lists of the MCP servers that are deployed by teams and operated by teams, also as part of our backstage deployment. And we also see that some teams are adopting a pattern where they have a proxy in front of multiple MCP servers. So the team members only configure the proxy URL and they get a combination of all the tools.

And they do not need to configure five servers individually, which makes the setup easier. But also when we move to the hosted world as part of our agent platform, we will have like a good default for everyone that gives you the main MCP servers that we recommend teams to use. And how we also distribute MCP servers is through then through the agent skill marketplace. Right?

Hmm.

So you have a skill marketplace with different plugins.

We divided them by languages, by disciplines like engineering and design and product. And where we also have for some of our domains MC not MCP servers, but skills that basically help you interact with this domain. Right. So if you have content that is online, you have the idea of the content, you have a skill that helps you debug quickly. What is the content? How do I access it? What is the link? And so on and so on.

And in some of the plugins we have MCP servers defined. So if you install the engineering plugin, you will get a default set of MCP servers that are most useful for you.

I hear a lot of developer experience configuration that's pre-configured for the target group. And that also gives you the chance to be more in control, but also exchangeable and flexible when the landscape changes. Is that the core reason for that? Or is it more like you don't want to have like tickets from end users and make it as bulletproof as possible? What's your reason behind it?

I mean, if you

move to allow listing, then you have a challenge that you will get a ticket every few hours. Right?

Sure.

And if you do this too early without knowing what is the landscape that you want to support, without understanding, hey, what are people using, you will be too restrictive. And you will kill the momentum of adoption. What we look for with the LiteLLM deployment is that we have a way of being flexible.

Right, models are changing very quickly when it comes to the capabilities. And if you have a proxy, you're more flexible because you can route people to different tools, they can experiment with different setups. And we see, for example, that also model availability plays a role, the latency that you have on the proxy plays a role. we still have like the GitHub Copilot subscription that that teams are using.

And some of the tools allow you to configure both the subscription and the proxy as the model sources. For example, open code or Pi. And with this, teams can use some of the models that they have available on Copilot through the planning stage. And then for the execution stage, they can direct to another model that we're offering in the

Interesting. I hear Copilot, OpenCode, and I'm sure you also have Cloud Code and things like this. it only on a client for you or do you also see agents in the cloud in sandboxes running in sandbox? Do you guys have any infrastructure where you have some sort of an agent platform or something like this?

you know, like you can host an agent in a Kubernetes pod and it will work, right? So you basically have a PydanticAI and as a dependency on the application, you will have the agent loop, you trigger it over an API, and depending on what is the risk of the actions that you execute there, if you're subject to untrusted input, then you have prompt injection risks.

But if this is input that you prepare in your application and you're in your use case, you can be more trusted. So we have teams who deployed agents because they just treated them as applications. Of course, they go through more hoops in order to deploy that. We do work on an agent platform where we will use KAgent as runtime. And this runtime will make sure that there is sandboxing in place, that you can schedule some work.

that will be picked up at the right moment in time and there is an automatic cron. But on Kubernetes you also have a cron, right? So you can have a cron job that will trigger an agent. So for some teams it's very flexible right what they are going to use and as we get more use cases on our agent platform it will just become easier in a similar way as we have enabled open the move to the cloud with microservices where we provided templates and nice building blocks on how you can basically deploy services.

And what is your deployment manifest as part?

Yeah, so you mean you are in the building of an agent platform experimenting, but not the standard way where you run all your source code development on a cloud. Most of them is still on the notebook in front of you with hands on the keyboard.

Yes, exactly. We

have like local development environments. We're looking to see like what do we standardize and how do we standardize. for now we look into standardizing configuration. but of course, we have experiments on how do you run a local sandbox. and their restricting is also a tricky piece. Some of the agent harnesses come with sandboxing in place. So for this we have like a reference configuration. Hey, these are good defaults that you should be applying.

But of course, it's very much depending on the different use cases that teams have, what is allowed and what is not allowed. And with the different needs of teams, it's hard to provide a good baseline.

that's why we haven't done it before. Now we have of course more incentives with code being created by agents and also you willing to be independent of the time when your machine is on to schedule something in the background.

We do have tools where we can do mass changes across microservice repositories, where all you need to do is you need to define the transformation. Transformation is run in the Docker container, and you define basically these are the target repos that I would like this to run. And it runs on our hosted infrastructure on EC2 instances. And this is like a way for teams to schedule actions that is more prepared as a campaign that then crunches through hundreds of thousands of repos.

And we ship platform migrations this way, right? Where

Interesting.

basically the platform team is shipping those code changes, teams are receiving a PR with a reference on why this change has been made. And then we recently were shipping some of AI based transformations where you basically have like a Claude SDK or Codex SDK that is running to do the transformation, and we had a 95% acceptance rate on those code changes.

That sounds fantastic.

And these were like

non-trivial transformations of deployment descriptors in order to migrate from one pattern to another pattern. And it also gives like the platform team an emerge an enormous leverage. Because in the past they would be able maybe to do half of those changes that fill a certain pattern. and the automatic PRs would be eligible to be applied only on half of the code base, whereas now.

with DLM based transformations, they can really cover one hundred percent of the footprint and also prepare them for teams and avoid a lot of the coordination overhead that you have to ask teams, hey, when you will be able to migrate, and so on and so on. And often those migrations are like blocking shutdown of a system that really costs money and blocking cost savings that you need because you want to invest the money elsewhere, like

And these are like framework changes and code changes in a mess. You guys have a monorepo or like repos all over the place and the service go through all the repos. How is this organized?

we have services that have individual repositories. we have some monorepos, we have a monorepo for our web platform, for Zalando and for the mobile apps. they also have each a monorepo. with of course some dependencies, but the core code is in this big repository and those are repos that have like hundred over a hundred of contributors.

if a service is big enough, it's also a repo a monorapper on its own, right? So our GraphQL aggregator that has like a hundred different connectors, this is also something that is like big enough to call it a monorapper as

And these changes that are made by services or even by AI, are they only for one huge repo or are they for all repos you?

no,

so you define basically the transformation function, and you define a search term over all GitHub, or you define a list of repositories that you found out of Ben or Bend over search. Right? So you

Okay.

can put all repositories there, and then the transform transformation function will be called on a checked out version of this repository. And then it will be verified: do I apply this transformation? Because the Repo is eligible.

And then if yes, or if there was a result of the transformation, then a PR will be ready.

Yeah, interesting. And speaking about PRs, I assume you guys see a lot of increasing PRs. How do you deal with it? I mean, we spoke previously about that. You investigated some time to analyze PRs and optimize it. Can you share a little bit? What has your work on a PRs brought to life?

yeah, so we have an engineering conference that we run yearly, and in my opening keynote for this conference, I was like preparing some materials on where do we see impact of agentic engineering and LLMs across our code base. One of the things that I did was looking at PR sizes and the size distribution. And we could basically pinpoint the moment in time when we started making LLMs available, where the PR sizes

and the distribution moved to the larger sized PRs. Right? So smaller PRs, 50 lines and less. They basically quarter by quarter lose share, and we see the share all over the all over the place of PRs that are bigger than 100 lines, 500 lines, or thousand lines. And you know, the big question was: does this correlate with some changes on the code bases themselves? what we did is

We picked a few code bases where we knew that agentic engineering was used either from the beginning or somewhere later in the process. And we compared them with reference code bases that are a little bit older. And I wanted to see how cyclomatic complexity evolves over time. So I scanned basically every single commit for the cyclomatic complexity and plotted this. There are some nice graphs in the blog post that you can link later in the show notes where

Sure, man.

you can see that.

At the moment in time where code commits have attributed LLM usage with co-authored by, there is an inflection point where the complexity accelerates, right? So there is like a regular pace on how it accrues over time, commit by commit, and then suddenly when agentic changes come in come in, you see that this pace is accelerating. Right. And the big question is, does this mean that we are accruing more complexity?

Or does this mean that we are building more? Right. And we have one code base where we started doing spec-driven development from the very beginning as an experiment. And we see that the shape of accruing complexity has changed over time. And our hypothesis is that basically the time of developing a product has shrinked. So you have the same complexity that you would have in the end result.

If you are a little bit careful with the guard rails and the constraints and you still look a little bit about how the code looks like, but you have a quick build-up of this complexity early on. And then it accrues slower over time, right? So

Okay.

this shape is something that we will be watching over a larger sample of repositories. And then what we also see is that if teams are conscious about the complexity growth, they will do refactoring, and then there is a big drop of complexity and you're back to the trends as you were before with human-only changes.

You were mentioned cyclomatic complexity for those listeners who are not familiar with that. Can you share a little bit what is, what that's actually is and what you were looking at?

so it's basically a metric that allows you to understand like how is the code structure evolving over time. And you can say it's like another perfect metric because you can easily game the complexity by structuring your code in a different way. It's like typically you would have higher cyclomatic complexity if you have nested structure.

your conclusion is with AI you get a steeper curve at complexity but it stays at the same level for a longer time. Did I get this correct?

what we basically see is that it accelerates very quickly when you introduce agentic engineering. And what our hypothesis is, is that you need a lower amount of commits in order to get to a point where your product will have a feature set that is full. Right? So because you're accelerated in terms of the amount of change that you do, the incremental

complexity is growing and you reach the target state of the repository quicker. Right? But the time that we have and the evolution of the product, because we're still evolving them, doesn't allow us to tell this is the final conclusion or not. The

Mm.

comparison with like a code base that we have seen over like 10 years of Java code was basically that it was an asymptote of complexity slowly accruing and then plateauing. What we see with the

agentic spec-driven codebase is that the complexity accrues very quickly because you can express with specs how you want the product to work and you get quicker to a result where you have a working MVP. But later when you do incremental changes, the complexity will not grow that that high anymore.

And you're still watching it and to find out how it's impact engineering, right?

you know, like it's a it's a multi-year challenge because often code bases have like years of shelf life. you cannot replace them that quickly. So by looking at the amo larger amount of repositories, and this is something that we would be interested in is to see based on the four we have analyzed, do we see similar effects there? so far by looking at the three code bases, we

We're quite certain because we could pinpoint the co-authored by to an inflection point with increase of complexity. But by looking at the spectrum code base, the hypothesis came that hey, probably we are accruing complexity quicker because we are building features quicker.

Mm.

And based on the MVP of the product, this also is true. Based on the migration where we are building the new servers over a course of two years, we see that in the last half year, the migration pace.

greatly accelerated and the team says, hey, we did more than before with the same amount of people. But of course you have to see when do you really stop adding features to your system. Because

Yeah, sure.

then you can expect that the complexity will accrue. Or when are you going to refactor the architecture in a way that you have the right modules and the right extension points, because then also complexity will stop growing.

That's interesting. You also mentioned that in the beginning you could clearly pinpoint the time where you introduced LLMs and coding agents because the pull requests gained were getting larger. Do you cap it at some point where you say like, this is 50,000 lines, come on, reject it just by the

Well

fact that it's so many lines or is there a rule

how you deal with large pull requests or is it just the new way?

You know, the fifty thousand lines is probably like an edge case. if you really need in a monolith to change a framework because you migrate from one version to another and you need to have a byte size changed, maybe as an edge case you will end up with something like that. If it's incremental feature and product changes, then it depends on what your what your ambition is.

Right, if you have a very good RS, if you have very good test ver verification steps as part of testing, then you could argue, I will pick what I want to review. Right. So you will review some of the design decisions that a person has chosen. You will review the data objects and the schema changes, you will review things that matter for the risk of deployments, and maybe then you're fine with having two K.

lines of code that are changed on a regular basis, and you can deal with the pace of how the complexity and how quickly features are accrued in the code base. We have some teams who basically struggled with large reviews, and they said, Hey, we will impose a cap, but we will not force it somewhere in the code base. we will basically force it on ourselves with the good discipline. And here we come back again to good engineering practices. You can make code changes in a way that

Slowly increase the amount of features that slowly build out in small changes, a feature that is larger, because you're able to hide it behind feature flags, because you're able to have a little bit of thought how you want to design things and you design them over time. Not every pull request has to go live and immediately be used by users. That's very non-intuitive for people.

But you can prepare and collect some data with a pull request that you're shipping because you add a log line. And then you will learn how often is this actually called, or you have an edge case that you want to verify. So you will again have some metrics that you maybe push to your telemetry system in order to have a counter that will tell you how often this edge case actually happened. And after 24 hours, you will check was the counter one, zero, or maybe a hundred. Right? And this level of development with the small type of changes, that's what more

comes with experience and maturity. And of course you have to teach it to people and you have to say what is okay for you as a team, what do we agree on, what our conventions are. And it can vary per Repo because of your verification measures and because of the risk that the code bases have.

Speaking about risk, you also had some risk based auto approval things implemented. So you scan the code based on the risk and then you decide if there's still, it has to be human in the loop. Can you share some about that?

yeah, so we have a four-eye principle on pull requests, which basically means it's two people that need to take a look at the pull request and then one of them can merge or a third person can merge. From a point of view of us willing to reduce lead time overall in our pull request reviews, we looked at like what could we do in order to help teams, right? And we have some

default labels that we have in every repository that explain to others is this like an architectural change, a major change where a new feature or an API contract is introduced, or whether this is just a documentation change or a minor change. And it's difficult for teams to align what is minor in this repository, even though there is like some guidance on that. So you always need to have a system that is going to do the labeling. we did not decide to reuse the same labels.

but rather we build a API based service that is triggered through a webhook on GitHub. And this service is going to give you feedback on your PR. Right? So the change set is going to be analyzed with a nicely built prompt, and you get a result which is low, medium, or high. Right, and you have also the reasoning recorded with the GitHub.

native tooling. So you can always go back when you have an incident and see, this PR was high risk. So it's no wonder that it caused an incident. Right. And the rules set has been built by one of our senior principal engineers from the builder infrastructure team. And it's based on all the incidents that we had in the past one and a half, two years, and the lessons learned from what changes are risky and what changes are low risk.

Right, so example you would have documentation changes would be always low risk, right? Change in the README, change in the docs, because it has no influence on something running in production. However, if you change scaling parameters, if you change resource requests on Kubernetes, this is likely going to be a high risk change because it will change the load profile of your application, and it may have a detrimental effect on the performance of the application or

it will cause load to be distributed differently in the system itself. And then there is like the middle range somewhere where you have some changes that maybe you have a configuration change and you want to explicitly say that hey, all configuration changes should be reviewed by humans and you do not accept that an agent is saying that this configuration change is okay, because it's so difficult to just have the context context to be able to judge that.

Right, and with this classification system, we basically have one-third of the PRs that are graded as low risk. And we manage to agree with our compliance teams that for those PRs, the risk assessment from the bot counts as one approval. So for those

PRs, the person who is the author will get a risk-based approval, and then they can add a plus one and merge the PR with only one single human approval.

Right, and we see that those changes that are self-approved by the author and self-merged are the ones that are going quickest through the system.

did I get it correct that 30 % of all pull requests is auto matched today?

so they are not auto merged. They are assessed as low risk and

Okay.

the person decides when to merge them.

Okay, so you still have for everything that goes to production at least in check to tick. Okay.

Exactly. Right.

And basically you have the check that you tick. This is the compliance requirement. And then you also pick by clicking the merge button when it is going to trigger the pipeline for production deployment. And here again it depends on team practices, right? You can have a practice where you enable auto merge on GitHub when you raise the PR because you think it's like, it's such a small change that you're ex that you're okay with that.

And then you do not care about the exact moment in time whether it will go to production because you rely on your capabilities that you have. One of the capabilities that we build is you can define guardrails based on SLOs for your application. And as long as those guardrails are green and met, the traffic will automatically be increased on this application until it reaches 100% and you define a moment or

intervals at which you want the probing to be happening, you define the maximum interval for the deployment and based on this the system calculates how many switches it needs to do.

that's interesting.

You can build something similar with Flagger.

Okay. Didn't know about that. that means PRS and, when we go back to AI and the code that we built, we have now a powerful tool that allows us to build everything. How do you decide what should we build and what should we not build? So what is activity and what is really outcome you measure a lot? Do you have a feeling that you guys built the right things and,

You see the right quality with AI.

I mean, if you build products, you always have a product manager that is looking after the customer facing impact of those products. So backlog refinement is still fair practice that is not changed by agentic engineering, and you need even more rigor on that. We do have like in community-based repositories, also activity that is not subject to a product manager. It's more the engineers who are deciding things.

And we do see PRs there that are just raised without having a conversation before. Is this something that is useful? Do we want to accept this in this code base, or should we rather have it elsewhere? And I think it's like a new power where people are just excited that they can finally contribute to something that is community community-based, that they can build something that will be useful also for others. And we've also seen good examples of like.

Those contributions being prototypes that allow us to have a structured conversation about the outcome. And it just requires people to calibrate a little. Right? If based on this PR and the prototype, we have a good discussion that allows us to understand do we want this? How do we want to change it? And is this a feature that we would like to have built and accepted? It's also a good conversation to have.

Mm.

And not with an issue description. And there will be good good outcomes that come from PRs like that. There will be also bad outcomes because you will hit a team with a flood of PRs and they are unable to even read them because their capacity does not allow for that. Right? So you really have to have a feel on when is it something that is good to contribute and when it is something where you really need to seek to have the conversation first. We are not the

not discovering internal contribution models with Agentic Engineering. We've been doing this for at least twelve years as we moved to the cloud and accumulated many of the open source repositories. And also because many of our teams have the mandate to do changes across the stack.

Though this is common practice for you to contribute even outside of your team, just find some code base. And if you have an idea or want to change or need to change something, you just contribute to other teams repositories.

So the the

need is the important part, right? If you're building a customer facing product and you contribute to the web platform, the web pages or to the graph call aggregator, it's like daily work, right? That you contribute to a repo that you do not own, that you do not operate, because it's a central service that is provided by the organization. There will be additional guardrails there when it comes to

For example, on web, you get preview deployments, right? So you put a PR, you have a preview deployment. Based on the preview deployment, you can decide: hey, is this really something that I had to do? And then you flag this, hey, this is ready for review. Code owners will be notified. And based on this, you have an expectation that your code change will be reviewed. In other cases where you need to extend back-end services, we advise teams to seek the discussion first, to basically say, hey.

we w we have an idea or we have a dependency on your code base, how would we go about that? And in many cases, a team will say, our backlog is full, but we will accept the PR. Right?

Sure. Okay.

And when they accept the PR, then there will be repositories where this is easier where you can just do this with a coding agent without having prior knowledge about the application. There will be other cases where maybe you need to have an engineer embedded in the team for two, four weeks.

in order for them to get a feel how the software is developed and they will then carry this knowledge back to their team that they're working on.

Interesting. You mentioned something to sentence before that people can easy prototype something and prototyping is equal to wipe coding for me. What are your guidelines or rule sets to bring something wipe coded into something more serious, close to production?

I mean it depends, right? Because if you have a vibe-coded application that is just a static website, we have a documentation hosting system. In this documentation hosting system, you just say which is the folder that you want to have uploaded. And we have an S3-based deployment that will protect it with our employee authentication. And then you have your prototype that is online. Right? We see many applications using this.

Way of deployment with some dashboards, with a cron job that is going to refresh a data file that is adjacent or something that is collecting information from our data lake. And then you have a visualization of this information in different forms that existed before, which is like simple to adjust. The bigger question is like, what what do you do with citizen developers? Right. And what is the base that you will give them today, in our case for documentation hosting.

You start with a repository, which is an engineering journey. And you do not start with, I want to share something, which is more the product manager or designer journey. Right. So we look to enable this journey better by probably building a skill that basically is called share. And the skill will then take whatever you have and find a way of how to deploy it. Right. And we will start small with all those static websites and we will figure a way.

How we are going to leverage some of the hosting services that we created in order to keep a registry of what are those prototypes that are deployed. Today this registry is usually given by the repository or by the docs hosting system that is allow allowing you to find the documentation sites and there is a registry of all of

Okay. So it's all training based and, based with the right tooling to, to, get the people into the right mindset.

Exactly. And you know, like it depends also like what your setup is, right? If you today use an agent to build an application, it will use whatever training data it had or whatever tutorials it will find on the web. And then if you ask it, I want to share it, it will tell you, you can use GitHub pages, Netlify, or whatnot. And then you end up with this data being put on a third party that is not really used in the company. So by providing the skill marketplace and shipping some of those default skills.

This is like our way of distributing some of the company knowledge and company defaults. And we are on this journey, so we're figuring out like what is the best way of enabling non engineers to also contribute engineering code.

Some of the companies have something like the map of the world where they basically have a big wiki or a big craft database where they have all their processes documented, how the company operates to give the agents more information. Do you guys have also something like this?

So we have a hosted RAG system where we ingest all the documentation sites. plus teams can also add some of the Google Docs, for example, that they would like to index. And then we have a managed service that one of the teams has built where you can, with an API call, basically request a RAG-based assessment on your documentation sources. You can filter them in a way that you have like a namespace for your team or for your snippet.

Knowledge that you put in place. And we made this easily available as an MCP server. And we have also a website where you can ask questions, and questions will be answered using this right-based system. And we're looking into ways how can we expand different types of knowledge to be included, which will be like more curated institutional knowledge, which would be including some of the dependencies and application metadata that we have already captured in our other systems.

And it's basically like finding a structured way of how you make it available. And do you query all at once? Do you query this one by one? Or do you provide separate skills in order to query this? We are at an exploration here, trying to figure out what makes sense at scale for our teams. This documentation base indexing has been very well picked up by teams. And now they come with their own use case where use cases where they say,

hey, we have like a knowledge base that we use for our processes, or we have some of our employees who need access to this knowledge and we would like to have a chatbot and they are willing to have more of those documentation sources indexed there with potentially configuration on the retrieval. And we want to find ways of how do we avoid every team building a rag on their own,

Hmm.

because this will not be efficient from the point of view of operating these.

But it's funny to see that for years we were fighting for do some documentation, write down everything, give us process documentation, and all of a sudden for the AI world, want the agents to be smarter. they optimize for good documentation for their AI. Yeah, true.

Because now you have incentives. Right?

For documentation the incentives were not always there because it's nice to be asked. It's nice to be the person that has knowledge because you have a lot of validation through that.

Yeah, that's true. Another huge topic that rises more and more is AI costs. Are you guys looking into measuring the costs, optimizing for costs? What are you doing? Can you share something about it or is there something that you cannot speak about?

So because

of the early beginnings where we had the LiteLLM proxy, we also added cost tracking in an anonymized way. the way we want to understand costs is basically what is the distribution? Do we have users that are having a spike in the costs and then talk to them? Understand like, hey, what is happening? Right. So having many of those conversations to

Yeah.

understand like is there ROI?

Is there a pattern that we are missing? And there are like fascinating things that you discover through the conversations about setups that people have, about the skills that they build, that we can then ask for being contributed back to our collection. But also some of things that we missed early on, right? So, for example,

Hmm.

in LiteLLM, you can configure prompt caching. And we discovered that while the harnesses have an awareness of caching.

The use cases that people build against the API have less awareness of that, right?

So.

Because you have engineers who basically say, I built like my personal agent or an agent for our team for our use cases, I bundle some prompt templates together, and maybe I will have an MCP server that I fetch data from. And then they just call the API, right? And if they themselves do not look at the costs of this use case.

Yep.

then it basically means that they may not know that it costs a lot. They may not know that there is an optimization potential. And we did like multiple things there. one thing is you can turn a configuration knob in light llm that will activate prompt caching, and you can specify where you want the cache points to be inject injected. So you can inject it after the system prompt, you can inject it in the last message.

And then every time you have use cases that basically build up conversation history, you will benefit from prompt caching if you hit the cache retention times. And this helps the user, but still does not increase the awareness. Right? So for those use cases where people call the API directly, where they cannot rely on data, usage data that is that is automatic automatically collected by the coding harnesses.

Where you can use like a tool like CodeBurn or AgentsView in order to inspect this well-structured information. We ask those agent builders to use a proxy service. Right? So they start locally a process. This process is the interface between the LLM proxy that we have and their application that they have built. And it has a small TUI. And in the TUI, you see request by request.

How many tokens were used, what was the cost for this request, and you have a counter. And the counter shows you basically today what were your costs. And with this, you can easily like build an intuition for your use case. But also, what we added there is if prompt caching and if cache hits were zero, we make it red, right? So you

Okay.

can very quickly see, something is red, it's not working properly.

Mm.

It's a very easy visual feedback.

And we have a nice tutorial in our Tech Radar on how prompt caching works. We did like a knowledge sharing session about that to explain when is it the case where you have to take care of this yourself? And where is it something that it is going to be done for you? And the complexity of this topic is like also high because if you use bedrock, there is a different scenario. If you use open AI models, there is a different scenario, then.

The GPT-5.6 models have added now explicit prompt caching, have added costs for writing the cache where those costs did not exist before. So it's like an art on its own how to use prompt caching. Well, but what we definitely want to teach is that you can structure your code, you can structure your prompts of the name of the agent in a way where you have a higher likelihood of making use of the prompt cache on your own. And we had like also a fascinating case of a user that was running auto research.

with a cheap model, but they had like spikes of a lot of requests because they had 3,000 threads doing things in parallel. And their the users as well were like super excited and cost conscious, but they did not think about structuring the threading in a way that would also benefit from prompt caching, because they cycled through the experiments one by one and did them.

all with different prompts and different personas that they were trying in their use case instead of going persona by persona and then doing the different experiments one by one. So just by inverting the loop sequence.

Mm-hmm.

Similar way, like you make use of caching in your CPU and cache lines there, you now have another knob and an engineering paradigm to learn.

Great. Do you incentivize saving tokens or do you just let people experiment right now at this point?

So at this stage, we want to understand like where the higher spend is going. We want to see what we can do to help users to tune their setups. And as sometimes it's also like a bug somewhere in open code or whatever, where there is a specific scenario where the cache is going to be cleared. Or it's sometimes prompting patterns and Pi has a setting where you can activate notifications that will be.

output during the conversation where it will tell you, you missed your caching window by a minute. That's why we are rebilling you with that many tokens. It's like a beautiful tool that helps users basically understand what and how they are doing. And the case with limits is like if you set the limit, a lot of people will be asking, what is the limit?

And then they will be afraid, or will something happen if I do not have the limit? are you looking at that? Right. So you absolutely do not want to influence behavior of people to the best way that is possible. You really want to see what they are building, what is the arrow eye of what they have built, and what can we learn from this one person if we extra extrapolate it to a hundred, right? So, most recent insight that I have is.

for changes that go across multiple repositories. we have users who are basically checking out those repositories in the folder. And then if you look at the costs, those costs are then concentrated in this specific folder. And there are a lot of graphs across the code base in order to figure out how do I make the changes in the code base? And you have a lot of overfetching happening.

Because for some explorations, you may not need to go through the fourth or fifth code base, because the second was the one that already contained all the information for the task. And I do not have a good answer yet. That's like a finding from yesterday. to basically understand how to help those users in order to reduce their token spent in those cases. But by being able to still set some safety cap and then have a few users talk to you that have those edge cases.

is basically something that makes it easy to understand where do you need to invest in.

And it's also a great example how deep you look into the data and rely on data-driven decisions.

yeah,

like w without having the data, you will not be able to do this. but also like you want to be conscious about what you do and you need limits. Without limits, you cannot meet budgets. Without limits,

Thank you.

people will not learn that they have high usage, right? So

Mm.

some users basically say, I did not know that it costs so much. What is the tool that I can use? Right. And then we tell, hey, install CodeBurn, take a look at what projects, what sessions, and CodeBurn also has some optimization tips.

Where you can take a look, like one common pattern that we also see is command line execution. If the agent decides to run some command, run tests, it's very easy to have an output of the test end up in your context window, whereas it's completely not required. And one of the edge case that I also have in the blog post is we found a commit message that included test output. And the commit was authored by the agent, the pull request was created by the agent.

And that's probably where you really want to have a guardrail that your commit messages should not be bigger than X, because there is just no point and no value in there.

Yeah, sure. And highly recommended installing CodeBurn. It gives you so much insights. It was basically your recommendation when we last met. And it's really exciting to see what session information are stored on the client and how you can work with that. yeah, that highly recommend. So install CodeBurn and find out what the agent is working on and how much it would cost you. Before we wrap up,

We started with your role as an executive principal engineer. And now you said you got promoted, you got a team. How do you lead the team and what is the role of the team? Share us a little bit more about it.

yeah, so I have a team of experts. there is three of them, and we look to see how we can support the organization best. And we sp sometimes sponsor projects. we sometimes part are part of the steering committees where we help make the right decisions. In other cases, we are the drivers and we basically help make decisions on where to invest, what products to build.

Help understand some misunderstandings or tensions between departments because their goals are set in a way that create these. Or we're trying to unwind some of the complexity that has accrued in systems over time. And basically helping the organization navigate, unpack, break down, understand where those problems are, and help executives understand what type of projects would they need to prioritize, what type of work is coming the team's way.

we also do like horizon scanning to try to understand trends, what is changing. We support the AI enablement journey by basically using the tools, understanding constraints, understanding patterns

that we have across the organization. we recently hosted an un-conference internally with the principal community, but also with some of our senior engineers.

who have hands-on experience on a day-to-day basis in a bigger capacity than the principal engineers can have. And we had like 30 sessions on Identic Engineering to basically understand what is the pool of the organization, where is our knowledge, what are our coverage gaps. And with the team, we also rely on contributions from the community. So we scale through the community, both the managers but also the individual contributors. So we have some processes that we are

responsible for with technology governance, with the Tech Radar. We look after these open source things that I mentioned before, trying to and trying to help the organization understand what the value of our assets is that we have created in terms of software, in terms of know-how. And we also like host some community events on a monthly basis where we are helping the principals grow and their job.

where we are trying to understand what are the topics that are important for them to learn about, to understand, and help them solve some of the challenges that they maybe have on a day to day basis, help them understand the role, help them understand how to deal with situations that they have encountered on their day to day basis, and also be a helping hand if they

Mm.

have challenges and setting expectations.

So we have

like a guide for principal engineers who are new in the role, where we are explaining these are the hundred days, these are the things that you should plan when you're new in the role, these are the conversations with your manager to have that are going to help you understand what your mission is, but also

to connect with people from product to get more understanding about the business domain and about the business objectives that the organization has.

And is this always received as a helping hand? mean, if I'm a team, I'm in the trenches, I have a full backlog and I just want to ship on, I have on call and things like this. And then all of a sudden there comes the principal from somewhere and he says, no, you can't do that. How is this in your organization implemented so that you, even if you are not directly responsible for the team, they still work with you, align with your thoughts and.

I mean if you speak about principal engineers who are like supporting three, sometimes five teams, they have to understand how to build credibility within the team. Right? Some credibility is built by doing hands-on work. Some credibility is built by helping to show that you have context that is important for the team to understand. And one of the key skills is whom do you talk to?

Okay.

Right.

Do you talk to a trusted engineer in the team in order to understand what the health of some of the systems is, where the challenges are hidden? Or do you talk to their manager to understand, hey, how do I help you as a peer? Or are you talking to your manager who's the head of engineering, who would be the one having some goals and aspirations where they want to take the team, what product they want to run?

but maybe they see challenges in the team when it comes to skill gaps, when it comes to processes that are running slow or that are outdated, that need to be improved, or where maybe the operational health of a team is not where it has to be, where the principal can really make a difference, which is going over time to give them the credibility that they require.

So what is your favorite way to lead in such situations?

Mm I mean, I don't think that there is like one single recipe that you can apply there. there are different things that work depending on the situation. I think the best outcomes are coming when you get a shared understanding of what the challenge is and you get to set some of the guardrails and you let the team come up with ideas and you challenge the team.

You give them a different perspective that they maybe have not thought about. but it's like highly collaborative way of solving the challenges and letting other people grow, right? Because that's the most important function of principal engineers, that you help to create an environment where others are able to grow.

I think that's great. Final words. Thank you so much. Before we finally close, where can people find you on the internet? I heard some blog posts that we might want to link. Is there anything that you want to share?

yeah, sure. Like I have a LinkedIn account where I link to the stuff that I write. I have a personal blog with my last name.net that I put on some of my things that I'm frequently repeating to others and I get tired of repeating them, so I wrote a write a post so that I can link to this post.

Or things that are part of my explorations, especially around agentic engineering, where I've been figuring figuring things out and sharing some of those lessons learned. On the corporate blog of Zalando, I have also posts that speak more to some of the projects that I was driving internally, which are often good references on hey, how does principal engineering work at Zalando?

the post back then was written because we didn't have that many principal engineers and we learned about the structure. So we needed to think about the next generation. And the next generation needs to be bred in other companies, right? So having a reference that we can also share with others was helpful to then have people who can at some point in time decide to continue their career with the Lando.

Fantastic. So I make sure that I put on the links in the show notes. We started the conversation with a question that I brought to you over from Ralph. So now I ask you, do you have a question for my next guest?

Yeah, I would like this question to be like engineering heavy. so the question is what's your favorite data structure that you implemented from scratch yourself?

that's a fantastic one. So what's your favorite data structure that you implemented from scratch? I like sure that I take this over to the next guest and I hope you will tune in and listen to But thank you so much for joining me on a podcast. It was a really great conversation. Thank you so much.

Thanks for your questions and for the great conversation. I enjoyed it a lot.

Thank you.