Wix's Josef Goldstein on building a multi-lane streaming architecture that powers 40B daily events and AI agents at scale.
Real-time data and AI are converging—and companies that have already solved the data pipeline problem are pulling ahead fast. Wix processes over 40 billion interactions every day across hundreds of millions of websites, and the architecture behind that scale didn't happen by accident. It was built, lane by lane, around the principle that your upstream data must be at least as fast as your fastest use case.
In this episode, Josef Goldstein, Head of R&D for the Big Data Platform at Wix.com, joins Joseph Morais to unpack how Wix evolved from a batch-based Hadoop architecture to a fully streaming-first platform on Confluent Cloud. The conversation covers Wix's multi-lane data architecture—from petabyte-scale data lakes to sub-second algorithmic decisions—how they approach data contracts and governance at distributed scale, and why Confluent Freight Clusters became a strategic unlock for cost and elasticity. Josef also explains how Wix is now wiring real-time stream processing directly into context layers for agentic AI systems.
You'll Learn:
About the Guest:
Josef Goldstein is an Israeli software engineer and engineering manager specializing in big data infrastructure and real-time analytics. With over 15 years of experience in developing data-intensive SaaS applications, Goldstein has built and led high-performing teams in the technology sector. Since 2021, he has served as the Head of R&D for Wix's Big Data and Analytics Platform, where he oversees the infrastructure and tools enabling data-driven decisions for the company's users and operations.
Guest Highlight:
"You understand that you need to be on the upstream side of the house as fast as your fastest lane. Otherwise, none of that is possible."
Chapters:
[00:00] Wix.com Overview and Customer Base
[07:08] Segment 1: Data Streaming Goodness
[24:24] Segment 2: Beyond the Stream
[45:37] Segment 3: Quick Bytes
[47:08] Joseph’s Top Takeaways
Dive Deeper into Data Streaming:
Get Connected:
Our Sponsor:
Your data shouldn’t be a problem to manage. It should be your superpower. The Confluent data streaming platform transforms organizations with trustworthy, real-time data that seamlessly spans your entire environment and powers innovation across every use case. Create smarter, deploy faster, and maximize efficiency with a true data streaming platform from the pioneers in data streaming. Learn more at confluent.io.
0:00:00.2 Josef Goldstein: If you want your agent to feel smart and not like my information stops at 2024, whatever, then the data also needs to be fresh. And we have a lot of focus on this right now, which is, how do we take these curated streams, we slap more stream processing on top of them and then we take those insights and now we're able to serve it as a context layer to agentic systems. If you don't know about all this stuff happening behind the scenes, it just looks like magic. It looks like it's reading your mind.
[music]
0:00:34.7 Joseph Morais: Welcome to Life Is But A Stream. Today I'm gonna be talking to the head of the Big Data Platform at Wix, which you may think of as just a website builder, but they are much more than that. They're making dreams come true on the Internet, processing over 40 billion interactions a day. And I'm gonna be talking about how they're using real time analytics and microservice decoupling to make the best possible decisions for their customers and their company based on real time data. I'm Joseph Morais, let's get started.
[music]
0:01:10.6 Joseph Morais: Well, thanks for coming on the show today Josef. Let's jump right into it. What is under your remit as the head of the Big Data Platform at Wix.com? And tell us a little bit more about Wix. Again, I'm a kid of the Internet, so child of the Internet, so I know Wix pretty well. But for the uninitiated, why don't you tell them about what you guys do?
0:01:27.8 Josef Goldstein: The way we like to describe it at Wix is we are... Our business is helping anyone make their dreams come true online. Right? Which I really love that title. More specifically, we're a platform for building websites and online businesses which is obviously just getting more and more important. Obviously there's a big question of okay, fine, but where does big data has anything to do in that? So the way I like to think about it is, imagine that on top of Wix there are hundreds of millions of websites. There's billions of users using them on a day-to-day basis. And what we're trying to do in my side of the house is basically try to track all that. So we're trying to be like the eyes of everything that's happening inside of this from the users that build websites or building their business using our platform to their users that are using their websites and hopefully having a pleasant experience. So all of that sums to, as you can imagine, quite a lot of data which is kind of why we need like an entire team to just manage that.
0:02:25.2 Joseph Morais: So I know we'll go through this as we get through the episode, but at a very high level. You know, I think the first thing that's probably most people are curious about is why a website builder, and I know Wix is a lot more than that, but that's its core functionality, needs so much data. Right? And I think the answer, you already gave the answer, right? It's to ensure that everyone's having the best possible experience. Is that kind of the core of why data is so important to Wix?
0:02:50.8 Josef Goldstein: Yeah. So the buzzword obviously is data driven. Right. In our philosophy we break this into three. The first one, and that one is kind of the obvious one is a, we call it data driven Wix. Right? So Wix is a huge company, but it's a pretty big company and ideally we wanna make business decisions based on data rather than opinions. Because if you have an opinion, I have one, we don't have data, let's go with mine type of thing. So a lot of this has to do with basically how to make a decision, how to move the business. Obviously when you're serving millions of people, you can't just go and ask them. You need to have your eyes, you make your eyes using data. So that's kind of an obvious thing. That's the world of internal analytics, you would call it. Then you have the second pillar, data driven users. If you're building your business on top of Wix, you also need to have those eyes. So we want to provide you with the data to basically understand how to manage your business in order so you will be successful. So we will be successful and have a pleasant experience.
0:03:52.2 Josef Goldstein: So that's data driven users, how we help our users make data driven decisions. And obviously now in the age of AI that's even more important and there's a lot of opportunities in that world. The last one, and that's kind of the invisible one, is a data driven product. How does the product itself customize itself, become smarter, behaves differently to accustomed to you as a user. Right? So maybe you are interested in building an E-commerce business. So your user experience is gonna be slightly different than someone that's looking to build a booking type business. So that's obviously now again also going through kind of a revolution through AI because our definition of what is a data powered product changes when we're talking about agentic systems, but that's the third pillar. So what we're trying to do is take essentially the same source of the data. So you can imagine the interactions that are happening across all the sites, user experience, transaction stuff that's happening in the back end. Right? So we're collecting, can say around 40 billion of those interactions a day.
0:04:55.5 Joseph Morais: Wow.
0:04:56.0 Josef Goldstein: And how to kind of organize them and serve them in a way that can provide the best possible experience and value for each one of these pillars.
0:05:03.6 Joseph Morais: So you have, you're using data to drive the company, you're using data to help the users enhance their outcomes, you know, whatever their dream that they build is, you wanna make sure it's optimized. But then also data driven product. Can we take signals from our user at an individual level and shape the Wix to them to make it optimized for them? I think it's very, very good trinity of data usage. So before we close out this segment, tell me more about Wix's customers. Right? Are we talking just, you know, people that are creatives that are trying to promote a brand, are people doing E-commerce through Wix? Tell me all about what a Wix customer looks like?
0:05:47.2 Josef Goldstein: So this is interesting because it's also changed over the years. I think that originally Wix started addressing either client or a small business. So obviously you have your own mom and pop shop. You can't afford to actually bring someone to build your own website. You use Wix, you do it yourself. I think that category is slowly expanding. So we see a bigger size businesses using this for various use cases. So for example, now with Base44, which is kind of a new thing in our house, we are seeing companies building internal products and tooling using for that. And then slowly we are seeing like more and more enterprise customers coming in, leveraging the same platform just to solve similar problems just at a different scale. So we're kind of across this entire thing from like a person have I wanna build a website for my, you know, art portfolio. So that's just me showcasing it all the way to like a big enterprise customer that just wanna spin up websites for all the branches of their business across the state, for example.
0:06:46.6 Joseph Morais: I imagine there's probably some relation there. Right? You have people that maybe grew up building websites with Wix for their own personal reasons. Right? And then they go on to do bigger things and eventually end up in an enterprise and say, well I'm not gonna write HTML and CSS on top of ColdFusion, I'm just gonna use Wix. [laughter] That's gonna be so much easier. That's fantastic. Thank you for that.
[music]
0:07:16.1 Joseph Morais: So we've set the stage. Let's dive deeper into the heart of your data streaming journey in our first segment. So we know kind of the why or the what data streaming is, but tell us more about specifically what you and your team built to give you all this visibility?
0:07:30.9 Josef Goldstein: The data journey at our platform actually starts in the end product. So to make it simple, let's imagine a scenario. We have a customer and the customer build an E-commerce website, this very basic one. And we want to track the shopping cart experience. Right? And it's kind of an interesting thing. We need to track the experience of you building the website with the shopping cart and also later to understand whether people using that experience on the end website have a pleasant experience. Now you can imagine that throughout that you have quite a lot of interactions and business transactions happening. So there is the act of a user going to the UI and clicking and navigating and trying to find their way. If we're managing a shopping cart, then behind the scenes there's this inventory that's just being managed by several microservices, several databases. Maybe if we're going into the act of actually buying the stuff in the shopping cart, then there's a transaction happening, there's work that's happening with some kind of a service provider in the payments. Throughout that entire experience, what we're trying to do is to collect what we call analytical telemetry. So obviously we didn't invent that.
0:08:40.4 Josef Goldstein: That's something that's very... It exists everywhere. Right? Google Analytics, for example, is built on top of that. There's platforms that do that. Now we kind of build it from scratch by ourselves for two reasons. One, when we started this journey, which was years ago, there wasn't really anything in the market that did it, especially in our scale. And the other reason is we have the privilege to make it extremely bespoke to our needs. So what it allows us to do is essentially, if you would imagine, we collect the click in the UI, we collect the backend change that happened in the database, and then later we track the response maybe from a transaction happening with a service provider. And we can place all that into a single stream of data that we can later process as a single unit. So you can compose something that involves user experience or that's in the user experience analytics world, or you can track stuff that's more like CDC like with what's happening with the data, but also collect stuff like integrations with third party consumers. Right? And then put everything into a single stream. When you sum this up on all what we call the verticals, the various things that you can do at Wix, on the number of users and all the transactions happening, that sums up, like I said before, like around 40 billion of those events. They're coming from either your browser or from a backend service at Wix, et cetera. And that's kind of where the journey starts. So all of this data, we're trying to collect it obviously as fast as possible, as real time as possible. And I assume we'll talk about that more later into, you can think about it like a delta, like a stream that basically tells the story about everything that's currently happening in the business. So that's where it starts.
0:10:19.6 Joseph Morais: Now, I'm curious, and I generally don't dive too deep, but of course I'm a Kafka guy, so I hear these things and I'm curious about how it works, at least at a high level. It sounds like all of these different data functions, whether it's building the dream, was the dream optimized, was the experience good? These are all emanating, I imagine, as events and they have some type of session ID or something like that, some identifier that allows you to then string them together, probably using stream processing, something along those lines. What were the underlying data or technology challenges that led you to adopt data streaming?
0:10:56.9 Josef Goldstein: So maybe we need to talk a little bit about our history, right. Because I mentioned that we built that way before there were system available to do that. So the history of this entire product that I'm talking about right now, obviously in much smaller scale, started roughly almost 10 years ago. So if you go today to historical data that we store, we have data that's from about 10 years ago. Back in the days, obviously Kafka was around, but it definitely wasn't what it was today. And also the thought process wasn't real time. It was just like how do we get that? And the way it was built was, as you would imagine, batch processing, Hadoop like that stack. Two things have happened since then. The first one was slowly the realization that we need the data to be as fast as the fastest use case that we have. So once we got into things like personalization and AB tests that calculates cohorts in real time, then you kind of realize that batch processing doesn't really fit. So you need to reduce the latency significantly. And that's the first place where the architecture doesn't work as well.
0:11:53.6 Josef Goldstein: And that was the movement into this kind of a more of a streaming type Architecture and that's I think where we introduced Kafka into the stack. I think the later phase that came after that was as scale increases, running Kafka or in general running stream processing at this scale, especially as the J curve becomes more and more steep, is significantly harder. So just to give an example, when we are, I don't think we finished with the J curve, but we're not running as fast as we did in the COVID period. Right? The traffic that we see on these systems are still growing in double digit percentage year over year. So being able to maintain the backbone for dealing with this is not our core business. Like we want to do our bespoke logic. We want to deal with integrating into the different systems, make value from this data not necessarily working on the pipe that needs to run it.
0:12:47.1 Joseph Morais: Right.
0:12:47.6 Josef Goldstein: So...
0:12:48.5 Joseph Morais: Right. You want the data, like let somebody else handle the pipe.
0:12:51.5 Josef Goldstein: Exactly. And that's basically what eventually led Wix, not only by the way for analytical use cases. Right. Because we use Kafka very much so also on the operational side of the house for asynchronous communication between microservices. So that was also a big thing during that time. So that was kind of what led us to move to Confluent eventually. And that's where we're running our Kafka and stream processing today as well.
0:13:13.5 Joseph Morais: All of this really, you know, the underlying use cases to make these best possible decisions on the freshest possible data. So walk me through how data streaming addressed microservice decoupling and real time analytics at Wix. So I know we kind of, we've talked about all these in, you know, kind of different, partially across different questions. But let's bring it all together. You know, obviously you mentioned this turning point, which is, I think it's very interesting. Right? It's the first time someone mentioned on the show that the data had to at least be as fast as your fastest use case, timing into... So that's like the, that's the operational side. So tie me into the, you know, the real time, the analytics side and some of that stream processing that I kind of inferred about earlier?
0:13:58.9 Josef Goldstein: I like to think about it like lanes as you think in, like when you plan a highway, right? You don't plan the same... All the lanes from the same speed. Right? The slowest lane in analytical data is usually the data that ends up in the data lake. So we obviously manage everything in the data lake. As you can imagine, 10 years of data, that's about a data lake of over 10 petabytes of data. So that's quite a lot of stuff. But that's relatively slow moving data. So even if we're streaming the data into the data lake, we're still talking minutes, which is still very, very fast in data lake or data lake house at this point actually terms. But it's not real time as you can imagine. So that's the slowest lane that you can have. But it's still real, it's important. That's where we store the source of truth, right? The history of everything. Then we go up to other use cases. So another use case that we have is, we are using this data as a way also to track the liveliness of the business. So if you'd imagine classically I would monitor the CPU and memory of my microservice, which is still super important, right? Or API calls and error rates, et cetera. In Wix we also have the concept of measuring business KPIs. If we... We have a fairly big production team and part of their job is to make sure that Wix is alive, which is a very weird term, right? But how do you measure it, right?
0:15:16.0 Joseph Morais: It makes a lot of sense to me, but it is weird.
0:15:18.9 Josef Goldstein: And some of the way of knowing that Wix is alive is saying, okay, the Kubernetes cluster is not in the red, which is very important. But another way of saying it is, are payments running through, right? Are people able to subscribe and upgrade to a premium, right? Are people able to publish their websites? And those are more of telemetry from behavior and things that need to happen in a healthy system. Quite a lot of the way we are kind of checking the health of a system is actually based on the same type of analytical telemetry. So we take this data, we kind of compress it, aggregate it, we put it into ClickHouse and then we monitor it from there. So how fast should that lane be? That lane needs to be up to a minute because anyway we're probing it one to a minute, right? Ideally we wanna know about an issue before that, but that's as fast as it needs to be. We're still not real time at this point, right? But it's a faster lane than the data lake house.
0:16:13.7 Joseph Morais: I like this...
0:16:14.2 Josef Goldstein: Slowly...
0:16:15.2 Joseph Morais: I like this. The first time I've heard this analogy, like different lanes. I like it, but yeah, please continue.
0:16:20.8 Josef Goldstein: So you continue down lane. Then another lane is our customer care team, which is a fairly big team. They need to be able to help a user without necessarily having to just log in into his name or asking him to share a screen or things that are very uncomfortable. So we have a system that enables them to at least partially see what the user is currently doing and have done in the last maybe few minutes to help troubleshoot, right. So now we're talking about something that needs to happen in many seconds, because otherwise that's not an interactive tool. A person can't actually use that for debugging. But that's still human speed, right? So that's what I am able to consume as a human. I wouldn't see the difference between one, two, three or four seconds at this point. Then you go to the fastest lane. And the fastest lane is the pillar of what I mentioned about data driven products. When we're talking about computer speed, about algorithm speed, you need to be as fast as possible. So for example, if I'm going to change a user experience based on a decision, now that decision can be, I've summed up and four turned into five and five is the magic number.
0:17:24.9 Josef Goldstein: And when five hits, then I do an X. Or on the more complicated side of the house, which enough features in a certain feature vector have aligned, and I've run inference on certain machine learning model and the machine learning model spewed one instead of zero for the first time, and now I need to make a decision. That's where you really wanna be as fast as possible. And that's kind of the fastest lane, right? So the fastest lane in this specific case behind something like I mentioned right now, will be a system like our feature store. So we'll be computing all of these matrices in real time and serve them either directly for the UI or subsystem to make decisions in real time as the user interact with the system. And there we wanna be as fast as possible. So when you take all these lanes, you kind of realize that you on the one hand need to architect to be able to serve all of them. Right? And obviously each one of them have different needs and different systems et cetera. Part of the complexities of building a big system is the heterogeneity of tooling. But you understand that you need to be on the upstream side of the house as fast as your fastest lane, otherwise none of that is possible.
0:18:31.3 Josef Goldstein: What we're trying to do all the time is to optimize for these two things. One is, how do we optimize for having the nice lanes not interrupting with each other, but eventually also how do we make sure that everything that's happening upstream, so basically from the point the user clicked the button to the point that we can calculate an insight, we are meeting the SLA of our fastest lane, which is in this case basically having the system react immediately back to the behavior that happened by the user. And usually what we wanna do that is to be in the very few seconds and ideally sub second, if we can even make it.
0:19:01.0 Joseph Morais: I love... Like again I had... Sorry I interrupted you, but the lanes I really like. So it really comes down to it's starting, knowing that the entire system now is built on telemetry. It's about having all that data collected as fast as your fastest use case. And then at that point, once it's into your data streaming platform, then you can really decide how fast do I need to emit this data? If it's a lakehouse, minutes are okay. If it's troubleshooting in real time, seconds are okay. But if it's algorithmic speed, microseconds are the key. And I know you mentioned, so we have the streams, we have the stream processing. I'm curious, is data governance like Schema Registry, is that an important part to the puzzle as well?
0:19:43.0 Josef Goldstein: So yes. So we're not using like strictly the known Schema Registry, like the one that usually everyone working with, when they use Kafka. Again, by the way, for the same reason that I mentioned before, we had the problem before the solution existed, so we kind of build it ourselves.
0:19:56.6 Joseph Morais: Sure.
0:19:57.2 Josef Goldstein: I actually like to talk about it from the position of data contracts and data products. I know that, so I know that data contracts, as a term was incredibly sexy a few years ago. It's kind of dying down like data mesh, it becomes uncool to say that. I think it's still real. Like the reason why it's kind of losing traction is because as usual, there's a good concept and then people pile on top of it other meanings that kind of make it be less cool, but I think it's still important. So basically when you're running this type of distributed data operation, but when I say distributed, I'm not talking about the number of machines, rather the number of sources and owners. Right? So if you can imagine if we're trying to collect information about all the domains that exist at Wix, behind the scenes, there's hundreds of data people, people that their job is to say I want to collect this data point, make sure that the data is instrumented, make sure that there's actual meaning behind this. And obviously all the continuity of how do I wanna process this data to reach the insight that I wanna get from it.
0:20:53.2 Josef Goldstein: So that doesn't happen by a single person, that doesn't happen by a single team, that happens distributed across the entire organization and not necessarily the same person that's gonna create the data is gonna consume it. So potentially I could have one person instrumenting data about the booking system, but I have another person, their responsibility is to aggregate information from a few systems and serve it to some product. They're not working at the same team. So the only way for that not to turn into a complete mess is to come and say, okay, so one thing is, first of all we need to understand what data we have here and we need it to have certain guarantees. And that essentially is a data contract, right? Like you can go into the specificities, but in the philosophy of things, that's what it is. When you wanna apply that to stream processing systems, you start with the very, very basic thing which is what's the gate of controlling what goes into the pipe and what's the gate of making sure where does the data go when it comes out of the pipe? So the way we do it in our house is, and actually I think I gave a talk about this a few years ago, is we build our own system to kind of manage this.
0:21:52.6 Josef Goldstein: So the way it works is kind of in reverse. You first define what data you wanna collect. We have this kind of events catalog and you kind of say, okay, I want an event for the shopping cart and this is how it's gonna look. And it has an owner and the owner is usually a team, not a person. And the description, it means what it does. And then an actual schema that defines, okay, this is what this field mean, that field mean et cetera. Not a single line of code has been written at this point. Not a single event has been fired. But the schema was defined, the process goes on from there. So we do code generation, we create SDKs. The SDKs ensure that the data comes out of the source already structured, and then obviously along the entire chain in the pipeline later on we can use that to verify things so we can make sure that the data actually meets the criteria that it's supposed to be. Right? So the string is a string, the UUID is a UUID, et cetera. We can do all the quality testing we need. And again in a unified way, right?
0:22:44.4 Josef Goldstein: We don't need to talk to like hundreds of people to be able to do that. And then later on we can also guarantee on whoever wanna consume this data down the road like this is what you get, what you see in this catalog, this is what you can get. You can trust it. And we are making sure that this is happening. So this is kind of like our flavor of Schema Registry and governance to those streams.
0:23:02.1 Joseph Morais: I like it. I mean a lot of the things that you guys did is what we ultimately built into our data portal and our governance package and Confluent Cloud is, you know, providing that kind of that metadata about your data. So it's not just the schema, but who's the owner. Right? It's like something you hit the nail on the head is that very rarely, if ever, well say very rarely, groups that are producing the data are the ones actually consuming the data. So it's something we talk about here a lot, we call it Shifting Left. The idea of processing the data as close to the source as possible, because that's where the people that own the data, that know the data the best would know how to process it. A lot of time people dump everything into a data lake. Right? And you got these poor data scientists that are like, what is this data? Right. I'm glad Wix already has that figured out. Next, we're gonna dive into how your partnership with Confluent solved your data challenges. But first, a quick word from our sponsor.
[music]
0:23:58.0 Speaker 3: Your data shouldn't be a problem to manage. It should be your superpower. The Confluent data streaming platform transforms organizations with trustworthy real time data that seamlessly spans your entire environment and powers innovation across every use case. Create smarter, deploy faster and maximize efficiency with the true data streaming platform, from the pioneers in data streaming.
[music]
0:24:26.4 Joseph Morais: Now, let's go beyond the stream on why Confluent was the right fit for Wix. All right, so again, this whole conversation has been mired with data as it should be. The show is called Life Is But A Stream. But we know data streaming was the answer. But as you mentioned, you know, you wanted the data. You didn't really care about running the pipeline because as you mentioned again, just quoting you, as the scale of your data streaming grows, the complexity in running it gets massively more difficult. What made Wix choose Confluent as their data streaming platform?
0:25:03.7 Josef Goldstein: I think it starts with Confluent is kind of the house of Kafka. I think that's everyone's perception on this. And that was a decision that was even before I was at Wix. But that was kind of... I assume that the same thought process went, which is kind of, that's the go to, you go to the source. And besides the idea of being as close as possible for the maintainers of the original projects and the people that kind of spearhead this entire thing, which is always important, I think that at the end of the day, streaming is not just Kafka, it's an ecosystem. And the agenda at Confluent always was to turn this into like a streaming platform. So for us it means besides the fact of I think we have really good partnership with Confluent and stability and being production grade and like five nines and everything that we wanna achieve is always on the table and it's already kind of a common understanding that that's table stakes and that's really good to be at this position because we can't always say that about all the companies we partner with. So it's good to have this mutual understanding that that's where you start.
0:26:08.1 Josef Goldstein: But then later on when you understand that you have more needs coming in this realm, that it's a good idea to kind of know that there is this progress that's happening that you can tap into. Now sometimes we're front running it, as I mentioned before, like for example, when we talked about Schema Registries, but sometimes we do not. Right. So a good example for that, we leverage KSQL very liberally. And also once Flink became available we kind of jumped on that. I think we were kind of like one of the first design partners for running Flink in production, which was really was great, like actually working with the product managers responsible for that and I think most recently leveraging Freight Clusters, which I think is kind of a lot of the ecosystem of Kafka is going towards, especially given everything I mentioned before about lanes and everything. So not having to deal with innovating in a place where again, it's not our core business and just saying, okay, we can take this off the shelf and even if we need to be like the, some of the first people there in line to try it out and help shape it as a even better product, and that's always great. I think that's part of the partnership and that like that lockstep of okay, let's see for example first of all what's coming as part of this platform offering and see if the solutions to the problems that we have are there or in the roadmap. I think that's something that's very important, important for us. And we keep trying to be in close communication, understand where things are going.
0:27:28.1 Joseph Morais: Okay, so you know, Wix viewed Confluent as the house of Kafka. I love that. I wanna say that, I mean everything you said makes sense, right? You realize that this is your core competency. Your core competency is unlocking dreams on the Internet. Is there, is there a provider out there that we trust to help run this? Right? And if, you know, you identify that as the source of this original technology. And again, Confluent obviously doesn't own Kafka or the ecosystem, but fairly contributors to it, you make those decisions to say, okay, we're gonna go, we're gonna let them run that for us because we know there's other features that are gonna come along we can jump on. And I really appreciate your early adoption of some of our newer services. And let's talk about Freight a little bit. So for the uninitiated in Confluent Cloud, we have a number of different cluster types that help you unlock these different lanes. And one of our newest is called Freight and Freight is built exclusively on top of object storage. So the idea of Freight is you're trading a little bit of latency. Things are a little... The latency is a little higher for reduced costs at big scale.
0:28:33.4 Joseph Morais: And again, it perfectly kind of fits into the concept of these lanes. You know, there are certain data that's gonna go through Freight which is, it's fine for there's other data that's not gonna go through Freight Cluster because you need those lower, you need that even more real time, so you need these lower millisecond end to end times. Can you maybe tell us a little bit more what was specifically the use case that you have? Which of the lanes are you using with Freight?
0:28:56.8 Josef Goldstein: First of all, I'll just say that I think that the motion towards Freight and the whole concept of basically giving I would say flavors of Kafka is a very important concept. I think that historically Kafka was built by LinkedIn for running on bare metal. And it has a lot of values, but there's all kind of inherent complexities to the architecture that leads to, up until I would say about a year ago or a year and a half ago, there were very few levers to play on trade offs between cost and latency. Now, if we go back to our story, I talked about the lanes and this is exactly where it fits. We would take the pie of, like I said, around 40 billion events a day. They're not all the same. So like I said, we assume upstream that they're all the same, because we want to be flexible in the ability to say, okay, I want this data on a different lane. Yesterday I only needed that for offline analytics. But today I need this data point to make real time decision making. So we always wanna have this flexibility. But the reality is, if you look at that pie, I would say about only 5% of that pie is really truly used at any given point in time for that low latency real time analytics.
0:30:03.0 Josef Goldstein: Then another 10 or 15% for the more like middle lane type of things where you can go up to a minute and then the rest, it's just, it's analytical telemetry that's important. We need it very reliable. But nothing happens if it lands minutes, right in terms of latency even. So, when Freight became part of the offering for us, it was a great thing because it says, okay, if we are able to rewire our architecture to be able to leverage that, then we can not only have these lanes as something that happens downstream from Kafka, but we can actually integrate that same concept and architecture into the Kafka, basically take it upstream. In practicality, what it means, and that's kind of how we build that, there is a decision point that we're propagating. So basically, because we have all the governance and we kind of know what every event is being used for, right? We have the governance information, we have ownership, we have lineage, so we know what depends on what. So we kind of have the ability to have the insight of what pieces of the pie is needed for what. So what we end up doing was propagating that information as upstream as we could to say in real time, okay, this event needs to go to the classic Confluent Cluster and this one needs to go to the Freight Cluster.
0:31:23.7 Josef Goldstein: So that obviously helped us tremendously with cost optimization, which everyone is obviously thinking about all the time. It also helped us with another thing which is kind of interesting and that's there is an additional inherent value to the technology behind Freight Clusters besides playing on that cost to speed scale. And that has to do with how fast you can scale. So because of that architecture, the brokers, not to go too technical, the brokers are stateless and that or semi-stateless. And that means that what used to be a pain point in Kafka in general, because it's part of the architecture, you can't avoid it. If now it's Black Friday and I get a burst of traffic, in old times what I would need to do is, okay, I put a reminder in the Calendar, we're gonna scale up the Cluster. We're gonna wait for it to scale up because it takes a while. We're gonna get the traffic on Black Friday. Okay, let's also wait for Cyber Monday. So yeah, we're gonna be overscaled for that time during, but fine, because we don't wanna touch it in the meantime.
0:32:26.6 Josef Goldstein: And then at the end of Cyber Monday, let's do this manually. Obviously still things that we need to keep in our mind, but Freight Cluster are inherently more elastic in their ability to scale. So in many ways we kind of expect, like the system is stable. Right? But our ability to handle big spikes of traffic actually magically increased while we lowered our costs. So it's kind of like again for this specific use case where we did have these lanes, it's like a magical solution.
0:32:50.7 Joseph Morais: I like that you kind of bound the lanes ultimately to the real timeness, right, to the their latency requirements because that kind of allows you to simplify things. Now you don't have to really worry about tagging the use case in terms of what it does from a pragmatic standpoint, you're just worried is this a fast, medium or slow use case? And then you just route it to the appropriate set of clusters or cluster, that way that really simplifies things. I like that you've also been able to kind of match and it was always the design of Freight. Right? Because you mentioned that this data, it's critical, you don't wanna lose it, but it's okay in minutes. But it also happens to be your largest bucket of data. As you were going through that, it was like 85% if I did the math correctly, falls into this, into that category. So it's your biggest source of data. It's while still important, it is not as latency bound. So it's perfect for this "slower swim lane", which was exactly what Freight was built for. Honestly geeked out that you guys are using the systems in that way.
0:33:49.0 Joseph Morais: It makes me feel very happy. This next question is traditionally for people that have kind of been there before Confluent came into the fold. I know Confluent was already present when you joined at Wix, so maybe we'll take this a different way. You know, generally I'm asking like, you know, what were things like before Confluent was implemented? How are things going now? Obviously you weren't there to kind of see that. So I'm curious with as tight of a partnership as we had and as you mentioned, you know, you guys have been early access and helping influence the design of Confluent Cloud as long as I've been here, which is almost six years. Are there any other like more novel interactions you can think of between your teams working with, you know, Confluent, whether it's engineering or product? I know you mentioned you were working with PMs on Flink. Do you think there's anything, any other novel parts of the partnership that are worth talking about? Let the audience out there understand how strong of a partnership we have together?
0:34:40.9 Josef Goldstein: I think in terms of... So we obviously have a lot of technological partners. I kind of mentioned it before, which is, when you build a Big Data Platform, it's like a patchwork, blanket type of thing because it's a mix of stuff you buy, stuff you can't buy, stuff you don't wanna buy. So we do have our vendors that we are working with because again as part of philosophy at least what is not our expertise and we can buy, we much rather buy. I think we can comfortably say that Confluent is our most reliable partner out of that bunch. I won't name the names of the others, but I think it is both in terms of veterancy, like the time, but also how it looks in practice, I think there's this good understanding of what we're trying to achieve. We have this very close communication where to me at least I think it's always important to share where my mind is on where we need to be as a company, but in general as an industry, six to 12 months ahead, for example, conversations about this class Kafka and this new trend and everything like that's something that the team at Confluent working with us knew that we're interested in way before Freight became an official offering.
0:35:54.1 Josef Goldstein: So the day that it came out it was kind of like when can we start? Which is very powerful to us. I think we also have a lot of really good knowledge sharing. I think doing these type of things is also kind of a way to share each other's stories, which I think is very, very important because I think a lot of the innovation that's happening in this space is always a collaboration between different factors. I can tell the story about how we are kind of rearchitected and maybe slightly innovated in the way we architected into a multi-lane architecture, but I can't tell that story without mentioning the technology coming from Confluent. So we also try to give the stage to each other in that sense. So I think it's part of that. Cool guys. Cool guys to meet with eventually. We're all people, so haven't met a person that didn't wanna work with, which is always good.
0:36:40.3 Joseph Morais: Oh, that's fantastic. That's the best feedback I think anyone can hear. That's really wonderful. And I couldn't agree with you more. You know, our engineers, we... They build things, right? They build features and they ship the features. And of course they wanna hear customers like yourself are ultimately using those features to, you know, make their lives easier. You know, I mean, I'm an operations guy. I had a pager for more often... More years in my career that I didn't. And you know, just, your Black Friday. So I worked at a retail company for years and I remember getting into the Black Friday, getting in the huddle and looking at all the KPIs getting ready to fail over in case everything, you know, blew up. So I can appreciate that example of just not even having to worry about it. You know, just, you know, I'm gonna go to bed Friday night and I'm not even gonna think about this thing. And then Monday I wanna go look and see how much money we made. So that's awesome. This next question, it's pretty general. It's, what is the final... What was the final impact of adopting data streaming?
0:37:32.3 Joseph Morais: And I think we've kind of talked about a lot of that throughout today's episode. But I'm curious, you know, are there any particular things that really impress you yourself? Automatic remediation or the changing of the app or the service itself based on these signals? Are there any particular things that you're doing now in 2026 that you can't believe? Like if you went back seven years, you couldn't even believe that these things are even possible and they're built on top of data streaming? I'm curious if you have any examples of that.
0:38:01.3 Josef Goldstein: I think it always surprises people, especially people that haven't been around to what you can leverage analytical data tool for. Because a lot of people have this, their notion of analytical data is these like slow dashboards that update daily, et cetera. And they don't realize that we are at a point where you can create these analytical insights in seconds or even sub seconds and then use it to power a product. So it's always interesting for me to find a new feature that was built at Wix that hinges on that because obviously we're using Confluent as a platform, but we are also serving this as a platform internally as Wix. So I don't even know about all the things that are being built on top of what we do.
0:38:39.0 Joseph Morais: That's cool.
0:38:39.5 Josef Goldstein: I think now where we are at the biggest unlock around AI is the understanding of... So the buzz that now is very interesting is this context, right? Everyone talk about context, context, context, context to power agents and like how do you make them smart? And what slowly people are gradually understanding which is something that we internally at Wix kind of understood already a while ago, but it took people a while to get, calm down from, okay, it's magic, LLM is magic, it generates text. But how do I make it actually functional for something that I wanna do? How do I make it actually look smart not for Googling the data for me, but actually for like an enterprise type of solution. And then that's when you realize that the only way you do that is by serving the data that an agentic system needs. If you want your agent to feel smart and not like my information stops at 2024, whatever, then the data also needs to be fresh and that, we have a lot of focus on this right now and we're kind of seeing like a renown... A renewed renaissance around stream processing and understanding of, we need to find ways to serve these analytical insights to these agentic systems.
0:39:52.9 Josef Goldstein: And again goes back to the lanes. We need to build it so it will be able to serve the most recent up to date agents. So if I want an agent to be smart enough to react to an action that I have just done in the product because otherwise it's a lame product. You feel like your AI agent is not as smart as it should be. Behind the scenes what it means, you need to rewire the system to be stream processing. And we have a lot of focus on this right now, which is how do we take these curated streams that we already have and we work very hard to have we slap more stream processing on top of them or reuse the ones that we already have and then we take those insights and now we're able to serve it as a context layer to agentic systems. If you don't know about all this stuff happening behind the scenes, it just looks like magic. It looks like it's reading your mind. And I think that's the thing that's gonna surprise people the most in the next six to 12 months is the companies that are actually smart enough to build these things, they're gonna have product that just blow your mind by how reactive and smart they are.
0:40:49.4 Joseph Morais: I mean, I can't argue with that. AI is just another data problem. And... But it's one that's exceptionally visible, right? For a long time, you know, people were able to get away with batch or maybe incomplete data or you know, not having schemas. But now with agentic AI you just, you cannot get away with it because one, everyone is aware of AI. You know, you could be a CEO at a flower company and you're asking about, how do we get AI agents to make this work and do this better? And the first part... The first challenge you're gonna run into is data, your pipelines. And if you already, and you just said it, if you already kind of have it figured out, layering agentic AI is not gonna be painful. But if you haven't, you're gonna have to start with re-plumbing your data estate and then finally sometime in 2027, getting some value out of AI. Can you share some advice or lessons learned for leaders like yourself that are just starting to tackle data streaming?
0:41:47.3 Josef Goldstein: I gave a talk several years ago talking about the evolution of data architectures and I'm very happy because it's aging well. Every time that I put something in front of people, I ask myself, will someone look at YouTube a year from now and think I'm an idiot? And one of the things I stated there, which again goes back to exactly what we talked about right now, it becomes more real, is that one of the biggest jumps that usually happens when people are building towards data architectures is the jump between batch and streaming. And it's kind of a hard jump and whoever can afford this... And I think now things are becoming much more affordable, not necessarily in terms of money, also in terms of skill and capability, is to architect for stream first. And the reason for that is you can always go from the fast lane to the slow lane. Going from something that is only the slow lane, like if you only have a lane for Freight trains, it'll be very hard for you to go to the bullet train. Like you will need to build a new track. So I think that's a big one.
0:42:40.7 Josef Goldstein: The other one, and that also touches a little bit on what we talked in terms of governance and metadata, but it also kind of touches on the future which is, the cliché is data is the new gold and it's been true and it continues to be true. It's gonna be even more true now because as you said, AI is a data problem, essentially. Metadata is kind of, I think that's the next big unlock. And that has to do both with data that stream and data not at stream. And that's basically the data that we have about our data and our ability to curate it and our ability to make it clear. Up until two years ago, that was important so our people will be able to nicely collaborate and understand and not waste their time talking to each other, trying to figure out what is happening. I think now we're at a stage where you're kind of understanding that we're not engineering any more, or not only for our people to talk with each other, but to our people's agents as utopic or dystopic as it sounds, that kind of depends on your personal view.
0:43:28.0 Josef Goldstein: But if you think about it as an extreme case, if tomorrow you want to have an agent that can autonomously consume your organizational data, makes decisions and even build its own tooling around that, it needs to start with understanding what exists. It cannot have a conversation with someone, unlike the person sitting next to you in the room. That's the second thing, right, is metadata and governance first. I know it's not sexy. No one likes to deal with that. It's very gray. No one likes to have that in their title. But it's becoming increasingly more important.
0:43:56.5 Joseph Morais: I mean, you've definitely hinted on it through the agentic lens, but what do you think is the vision for data streaming at Wix two, three, five years from now?
0:44:05.2 Josef Goldstein: Hopefully we're all here and not all replaced by agents, and there's still problems to be solved, but I still think there will be problem to be solved. Besides the obvious, which is scale and heterogeneity of data, obviously this entire trend of feeding data into agentic systems and how to make this more and more sophisticated and more autonomous, I think it's a problem today, it's just gonna be more of that. One of the other problems that I think we're not dealing with enough and is gonna become a trend. I can't tell five years from now, but definitely in the next year or two, is the understanding that data has become gradually more unstructured. We're actually going full circle on this, which people used to throw trash into their data systems and then we figure out, okay, we need to kind of be schema full and we need to structure it and governance, et cetera. But now we're getting to the point that our systems are unstructured by nature. We used to track clicks and funnels and CDC et cetera. These are slowly turning into the new generation systems into just conversations. So leveraging what we so far were able to leverage using arithmetic computation on top of strings and numbers, et cetera and that's how we saw data, I think the world of using AI to process what's coming out of these agentic systems is gonna be a big trend. I know there's work on that. I know there's still a lot of unsolved problem in that world, but basically trying to compute the uncomputable, I think that's a big trend and we'll still be fighting and solving these problems at scale again in the next several years.
0:45:33.0 Joseph Morais: I like it. Compute the uncomputable. That is quite the vision.
[music]
0:45:44.3 Joseph Morais: Before we let you go, we're gonna do a lightning round Byte-sized questions, Byte-size answers that is B-Y-T-E like hot takes but schema backed and serialized. Are you ready?
0:45:53.8 Josef Goldstein: Yes.
0:45:54.6 Joseph Morais: What is something you hate about IT?
0:45:56.5 Josef Goldstein: The need for waiting for someone to do it.
0:46:00.4 Joseph Morais: I can relate. What is your hot take on the future of artificial intelligence?
0:46:04.6 Josef Goldstein: People in the loop, superpowers around them.
0:46:07.0 Joseph Morais: I like that. What's a non-tech activity or hobby that's impacted about how you think about data?
0:46:12.7 Josef Goldstein: Computer games.
0:46:14.0 Joseph Morais: Computer...
0:46:14.6 Josef Goldstein: Computer games.
0:46:15.3 Joseph Morais: Where are you getting outside inspiration from? Are there any books that you can recommend or perhaps a thought leader?
0:46:20.1 Josef Goldstein: It's a hard one. I would say, find some people that you chime with in Twitter, follow on them. It's a really good source. Books unfortunately are becoming more and more obsolete. Ask your favorite agent, he has information, or rather it has information.
0:46:34.7 Joseph Morais: Let the agent read the PDF, I did it. I'm already doing that. Any final thoughts or anything to plug?
0:46:40.0 Josef Goldstein: No. Thank you for having me. I'll just mention that hopefully if everything goes well and this airs before that then I'll probably be speaking in current this year in London. So if you're around come over. We will be talking about fast lane, slow lanes and Freight Clusters. So I will be there and I'll be happy to meet anyone that wants to hear more about our story at Wix.
0:47:00.4 Joseph Morais: Thank you so much for joining me today Josef. And for the audience, stick around because after this I'm giving you my top three takeaways in two minutes.
[music]
0:47:17.6 Joseph Morais: So my first takeaway is really Wix's mission statement of making dreams come true on the Internet. Right? Again, I've been around long enough, I remember them just as a classic website builder but they're obviously much more than that. And the numbers back at 40 billion interactions a day, right across running Wix, across the user experience and then across all of the data driven nature of being able to tune the service to the user. The next takeaway is, I love Yousef, he said that ultimately Wix came up with the slow realization that the data needs to be as fast as their fastest use case and that's when they realized that batch didn't fit. You further reinforce that with this idea is if you start fast, if your data's fast, you can put it in the slow lanes, right? Or fast lanes. But if your data is slow, you can't do that. So you have to start at least upstream with that fast, real time, high quality data and then you can send it to whatever lane you need, even the slow ones for analytics.
0:48:15.9 Joseph Morais: And then the last thing to kind of tie this all together, my last takeaway is the idea of having different lanes for different types of data. You have data that needs to be near real time, you have data that's okay with emanating or emitting within a few minutes, that's all okay. The nice thing is Confluent provides you these different cluster types like Freight that allow you to fully realize the auto scaling, the cost savings and allow you to make those trade offs. I'll tie that all together with something really fascinating that Josef said and that if you already have your data, your context, your schemas figured out, AI can be magic. It's gonna be almost like it's reading your mind. But if you don't have any of that, if you have nothing but slow data and you have no lanes, AI is gonna be a real challenge to actually make, you know, productive and beneficial to your enterprise.
0:49:07.0 Joseph Morais: That's it for this episode of Life Is But A Stream. Thanks again to Josef for joining us and thanks to you for tuning in. As always, we're brought to you by Confluent. The Confluent data streaming platform is the data advantage every organization needs to innovate today and win tomorrow. Your unified platform to stream, connect, process and govern your data starts at confluent.io. If you'd like to connect, find me on LinkedIn. Tell a friend or co-worker about us and subscribe to the show so you never miss an episode. We'll see you next time.
[music]