Jeffrey Johnathon Jennings (J3) of signalRoom shares how to make your data streaming initiatives a success by using impactful proofs of concept, establishing strong data governance, and fostering cross-team collaboration to drive business outcomes.
Want your real-time data streaming initiative to stick? Success hinges on more than pipelines—it’s about people, governance, and business impacts. Jeffrey Johnathon Jennings (J3), managing principal at signalRoom, shares how to bring it all together.
In this episode, J3 shares how he’s used impactful proofs of concepts to demonstrate value early, then scaled effectively through shift left with governance and stronger cross-team collaboration.
You’ll learn about:
If you’re building or scaling a data streaming practice, this episode goes beyond the technology, showing you how to drive real impact.
About the Guest:
Jeffrey Johnathan Jennings is the managing principal of signalRoom, a dedicated father, avid traveler, and EDM enthusiast whose creativity and energy shape both his personal and professional life. As a cloud-native data streaming expert, he specializes in integrating ML/AI technologies to drive transformative change and improve business outcomes. With a focus on innovation, he designs scalable data architectures that enable real-time insights and smarter decision-making. Committed to continuous learning, Jeffrey stays ahead of technological advancements to help businesses navigate the evolving digital landscape and achieve lasting growth.
Guest Highlight:
“We need to speak the same language. The only way to speak the same language is to have a Schema Registry. I don't think there's an option. You just have to do this. We share a common language and therefore we build common libraries.”
Episode Timestamps:
*(01:13) - J3’s Data Streaming Journey
*(07:07) - Data Streaming Goodness: Strategies to Demonstrate Value
*(26:38) - The Runbook: Data Streaming Center of Excellence
*(37:00) - Data Streaming Street Cred: Improve Data Streaming Adoption
*(42:35) - Quick Bytes
*(45:00) - Joseph’s Top 3 Takeaways
Dive Deeper into Data Streaming:
Links & Resources:
Our Sponsor:
Your data shouldn’t be a problem to manage. It should be your superpower. The Confluent data streaming platform transforms organizations with trustworthy, real-time data that seamlessly spans your entire environment and powers innovation across every use case. Create smarter, deploy faster, and maximize efficiency with a true data streaming platform from the pioneers in data streaming. Learn more at confluent.io.
0:00:06.4 Joseph: Welcome to Life Is But a Stream, the web show for tech leaders who need real time insights. I'm Joseph Morais, technical champion and data streaming evangelist here at Confluent. My goal, helping leaders like you harness data streaming to drive instant analytics, enhance customer experiences and lead innovation. Today I'm talking to Jeffrey Jonathan Jennings, founder of signalRoom. In this episode, we'll dive into J3's journey with data streaming. Everything from adoption to architecture and governance. We'll cover how to get teams on board, the keys to quality data, and what it takes to build a data streaming center of excellence. But first, a quick word from our sponsor.
0:00:43.4 Announcer: Your data shouldn't be a problem to manage. It should be your superpower. The Confluent data streaming platform transforms organizations with trust. Trustworthy real time data that seamlessly spans your entire environment and powers innovation across every use case. Create smarter, deploy faster and maximize efficiency with the true data streaming platform from the pioneers in data streaming.
0:01:15.4 Joseph: Welcome back. Joining me now is Jeffrey Jonathan Jennings, aka J3, managing principal signalRoom. How are you today, J3?
0:01:21.8 J3: Doing well. How about yourself Joseph?
0:01:24.3 Joseph: Been doing great. I'm trying to think, when did we first meet? Was it back at Reinvent 2021?
0:01:28.6 J3: Was it 2021 or 2022? Oh my God, I'm getting so old. I don't remember anyone.
0:01:33.4 Joseph: 2021 when I did the Edge. The sessions about the edge. 2022 was about... We did serverless, so I don't know if that jogs your memory?
0:01:42.2 J3: I think it was 2021. Yeah, it was 2021. Yeah.
0:01:48.8 Joseph: So going on four years now. That's crazy.
0:01:51.2 J3: Oh my God. I remember when I first met you and we were all in that conference room and then you got up and said, "Oh, let's sit back down."
0:02:02.0 Joseph: That is so funny. Well, let's jump right into it. What do you and your team do at signalRoom?
0:02:07.6 J3: We do two things. One, we're developing the AI data mesh service for business. I been building data warehouses, data platforms for a very long time. And I just realized that now with AI, why can't we just do it automatically? Just build it automatically and then give the power back to business. Business could actually say, "I want to do this." And essentially I actually curate... The AI system actually curates the data product for business. And actually that's what I've been working on. I'm not getting close to actually having a MVP that I can put out there and have someone try to break it or bend yet... Just try to say it's there. But if you could just do this. And that's what I'm actually doing.
0:03:06.6 Joseph: Just to summarize, and we'll talk about this a bit more later, what you're building ultimately will use generative AI to build data products.
0:03:14.5 J3: That's correct.
0:03:15.5 Joseph: So if I have a stream of data and I have three streams of data, I have some common field, I can ask your system to then join those and then that merge data might be like a Customer360, for example.
0:03:29.6 J3: Absolutely. And in addition, because the second part is actually a consulting practice to actually help organizations that I know who want to take advantage of some of the core technologies that we're using, namely Confluent, Flink and where I have to say, I've become very good at Apache Flink and really taking advantage of that and then showing other organizations how they too can take advantage of it.
0:04:00.3 Joseph: Well, this is a great transition for our audience. I'm here with J3 today, but if you haven't watched and you're curious, what is Flink about? Watch our first three episodes. Episodes one, two and three. They are our foundational episodes that talk about what is data streaming, stream processing, governance and integration, which we'll be going into. I think this is a great segue into who is signalRoom's customers and who aren't your customers.
0:04:22.0 J3: This is a great time to be alive right now in this data streaming world because we now have the means to actually live up to the promise of providing fast data to not only our analytics, but our operational system. Where we didn't have that before and we didn't... I would actually dare say it wasn't easy, where we could just look in our tool shed, pull out a tool and put it in action. We can actually do that now.
0:04:56.7 Joseph: I like that. I know you've worked at multiple organizations or with multiple organizations and for multiple organizations that have implemented data streaming. Curious at a very high level, quick, like you're going to go into too much depth. But what is your data streaming strategy?
0:05:13.3 J3: Let's talk about the business outcomes, let's not talk about the technology because I speak to people and sometimes they go right to the solution immediately. And so no, let's speak about the problem. And then what I find is typically they want to have a conversation, their systems want to have a conversation with one another. What I found with one customer for instance, is that we need to usage information, let's say. You're utilizing the services and so when usage server is about to max out, we then need to update our payable system. But before we update our payable system, we want to start sending updates notifications to our customers, say, "Hey, you're about to top off, you're about to get up. Would you like to pay a little bit more to go to the next plan?"
0:06:08.6 J3: Giving them that opportunity. And then if they say, "yes." going through their portal or their mobile device, you want to then take that information and then upgrade their system automatically. So having a conversation... And then update our accounting package to of course update their billing status, you want to have that conversation. So it's having that conversation is what we want to discuss because now like I said, great time to live, so many possibilities to solving their solutions.
0:06:42.6 Joseph: So if I heard that right, it's about one, identifying the pain and talking about it. But it's also, and I've heard that before on the show with other guests. But one thing that you honed in on, I think it's interesting is discussing the stream of data end to end and understanding that journey. Not just looking at the data streaming itself, but where's the source of the data, where do you ultimately want that data to go and then treating that strategy with that end to end view in mind. I like that. It's a really strong one.
0:07:14.8 Joseph: We set the stage. Let's dig deeper into the heart of your data streaming journey in our first segment. Data streaming goodness. Thinking about your past and current experience J3, what was it like before implementing data streaming? Let's talk about the logistics tech company you worked at. What were things like before you guys implemented data streaming?
0:07:36.6 J3: Things were waiting to happen and so we had to pull to find out, are you ready? When we did find out you were ready, we would just have a solo conversation with that one system. But it turns out other systems needed to know I'm ready and the data is ready to use. You had to, as a developer had to think about all those other systems out there that needed to know about what just happened. Not just you let those other systems know this is a human thing. So you needed to talk with those other teams to know what they needed to know. Okay. It's really about telling other people. The problem was one, it was more of a, I would say things were going in slow mo and just waiting to happen.
0:08:36.5 Joseph: So everything was slow. You finally convinced or worked with the team to convince this logistics firm, "Let's do this." What were your biggest challenges?
0:08:45.4 J3: It was a challenge and it's always a challenge with people and getting people to appreciate what's happening and then in many respects to prove things out. Even if you did a small POC, you always roll up into another issue and then therefore you needed to resolve. In the beginning it was just me, so I was championing the solution and then I was building it out. Your support team was my team in waiting at all times. And so whenever I have this issue, I can always give you guys a call and I know no one is listening, but I can actually send them some of my code and they would actually tell me, "Jeffrey, correct that line." and you'll be good.
0:09:34.1 Joseph: As these different organizations you've worked with kind of the maturity progressed. Who in the org is handling data streaming and stream processing? Is it ops people? Is it data scientist? Or is it somebody else completely? Or does it really vary from organization organization?
0:09:53.4 J3: Well, I see it's very small organization to organization. However, I am finding a common thread and that's typically in the integration group. That's actually, because the thing... [0:10:05.4] ____ Kafka go, okay so I'm going to be able to ingest and then I'm going to be able to sync out my data to those other systems. That's what people automatically think and it's not a problem. But now what they have to now consider is now you have this streaming processing processor that you have to deal with and you can look at it, what you have SQL? Isn't that what the data team does? Who should be doing this work? And then, so what I'm finding is that we're doing more collaborative. Like in the other company that I worked at, I end up becoming both the data guide and the integration guide. And they put it in both. Now I have to say that in a lot of large organizations, most organizations when they establish their departments, they do it by technology or they do it by process. That won't be a option for many organizations. Okay, yeah, you handle them both. You handle both the integration and data team. That's where I see what has to start developing some sort of data stream center of excellence has to be developed.
0:11:15.1 Joseph: All right, I do have a question about that later. Let's [0:11:18.7] ____ some of those details. I don't want to give it all away yet, but it's interesting that you mentioned it's the integration space and I've noticed that a lot myself. But many people, when they start in their data streaming journey, they want to build a pipeline between one thing and another thing. And while that is valuable, that's not really the core function of Kafka or confluent. It's all about that's one piece of it. But it's, hey, let's use that data from some database and make it part of our operational state. We don't just want to transit the system. There's other things that should be interacting with those events, whether it's a stream process or consumer producer model. Once you've helped these organizations adopt data streaming and you've done this multiple times now, tell us some outcomes you've seen or were aiming for with leveraging stream processing and integration.
0:12:02.9 J3: What actually excites me because I had the opportunity for my team to grow and where it was just not me. And then my teammates actually came in and said, "This is how we should do it. We should do it this way." where hey wait, I'm bringing in that data stream and then you're bringing in that data stream. I can combine those two data streams and make a brand new baby and actually be able to do. And they actually came up with it themselves. And then even my upper management started to see the value. Actually saw the value. And wait a minute, you can do all of that in one place? Because if you think about it, a topic is a table. So why can't you just join the multiple topics together and you come up with a new transformation and then another system can use that data. People start to see it, but I hate to say you build it and they will come, but is in the sense... In many respects you have to build it and show them away.
0:13:06.4 Joseph: That must have been such a big moment for you. I'm thinking of, there was this movie, I don't know if you ever saw it. It had Rowdy Roddy Piper, it was called 'They Live' in the '80s. And the idea was there was these aliens. And they had infiltrated the earth and there was a bunch of things that they had hidden in sight. But you could only see the reality with the glasses. It reminds me of like that moment for you. There was this one famous fight scene where the two guys are fighting and finally Rowdy Roddy Piper puts the glasses on the other guy and he can now see for the first time. And I imagine it was like a similar feeling, like, "Oh my God, these people get it now."
0:13:40.8 J3: Oh my God, they actually see it and it becomes... It's actually a, aha moment and everyone has to come at it at their own pace. What I realized thinking of that is that I can see it and I struggle to like get other people to see it. And then I realize Jeffrey, you just need to relax. They will come when you start adding things and they have to actually start seeing it. Sometimes we have to just go through the journey ourselves.
0:14:16.0 Joseph: I'm curious as this maturity progressed, have you seen any of these organizations take data products that may be traditionally found in an analytical data state and actually move that into their operational data state using stream processing and then take advantage of that quality data?
0:14:35.2 J3: Well, I end up doing that. Hopefully, we're not going to mention the date of when this took place, but I was on a call with my customer and we were talking about using a third party product and getting other data and they were just saying, "Jeffrey, we just have to just wait until we get that other data flowing through Kafka before we can do that and then we could serve those other systems." But he soared. He actually understood, yeah, we need a shift left, we need to get... And that's like, "When are we going to get that data?" I'm like a little kid that's like, "When are we going to do this? When are we going to get there."
0:15:16.8 Joseph: Let's not revert DTL it. That just feels gross.
0:15:21.5 J3: Let's just do this. And he got it. That's what I appreciate. He got it.
0:15:26.3 Joseph: I love that. What is the data streaming architecture? Considering all the history that you have with data streaming, what is the architecture you're most proud of building, or maybe are building currently?
0:15:37.7 J3: The one that I'm most proud of is actually building a streaming API, meaning we had a customer who was using Costco already and we needed to get orders from them sent to us and then we needed to send them back updates. Now we were working on a process, a webhook strategy to send them updates, but we weren't there yet. I was thinking that's producers and consumers, would you mind producing your order to us and then you could consume your orders from us. And we actually shared a public cluster together. But even more, I think more powerful is we had the Schema Registry. We actually collaborated on a data contract between one another, which we're doing everyday business and then we actually figured out, okay, what your system needs to supply to us in terms of an order, and then what we need to provide you in terms of an update with the Schema Registry. And then if we ever needed to make any changes, we can make that through evolving schema, which made it actually faster.
0:16:58.7 Joseph: Was this at the logistics company?
0:17:00.7 J3: Yes, it was.
0:17:01.6 Joseph: Okay, got it. So this API, this was customers that own fleets of vehicles, and they were looking to you as a technology provider for orders, for logistics tracking, things like that. I just want to make sure the audience understands exactly what it is.
0:17:13.5 J3: Right. They were looking for essentially for delivery... Us bringing cars to their customers. And so this is how they would go back and forth. And I found that so powerful because, one, we didn't have to do restful APIs any longer, and we didn't actually have to do webhooks. And it was just cleaner because it just you produce to us and you consume from us. It's just the language in the conversation just flowed easier. And then, of course, because you do want to do some synchronous operations, well, then you could easily do blocking in your choice of language. That's actually not a difficult thing to do. I just found that to be so powerful and so clean.
0:18:00.4 Joseph: I love that. Tell me, what was the outcome? Were the customers absolutely thrilled? Was it hard to onboard them? Did it all work flawlessly?
0:18:06.3 J3: Oh, no. The customer was actually extremely happy about it. One, they got updates in real time when they needed to make changes. Because it turns out most organizations, you have your order ID, so this is your order ID, but they say, "You know what, Jeffrey, we need to talk to multiple systems internally. We're going to send you four extra fields that will identify that order, and I want you to send it back to us when we do an update. Then our various systems internally can be able to figure that out, can be able to associate." Well, since we're sharing the Schema Registry, we could collaborate, we can update the Schema Registry. Matter of fact, they updated the Schema Registry themselves. I just consumed it.
0:18:54.0 J3: I actually updated my code and it was a nice easy flow and upgrading it was a very collaborative approach. They appreciated that. And just one quick story. During our initial integration, we had promise over the weekend we won't process any physical orders, but we can still take in orders into our system. We want to process any orders over the weekend. Well, I'm thinking I can update the system.
0:19:25.8 J3: I can make some updates. I broke it over the weekend. I broke the system over the weekend, but I was able to fix it. But then, of course, orders were coming in. But since we were sharing Kafka, and remember I said, Kafka is four things. It's a message queue, pub subsystem, commit log database, data streaming. It's the third thing, commit log database. I can go back in, pluck out that record, send that order through the system, and everything worked perfectly. The customer never knew.
0:19:57.3 Joseph: [0:19:58.0] ____ goes back to that offset and you're set.
0:20:00.3 J3: Boom. And that's powerful. Think about that. If you normally have a RESTful API system, you would actually have to build in that system to actually store that data. Because we also had another issue. First of all, that was good. One Kafka saved my weekend and to save the extra stick of deodorant. So that was very important. And then also, if we just think about that, power is sometimes people say, "Hey, I sent something to you via the API. Didn't you get it?" And because we had that situation back and forth, I said, "Well, you know what, let's look in Kafka. Let's see if you did." And then, "Oh, I didn't get that update. Let's see if I sent it in Kafka." Kafka it automatically stores it. It's a byproduct of the solution.
0:20:48.5 Joseph: Having that substrate is very powerful for compliance reasons, for replay reasons. There's a whole bunch of them. You already touched on this. You talked about Schema Registry. You also talked about how you can track events as they flow through systems in Confluent. We basically roll that under this package of data governance. And it's something that a lot of people skip. They decide to go into data streaming without Schemas, they're just, "I will do it all inside the app." which breaks when you evolve things and add fields and all that. How do you approach data governance? How do you ensure that the organizations you're working with value and go through the efforts of getting it right the first time?
0:21:29.4 J3: We need to speak the same language. The only way to speak the same language is we have to have a Schema Registry. When I approach this, I don't actually think there's an option. You just have to do this. Let's build the Schema Registry. But then the second thing, it's really, to really broaden it out where other teams could come in and actually utilize the topics that you created is to bring in something, cloud event specification is something that I discovered. Cloud events is a specification for standardizing how events should look and how they should be described. That makes it easy for other teams to come in because they know how to use it. We share a common language and therefore we could build common libraries, which I did. Because one of the things that we want to do is lower the cognitive load on other teams coming in and actually using Kafka. They don't have to understand all the nitty gritty, if in many respects it's all really the same process. Why don't we just build libraries? And I even built the libraries for myself. So when I had to build another microservice that needed to do consuming or producing, I just used the same standard code.
0:22:47.3 Joseph: I love it. This is a follow up directly to data governance. But how important is tracking and enforcing the flow of quality data as it enters your data streaming system?
0:22:58.1 J3: Well, it comes down to all about trust. People will not trust the system if people don't know that you're not doing certain verification and certain checking of the data. That's actually where Flink comes into play. Because then you can use Flink to actually do set up validation rules and actually note so if all of a sudden, oh, this record doesn't contain at least an email address or a phone number when it should. And this is before of course we came out with data contract within Confluent.
0:23:32.5 Joseph: Right. Before you could do it at the governance level [0:23:33.2] ____.
0:23:34.7 J3: At the governance level.
0:23:36.2 Joseph: And probably you would put that event in some type of topic for remediation.
0:23:40.1 J3: Exactly. I would put it in the deadline queue to get processed. And I have another process that will read in now with the advent of data contracting which I say everybody should utilize because you could put data roles to make sure, "Wait, this doesn't have an email or a phone number. No, we're not exact."
0:23:58.0 Joseph: And then they can't even, submit it in that case.
0:24:01.5 J3: Can't even submit it. And that's actually absolutely important. Or the format of the phone number should be a certain way. Does it have this? Or the email address should be a certain way, doesn't have this? Or the length. Did you know sometimes we only accept 100 characters at the end? Well, let's do that check before we even allow it to go through.
0:24:23.1 Joseph: The example I like is a Social Security number because in the US it's always nine digits. It's never longer, it's never shorter. It's never alphanumeric, it's just numeric. And that's a good one to be like, "Okay, that's 10. That's wrong. Get that out of there."
0:24:35.8 J3: Right. Or even the case of our logistics, it's the VIN, the vehicle identification number, the same thing, depending...
0:24:43.2 Joseph: That's a good point. That's very consistent too. Yes.
0:24:44.8 J3: Absolutely.
0:24:46.3 Joseph: Just double clicking into signalRoom. We know that the initial idea is to work with data products. To generate data products through generative AI. What's the future? What's the next feature or two that you're looking to launch with?
0:25:01.4 J3: I would say a data product linking. Because you're going to have all these multiple data products out there. How do you know to link the data products to one another? How are you going to match, how are you going to do that discovery of those data products? So you need that. That's going to be something that we're going to offer because I believe for customers that don't say, "Okay, we don't want you to build our data products. But we already have data products." and they need to talk to one another. How are people going to discover those data products?
0:25:32.5 Joseph: It's almost like a relationship discovery, something like that.
0:25:35.8 J3: Absolutely. Yeah. That's going to be something... This is a fascinating world, especially with AI, with general intelligence coming out and how we could take advantage of that. Because right now for the most part we utilize a lot of shallow learning, basic statistical. But it's really that looking at a Schema and sometimes organizations don't use foreign keys, they don't use primary keys the way... So how do you make sense of that? And so that's really some of the challenges that we're trying to figure out now.
0:26:12.3 Joseph: I really like that because you're basically saying it does not just limit it to your use case, but as people start to build some of these gen AI systems and they feed them data, they can get into this pattern of, "Tell me what I don't know." And that's really what you're saying. It's like, "Hey, I have these data products, I don't know how and if at all they relate to each other. But if they do identify that for me and now I it unlocks these new opportunities for innovation." Our next segment is the runbook where we break down strategies to overcome common challenges and set your data in motion. J3, today let's specifically talk about the organizational challenges of building a data streaming center of excellence. My first question to you is, were there any roadblocks you faced when setting up a data streaming center of excellence outside of the tech itself? And I know you've done this at different organizations.
0:27:06.5 J3: You have to prove the worthiness of this technology in many respects. Because, database technologies, client server technology has been around for a very long time. So people say, "Well, why do we need this? I already got it working." Matter of fact, that's probably one of the common refrains that I get most of the time.
0:27:29.6 Joseph: Why data streaming?
0:27:30.6 J3: Yeah, we already got this. It's not the greatest, but it works okay. I love the fact that I didn't have to commit hundreds of thousands of dollars to Confluent. I could literally put a lot of my stuff on a credit card initially before I can build it up to show business value to the organization. That's something that's very powerful, is you have to go bit by bit by showing them examples of how, "Hey, you know what? I'm ingesting that data, do a little manipulation that data, and then I'm syncing it to either Snowflake or syncing it to another operational system." showing that example and doing that very quickly. And then it's, okay, great. Can we bring in some more data? Sure, we can bring in some more data. And then once we bring in one stream of data, two streams of data, three streams of data. And what's key is one should be thinking how these three strings need to connect to one another.
0:28:34.1 J3: You need to be strategic about the streams that you're picking in. Because the second thing you want to show off is how, wait a minute, you realize I could combine those streams and then push it into the system where before we were doing that work in the analytics system, and it turns out we were doing a little of that work in the operational system as well. Hey, we're now showing how we can shift left where we do it once and we do it right here upstream, and then we push the results down.
0:29:08.5 Joseph: Because then you have it operationally and you have it analytically, and you've probably done it in the most efficient way. Instead of letting that data pass through one and then reverse back. If you can handle it there, do it there.
0:29:19.4 J3: Absolutely. One other point that I just want to mention is that I had a customer come to me, said, "Wait our data science people, they want to see all the data. They want to have access to everything. So they don't want any manipulation." Okay, that's fine. It's all in Kafka. All the original data is there.
0:29:37.3 Joseph: You can go back to the raw data.
0:29:39.5 J3: The raw data is there, untouched. And as a matter of fact, I can't wait till TableFlow comes out because then I can actually just sync it to a topic, dump it into an iceberg table, link it up as an external table of snowflake. Be happy with, "Hey data science team, here's all the raw data you want." And eventually those data science people say, "Wow, you know I'm doing all this stuff." because as we all know, 70% to 80% of all their work, it's cleansing and getting the data just right. You already did that. Now that I see it and I see what you did over there and what I've been doing, I'm going to use your stuff. Then you can use my new topic.
0:30:19.9 Joseph: Great. You have like a pretty common theme there when it comes to establishing a data streaming center of excellence, one you have to convince people why you want to change, because change is hard. And I get that. And you convince them by showing them the possibilities. You take that work on, you add a little piece, you add a little piece, you add a little piece and suddenly you got buy in. And then the next thing you know, you realize there's some redundancies between your analytical and operational state. And now you covering data holistically and suddenly you now have the formings of your center of excellence.
0:30:51.3 J3: Absolutely. And another thing that I think is extremely important is always keep in mind what is the business outcome.
0:31:01.4 Joseph: True.
0:31:02.1 J3: Ultimately, what is business really looking for and actually seeking that out. When I'm talking about being strategic about the data streams, it's really the combining of that data, not only did you just make it more efficient, but you actually created something to alert them. Because that's why you want to bring it together, because you're alerting what they otherwise probably wouldn't know until too late. That's why I'm actually not a big believer in dashboards and reports. The dashboards and reports say it already happened. It's too late, the damage has been done.
0:31:39.8 Joseph: Tell me what's going to happen.
0:31:41.8 J3: Tell me what's going to happen. So if I can actually fix that further upstream and then alert went, "Oh, you realize we're getting bad emails coming in all the time." Notified me so I could talk to the source and say, "Hey, you're not entering the email addresses correctly." That's one silly example. But it's nevertheless, a very speedy and fast example.
0:32:09.7 Joseph: That makes sense. A lot of teams struggle with the human side of setting up a data mesh. What actually needs to happen to make it work?
0:32:15.8 J3: We need to stop blocking ourselves. We meaning technical folks. This is really for the business and really thinking about the business. So not getting them to understand how we need to clean up data and make it right. Because the reason why we're doing data warehouses, because we want to build these fact and dimension tables to clean it up and to build this really pretty generic pipeline. No.
0:32:41.1 Joseph: Does that really help the business? Or is that just a cool thing to do?
0:32:45.0 J3: That is our work, that's what we want to do. So actually give them a clean interface that they can use to talk to the data product that talks their business and provides that information to them. So it's understanding their domain. That's what's very important. We need to make sure as partners we understand their domain.
0:33:07.7 Joseph: That's a really good answer. Because the business requirements what fuel the entire organization. I mean sometimes the technology is the business, so that's okay. More oftentimes than not, it isn't. It's a function, a tool set for the business. Let's make sure we keep the business in mind. That's a really excellent answer. If a company wants to set up a data streaming center of excellence, what's the most important thing they need to get right when it comes to the organizational side of things?
0:33:37.2 J3: You need to have someone who... If you're product oriented organization is getting one of those people involved in the team because they understand how to communicate to the rest of the business. That's extremely important. Then getting internally tech people who actually understand how to manipulate data. Understanding it's like for me, how am I building my signalRoom consulting practice? What kind of people do I need? Well, I need people who know SQL. Okay, so Python and Java, one of those two, if you know those... If you know essentially the language and SQL, so you know the declarative and imperative language. I can actually teach you Flink. I can do that for you, I can build that team. But I think that's important that you want. And then I think getting someone who knows how to visualize and present information is key as well.
0:34:40.6 Joseph: Okay, so really it's about these particular skill sets and strengths. Do I have someone that understands perhaps operations, someone that understands data stream, someone that understands at least the underpinnings of stream processing, something like Java or Python or SQL. And you really can kind of pick your flavor of those. Just having all the right pieces together to get that holistic view of data streaming and stream processing. I really like that.
0:35:04.2 J3: And I would also add, and this is one I got from my friends who were working at AWS, so thank you AWS. Is being document centric. Documentation is our friend. We need to work backwards, which is something famous that AWS does all the time.
0:35:26.6 Joseph: I used to work for them. I used to say working backwards all the time. I still do.
0:35:29.9 J3: It's so important. And put it down on paper. So I do a ADR architecture design record. I do that. Where I actually explain what's the context, what's the problem and then the why statement. What's the why? It's to frame it up that way and then have the positive and negative. What we do is all trade offs. Everything we do is trade off. So some are not as good or okay and some are bad. We just need to know those. And then from there, then we start the code. But it's important that we do that because we need to understand and other people who come onto the team, they need to understand why we chose this, because we write all these design documents that talked about the how but never talk about the why.
0:36:22.4 Joseph: Oh, I talk about this all the time. Why are we doing this? Why do we want a data stream? You got to start with that before the, "Oh well the tech works like this." You have to start with the pain. Start with the pain.
0:36:35.1 J3: I can't remember his name, but he was a co-founder of Wave Google bought them. He has a book called 'Fall in Love with the Problem' You need to love the problem, understand the problem. Because after all, that's the reason why we're here. We're here because of problems. I love problems. Absolutely.
0:37:03.6 Joseph: We talked about the tech tools and tactics, but none of that moves the needle without the right people behind it. Let's dive into how you get your organization fully committed to data streaming. Again, considering in your huge bevy of experiences and data streaming, are there any tactics that helped improve the adoption of data streaming across the organization?
0:37:24.0 J3: What helped me in proving it out. I actually ended up working with the person who was in the other vehicle company, who was a product person, essentially the strategy person, really understanding what are you trying to solve and just talk to them and understand what you're looking to do. And then I think it's very important that we need to communicate with others. We need to reach out to them. No one is going to come to us. That's very rare and we should be okay with. They never tell us what they're doing. They never going to tell you what to do. Has anyone ever been married before? So the feel is, is that you need to go out and you need to find out what they need. And then just say, "Okay, let me come back to you." and then actually show them. Don't make it perfect. Never let the perfect become the enemy of the good. Just come up with something that actually does the trick and then show them to know that you're on the right track. I think it's very important.
0:38:34.6 Joseph: Getting closer to the business, I think is an excellent tactic. It's right [0:38:37.4] ____ true objectives and working as a partner to the business. I think that really goes a long way in saying, okay, this person, in this case you really cares about our outcomes and really wants to build something and they think that this is the right technology. So let's take them seriously. Let's give it a shot.
0:38:57.3 J3: One of the things that I did is they were... Initially, when we were working with a customer at this other company, they were trying to figure out, okay, how best to charge a certain price for vehicle being used storage, for instance. How to do that. What I did is I actually created the Excel model. I think I know how to do it and I did it in the Excel model. And it's okay, you enter your number here, that will change. Is this what you're looking at? And I use that to actually demonstrate if that's what they want, then I can actually, then go... I can go back into the system and actually make it real. But showing that to them, which she was so happy because she was like, "Okay, yeah, this is what I'm talking about, this is what I want to see." And then she could actually use it with her other counterpart to make sure that it actually did what it's supposed to do. Something that they could work with.
0:39:57.0 Joseph: Yeah, to simplify that consult and show. Really simple, I love it.
0:40:00.4 J3: Consult and show. I love that.
0:40:04.2 Joseph: This is also related and ties together the question of the organizational challenges. In your past experiences, how did you get buy in from your CTO or CIO and/or leadership to push for the implementation of data streaming? You kind of got everyone kind of next to you and maybe reporting to you bought in. But how do you get up to that upper level that's still maybe not convinced?
0:40:26.5 J3: If a customer says this needs to be done, it will get done. If you say it needs to be done, maybe. But having the customer...
0:40:33.1 Joseph: By leveraging the relationship.
0:40:39.0 J3: Relationship with the customer, because I would actually have the customer said, "You know what? Can you just send an email." I'm gonna, and just reply and actually say, "This is what you need." And that actually made things happen. And I had the customer to actually be the cheerleader of the solution because it was for them, after all. And then even doing that internally, making them be the cheerleader, making them, being the advocate is actually very important. And I think that... So you have to align with the folks that need it and that fixes their part. Because at the end of the day, if they don't have to deal with the problem, they don't want to deal with it. So figured it out. And then from there then the leadership actually became easy because then the leadership, like that makes sense. And then when they had to do a big commit, it was elementary. It's like of course.
0:41:35.6 Joseph: Right. Because they already get it, they understand what you're doing and...
0:41:37.8 J3: Already get it.
0:41:39.1 Joseph: I really like the idea of leveraging external experts. Because it could be Confluent, but it could also be someone else who's just big in the industry that showed this pattern of something to get done. So just by leveraging those experts outside of the organization can get that leadership buy in. I think that's a really good strategy.
0:41:56.3 J3: We just need to let go of our ego. It's not all about us. Just actually bring somebody else in and let them validate. Because you know what, if you're wrong, somebody show you that you're wrong and you can fix it. Because who wants to be wrong? The goal is I don't want to be right, I want to do right. That should always be the goal.
0:42:20.2 Joseph: I don't want to be right, I want to do right. I think those are great words to live by just in general. We cracked open your runbook, we unpacked the tools and tactics and we explored some real world wins with data streaming. Now let's shift gears and dive into real hard hitting content J3, the data streaming meme of the week.
0:42:43.0 Speaker 4: And what subject do mathematicians specialize in?
0:42:51.6 Speaker 5: I don't really know.
0:42:53.7 Joseph: So what are your thoughts on that J3?
0:42:55.0 J3: First of all, I love his expression.
0:43:00.5 Joseph: I know. The expression is the best part. I know.
0:43:03.8 J3: [0:43:04.6] ____ It's our job to come in and just explain and be okay with just letting them know what it is. Maybe they don't know what it is. Who cares? Who cares if they don't understand it? I think it's not important that people come to our way of thinking. We need to come to their way of thinking. We need to help them along and if they need to use other language because typically people always say in other words, they need to figure out other analogies that make sense to them and be good with it.
0:43:44.9 Joseph: J3, before we let you go, we're going to do a lightning round. We call this quick bites. Bite size questions at byte bite size answers like hot takes but Schema backed and serialized. Are you ready?
0:43:57.4 J3: Yeah, absolutely.
0:43:57.9 Joseph: All right. So just whatever's top of mind. Quick answers. What's a hobby you enjoy that helps you think differently about working with data across a large enterprise?
0:44:06.2 J3: EDM. I love dancing.
0:44:09.0 Joseph: I knew that about you. Okay. I like that. Okay. Keeping those feet in motions, like keeping that data in motion. I love it. Can you name a book or resource and you did already touch on this so you can reuse it if you want. Can you name a book or resource has influenced your approach to building event driven architecture or implementing data streaming.
0:44:27.1 J3: I'm gonna have to send you the name of the person but it's Fall in love with the Problem by the Co-founder of Waves.
0:44:36.2 Joseph: Excellent. Listen, this audience, they all know how to Google. They'll find it. That's a great... Just the very concept of that book is great.
0:44:42.2 J3: Oh absolutely.
0:44:44.2 Joseph: What is your...
0:44:45.2 J3: One more.
0:44:47.5 Joseph: Go ahead. Yeah, please.
0:44:50.0 J3: 'Ask a Developer' by the founder of Twitter. That's actually another great book.
0:44:54.3 Joseph: Ask a Developer and Love the Problem.
0:44:58.0 J3: Oh yeah, absolutely.
0:45:00.6 Joseph: Those are just good pieces of advice anyway. What is your advice for a first time Chief Data Officer or someone else with an equivalent impressive title.
0:45:09.9 J3: Super open mind. You'd be surprised where your solutions will come from.
0:45:15.7 Joseph: I like that. Any final thoughts or anything to plug J3.
0:45:19.4 J3: Let's just say embrace this world. We are just so lucky. Of all the different avenues and different pieces, look at the partnership that's been developed between Confluent and Databricks for example. You would have never thought that would happen. And it happened. And just understand how you can put these different pieces to build a solution that you never thought was available to you. Embrace the new world and just enjoy.
0:45:54.5 Joseph: I love that. That's fantastic. That's a perfect way to end this conversation. So thank you so much for joining me today and for the audience, stick around because after this I'm giving you my top three takeaways in two minutes. That was a great conversation with J3 and here are my ironically top three takeaways. So look, for that 'They Live' moment, I highly encourage you to check out at least a summary of that movie or a couple clips from it.
0:46:27.1 Joseph: The fight scene in the alleyway is just one of the greatest moments in film history. But this concept that other people can finally see what you've seen. So if you're that initial pilot light at a company that's pushing for data streaming and stream processing, keep building things, build things that relate to this group or that group or to this individual or that individual to build that cohort of others so they can finally see the value in stream processing and data streaming like you already do. There was something that J3 said when we were talking about governance that I really liked. He said we need to speak the same language, which is a pretty incredible way of talking about Schema Registry and data contracts, which are essentially a contract between someone that produces an event and someone that consumes an event and make sure that we expect all the data fields and everything to be consistent from the producer to the consumer.
0:47:16.5 Joseph: But this idea of treating it like a language is really fascinating to me. If you think of adding a new dialect and suddenly I'm speaking Brazilian Portuguese versus a traditional Portuguese, we gotta make sure that we pre specify that early on so that there's no errors in communication downstream. So speaking the same language, a fantastic way to think of governance. And the last One is when J3 was talking about the future of his product at signalRoom. Tell me what I do not know. He's building this product to use generative AI to make it more simple to build data contracts, which we just talked about. But the idea that I can ask that same system to look at all of my data contracts and look for relationships that I haven't identified myself is really just kind of mind blowing about what and how that that could spark innovation in any given company.
0:48:08.2 Joseph: Just a fantastic conversation with J3. I'm so glad he was able to join us. That's it for this episode of Life Is But a Stream. Thanks again to J3 for joining us and thank you for tuning in as always we're brought to you by Confluent. The Confluent Data Streaming platform is the data advantage every organization needs to innovate today and win tomorrow. Your unified solution to stream, connect, process and govern your data starts at confluent.io. If you'd like to connect, find me on LinkedIn. Tell your friend or coworker about us and subscribe to the show so you never miss an episode. We'll see you next time.