Kafka for Administrator
language: EN
WEBVTT
Everybody have completed?
Any questions on there?
Are any questions that you want to discuss?
Then I have fixed the cluster on the topic yesterday like there was an issue with the
AWS where it is not able to accept the request.
Now we give you the link where you can try out the game.
Once you try it out, we'll check the monitoring and cloud and see how the topics are getting
the messages.
on 2025-01-06
language: EN
WEBVTT
We will check the monitoring and see how the topics are getting the messages.
And we will go ahead and install the Grafana and Prometheus to see in our own system.
If you are able to open the one which I have picked, you should be able to see dashboard like this.
When we go through, I will check the topics and see.
First I will go and check the user name.
See the messages from there, from email.
You see for every 20 seconds we get a user with the score in the lives and the weight the level is.
This is one of the topics which listens to the messages coming in from specific users.
So let's see the stream lineage how actually the whole workflow goes through.
See there are 349 producers.
Why there are 349 producers?
Basically every time you do play, there is an asynchronously happens using the lambda function.
It triggers the serverless option and inside the serverless it will push the message to the topic.
These are all different producers producing data.
So ideally what happens is in this case, every time a new producer comes in, he will actually produce only one message.
He will not produce multiple messages because every time it triggers a new event in the internal links from the application point of view.
So then those producers, whatever it is producing, it is going to the topic user game.
And we can see the number of partitions here and what is the bytes in and out.
What is the bytes we are getting in and what is it's getting out.
And also the message is in and message is out from the topic.
And we can see the retention time, how long we can keep.
As of now, I kept it as one hour and retention size I have set it to infinite.
It depends by comparing whatever the value we want.
And we have the trainer policy as we made here.
From here we have two ksql queries that we have added.
One is stats per user.
That means we need the details of every user, highest score, but we will take it off the here.
And the summary stats is basically like the whole whoever has still played till now.
It will be a summary of all those details.
Is what the summary stats has their query has.
From here we have two more topics.
One topic is primarily push this stats per user and another is to do the summary.
These two are streams and these two are topics.
See in general the Kafka uses internal topics to manage its offsets and all those stuff.
Those also you can see and observe here the internal topics.
So it is going to manage them internally to handle it.
So now as you play along you see here we can see the throughput coming in.
What is the consumption?
But as soon as we get what is it getting consumed or not?
Basically it gives us the producers versus the consumption.
And as you do continually play around there is a storage occupied.
Now you see there are only few of the matrices.
There are more matrices we can do it.
So what I will do is I will explore these matrices to Prometheus and push them to Grafana.
I will do it in my system along with me.
You can also try to do it basically like using this Prometheus configuration.
We can scrape all the matrices from there and put it in our own Grafana dashboard.
Such that if you want to do monitor more details or more information we can do by our own cloud.
Since we are not using any cloud we have our own VM.
I will walk through the steps and how we can do it.
So what are the report that?
I think everybody is able to see the dashboard.
Let's see.
Anybody have the USB-C from the HDMI to USB-C?
Can we get it from the IT?
This is a sample like one of the illustrative how we can use the real time data that is coming in.
Stream it and kind of use it for different purposes.
What we will do is we will take these matrices that is coming from the same.
This is going to be the cluster.
We are going to monitor all of them.
Let's say this is a schema.
This is the cluster with all the Grafana nodes.
But generally what happens is the methodic part of this will be the Grafana and all that stuff.
We will use these methods to kind of check in the Grafana.
What I will do is I will install Prometheus and start it.
Prometheus is basically a scraping open source tool where it will scrape the information from any logging framework.
And it will store it in the local.
And from there we use the Grafana graphize an open source matrices.
We can do input from different sources and dashboard will be very good to manage and monitor.
Like kind of monitor from there you can manage better the cluster.
I have the file in this one.
We go to VM in the network.
On my drive I have the install pro file.
The first step is to open any terminal anywhere.
Generally I would say keep it somewhere maybe in your student folder.
Open the terminal.
You need to use the terminal to get all these things.
So what I mean by step is I am going to get the Prometheus downloaded to my local.
Once it is downloaded.
I am going to use the next step like basically unzipping it to my same local.
So if you do unless you basically listing it.
I can see the Prometheus folder created.
I mean it got extracted.
So I will do cd into that.
I am going inside to that.
So we have different files.
So we will modify the Prometheus OML as per the given OML here.
We do this on it.
It is already given in the file.
The details.
As of now I am using the username password as mile to connect to the cloud.
So you can try using the same.
Since you don't have it you can use mine.
So once we are in.
What you can do we do is remove the Prometheus to the local bin folder.
So just to make sure it is part of running as a back up.
We will do the steps one by one.
So you do sudo move it to the Prometheus bin.
Let me know if you are stuck at any one of the step.
It will help you out.
Then I am moving the Prometheus tool.
I am making two batteries.
One is Prometheus to get the kind of label and all that stuff.
You just need to copy paste it.
And we can try to do it.
If you miss any one of the steps let me know.
Once we have it what we need to do is modify the file that is in the etc.
Prometheus folder.
What we can do is go to the file system.
We can copy the job from this file.
What I do is I get all the resources.
I am giving you here the.
In the file we just have one of the resources.
Let me save them.
To get all those.
You can just copy it from the.
Are you able to copy from the chat?
You can just copy whatever is there in the existing file.
What you need to do is take this job.
And paste it in the Prometheus.aml.
We already have one job which is Prometheus.
We will create one more job.
Make sure you make it formatting right.
Now I added one more job called content cloud.
I am going to pull the details.
I am going to use this API.
This username and password is mine.
You can put it there.
Then you save this.
This one I just modified in the same way we have the Prometheus.
The initial file.
What I am going to do is I am going to move this file.
That is there in the next step.
In the line number 37 what I am going to do is I am going to move this.
To the VTC1.
Then once it is moved you can go to the file system.
on 2025-01-06
language: EN
WEBVTT
then click on the targets, then you should see the target
connecting to the client.
From the same system you can open the local course.
Were you able to see?
If you see any failure, just do the four steps again, then it will be up.
Go to status, targets,
I want you to run it.
Go to your file, click file, file, and then copy where the student has been.
Open file, step.
But you just maybe you have to do it.
You have to do it.
If you see the content in the
If you see, open the
So it's not how much it's about
Open this arrow.
Oh, you don't have the content.
That's why you have only that you can find what you can do is
You need to copy this.
Just copy the job.
After the file, just go to open it first.
Make the job.
Now, just go to the terminal.
Restart the program.
One more command to restart.
It's something from here.
Restart the program.
Just say failure.
Can you try using this one?
Copy the password.
Should I?
But
You're giving me a new idea.
So on base.
Yeah, yeah, yeah.
Yeah, some issues like this.
You can use both anything.
Let me generate another one.
I'm giving a new one.
If you want to.
Then you have problem.
Not.
But this is the one that's.
Yeah, yeah.
So it's based on the.
Find this.
Oh, that's like this.
Or.
Within the line and.
No, yeah.
Yeah, no, name it somewhere.
Just read it.
Oh, okay.
What do you mean?
It is.
Okay.
The.
Then restart this.
Let's say.
Why?
You know.
Yeah, I'm just restarting.
Maybe the restart comments.
Please stop.
I'll take the status.
How do you have to choose the.
I will format and give you the latest family.
Sometimes we copy paste.
I will give it the working one.
Then it should be.
So this is mine, which is working.
No, yeah.
Yeah, I didn't.
It's a place it on this.
Let's take a break.
And we'll try.
No, you're going to.
If you have any issue with the family, use the family, which I have added just now.
Just replace that one.
Then it should be.
What is this thing?
Yeah, I think.
How do you pronounce the name?
Yes, yeah, there is is up.
If you see.
If you see something like this, once it is up.
So these two there is this stuff.
So once you have it, it will show something like unknown at the first.
Because it is trying to get all the details.
If you restart it, reload it.
If you see now, it is up.
So up in that is connected.
Now it is getting the mattresses.
Then the next thing we'll do.
Install the phone and connect it and see the mattresses.
Try to do complete this in here.
We'll take a break and we'll come back.
What do you know?
Yeah, we'll come to 11 again.
Anybody having issue?
Let me know.
I'll help you.
So sometimes what happens?
We probably get straight from.
That is line X.
This is Windows.
Yeah, characters.
Yeah, that's a problem.
Thank you.
So cable is having some issue.
We're going to change this cable.
Somewhere there is a loose connection.
Somewhere there is a loose connection.
If you have installed and the.
on 2025-01-06
language: EN
WEBVTT
I just want to do one more thing.
You see here the query or the metrics browser.
So usually they use count, they do all the stuff.
If you want to explore more or more matrices on how to handle it, we need to learn about
how we can handle a graph on a, but I'm giving you a sample example of one of the metric.
What is a partition count of this?
How many partings are there on this server?
Okay, since this is saying like in this clusters.
I have only one broker.
That's where it is going.
Okay, one server.
Yeah, if you see previously there was no request.
Now since there are.
You want to see the data in your thing.
Okay.
Dennis is able to see it.
Yeah.
Once we do some action on the topics, all this stuff, then you can see the data coming
in the flow.
Okay.
If you see any error like this, just click on it.
And you take a new visible.
Try it out.
Let me know if we have issues.
Everybody able to see the data coming in?
Okay, so if you do any operation there, like kind of getting the messages in and out,
then the data will appear.
There is a delay in one to two minutes.
But monitoring is primarily.
So any questions, any doubts or any issue while you are doing it?
This is actually coming from the.
Yeah, that's my throw this from here.
Yeah.
Right.
Which one?
The.
No, this one.
Yeah.
That's what this one is in your local.
But data is driven from the cloud.
Yeah.
You know, you're sitting there.
My dashboard.
Oh, you see what access this dashboard.
It should be in the cloud because this is my local.
Yeah.
If you want to do it kind of like somebody you want to show, right.
Or if somebody has to access, then you need to put a different cloud.
The graph on then type it up.
Then people can say from there you can.
But this is my local.
You cannot be able to.
You will not be able to access it.
But.
Those are open.
Those are open source.
The only thing is you need to have the hardware ideally a VM to run.
You can run both of them on the same server.
But in general practice, right, they run from it is on one server and graph on another server.
To make sure if something was at least you have the data on the graph.
Yeah.
This is in general, like you can use.
There are so many tools to visualize, but graph on is the most popular one for visualizing
because it's an open source.
It has you can see like people building all the dashboards.
We can utilize them or you can take an inspiration from them and build our own build our own
dashboards for monitoring it and trying to see if something is going wrong.
Yeah.
So in my previous order, we used to have the how many orders are coming in, how many orders
are going down.
So on a high level, we used to use that.
I mean, there's like no direct simple way.
Yes.
Just for people from the.
Yeah, there will be no simple thing.
But yeah, like if you want to get something, you have to use something like this.
Or you utilize the conference load itself, but they will have limited number of them.
They will not have the more like to see this one.
Here you have a limited list through poor production.
The major ones usually what they do, but not every matrix will be there.
So it is very limited.
The audit loss.
No, you will not be able to get audit logs from the.
Confluent cloud on the conference, because they will not allow you to get it.
Unless you have your own cloud.
I mean, this one you installed in your.
There are two ways to do it.
Get the cloud from the conflict or use the conference cost cut to install in your own
on print.
If you do it on time, then you'll able to do it because you are holding the loss.
If you have those logs.
No.
Yeah, if you are, that's what I'm saying.
So yeah, that we will do.
Actually, that is another session.
We can do it, but conflict.
I'm not sure how they do it or how they pull it.
But if we have our own Kafka broker in our on frame or in our cloud.
By default, Kafka emits some of the matrices.
I will show you to the next session.
So basically you can install the.
Have added for the next session.
It's called.
J.M.X.
Basically, Java emits some.
So Kafka is built on Java.
Since it is on Java, what we can do is we can use this J.M.X.
Prometheus Java agent to publish to one of the thing.
And from there, we can put those details to the permit us and we can manage.
Whatever you are saying, right?
How many people I mean, how many topics got created, whether we have more to buy something.
But that we need to use.
You need to have to write the rules.
Saying that what metric you want to capture and from there, what you want to take it up.
There is no direct way as of now.
So we have to do kind of like this.
Let me show you.
Yeah.
Yeah, Melon maker is helpful to replicate or.
Let's say I have one cluster of.
Servers in one location.
And I want to replicate it.
Let's say in an on frame, I want to take it to the whole.
We can use a mirror maker to replicate the whole thing to the other cluster.
Like the whole you can copy the whole content to be another cluster, another server or another cluster.
So replication is done as part of in.
OK.
Now, what I understand for our server, we're using this.
OK.
And from now, we are going to be.
You we do mirror.
You do mirror.
In the sense inside a cluster, let's say I have five.
Brokers knows how to replicate.
If it is a job, people do people says if it is fast, they internally know how to replicate.
But if this all to the different cluster, which is residing in.
Maybe in different cloud or in different data center, then you want to replicate from here to here.
Then the bridge is the middle.
The maker is useful to do between two places.
All together in a different.
I mean, somebody's I mean, one group is managing, let's say one group is managing five circles.
The other group is managing a different region or a different data center or a different flow.
But if you want to make up both of them same and use kind of.
The replicator is helpful in.
I mean, the replication, whatever happens in between the brokers, it is done by or usually taken care by Jupiter or.
But if you want to do it in a different location or something like that, this cannot manage.
Yes.
You talked about the how many people are the matrices in terms of audit, right?
You can do that, but.
You need to kind of write rules.
What type of data you want to take it out.
And how you want to do it, then based on the rules.
The GMX will emit those to some from a server and from there you can put it into the graph.
So we need to kind of add similar to this.
As of now, like for the audit logs to the graph on, I haven't done anything.
In my arms, but that we can kind of take it and do.
But in order to do that, you need to make sure that the audit logging is enabled and not it.
Logging is that I'm not so confident is giving you the audit logs.
But they will have their own satellite.
They can actually go into the server and it is being administered by.
They can like go to their own Kubernetes cluster, log into it and see what is happening.
But I don't think they will give you the.
I think we are asking for the control.
We are just asking for the.
For the activity.
Oh, I've been actually.
And that should be my I think I see.
Yeah. Those logs, I don't think we have it.
I mean, since from my side, I don't have it.
But generally, if you still want to know, like what is happening on this, that is basically on a user management.
Right. So we need to.
For two, I will see what does.
For the clusters.
And for me, I'm sorry, there are two different audit logs.
One, what is happening on a cluster?
The other thing is who is actually doing right?
Yeah. Yeah. So which one you are checking for?
You want both?
So.
They will like we need to check them like how they do the audit logs, all this stuff we can ask them to get it.
In general, like.
They have the.
General logs like what is happening on top of the cluster.
But if you want to know which user is doing what?
I'm not sure.
You can ask conflict to say, hey, can you give me the user audit log?
Then probably that they may probably.
But general logs will be audit logic.
What is happening for a topic?
What is happening for a cluster with a modified all the stuff?
But who is managing it and who is using it?
Probably they can't provide.
But.
I don't know, like, which.
Plan you're using based on the plan they may have or may not have.
Many.
Yeah.
Yeah, based on the plan, they may give you the.
You know, if you see if you want to do it.
I'm just using this for.
But if they are using any standard dedicated premium kind of thing, then just definitely they should be having the logs and probably they will.
So we can ask them to emit them to some our own.
So with this or the final server, then we can take it kind of give it what is happening and put a lot of it.
I also want to show how we can do the other things here.
So let's say in this dashboard.
OK, if something is going, we want to specifically do some other thing mechanism like send a mail kind of thing.
So we can do add other tools.
Saying that what type of data source you select and we want what data dashboard.
Then choose what is the point and search something like that and create a rule and add it.
Let's say you want.
Let's say I'm saying this is my data source and I want this metric to be calculated.
Then click on that.
Then you get the query and add it.
And on top of it, you select the room and kind of add it.
Then do what you want to do with them.
OK, whether you want to do say between something else or kind of them.
Then.
The Raffana has its own inbuilt SMTP server.
You can actually send them a message to any one of the configured units that also can be done on the phone.
If you are using Raffana, it is like a useful.
Allergy mechanism.
What are we getting from the confluent dashboard?
Those are very limited.
But if you still want to go ahead and further dive deep into more matrices.
So this is where you are doing like the document, which I have added right monitoring of COVID-19.
This emits a lot of matrices like broker matrices, which I have discussed yesterday.
Some of them.
And you can put alerts on top of it.
These are all the brokers are a meter.
There are a number of matrices, which is like you can check.
I should I was saying like how we do the bad size average, bad size max.
We using this matrices when you do any performance test.
By using this matrix, we can increase the best size, decrease the best size, and you can select how many partitions you need.
What's the throughput in general is what?
If you're doing any performance testing on the stuff, we use these matrices to manage.
You see here we have odd, other than matrices like you asked about this one.
Right. Probably we can ask them to see what is happening on.
But this generally what is happening on a high level, but not sure in detail at user level.
I am not sure.
In the next session, we will see about the general issues.
What happens to the Kafka broker and Kafka.
And see how we can mitigate them or how we can fine tune the Kafka servers.
Is everybody able to get the matrices whatever I am seeing.
Are we good or do you still have any issues?
Are you using any kind of these dashboards or are you just using the default?
They're just maybe like shading on a high level.
I think there are some matrices like the circle.
Give me a number.
OK, you don't show it.
So actually, there is a user management thing.
Probably they can add users, but if they add users, there is a cost of society.
So probably they are doing.
Yeah.
This is the general doing it.
They should be giving you access at least like read only dashboards.
Maybe to see.
Yeah, but we want it to be on the same.
on 2025-01-06
language: EN
WEBVTT
So, memq has been built on top of Kafka by the Pinterest.
So they were trying to do the large file management using Kafka.
So basically like which is a better management pops up platform compared to Kafka, which
What they did is on top of this Kafka framework, they added the storage.
You can add Amazon S3 or GCP or anything.
Now this is managed by itself in the memq.
So you can go through this whenever you have some time to understand, if you want to better
understand how the other things work.
Similar to what we have Zookeeper, they have NetEam, which handles all this stuff.
N number of things that came on top of Kafka.
But Kafka on a core.
Any other questions?
No?
on 2025-01-06
language: CY
WEBVTT
Yn ystod y cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
Mae'r cyfle hwn yn ystod y cyfle hwn.
on 2025-01-06