java - How to monitor application processing of Kafka messages for load testing -

-

there application (not mine) reads messages kafka, processing on them, , stores records in database. i've put program in java writes messages queue @ given rate. right now, simple measure of performance querying database @ end of test run ensure records in = records out. however, i'd expand periodically check queue see how many messages pending application hasn't yet processed see if it's getting backed up.

i figure can check offset of application's group id in zookeeper. looked @ kafka documentation, gives basic consumer examples , api documentation sparse @ best, i'm not sure how go finding information.

what apis need call in order find out in queue application at, , how many messages in queue behind position?

i'm using kafka 2.10-0.8.2.1 single zookeeper instance , 3 kafka instances, , load tester using 0.8.2.1 java api. topic in question has 3 partitions (one on each kafka server), purpose of test there single consumer.

i suggest looking @ provided tools in kafka (code available in src if need call api directly). in particular,

$ bin/kafka-run-class.sh kafka.tools.consumeroffsetchecker --group consumer-group1 --zkconnect zkhost:zkport --topic topic1

will show offset , lag:

consumer-group1,topic1,0-0 (group,topic,brokerid-partitionid) owner = consumer-group1-consumer1 consumer offset = 70121994703 = 70,121,994,703 (65.31g) log size = 70122018287 = 70,122,018,287 (65.31g) consumer lag = 23584 = 23,584 (0.00g)

references:


Comments

Post a Comment

Popular posts from this blog

javascript - Karma not able to start PhantomJS on Windows - Error: spawn UNKNOWN -

-
i lost "unknown" error: karma.conf.js: frameworks: ['mocha', 'should'], plugins: ['karma-mocha', 'karma-should', 'karma-phantomjs-launcher'], ... browsers: ['phantomjs'], error: 23 07 2015 14:35:37.691:warn [karma]: no captured browser, open http://localhost:9876/ 23 07 2015 14:35:37.701:info [karma]: karma v0.13.3 server started @ http://localhost:9876/ 23 07 2015 14:35:37.707:info [launcher]: starting browser phantomjs 23 07 2015 14:35:37.738:error [karma]: { [error: spawn unknown] code: 'unknown', errno: 'unknown', syscall: 'spawn' } error: spawn unknown @ exports._errnoexception (util.js:749:11) @ childprocess.spawn (child_process.js:1093:11) @ exports.spawn (child_process.js:933:9) @ object._execcommand (c:\users\jinga4x\git\mdl-sample-project\node_modules\karma\lib\launchers\process.js:63:21) @ object._start (c:\users\jinga4x\git\md...
Read more

Nuget pack csproj using nuspec -

-
i want create nuget package contains specified in nuspec file, still version csproj. in order use token have pack like: nuget pack myproj.csproj but when adds dependencies , creates unwanted folder in nuget package. nuspec file is: <?xml version="1.0" encoding="utf-8"?> <package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd"> <metadata> <id>test</id> <version>$version$</version> <title>test</title> <authors>test</authors> <owners>test</owners> <requirelicenseacceptance>false</requirelicenseacceptance> <description>test</description> <summary>test</summary> <releasenotes>test</releasenotes> <copyright>test</copyright> </metadata> <files> <file src="bin\debug\*.dll" target="lib\net45" /> <file s...
Read more

c# - Display ASPX Popup control in RowDeleteing Event (ASPX Gridview) -

-
i have aspxgridview includes new , delete button. delete button includs gv_rowdeleting event. want show aspxpopupcontrol (id="pupconfirm"). want user have confirm password delete item. tried: protected void gv_rowdeleting(object sender, devexpress.web.data.aspxdatadeletingeventargs e) { pupconfirm.showonpageload = true; } but nothing happened. please me. !!! you can create template column javascript prompt confirmation. <itemtemplate> <asp:linkbutton id="btndelete" runat="server" text="delete" commandname="delete" onclientclick="javascript:if(!confirm('are sure want delete record?')){return false;}"> </asp:linkbutton> </itemtemplate>
Read more