How can I define or solve this error for hadoop streaming? -

-

i got errors hadoop mr job, how can define problem hadoop streaming?

error: java.io.eofexception: unexpected end of input stream     @ org.apache.hadoop.io.compress.decompressorstream.decompress(decompressorstream.java:145)     @ org.apache.hadoop.io.compress.decompressorstream.read(decompressorstream.java:85)     @ java.io.inputstream.read(inputstream.java:101)     @ org.apache.hadoop.util.linereader.fillbuffer(linereader.java:180)     @ org.apache.hadoop.util.linereader.readdefaultline(linereader.java:216)     @ org.apache.hadoop.util.linereader.readline(linereader.java:174)     @ org.apache.hadoop.mapred.linerecordreader.next(linerecordreader.java:209)     @ org.apache.hadoop.mapred.linerecordreader.next(linerecordreader.java:47)     @ org.apache.hadoop.mapred.maptask$trackedrecordreader.movetonext(maptask.java:199)     @ org.apache.hadoop.mapred.maptask$trackedrecordreader.next(maptask.java:185)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:63)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:432)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:175)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:170)  error: java.io.eofexception: unexpected end of input stream     @ org.apache.hadoop.io.compress.decompressorstream.decompress(decompressorstream.java:145)     @ org.apache.hadoop.io.compress.decompressorstream.read(decompressorstream.java:85)     @ java.io.inputstream.read(inputstream.java:101)     @ org.apache.hadoop.util.linereader.fillbuffer(linereader.java:180)     @ org.apache.hadoop.util.linereader.readdefaultline(linereader.java:216)     @ org.apache.hadoop.util.linereader.readline(linereader.java:174)     @ org.apache.hadoop.mapred.linerecordreader.next(linerecordreader.java:209)     @ org.apache.hadoop.mapred.linerecordreader.next(linerecordreader.java:47)     @ org.apache.hadoop.mapred.maptask$trackedrecordreader.movetonext(maptask.java:199)     @ org.apache.hadoop.mapred.maptask$trackedrecordreader.next(maptask.java:185)     @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:63)     @ org.apache.hadoop.streaming.pipemaprunner.run(pipemaprunner.java:34)     @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:432)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:343)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:175)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:170)

(unfortunately don't have permission post source code)

these error logs not helping much. since dont have permission share code, can try following steps.

  1. check dependent libraries used in code present in nodes of hadoop cluster. required because task may execute in of worker nodes.

  2. get sample input file , execute code locally before running mapreduce program. can execute locally in following way.

    cat sampleinput | python mappercode.py


Comments

Post a Comment

Popular posts from this blog

javascript - Karma not able to start PhantomJS on Windows - Error: spawn UNKNOWN -

-
i lost "unknown" error: karma.conf.js: frameworks: ['mocha', 'should'], plugins: ['karma-mocha', 'karma-should', 'karma-phantomjs-launcher'], ... browsers: ['phantomjs'], error: 23 07 2015 14:35:37.691:warn [karma]: no captured browser, open http://localhost:9876/ 23 07 2015 14:35:37.701:info [karma]: karma v0.13.3 server started @ http://localhost:9876/ 23 07 2015 14:35:37.707:info [launcher]: starting browser phantomjs 23 07 2015 14:35:37.738:error [karma]: { [error: spawn unknown] code: 'unknown', errno: 'unknown', syscall: 'spawn' } error: spawn unknown @ exports._errnoexception (util.js:749:11) @ childprocess.spawn (child_process.js:1093:11) @ exports.spawn (child_process.js:933:9) @ object._execcommand (c:\users\jinga4x\git\mdl-sample-project\node_modules\karma\lib\launchers\process.js:63:21) @ object._start (c:\users\jinga4x\git\md...
Read more

Nuget pack csproj using nuspec -

-
i want create nuget package contains specified in nuspec file, still version csproj. in order use token have pack like: nuget pack myproj.csproj but when adds dependencies , creates unwanted folder in nuget package. nuspec file is: <?xml version="1.0" encoding="utf-8"?> <package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd"> <metadata> <id>test</id> <version>$version$</version> <title>test</title> <authors>test</authors> <owners>test</owners> <requirelicenseacceptance>false</requirelicenseacceptance> <description>test</description> <summary>test</summary> <releasenotes>test</releasenotes> <copyright>test</copyright> </metadata> <files> <file src="bin\debug\*.dll" target="lib\net45" /> <file s...
Read more