-
Notifications
You must be signed in to change notification settings - Fork 1
/
YAHOO-CHANGES.txt
1402 lines (950 loc) · 55.2 KB
/
YAHOO-CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Yahoo! Distribution of Hadoop Change Log
Patches from the following Apache Jira issues have been applied
to this release in the order indicated. This is in addition to
the patches applied from issues referenced in CHANGES.txt.
Release 0.20.104.0 - unreleased
MAPREDUCE-1682. Fix speculative execution to ensure tasks are not
scheduled after job failure. (acmurthy)
MAPREDUCE-1914. Ensure unique sub-directories for artifacts in the
DistributedCache are cleaned up. (Dick King via acmurthy)
MAPREDUCE-1538. Add a limit on the number of artifacts in the
DistributedCache to ensure we cleanup aggressively. (Dick King via
acmurthy)
MAPREDUCE-1900. Fixes a FS leak that i missed in the earlier patch.
(ddas)
MAPREDUCE-1900. Makes JobTracker/TaskTracker close filesystems, created
on behalf of users, when they are no longer needed. (ddas)
HADOOP-6832. Add a static user plugin for web auth for external users.
(omalley)
HDFS-1007. Fixes a bug in SecurityUtil.buildDTServiceName to do
with handling of null hostname. (omalley)
HDFS-1007. makes long running servers using hftp work. Also has some
refactoring in the MR code to do with handling of delegation tokens.
(omalley & ddas)
HDFS-1178. The NameNode servlets should not use RPC to connect to the
NameNode. (omalley)
MAPREDUCE-1807. Re-factor TestQueueManager. (Richard King via acmurthy)
HDFS-1150. Fixes the earlier patch to do logging in the right directory
and also adds facility for monitoring processes (via -Dprocname in the
command line). (Jakob Homan via ddas)
HADOOP-6781. security audit log shouldn't have exception in it. (boryas)
HADOOP-6776. Fixes the javadoc in UGI.createProxyUser. (ddas)
HDFS-1150. building jsvc from source tar. source tar is also checked in.
(jitendra)
HDFS-1150. Bugfix in the hadoop shell script on top of earlier patches (ddas)
MAPREDUCE-1664. Bugfix to enable queue administrators of a queue to
view job details of jobs submitted to that queue even though they
are not part of acl-view-job.
HDFS-1150. Bugfix to add more knobs to secure datanode starter.
HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's
bindings (cos)
HDFS-1130. Adds a configuration dfs.cluster.administrators for
controlling access to the default servlets in hdfs. (ddas)
HADOOP-6706.FIX. Relogin behavior for RPC clients could be improved
(boryas)
HDFS-1150. Verify datanodes' identities to clients in secure clusters.
(jghoman)
MAPREDUCE-1442. Fixed regex in job-history related to parsing Counter
values. (Luke Lu via acmurthy)
HADOOP-6760. WebServer shouldn't increase port number in case of negative
port setting caused by Jetty's race. (cos)
HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem.
(jitendra)
HADOOP-6706. Fix on top of the earlier patch. Closes the connection
on a SASL connection failure, and retries again with a new
connection. (ddas)
MAPREDUCE-1716. Fix on top of earlier patch for logs truncation a.k.a
MAPREDUCE-1100. Addresses log truncation issues when binary data is
written to log files and adds a header to a truncated log file to
inform users of the done trucation.
Release 0.20.103.0 - unreleased
MAPREDUCE-1744. Fixed DistributedCache apis to take a user-supplied
FileSystem to allow for better proxy behaviour for Oozie. (Richard King)
MAPREDUCE-1733. Authentication between pipes processes and java counterparts.
(jitendra)
MAPREDUCE-1664. Bugfix on top of the previous patch. (ddas)
HDFS-1136. FileChecksumServlets.RedirectServlet doesn't carry forward
the delegation token (boryas)
HADOOP-6756. Change value of FS_DEFAULT_NAME_KEY from fs.defaultFS
to fs.default.name which is a correct name for 0.20 (steffl)
HADOOP-6756. Document (javadoc comments) and cleanup configuration
keys in CommonConfigurationKeys.java (steffl)
MAPREDUCE-1759. Exception message for unauthorized user doing killJob,
killTask, setJobPriority needs to be improved. (gravi via vinodkv)
HADOOP-6715. AccessControlList.toString() returns empty string when
we set acl to "*". (gravi via vinodkv)
HADOOP-6757. NullPointerException for hadoop clients launched from
streaming tasks. (amarrk via vinodkv)
HADOOP-6631. FileUtil.fullyDelete() should continue to delete other files
despite failure at any level. (vinodkv)
MAPREDUCE-1317. NPE in setHostName in Rumen. (rksingh)
MAPREDUCE-1754. Replace mapred.persmissions.supergroup with an acl :
mapreduce.cluster.administrators and HADOOP-6748.: Remove
hadoop.cluster.administrators. Contributed by Amareshwari Sriramadasu.
HADOOP-6701. Incorrect exit codes for "dfs -chown", "dfs -chgrp"
(rphulari)
HADOOP-6640. FileSystem.get() does RPC retires within a static
synchronized block. (hairong)
HDFS-1006. Removes unnecessary logins from the previous patch. (ddas)
HADOOP-6745. adding some java doc to Server.RpcMetrics, UGI (boryas)
MAPREDUCE-1707. TaskRunner can get NPE in getting ugi from TaskTracker. (vinodkv)
HDFS-1104. Fsck triggers full GC on NameNode. (hairong)
HADOOP-6332. Large-scale Automated Test Framework (sharad, Sreekanth
Ramakrishnan, at all via cos)
HADOOP-6526. Additional fix for test context on top of existing one. (cos)
HADOOP-6710. Symbolic umask for file creation is not conformant with posix.
(suresh)
HADOOP-6693. Added metrics to track kerberos login success and failure.
(suresh)
Release 0.20.102.0 - unreleased
MAPREDUCE-1711. Gridmix should provide an option to submit jobs to the same
queues as specified in the trace. (rksing via htang)
MAPREDUCE-1687. Stress submission policy does not always stress the
cluster. (htang)
MAPREDUCE-1641. Bug-fix to ensure command line options such as
-files/-archives are checked for duplicate artifacts in the
DistributedCache. (Amareshwari Sreeramadasu via acmurthy)
MAPREDUCE-1641. Fix DistributedCache to ensure same files cannot be put in
both the archives and files sections. (Richard King via acmurthy)
HADOOP-6670. Fixes a testcase issue introduced by the earlier commit
of the HADOOP-6670 patch. (ddas)
MAPREDUCE-1718. Fixes a problem to do with correctly constructing
service name for the delegation token lookup in HftpFileSystem
(borya via ddas)
HADOOP-6674. Fixes the earlier patch to handle pings correctly (ddas).
MAPREDUCE-1664. Job Acls affect when Queue Acls are set.
(Ravi Gummadi via vinodkv)
HADOOP-6718. Fixes a problem to do with clients not closing RPC
connections on a SASL failure. (ddas)
MAPREDUCE-1397. NullPointerException observed during task failures.
(Amareshwari Sriramadasu via vinodkv)
HADOOP-6670. Use the UserGroupInformation's Subject as the criteria for
equals and hashCode. (omalley)
HADOOP-6716. System won't start in non-secure mode when kerb5.conf
(edu.mit.kerberos on Mac) is not present. (boryas)
MAPREDUCE-1607. Task controller may not set permissions for a
task cleanup attempt's log directory. (Amareshwari Sreeramadasu via
vinodkv)
MAPREDUCE-1533. JobTracker performance enhancements. (Amar Kamat via
vinodkv)
MAPREDUCE-1701. AccessControlException while renewing a delegation token
in not correctly handled in the JobTracker. (boryas)
HDFS-481. Incremental patch to fix broken unit test in contrib/hdfsproxy
HADOOP-6706. Fixes a bug in the earlier version of the same patch (ddas)
HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
mappings(boryas).
HDFS-1012. Support for cluster specific path entries in ldap for hdfsproxy
(Srikanth Sundarrajan via Nicholas)
HDFS-1011. Improve Logging in HDFSProxy to include cluster name associated
with the request (Srikanth Sundarrajan via Nicholas)
HDFS-1010. Retrieve group information from UnixUserGroupInformation
instead of LdapEntry (Srikanth Sundarrajan via Nicholas)
HDFS-481. Bug fix - hdfsproxy: Stack overflow + Race conditions
(Srikanth Sundarrajan via Nicholas)
MAPREDUCE-1657. After task logs directory is deleted, tasklog servlet
displays wrong error message about job ACLs. (Ravi Gummadi via vinodkv)
MAPREDUCE-1692. Remove TestStreamedMerge from the streaming tests.
(Amareshwari Sriramadasu and Sreekanth Ramakrishnan via vinodkv)
HDFS-1081. Performance regression in
DistributedFileSystem::getFileBlockLocations in secure systems (jhoman)
MAPREDUCE-1656. JobStory should provide queue info. (htang)
MAPREDUCE-1317. Reducing memory consumption of rumen objects. (htang)
MAPREDUCE-1317. Reverting the patch since it caused build failures. (htang)
MAPREDUCE-1683. Fixed jobtracker web-ui to correctly display heap-usage.
(acmurthy)
HADOOP-6706. Fixes exception handling for saslConnect. The ideal
solution is to the Refreshable interface but as Owen noted in
HADOOP-6656, it doesn't seem to work as expected. (ddas)
MAPREDUCE-1617. TestBadRecords failed once in our test runs. (Amar
Kamat via vinodkv).
MAPREDUCE-587. Stream test TestStreamingExitStatus fails with Out of
Memory. (Amar Kamat via vinodkv).
HDFS-1096. Reverting the patch since it caused build failures. (ddas)
MAPREDUCE-1317. Reducing memory consumption of rumen objects. (htang)
MAPREDUCE-1680. Add a metric to track number of heartbeats processed by the
JobTracker. (Richard King via acmurthy)
MAPREDUCE-1683. Removes JNI calls to get jvm current/max heap usage in
ClusterStatus by default. (acmurthy)
HADOOP-6687. user object in the subject in UGI should be reused in case
of a relogin. (jitendra)
HADOOP-5647. TestJobHistory fails if /tmp/_logs is not writable to.
Testcase should not depend on /tmp. (Ravi Gummadi via vinodkv)
MAPREDUCE-181. Bug fix for Secure job submission. (Ravi Gummadi via
vinodkv)
MAPREDUCE-1635. ResourceEstimator does not work after MAPREDUCE-842.
(Amareshwari Sriramadasu via vinodkv)
MAPREDUCE-1526. Cache the job related information while submitting the
job. (rksingh)
Release 0.20.101.1 - unreleased
BUG:3611917. Fixes NPE in the distcp to do with handling of null Tokens
(ddas)
Release 0.20.101.0 - 7 Apr 2010
HADOOP-6674. Turn off SASL checksums for RPCs. (jitendra via omalley)
BUG:3524726. Replace fork of DF with library call. (cdouglas via omalley)
HDFS-999. Secondary namenode should login using kerberos if security
is configured. Bugfix to original patch. (jhoman)
MAPREDUCE-1594. Support for SleepJobs in Gridmix (rksingh)
HDFS-1007. Fix. ServiceName for delegation token for Hftp has hftp
port and not RPC port.
MAPREDUCE-1376. Support for varied user submissions in Gridmix (rksingh)
HDFS-1080. SecondaryNameNode image transfer should use the defined
http address rather than local ip address (jhoman)
Release 0.20.100.0 - 1 Apr 2010
HADOOP-6661. User document for UserGroupInformation.doAs for secure
impersonation. (jitendra)
MAPREDUCE-1624. Documents the job credentials and associated details
to do with delegation tokens (ddas)
HDFS-1036. Documentation for fetchdt for forrest (boryas)
HDFS-1039. New patch on top of previous patch. Gets namenode address
from conf. (jitendra)
HADOOP-6656. Renew Kerberos TGT when 80% of the renew lifetime has been
used up. (omalley)
HADOOP-6653. Protect against NPE in setupSaslConnection when real user is
null. (omalley)
HADOOP-6649. An error in the previous committed patch. (jitendra)
HADOOP-6652. ShellBasedUnixGroupsMapping shouldn't have a cache.
(ddas)
HADOOP-6649. login object in UGI should be inside the subject
(jitendra)
HADOOP-6637. Benchmark overhead of RPC session establishment
(shv via jitendra)
HADOOP-6648. Credentials must ignore null tokens that can be generated
when using HFTP to talk to insecure clusters. (omalley)
HADOOP-6632. Fix on JobTracker to reuse filesystem handles if possible.
(ddas)
HADOOP-6647. balancer fails with "is not authorized for protocol
interface NamenodeProtocol" in secure environment (boryas)
MAPREDUCE-1612. job conf file is not accessible from job history
web page. (Ravi Gummadi via vinodkv)
MAPREDUCE-1611. Refresh nodes and refresh queues doesnt work with
service authorization enabled. (Amar Kamat via vinodkv)
HADOOP-6644. util.Shell getGROUPS_FOR_USER_COMMAND method
name - should use common naming convention (boryas)
MAPREDUCE-1609. Fixes a problem with localization of job log
directories when tasktracker is re-initialized that can result
in failed tasks. (Amareshwari Sriramadasu via yhemanth)
MAPREDUCE-1610. Update forrest documentation for directory
structure of localized files. (Ravi Gummadi via yhemanth)
MAPREDUCE-1532. Fixes a javadoc and an exception message in JobInProgress
when the authenticated user is different from the user in conf. (ddas)
MAPREDUCE-1417. Update forrest documentation for private
and public distributed cache files. (Ravi Gummadi via yhemanth)
HADOOP-6634. AccessControlList uses full-principal names to verify acls
causing queue-acls to fail (vinodkv)
HADOOP-6642. Fix javac, javadoc, findbugs warnings. (chrisdo via acmurthy)
HDFS-1044. Cannot submit mapreduce job from secure client to
unsecure sever. (boryas)
HADOOP-6638. try to relogin in a case of failed RPC connection
(expired tgt) only in case the subject is loginUser or
proxyUgi.realUser. (boryas)
HADOOP-6632. Support for using different Kerberos keys for different
instances of Hadoop services. (jitendra)
HADOOP-6526. Need mapping from long principal names to local OS
user names. (jitendra)
MAPREDUCE-1604. Update Forrest documentation for job authorization
ACLs. (Amareshwari Sriramadasu via yhemanth)
HDFS-1045. In secure clusters, re-login is necessary for https
clients before opening connections (jhoman)
HADOOP-6603. Addition to original patch to be explicit
about new method not being for general use. (jhoman)
MAPREDUCE-1543. Add audit log messages for job and queue
access control checks. (Amar Kamat via yhemanth)
MAPREDUCE-1606. Fixed occassinal timeout in TestJobACL. (Ravi Gummadi via
acmurthy)
HADOOP-6633. normalize property names for JT/NN kerberos principal
names in configuration. (boryas)
HADOOP-6613. Changes the RPC server so that version is checked first
on an incoming connection. (Kan Zhang via ddas)
HADOOP-5592. Fix typo in Streaming doc in reference to GzipCodec.
(Corinne Chandel via tomwhite)
MAPREDUCE-813. Updates Streaming and M/R tutorial documents.
(Corinne Chandel via ddas)
MAPREDUCE-927. Cleanup of task-logs should happen in TaskTracker instead
of the Child. (Amareshwari Sriramadasu via vinodkv)
HDFS-1039. Service should be set in the token in JspHelper.getUGI.
(jitendra)
MAPREDUCE-1599. MRBench reuses jobConf and credentials there in.
(jitendra)
MAPREDUCE-1522. FileInputFormat may use the default FileSystem for the
input path. (Tsz Wo (Nicholas), SZE via cdouglas)
HDFS-1036. In DelegationTokenFetch pass Configuration object so getDefaultUri
will work correctly.
HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only if
security is enabled. (jitendra)
HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas)
HADOOP-6598. Verbose logging from the Group class (one more case)
(boryas)
HADOOP-6627. Bad Connection to FS" message in FSShell should print
message from the exception (boryas)
HDFS-1033. In secure clusters, NN and SNN should verify that the remote
principal during image and edits transfer (jhoman)
HDFS-1005. Fixes a bug to do with calling the cross-realm API in Fsck
client. (ddas)
MAPREDUCE-1422. Fix cleanup of localized job directory to work if files
with non-deletable permissions are created within it.
(Amar Kamat via yhemanth)
HDFS-1007. Fixes bugs to do with 20S cluster talking to 20 over
hftp (borya)
MAPREDUCE:1566. Fixes bugs in the earlier patch. (ddas)
HDFS-992. A bug in backport for HDFS-992. (jitendra)
HADOOP-6598. Remove verbose logging from the Groups class. (borya)
HADOOP-6620. NPE if renewer is passed as null in getDelegationToken.
(jitendra)
HDFS-1023. Second Update to original patch to fix username (jhoman)
MAPREDUCE-1435. Add test cases to already committed patch for this
jira, synchronizing changes with trunk. (yhemanth)
HADOOP-6612. Protocols RefreshUserToGroupMappingsProtocol and
RefreshAuthorizationPolicyProtocol authorization settings thru
KerberosInfo (boryas)
MAPREDUCE-1566. Bugfix for tests on top of the earlier patch. (ddas)
MAPREDUCE-1566. Mechanism to import tokens and secrets from a file in to
the submitted job. (omalley)
HADOOP-6603. Provide workaround for issue with Kerberos not
resolving corss-realm principal. (kan via jhoman)
HDFS-1023. Update to original patch to fix username (jhoman)
HDFS-814. Add an api to get the visible length of a
DFSDataInputStream. (hairong)
HDFS-1023. Allow http server to start as regular user if https
principal is not defined. (jhoman)
HDFS-1022. Merge all three test specs files (common, hdfs, mapred)
into one. (steffl)
HDFS-101. DFS write pipeline: DFSClient sometimes does not detect
second datanode failure. (hairong)
HDFS-1015. Intermittent failure in TestSecurityTokenEditLog. (jitendra)
MAPREDUCE-1550. A bugfix on top of what was committed earlier (ddas).
MAPREDUCE-1155. DISABLING THE TestStreamingExitStatus temporarily. (ddas)
HDFS-1020. Changes the check for renewer from short name to long name
in the cancel/renew delegation token methods. (jitendra via ddas)
HDFS-1019. Fixes values of delegation token parameters in
hdfs-default.xml. (jitendra via ddas)
MAPREDUCE-1430. Fixes a backport issue with the earlier patch. (ddas)
MAPREDUCE-1559. Fixes a problem in DelegationTokenRenewal class to
do with using the right credentials when talking to the NameNode.(ddas)
MAPREDUCE-1550. Fixes a problem to do with creating a filesystem using
the user's UGI in the JobHistory browsing. (ddas)
HADOOP-6609. Fix UTF8 to use a thread local DataOutputBuffer instead of
a static that was causing a deadlock in RPC. (omalley)
HADOOP-6584. Fix javadoc warnings introduced by original HADOOP-6584
patch (jhoman)
HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather than using
createRemoteUser(). (jhoman)
MAPREDUCE-899. Modified LinuxTaskController to check that task-controller
has right permissions and ownership before performing any actions.
(Amareshwari Sriramadasu via yhemanth)
HDFS-204. Revive number of files listed metrics. (hairong)
HADOOP-6569. FsShell#cat should avoid calling uneccessary getFileStatus
before opening a file to read. (hairong)
HDFS-1014. Error in reading delegation tokens from edit logs. (jitendra)
HDFS-458. Add under-10-min tests from 0.22 to 0.20.1xx, only the tests
that already exist in 0.20.1xx (steffl)
MAPREDUCE-1155. Just pulls out the TestStreamingExitStatus part of the
patch from jira (that went to 0.22). (ddas)
HADOOP-6600. Fix for branch backport only. Comparing of user should use
equals. (boryas).
HDFS-1006. Fixes NameNode and SecondaryNameNode to use kerberizedSSL for
the http communication. (Jakob Homan via ddas)
HDFS-1007. Fixes a bug on top of the earlier patch. (ddas)
HDFS-1005. Fsck security. Makes it work over kerberized SSL (boryas and
jhoman)
HDFS-1007. Makes HFTP and Distcp use kerberized SSL. (ddas)
MAPREDUCE-1455. Fixes a testcase in the earlier patch.
(Ravi Gummadi via ddas)
HDFS-992. Refactors block access token implementation to conform to the
generic Token interface. (Kan Zhang via ddas)
HADOOP-6584. Adds KrbSSL connector for jetty. (Jakob Homan via ddas)
HADOOP-6589. Add a framework for better error messages when rpc connections
fail to authenticate. (Kan Zhang via omalley)
HADOOP-6600,HDFS-1003,MAPREDUCE-1539. mechanism for authorization check
for inter-server protocols(boryas)
HADOOP-6580,HDFS-993,MAPREDUCE-1516. UGI should contain authentication
method.
Namenode and JT should issue a delegation token only for kerberos
authenticated clients. (jitendra)
HDFS-984,HADOOP-6573,MAPREDUCE-1537. Delegation Tokens should be persisted
in Namenode, and corresponding changes in common and mr. (jitendra)
HDFS-994. Provide methods for obtaining delegation token from Namenode for
hftp and other uses. Incorporates HADOOP-6594: Update hdfs script to
provide fetchdt tool. (jitendra)
HADOOP-6586. Log authentication and authorization failures and successes
(boryas)
HDFS-991. Allow use of delegation tokens to authenticate to the
HDFS servlets. (omalley)
HADOOP-1849. Add undocumented configuration parameter for per handler
call queue size in IPC Server. (shv)
HADOOP-6599. Split existing RpcMetrics with summary in RpcMetrics and
details information in RpcDetailedMetrics. (suresh)
HDFS-985. HDFS should issue multiple RPCs for listing a large directory.
(hairong)
HDFS-1000. Updates libhdfs to use the new UGI. (ddas)
MAPREDUCE-1532. Ensures all filesystem operations at the client is done
as the job submitter. Also, changes the renewal to maintain list of tokens
to renew. (ddas)
HADOOP-6596. Add a version field to the seialization of the
AbstractDelegationTokenIdentifier. (omalley)
HADOOP-5561. Add javadoc.maxmemory to build.xml to allow larger memory.
(jkhoman via omalley)
HADOOP-6579. Add a mechanism for encoding and decoding Tokens in to
url-safe strings. (omalley)
MAPREDUCE-1354. Make incremental changes in jobtracker for
improving scalability (acmurthy)
HDFS-999.Secondary namenode should login using kerberos if security
is configured(boryas)
MAPREDUCE-1466. Added a private configuration variable
mapreduce.input.num.files, to store number of input files
being processed by M/R job. (Arun Murthy via yhemanth)
MAPREDUCE-1403. Save file-sizes of each of the artifacts in
DistributedCache in the JobConf (Arun Murthy via yhemanth)
HADOOP-6543. Fixes a compilation problem in the original commit. (ddas)
MAPREDUCE-1520. Moves a call to setWorkingDirectory in Child to within
a doAs block. (Amareshwari Sriramadasu via ddas)
HADOOP-6543. Allows secure clients to talk to unsecure clusters.
(Kan Zhang via ddas)
MAPREDUCE-1505. Delays construction of the job client until it is really
required. (Arun C Murthy via ddas)
HADOOP-6549. TestDoAsEffectiveUser should use ip address of the host
for superuser ip check. (jitendra)
HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh)
HDFS-946. NameNode should not return full path name when lisitng a
diretory or getting the status of a file. (hairong)
MAPREDUCE-1398. Fix TaskLauncher to stop waiting for slots on a TIP
that is killed / failed. (Amareshwari Sriramadasu via yhemanth)
MAPREDUCE-1476. Fix the M/R framework to not call commit for special
tasks like job setup/cleanup and task cleanup.
(Amareshwari Sriramadasu via yhemanth)
HADOOP-6467. Performance improvement for liststatus on directories in
hadoop archives. (mahadev)
HADOOP-6558. archive does not work with distcp -update. (nicholas via
mahadev)
HADOOP-6583. Captures authentication and authorization metrics. (ddas)
MAPREDUCE-1316. Fixes a memory leak of TaskInProgress instances in
the jobtracker. (Amar Kamat via yhemanth)
MAPREDUCE-670. Creates ant target for 10 mins patch test build.
(Jothi Padmanabhan via gkesavan)
MAPREDUCE-1430. JobTracker should be able to renew delegation tokens
for the jobs(boryas)
HADOOP-6551, HDFS-986, MAPREDUCE-1503. Change API for tokens to throw
exceptions instead of returning booleans. (omalley)
HADOOP-6545. Changes the Key for the FileSystem to be UGI. (ddas)
HADOOP-6572. Makes sure that SASL encryption and push to responder queue
for the RPC response happens atomically. (Kan Zhang via ddas)
HDFS-965. Split the HDFS TestDelegationToken into two tests, of which
one proxy users and the other normal users. (jitendra via omalley)
HADOOP-6560. HarFileSystem throws NPE for har://hdfs-/foo (nicholas via
mahadev)
MAPREDUCE-686. Move TestSpeculativeExecution.Fake* into a separate class
so that it can be used by other tests. (Jothi Padmanabhan via sharad)
MAPREDUCE-181. Fixes an issue in the use of the right config. (ddas)
MAPREDUCE-1026. Fixes a bug in the backport. (ddas)
HADOOP-6559. Makes the RPC client automatically re-login when the SASL
connection setup fails. This is applicable to only keytab based logins.
(ddas)
HADOOP-2141. Backport changes made in the original JIRA to aid
fast unit tests in Map/Reduce. (Amar Kamat via yhemanth)
HADOOP-6382. Import the mavenizable pom file structure and adjust
the build targets and bin scripts. (gkesvan via ltucker)
MAPREDUCE-1425. archive throws OutOfMemoryError (mahadev)
MAPREDUCE-1399. The archive command shows a null error message. (nicholas)
HADOOP-6552. Puts renewTGT=true and useTicketCache=true for the keytab
kerberos options. (ddas)
MAPREDUCE-1433. Adds delegation token for MapReduce (ddas)
HADOOP-4359. Fixes a bug in the earlier backport. (ddas)
HADOOP-6547, HDFS-949, MAPREDUCE-1470. Move Delegation token into Common
so that we can use it for MapReduce also. It is a combined patch for
common, hdfs and mr. (jitendra)
HADOOP-6510,HDFS-935,MAPREDUCE-1464. Support for doAs to allow
authenticated superuser to impersonate proxy users. It is a combined
patch with compatible fixes in HDFS and MR. (jitendra)
MAPREDUCE-1435. Fixes the way symlinks are handled when cleaning up
work directory files. (Ravi Gummadi via yhemanth)
MAPREDUCE-6419. Fixes a bug in the backported patch. (ddas)
MAPREDUCE-1457. Fixes JobTracker to get the FileSystem object within
getStagingAreaDir within a privileged block. Fixes Child.java to use the
appropriate UGIs while getting the TaskUmbilicalProtocol proxy and while
executing the task. Contributed by Jakob Homan. (ddas)
MAPREDUCE-1440. Replace the long user name in MapReduce with the local
name. (ddas)
HADOOP-6419. Adds SASL based authentication to RPC. Also includes the
MAPREDUCE-1335 and HDFS-933 patches. Contributed by Kan Zhang.
(ddas)
HADOOP-6538. Sets hadoop.security.authentication to simple by default.
(ddas)
HDFS-938. Replace calls to UGI.getUserName() with
UGI.getShortUserName()(boryas)
HADOOP-6544. fix ivy settings to include JSON jackson.codehause.org
libs for .20 (boryas)
HDFS-907. Add tests for getBlockLocations and totalLoad metrics. (rphulari)
HADOOP-6204. Implementing aspects development and fault injeciton
framework for Hadoop (cos)
MAPREDUCE-1432. Adds hooks in the jobtracker and tasktracker
for loading the tokens in the user's ugi. This is required for
the copying of files from the hdfs. (Devaraj Das vi boryas)
MAPREDUCE-1383. Automates fetching of delegation tokens in File*Formats
Distributed Cache and Distcp. Also, provides a config
mapreduce.job.hdfs-servers that the jobs can populate with a comma
separated list of namenodes. The job client automatically fetches
delegation tokens from those namenodes.
HADOOP-6337. Update FilterInitializer class to be more visible
and take a conf for further development. (jhoman)
HADOOP-6520. UGI should load tokens from the environment. (jitendra)
HADOOP-6517, HADOOP-6518. Ability to add/get tokens from
UserGroupInformation & Kerberos login in UGI should honor KRB5CCNAME
(jitendra)
HADOOP-6299. Reimplement the UserGroupInformation to use the OS
specific and Kerberos JAAS login. (jhoman, ddas, oom)
HADOOP-6524. Contrib tests are failing Clover'ed build. (cos)
MAPREDUCE-842. Fixing a bug in the earlier version of the patch
related to improper localization of the job token file.
(Ravi Gummadi via yhemanth)
HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry
via cos)
MAPREDUCE-1186. Modified code in distributed cache to set
permissions only on required set of localized paths.
(Amareshwari Sriramadasu via yhemanth)
HDFS-899. Delegation Token Implementation. (Jitendra Nath Pandey)
MAPREDUCE-896. Enhance tasktracker to cleanup files that might have
been created by user tasks with non-writable permissions.
(Ravi Gummadi via yhemanth)
Release yahoo-hadoop-0.20.1-3195383006
HADOOP-5879. Read compression level and strategy from Configuration for
gzip compression. (He Yongqiang via cdouglas)
HADOOP-6161. Add get/setEnum methods to Configuration. (cdouglas)
HADOOP-6382 Mavenize the build.xml targets and update the bin scripts
in preparation for publishing POM files (giri kesavan via ltucker)
HDFS-737. Add full path name of the file to the block information and
summary of total number of files, blocks, live and deadnodes to
metasave output. (Jitendra Nath Pandey via suresh)
HADOOP-6577. Add hidden configuration option "ipc.server.max.response.size"
to change the default 1 MB, the maximum size when large IPC handler
response buffer is reset. (suresh)
Release yahoo-hadoop-0.20.1-3195383008
HADOOP-6521. Fix backward compatiblity issue with umask when applications
use deprecated param dfs.umask in configuration or use
FsPermission.setUMask(). (suresh)
HDFS-737. Add full path name of the file to the block information and
summary of total number of files, blocks, live and deadnodes to
metasave output. (Jitendra Nath Pandey via suresh)
HADOOP-6521. Fix backward compatiblity issue with umask when applications
use deprecated param dfs.umask in configuration or use
FsPermission.setUMask(). (suresh)
MAPREDUCE-433. Use more reliable counters in TestReduceFetch.
(Christopher Douglas via ddas)
MAPREDUCE-744. Introduces the notion of a public distributed cache.
(ddas)
MAPREDUCE-1140. Fix DistributedCache to not decrement reference counts
for unreferenced files in error conditions.
(Amareshwari Sriramadasu via yhemanth)
MAPREDUCE-1284. Fix fts_open() call in task-controller that was failing
LinuxTaskController unit tests. (Ravi Gummadi via yhemanth)
MAPREDUCE-1098. Fixed the distributed-cache to not do i/o while
holding a global lock.
(Amareshwari Sriramadasu via acmurthy)
MAPREDUCE-1338. Introduces the notion of token cache using which
tokens and secrets can be sent by the Job client to the JobTracker.
(Boris Shkolnik)
HADOOP-6495. Identifier should be serialized after the password is created
In Token constructor. (Jitendra Nath Pandey)
HADOOP-6506. Failing tests prevent the rest of test targets from
execution. (cos)
x
HADOOP-5457. Fix to continue to run builds even if contrib test fails.
(gkesavan)
MAPREDUCE-856. Setup secure permissions for distributed cache files.
(Vinod Kumar Vavilapalli via yhemanth)
MAPREDUCE-871. Fix ownership of Job/Task local files to have correct
group ownership according to the egid of the tasktracker.
(Vinod Kumar Vavilapalli via yhemanth)
MAPREDUCE-476. Extend DistributedCache to work locally (LocalJobRunner).
(Philip Zeyliger via tomwhite)
MAPREDUCE-711. Removed Distributed Cache from Common, to move it under
Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
MAPREDUCE-478. Allow map and reduce jvm parameters, environment
variables and ulimit to be set separately. (acmurthy)
MAPREDUCE-842. Setup secure permissions for localized job files,
intermediate outputs and log files on tasktrackers.
(Vinod Kumar Vavilapalli via yhemanth)
MAPREDUCE-408. Fixes an assertion problem in TestKillSubProcesses.
(Ravi Gummadi via ddas)
HADOOP-4041. IsolationRunner does not work as documented.
(Philip Zeyliger via tomwhite)
MAPREDUCE-181. Changes the job submission process to be secure.
(Devaraj Das)
HADOOP-5737. Fixes a problem in the way the JobTracker used to talk to
other daemons like the NameNode to get the job's files. Also adds APIs
in the JobTracker to get the FileSystem objects as per the JobTracker's
configuration. (Amar Kamat via ddas)
HADOOP-5771. Implements unit tests for LinuxTaskController.
(Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
HADOOP-4656, HDFS-685, MAPREDUCE-1083. Use the user-to-groups mapping
service in the NameNode and JobTracker. Combined patch for these 3 jiras
otherwise tests fail. (Jitendra Nath Pandey)
MAPREDUCE-1250. Refactor job token to use a common token interface.
(Jitendra Nath Pandey)
MAPREDUCE-1026. Shuffle should be secure. (Jitendra Nath Pandey)
HADOOP-4268. Permission checking in fsck. (Jitendra Nath Pandey)
HADOOP-6415. Adding a common token interface for both job token and
delegation token. (Jitendra Nath Pandey)
HADOOP-6367, HDFS-764. Moving Access Token implementation from Common to
HDFS. These two jiras must be committed together otherwise build will
fail. (Jitendra Nath Pandey)
HDFS-409. Add more access token tests
(Jitendra Nath Pandey)
HADOOP-6132. RPC client opens an extra connection for VersionedProtocol.
(Jitendra Nath Pandey)
HDFS-445. pread() fails when cached block locations are no longer valid.
(Jitendra Nath Pandey)
HDFS-195. Need to handle access token expiration when re-establishing the
pipeline for dfs write. (Jitendra Nath Pandey)
HADOOP-6176. Adding a couple private methods to AccessTokenHandler
for testing purposes. (Jitendra Nath Pandey)
HADOOP-5824. remove OP_READ_METADATA functionality from Datanode.
(Jitendra Nath Pandey)
HADOOP-4359. Access Token: Support for data access authorization
checking on DataNodes. (Jitendra Nath Pandey)
Release yahoo-hadoop-0.20.1-3195383006
MAPREDUCE-1372. Fixed a ConcurrentModificationException in jobtracker.
(Arun C Murthy via yhemanth)
MAPREDUCE-1316. Fix jobs' retirement from the JobTracker to prevent memory
leaks via stale references. (Amar Kamat via acmurthy)
MAPREDUCE-1342. Fixed deadlock in global blacklisting of tasktrackers.
(Amareshwari Sriramadasu via acmurthy)
HADOOP-6460. Reinitializes buffers used for serializing responses in ipc
server on exceeding maximum response size to free up Java heap. (suresh)
MAPREDUCE-1100. Truncate user logs to prevent TaskTrackers' disks from
filling up. (Vinod Kumar Vavilapalli via acmurthy)
MAPREDUCE-1143. Fix running task counters to be updated correctly
when speculative attempts are running for a TIP.
(Rahul Kumar Singh via yhemanth)
HADOOP-6151, 6281, 6285, 6441. Add HTML quoting of the parameters to all
of the servlets to prevent XSS attacks. (omalley)
MAPREDUCE-896. Fix bug in earlier implementation to prevent
spurious logging in tasktracker logs for absent file paths.
(Ravi Gummadi via yhemanth)
MAPREDUCE-676. Fix Hadoop Vaidya to ensure it works for map-only jobs.
(Suhas Gogate via acmurthy)
HADOOP-5582. Fix Hadoop Vaidya to use new Counters in
org.apache.hadoop.mapreduce package. (Suhas Gogate via acmurthy)
HDFS-595. umask settings in configuration may now use octal or
symbolic instead of decimal. Update HDFS tests as such. (jghoman)
MAPREDUCE-1068. Added a verbose error message when user specifies an
incorrect -file parameter. (Amareshwari Sriramadasu via acmurthy)
MAPREDUCE-1171. Allow the read-error notification in shuffle to be
configurable. (Amareshwari Sriramadasu via acmurthy)
MAPREDUCE-353. Allow shuffle read and connection timeouts to be
configurable. (Amareshwari Sriramadasu via acmurthy)
HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented.
(suresh)
MAPREDUCE-1185. Redirect running job url to history url if job is already
retired. (Amareshwari Sriramadasu and Sharad Agarwal via sharad)
MAPREDUCE-754. Fix NPE in expiry thread when a TT is lost. (Amar Kamat
via sharad)
MAPREDUCE-896. Modify permissions for local files on tasktracker before
deletion so they can be deleted cleanly. (Ravi Gummadi via yhemanth)
HADOOP-5771. Implements unit tests for LinuxTaskController.
(Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
MAPREDUCE-1124. Import Gridmix3 and Rumen. (cdouglas)
MAPREDUCE-1063. Document gridmix benchmark. (cdouglas)
HDFS-758. Changes to report status of decommissioining on the namenode web
UI. (jitendra)
HADOOP-6234. Add new option dfs.umaskmode to set umask in configuration
to use octal or symbolic instead of decimal. (Jakob Homan via suresh)
MAPREDUCE-1147. Add map output counters to new API. (Amar Kamat via
cdouglas)
MAPREDUCE-1182. Fix overflow in reduce causing allocations to exceed the
configured threshold. (cdouglas)
HADOOP-4933. Fixes a ConcurrentModificationException problem that shows up
when the history viewer is accessed concurrently.
(Amar Kamat via ddas)
MAPREDUCE-1140. Fix DistributedCache to not decrement reference counts for
unreferenced files in error conditions.
(Amareshwari Sriramadasu via yhemanth)
HADOOP-6203. FsShell rm/rmr error message indicates exceeding Trash quota
and suggests using -skpTrash, when moving to trash fails.
(Boris Shkolnik via suresh)
HADOOP-5675. Do not launch a job if DistCp has no work to do. (Tsz Wo
(Nicholas), SZE via cdouglas)