Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconcile terminology and method naming to 'used/unused segments'; Rename MetadataSegmentManager to MetadataSegmentsManager #7306

Merged
merged 67 commits into from
Jan 27, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
87556bc
Reconcile terminology and method naming to 'used/unused segments'; Do…
leventov Mar 19, 2019
e44e199
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Mar 20, 2019
25de70d
Fix brace
leventov Mar 20, 2019
fe13d9c
Import order
leventov Mar 21, 2019
050bd3b
Rename withKillDataSourceWhitelist to withSpecificDataSourcesToKill
leventov Mar 21, 2019
562f5b4
Fix tests
leventov Mar 21, 2019
a6138da
Fix tests by adding proper methods without interval parameters to Ind…
leventov Mar 25, 2019
cb6fe7f
More aligned names of DruidCoordinatorHelpers, rename several Coordin…
leventov Mar 28, 2019
164f783
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Mar 28, 2019
d959bb0
Rename ClientCompactTaskQuery to ClientCompactionTaskQuery for consis…
leventov Mar 28, 2019
01c4494
More variable and method renames
leventov Mar 28, 2019
06ec389
Rename MetadataSegments to SegmentsMetadata
leventov Apr 1, 2019
09fab95
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Apr 1, 2019
f4774e1
Javadoc update
leventov Apr 1, 2019
8d5200d
Simplify SegmentsMetadata.getUnusedSegmentIntervals(), more javadocs
leventov Apr 1, 2019
255f320
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Apr 9, 2019
f34eef0
Update Javadoc of VersionedIntervalTimeline.iterateAllObjects()
leventov Apr 9, 2019
2ebb23c
Reorder imports
leventov Apr 9, 2019
ba8ed62
Rename SegmentsMetadata.tryMark... methods to mark... and make them t…
leventov Apr 19, 2019
64b0404
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Apr 19, 2019
231a529
Complete merge
leventov Apr 19, 2019
a113231
Add CollectionUtils.newTreeSet(); Refactor DruidCoordinatorRuntimePar…
leventov Apr 22, 2019
abad3f3
Remove MetadataSegmentManager
leventov Apr 22, 2019
cb1f3b0
Rename millisLagSinceCoordinatorBecomesLeaderBeforeCanMarkAsUnusedOve…
leventov Apr 22, 2019
a5840b8
Fix tests, refactor DruidCluster creation in tests into DruidClusterB…
leventov Apr 26, 2019
f4283d2
Merge with master
leventov May 3, 2019
559f2c8
Merge remote-tracking branch 'upstream/master' into used-segments
leventov May 8, 2019
93c12cd
Merge remote-tracking branch 'upstream/master' into used-segments
leventov May 9, 2019
dfc0dbb
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 19, 2019
553b804
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 22, 2019
886e580
Fix inspections
leventov Jul 22, 2019
77fc5e8
Fix SQLMetadataSegmentManagerEmptyTest and rename it to SqlSegmentsMe…
leventov Jul 22, 2019
1258fa0
Rename SegmentsAndMetadata to SegmentsAndCommitMetadata to reduce the…
leventov Jul 23, 2019
15cb9ae
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 24, 2019
c2c7547
Rename DruidCoordinatorHelper to CoordinatorDuty, refactor DruidCoord…
leventov Jul 25, 2019
9cd239e
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 26, 2019
a723553
Unused import
leventov Jul 26, 2019
26a8381
Optimize imports
leventov Jul 29, 2019
afcd301
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 29, 2019
1557acc
Rename IndexerSQLMetadataStorageCoordinator.getDataSourceMetadata() t…
leventov Jul 29, 2019
96bcab7
Unused import
leventov Jul 29, 2019
d934853
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jul 31, 2019
c3e488e
Update terminology in datasource-view.tsx
leventov Jul 31, 2019
f65e654
Fix label in datasource-view.spec.tsx.snap
leventov Jul 31, 2019
5e8f3e4
Fix lint errors in datasource-view.tsx
leventov Aug 1, 2019
697f0d5
Doc improvements
leventov Aug 1, 2019
66a23f9
Another attempt to please TSLint
leventov Aug 2, 2019
d3882c4
Another attempt to please TSLint
leventov Aug 2, 2019
4b9d992
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Aug 2, 2019
b77a327
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Aug 8, 2019
d355aa6
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Aug 12, 2019
718bb23
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Aug 28, 2019
19b7f3a
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Aug 28, 2019
2a6bcff
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Sep 11, 2019
22e71d1
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Nov 14, 2019
c575e6e
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Nov 14, 2019
c5d22a0
Style fixes
leventov Nov 14, 2019
285ebc9
Fix IndexerSQLMetadataStorageCoordinator.createUsedSegmentsSqlQueryFo…
leventov Nov 14, 2019
53f572a
Try to fix docs build issue
leventov Nov 14, 2019
da7667c
Javadoc and spelling fixes
leventov Nov 15, 2019
91a5c8c
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Nov 20, 2019
a566b6d
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Nov 21, 2019
6da0b9d
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Dec 8, 2019
d0bf20a
Rename SegmentsMetadata to SegmentsMetadataManager, address other com…
leventov Jan 22, 2020
10587c1
Merge remote-tracking branch 'upstream/master' into used-segments
leventov Jan 22, 2020
2ad6dd5
Address more comments
leventov Jan 23, 2020
6745f4a
Fix a bug in DataSourceOptimizer
leventov Jan 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@
import com.google.common.collect.ImmutableMap;
import org.apache.druid.client.DataSourcesSnapshot;
import org.apache.druid.java.util.common.DateTimes;
import org.apache.druid.server.coordinator.helper.CompactionSegmentIterator;
import org.apache.druid.server.coordinator.helper.CompactionSegmentSearchPolicy;
import org.apache.druid.server.coordinator.helper.NewestSegmentFirstPolicy;
import org.apache.druid.server.coordinator.duty.CompactionSegmentIterator;
import org.apache.druid.server.coordinator.duty.CompactionSegmentSearchPolicy;
import org.apache.druid.server.coordinator.duty.NewestSegmentFirstPolicy;
import org.apache.druid.timeline.DataSegment;
import org.apache.druid.timeline.VersionedIntervalTimeline;
import org.apache.druid.timeline.partition.NumberedShardSpec;
Expand Down
5 changes: 4 additions & 1 deletion docs/content/operations/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,10 @@ These metrics are for the Druid Coordinator and are reset each time the Coordina
|`segment/unavailable/count`|Number of segments (not including replicas) left to load until segments that should be loaded in the cluster are available for queries.|dataSource.|0|
|`segment/underReplicated/count`|Number of segments (including replicas) left to load until segments that should be loaded in the cluster are available for queries.|tier, dataSource.|0|

If `emitBalancingStats` is set to `true` in the Coordinator [dynamic configuration](../configuration/index.html#dynamic-configuration), then [log entries](../configuration/logging.html) for class `org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger` will have extra information on balancing decisions.
If `emitBalancingStats` is set to `true` in the Coordinator [dynamic configuration](
../configuration/index.html#dynamic-configuration), then [log entries](../configuration/logging.html) for class
`org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics` will have extra information on balancing
decisions.

## General Health

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ public class CompactionTask extends AbstractTask
private static final Logger log = new Logger(CompactionTask.class);
/**
* Must be synced with {@link
* org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentCompactor#COMPACTION_TASK_TYPE}.
* org.apache.druid.server.coordinator.duty.CompactSegments#COMPACTION_TASK_TYPE}.
*/
private static final String TYPE = "compact";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
import java.util.Map;

/**
* An immutable snapshot information about used segments and overshadowed segments for
* An immutable snapshot of metadata information about used segments and overshadowed segments, coming from
* {@link org.apache.druid.metadata.SqlSegmentsMetadata}.
*/
public class DataSourcesSnapshot
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,6 @@
@BindingAnnotation
@Target({ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface CoordinatorIndexingServiceHelper
public @interface CoordinatorIndexingServiceDuty
{
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

package org.apache.druid.server.coordinator;

import org.apache.druid.server.coordinator.duty.BalanceSegments;
import org.apache.druid.timeline.DataSegment;

import javax.annotation.Nullable;
Expand All @@ -30,7 +31,7 @@
* This interface describes the coordinator balancing strategy, which is responsible for making decisions on where
* to place {@link DataSegment}s on historical servers (described by {@link ServerHolder}). The balancing strategy
* is used by {@link org.apache.druid.server.coordinator.rules.LoadRule} to assign and drop segments, and by
* {@link org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer} to migrate segments between historicals.
* {@link BalanceSegments} to migrate segments between historicals.
*/
public interface BalancerStrategy
{
Expand Down Expand Up @@ -77,7 +78,7 @@ default Iterator<ServerHolder> pickServersToDrop(DataSegment toDropSegment, Navi

/**
* Add balancing strategy stats during the 'balanceTier' operation of
* {@link org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer} to be included
* {@link BalanceSegments} to be included
* @param tier historical tier being balanced
* @param stats stats object to add balancing strategy stats to
* @param serverHolderList servers in tier being balanced
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
import com.google.common.collect.ImmutableSet;
import org.apache.druid.common.config.JacksonConfigManager;
import org.apache.druid.java.util.common.IAE;
import org.apache.druid.server.coordinator.helper.DruidCoordinatorUnusedSegmentsKiller;
import org.apache.druid.server.coordinator.duty.KillUnusedSegments;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
Expand Down Expand Up @@ -58,22 +58,22 @@ public class CoordinatorDynamicConfig
private final int balancerComputeThreads;
private final boolean emitBalancingStats;

/** If true {@link DruidCoordinatorUnusedSegmentsKiller} sends kill tasks for unused segments in all data sources. */
/** If true {@link KillUnusedSegments} sends kill tasks for unused segments in all data sources. */
private final boolean killUnusedSegmentsInAllDataSources;

/**
* List of specific data sources for which kill tasks are sent in {@link DruidCoordinatorUnusedSegmentsKiller}.
* List of specific data sources for which kill tasks are sent in {@link KillUnusedSegments}.
*/
private final Set<String> specificDataSourcesToKillUnusedSegmentsIn;
private final Set<String> decommissioningNodes;
private final int decommissioningMaxPercentOfMaxSegmentsToMove;

/**
* Stale pending segments belonging to the data sources in this list are not killed by {@link
* DruidCoordinatorKillStalePendingSegments}. In other words, segments in these data sources are "protected".
* KillStalePendingSegments}. In other words, segments in these data sources are "protected".
*
* Pending segments are considered "stale" when their created_time is older than {@link
* DruidCoordinatorKillStalePendingSegments#KEEP_PENDING_SEGMENTS_OFFSET} from now.
* KillStalePendingSegments#KEEP_PENDING_SEGMENTS_OFFSET} from now.
*/
private final Set<String> dataSourcesToNotKillStalePendingSegmentsIn;
egor-ryashin marked this conversation as resolved.
Show resolved Hide resolved

Expand Down
Loading