Opened 11 years ago

Closed 10 years ago

#23 closed enhancement (fixed)

crawler: prioritize authoritative servers for zones

Reported by: wander Owned by:
Priority: important Component: crawler
Keywords: Cc:

Description

Some auth servers for a zone throw timeouts or have other issues, e.g. hn. had a lame delegation in NS set (only from child, not root) for nicmx-anycast.rds.org.hn. (200.33.111.1) which was returning REFUSED.

We should skip servers which respond unhealthy.

Also: prefer low latency servers over others?

Change History (2)

comment:1 Changed 11 years ago by wander

Related issue: parameter set does not change on all servers at the same time. Try to find newest paramset via SOA and then skip servers handing out obsolete paramsets?

Example:

opening file ../../domaindata/domain_xn--mgbx4cd0ab.db
?Lucky hit: xn--mgbx4cd0ab.
?Lucky hit: xn--mgbx4cd0ab.
Assuming closest encloser xn--mgbx4cd0ab. -> 5RI9C0D5IMPC1OGKGR88LQNNNRK59LDS
Assuming wildcard *.xn--mgbx4cd0ab. -> 3COM8GJ78BEGD0HM8I27AMEF15390JS5
paramset: ParamSet[NSEC3, Algorithm=1, Flags=0, Iterations=10, Salt=eb9717
paramid: 2
Initializing Gaps for paramid 2
No ranges in database yet
Fresh run
?Old: ParamSet[NSEC3, Algorithm=1, Flags=0, Iterations=10, Salt=eb9717
New: ParamSet[NSEC3, Algorithm=1, Flags=1, Iterations=10, Salt=a1699e

comment:2 Changed 10 years ago by wander

Resolution: fixed
Status: newclosed

(In [998]) uses internal "badness" measure for each server to skip faulty authoritative servers

  • iterating servers is still sequential (not considering latency), but bad servers will be skipped for some rounds
  • badness values timeout=100, servfail/refused=1000, i.e. skip server for 100 rounds after timeout

closes #23

Note: See TracTickets for help on using tickets.