[MacPorts] #57720: preferred_hosts has no effect on archive_site_local

[MacPorts]

#57720: preferred_hosts has no effect on archive_site_local
------------------------+----------------------------
Reporter: ryandesign | Owner: (none)
Type: defect | Status: new
Priority: Normal | Milestone: MacPorts 2.6.0
Component: base | Version: 2.5.99
Keywords: | Port:
------------------------+----------------------------
With `archive_site_local`'s basic functionality fixed (#57717), we now see
that `preferred_hosts` has no effect on `archive_site_local`. With
macports.conf containing:


{{{
archive_site_local https://packages-private.macports.org/:tbz2
preferred_hosts packages.macports.org
}}}

We see that, contrary to intentions, nevertheless packages-
private.macports.org was tried before packages.macports.org:

{{{
---> Fetching archive for libpng
DEBUG: Executing org.macports.archivefetch (libpng)
DEBUG: euid/egid changed to: 0/0
DEBUG: chowned /opt/local/var/macports/incoming to macports
DEBUG: euid/egid changed to: 502/501
---> libpng-1.6.36_0.darwin_15.x86_64.tbz2 doesn't seem to exist in
/opt/local/var/macports/incoming/verified
---> Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https
://packages-private.macports.org/libpng
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not
Found
---> Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https
://packages-private.macports.org/libpng
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not
Found
---> Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from
https://packages.macports.org/libpng
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not
Found
}}}

[MacPorts]

#57720: preferred_hosts has no effect on archive_site_local
-------------------------+----------------------------
Reporter: ryandesign | Owner: (none)
Type: defect | Status: new
Priority: Normal | Milestone: MacPorts 2.6.0
Component: base | Version: 2.5.99
Resolution: | Keywords:
Port: |
-------------------------+----------------------------

Comment (by ryandesign):

In [changeset:"531720bba7af24204b507f26bdefbc60e58556c0/macports-ports"
531720bba7af24204b507f26bdefbc60e58556c0/macports-ports] (master):
{{{
#!ConfigurableCommitTicketReference repository="macports-ports"
revision="531720bba7af24204b507f26bdefbc60e58556c0"
Travis: Prefer public archives over private ones

I had tried to do this with preferred_hosts before, but it appears to
have no effect on archive_site_local.

See: https://trac.macports.org/ticket/57720

Instead list both the public and the private URLs in archive_site_local.
They appear to be tried in the order listed.

But then we need to fix the base bug with multiple URLs in
archive_site_local:

See: https://trac.macports.org/ticket/57718

See: https://trac.macports.org/ticket/54800
See: https://github.com/macports/macports-ports/pull/3099
}}}

[MacPorts]

#57720: preferred_hosts has no effect on archive_site_local
-------------------------+----------------------------
Reporter: ryandesign | Owner: (none)
Type: defect | Status: new
Priority: Normal | Milestone: MacPorts 2.6.0
Component: base | Version: 2.5.99
Resolution: | Keywords:
Port: |
-------------------------+----------------------------

Comment (by jmroot):

This is working as intended. The *_local variables are for a single URL
that is, as the name implies, local and thus expected to have the best
performance.

[MacPorts]

#57720: preferred_hosts has no effect on archive_site_local
-------------------------+----------------------------
Reporter: ryandesign | Owner: (none)
Type: defect | Status: new
Priority: Normal | Milestone: MacPorts 2.6.0
Component: base | Version: 2.5.99
Resolution: | Keywords:
Port: |
-------------------------+----------------------------

Comment (by ryandesign):

I can understand that point of view. However, what we need for Travis CI
and our Buildbot setup is to be able to communicate to MacPorts that we
want one URL checked first, a different URL checked second, and no other
URLs checked. `archive_site_local` seemed like a way to do that, but it's
becoming frustrating when the behavior is not defined/documented anywhere
and seems to change at random when I move to a different computer for
example.

[MacPorts]

#57720: preferred_hosts has no effect on archive_site_local
-------------------------+----------------------------
Reporter: ryandesign | Owner: (none)
Type: defect | Status: new
Priority: Normal | Milestone: MacPorts 2.6.0
Component: base | Version: 2.5.99
Resolution: | Keywords:
Port: |
-------------------------+----------------------------

Comment (by ryandesign):

I originally thought that multiple URLs in `archive_site_local` were tried
in the order specified, based on some initial testing. But my test wasn't
realistic, because I used the same hostname (but a different path) for
each URL. In that case, they are indeed tried in order, as far as I can
tell.

It appears that these hosts are still pinged; times for these hosts are
recorded in the pingtimes file. If we're still pinging them and trying
them in ping order, it should be allowed to specify one or more that are
preferred, using `preferred_hosts`.

However I cannot determine whether we are actually trying them in ping
order, or in the order specified, or in random order. It seems to change
every time I test it.