[Polipo-users] Offline mode + HTTPS does not seem possible

[Pottinger, Hardy J.]

Hi, I'm trying to use Polipo as a way to facilitate offline access to code repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox development environment. Our build process uses Grunt and Bower to pull in various pieces of front-end "stuff" and many of these URLs are https URLs to github, for example:

https://github.com/Modernizr/Modernizr.git

Now, I can have Polipo running, and the network on, on my guest machine I can successfully run

wget https://github.com/Modernizr/Modernizr.git

Which, I would assume, is caching the content. I then disable my network connection on my host machine, put Polipo in offline mode, and then re-attempt a wget of Modernizer... but I get the following:

Connecting to 10.0.2.2:8123... connected.
Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to establish SSL connection

Running a tail -f on /var/log/polipo/polipo.log does not show me anything during this error.

Here is the only additions I've made to my /etc/polipo/config file:

disableIndexing = false
disableServersList = false
dnsUseGethostbyname = reluctantly
dnsQueryIPv6 = no
dnsQueryIPv6 = reluctantly
proxyOffline = yes

Any advice would be much appreciated, thanks!

On a positive note, despite not getting offline caching to work, I *have* noticed a considerable bump in compile performance. Down from 5 minutes to 2 minutes, with tests turned on. So, I'm moderately pleased about that. But, i would like to figure out this offline proxy stuff. :-)

--Hardy

[Juliusz Chroboczek]

> Connecting to 10.0.2.2:8123... connected. Proxy tunneling failed:
> Couldn't connect: Network is unreachableUnable to establish SSL connection

Polipo cannot cache encrypted (https) content.

-- Juliusz

[Donovan Baarda]

Unfortunately, the enthusiastic push for better internet security with
https is breaking cacheability. None of the people serving this content
care, because traffic is one of their metrics for success, and they can
cache right up to their boundary by terminating the ssl at their edge, so
they don't care if it's non-cachable beyond there. Companies like Google
"solve" the caching problem by just pushing their "edge" further and
further out, owning more and more of the internet infrustructure between
customers and their servers.

I've been considering trying to create an unholy chimera by duct-taping
polipo and https://mitmproxy.org/ together with Python to fix this.

I'm getting sick of the 5 different android devices in the house insisting
on downloading all the firmware and app updates 5x over https. This was
particularly bad when I was briefly connected using stupidly expensive 4G
and found the android devices had significantly contributed to exhausting
my 5GB pre-paid quota in less than 2 days.

Things like mitmproxy are pretty bad from a security perspective, but
unfortunately that's the only way to work around this. Unfortunately the
push for more security is working on breaking them too. Cert-pinning is the
"magic solution" to breaking on-the-fly cert generation used by mitmproxy,
and it's already starting to appear in chrome etc. Not sure how easy it's
going to be to work around.


On 10 July 2015 at 04:38, Pottinger, Hardy J. <***@missouri.edu>
wrote:

> Hi, I'm trying to use Polipo as a way to facilitate offline access to code
> repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox
> development environment. Our build process uses Grunt and Bower to pull in
> various pieces of front-end "stuff" and many of these URLs are https URLs
> to github, for example:
>
> https://github.com/Modernizr/Modernizr.git
>
> Now, I can have Polipo running, and the network on, on my guest machine I
> can successfully run
>
> wget https://github.com/Modernizr/Modernizr.git
>
> Which, I would assume, is caching the content. I then disable my network
> connection on my host machine, put Polipo in offline mode, and then
> re-attempt a wget of Modernizer... but I get the following:
>
> Connecting to 10.0.2.2:8123... connected.
> Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to
> establish SSL connection
>
> Running a tail -f on /var/log/polipo/polipo.log does not show me anything
> during this error.
>
> Here is the only additions I've made to my /etc/polipo/config file:
>
> disableIndexing = false
> disableServersList = false
> dnsUseGethostbyname = reluctantly
> dnsQueryIPv6 = no
> dnsQueryIPv6 = reluctantly
> proxyOffline = yes
>
> Any advice would be much appreciated, thanks!
>
> On a positive note, despite not getting offline caching to work, I *have*
> noticed a considerable bump in compile performance. Down from 5 minutes to
> 2 minutes, with tests turned on. So, I'm moderately pleased about that.
> But, i would like to figure out this offline proxy stuff. :-)
>
> --Hardy
>
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> Polipo-users mailing list
> Polipo-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/polipo-users
>
>


--
Donovan Baarda <***@minkirri.apana.org.au>

[Pottinger, Hardy J.]

Oh, great, I followed that link to mitmproxy.org... I'm sure I'm on some government list now. ;-)

OK, seriously, that sounds like a fun hack. I'm not much of a Pythonista, but I'd love an excuse to play with it a bit. And this sounds like fun. If you need a co-conspirator, let me know.

--Hardy

________________________________
From: ***@gmail.com [***@gmail.com] on behalf of Donovan Baarda [***@minkirri.apana.org.au]
Sent: Thursday, July 09, 2015 8:00 PM
To: Pottinger, Hardy J.
Cc: polipo-***@lists.sourceforge.net
Subject: Re: [Polipo-users] Offline mode + HTTPS does not seem possible

Unfortunately, the enthusiastic push for better internet security with https is breaking cacheability. None of the people serving this content care, because traffic is one of their metrics for success, and they can cache right up to their boundary by terminating the ssl at their edge, so they don't care if it's non-cachable beyond there. Companies like Google "solve" the caching problem by just pushing their "edge" further and further out, owning more and more of the internet infrustructure between customers and their servers.

I've been considering trying to create an unholy chimera by duct-taping polipo and https://mitmproxy.org/ together with Python to fix this.

I'm getting sick of the 5 different android devices in the house insisting on downloading all the firmware and app updates 5x over https. This was particularly bad when I was briefly connected using stupidly expensive 4G and found the android devices had significantly contributed to exhausting my 5GB pre-paid quota in less than 2 days.

Things like mitmproxy are pretty bad from a security perspective, but unfortunately that's the only way to work around this. Unfortunately the push for more security is working on breaking them too. Cert-pinning is the "magic solution" to breaking on-the-fly cert generation used by mitmproxy, and it's already starting to appear in chrome etc. Not sure how easy it's going to be to work around.


On 10 July 2015 at 04:38, Pottinger, Hardy J. <***@missouri.edu<mailto:***@missouri.edu>> wrote:
Hi, I'm trying to use Polipo as a way to facilitate offline access to code repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox development environment. Our build process uses Grunt and Bower to pull in various pieces of front-end "stuff" and many of these URLs are https URLs to github, for example:

https://github.com/Modernizr/Modernizr.git

Now, I can have Polipo running, and the network on, on my guest machine I can successfully run

wget https://github.com/Modernizr/Modernizr.git

Which, I would assume, is caching the content. I then disable my network connection on my host machine, put Polipo in offline mode, and then re-attempt a wget of Modernizer... but I get the following:

Connecting to 10.0.2.2:8123... connected.
Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to establish SSL connection

Running a tail -f on /var/log/polipo/polipo.log does not show me anything during this error.

Here is the only additions I've made to my /etc/polipo/config file:

disableIndexing = false
disableServersList = false
dnsUseGethostbyname = reluctantly
dnsQueryIPv6 = no
dnsQueryIPv6 = reluctantly
proxyOffline = yes

Any advice would be much appreciated, thanks!

On a positive note, despite not getting offline caching to work, I *have* noticed a considerable bump in compile performance. Down from 5 minutes to 2 minutes, with tests turned on. So, I'm moderately pleased about that. But, i would like to figure out this offline proxy stuff. :-)

--Hardy

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Polipo-users mailing list
Polipo-***@lists.sourceforge.net<mailto:Polipo-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/polipo-users




--
Donovan Baarda <***@minkirri.apana.org.au<mailto:***@minkirri.apana.org.au>>

[Pottinger, Hardy J.]

It occurred to me that there might already be a Python-based caching proxy, and there is (kinda/sorta):

https://pypicache.readthedocs.org/en/latest/

It's specific to PyPi, but nothing we couldn't hacksaw out of the way.

--Hardy

________________________________
From: Pottinger, Hardy J.
Sent: Friday, July 10, 2015 11:05 AM
To: Donovan Baarda
Cc: polipo-***@lists.sourceforge.net
Subject: Re: [Polipo-users] Offline mode + HTTPS does not seem possible

Oh, great, I followed that link to mitmproxy.org... I'm sure I'm on some government list now. ;-)

OK, seriously, that sounds like a fun hack. I'm not much of a Pythonista, but I'd love an excuse to play with it a bit. And this sounds like fun. If you need a co-conspirator, let me know.

--Hardy

________________________________
From: ***@gmail.com [***@gmail.com] on behalf of Donovan Baarda [***@minkirri.apana.org.au]
Sent: Thursday, July 09, 2015 8:00 PM
To: Pottinger, Hardy J.
Cc: polipo-***@lists.sourceforge.net
Subject: Re: [Polipo-users] Offline mode + HTTPS does not seem possible

Unfortunately, the enthusiastic push for better internet security with https is breaking cacheability. None of the people serving this content care, because traffic is one of their metrics for success, and they can cache right up to their boundary by terminating the ssl at their edge, so they don't care if it's non-cachable beyond there. Companies like Google "solve" the caching problem by just pushing their "edge" further and further out, owning more and more of the internet infrustructure between customers and their servers.

I've been considering trying to create an unholy chimera by duct-taping polipo and https://mitmproxy.org/ together with Python to fix this.

I'm getting sick of the 5 different android devices in the house insisting on downloading all the firmware and app updates 5x over https. This was particularly bad when I was briefly connected using stupidly expensive 4G and found the android devices had significantly contributed to exhausting my 5GB pre-paid quota in less than 2 days.

Things like mitmproxy are pretty bad from a security perspective, but unfortunately that's the only way to work around this. Unfortunately the push for more security is working on breaking them too. Cert-pinning is the "magic solution" to breaking on-the-fly cert generation used by mitmproxy, and it's already starting to appear in chrome etc. Not sure how easy it's going to be to work around.


On 10 July 2015 at 04:38, Pottinger, Hardy J. <***@missouri.edu<mailto:***@missouri.edu>> wrote:
Hi, I'm trying to use Polipo as a way to facilitate offline access to code repositories (Maven central, github, etc) for my Vagrant-managed Virtualbox development environment. Our build process uses Grunt and Bower to pull in various pieces of front-end "stuff" and many of these URLs are https URLs to github, for example:

https://github.com/Modernizr/Modernizr.git

Now, I can have Polipo running, and the network on, on my guest machine I can successfully run

wget https://github.com/Modernizr/Modernizr.git

Which, I would assume, is caching the content. I then disable my network connection on my host machine, put Polipo in offline mode, and then re-attempt a wget of Modernizer... but I get the following:

Connecting to 10.0.2.2:8123... connected.
Proxy tunneling failed: Couldn't connect: Network is unreachableUnable to establish SSL connection

Running a tail -f on /var/log/polipo/polipo.log does not show me anything during this error.

Here is the only additions I've made to my /etc/polipo/config file:

disableIndexing = false
disableServersList = false
dnsUseGethostbyname = reluctantly
dnsQueryIPv6 = no
dnsQueryIPv6 = reluctantly
proxyOffline = yes

Any advice would be much appreciated, thanks!

On a positive note, despite not getting offline caching to work, I *have* noticed a considerable bump in compile performance. Down from 5 minutes to 2 minutes, with tests turned on. So, I'm moderately pleased about that. But, i would like to figure out this offline proxy stuff. :-)

--Hardy

------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Polipo-users mailing list
Polipo-***@lists.sourceforge.net<mailto:Polipo-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/polipo-users




--
Donovan Baarda <***@minkirri.apana.org.au<mailto:***@minkirri.apana.org.au>>