Hi, I;ve just activated HTTPS on my Elkarte forum using Let's Encrypt. :) The problem is if a post contains images that hosted on non-https sites there will be mixed content warning. I'm requesting a feature/addon for Elkarte to cache the non-secured image.
Some references:
http://www.simplemachines.org/community/index.php?topic=527996.0
http://www.vbulletin.org/forum/showthread.php?t=316826
https://www.phpbb.com/community/viewtopic.php?f=496&t=2271541
http://community.mybb.com/thread-162572.html
https://invisionpower.com/forums/topic/416988-image-proxy/
https://xenforo.com/community/threads/what-is-proxy-links-and-proxy-images.81020/
http://arstechnica.com/information-technology/2015/10/chrome-finally-kills-off-the-http-https-mixed-content-warning/
https://support.mozilla.org/en-US/kb/mixed-content-blocking-firefox
https://www.chromium.org/Home/chromium-security/marking-http-as-non-secure
Thanks! :)
I am not using https and I never test it, but this should work: http://addons.elkarte.net/enhancement/Image-Cache.html
thanks
@radu81 my search skill sucks. haha. :D
i will try it later.
I knew I wrote something!! xD
I wasn't able to find it as well... LOL
Time for a bump!
I think we finally need to pull this into the 1.1 core (and possibly for 1.0.10) as the Chrome mandate is just about here. As I understand it Chrome will begin to flag screens with password inputs, on non-https servers, as insecure.
Thats not really an ElkArte issue but a server / site issue. However once someone enables https, they will need the image proxy to still allow links to http sites from your https site, something ElkArte should support out of the box.
Agreed.
Quick&dirty we can simply package the addon. O:-)
I wish I were able to finish what I had in mind in terms of addons... actually... I may have already finished, but I don't remember.
This evening I'll give it a try, if I did what I wanted to do, pack the "installed" addon is the easiest thing.
If you get that in a branch somewhere that would be cool. I'm looking to beef up the proxy a bit so have been working on that.
Here is a first pass at updating this for 1.1 ....
I changed a few things in the proxy and the proxy-ed "link" which now both use a hash code, this should prevent someone from adding image and then being able to use the proxy from another site. Some restructure etc.
Found a kind of a bug in the Elk code due to an update in the way Imagic deals with a parameter.
Some things to consider ....
- Adding a scheduled task to purge old image cache files.
- Using a library to prevent some potential abuse here. One could to link a massive image file, which the site would have to download, try to cache, etc. There are smarter getimagesize functions that we could use to help prevent this.
- Adding a new db table to track last access (to help with scheduled task vs touching) but also to track success in downloading, a protected image may not download, and we don't want to continually try to get it each page load.
One other thing, where to put the settings (basically on /off) ... core feature page, attachment settings page, other suggestions ?
Core feature page should be fine, attachment settings I'd say no since the images are embeded and not attachments. As a suggestion I could say ACP->Forum-> Posts and topics since is related to messages and images embeded with IMG tag.
My doubts regarding the proxy image:
1. how will deal with external avatars?
2. Gravatar is using an ssl connection? On elkarte.net existing gravatar images are not https
3. what about images inserted into signatures?
I'm planing to switch next year to https connection and I will use a free certificate from let's encrypt
Right now the proxy only works with images wrapped in [img]...[/img] tags ... so in theory I think it would work fine with signatures. Gravatar using http is frankly a bug that should be fixed, no reason not to use https. External avatars have not looked at, have to leave something for eman to do ;)
I've added the basics for retry on fail for images that don't fetch properly. It will try 10 times to get the image on a ^4 based timer curve. Nothing fancy but basically waits longer each time it fails until it trys again. On failure it just shows the default mime thumbnail.
To do the above I added a new table to track the image / time / failures to the db, so 1.1 will need a new table for this. This same table will be used for prune operations so "old" images (ones that have not been accessed in X time) can be removed from the cache. Thats the next bit to do.
I don't think https image shown in http site is a problem, however, the vice versa is i.e. http image in https site.
Then, I'd say let's do the same we did with the likes stats: 1.1.1?
I thought like stats are already in 1.1.0, did I miss something O:-)
I'm good with waiting until 1.1.1 for this proxy (since I know how to incorporate this O:-) ) although it should probably be addon-ized for 1.1.0 so it can be used.
I was thinking about the adding of the new table, unless... well, we can create it when the feature is enabled. True.
Pretty much finished for the time being.
I added two maintenance tasks, one to empty the image cache, and a scheduled task to remove "old" image files. Old being ones that have not been accessed in the number of days defined in the admin panel.
I also did this as a module, so its in the core features. Enable it will add the admin setup menu for it under the configuration menu.
Also added the option to use this for all BBC IMG tags, not just those that would trigger a mixed content warning.
This will not work on 1.0 as is. I'll do my best not to delete the branch (I'm a tidy freak with a poor memory !) while we wait for 1.1.1 consideration.
can this be disabled? as in i want to the image saved indefinitely.
is there a screenshot how it looks like? will be there a link or description to the image source? while it may be under fair usage but credit to source is nice too if we gonna rehost it.
thanks.
Just set the keep days to 0 which means forever or alternatively you could disable the schedule task.
Below each cached image is a footer line which links to the original image.
Sorry have not taken any screen shots of the updates.
Not with how it is currently implemented, but thats not different from any other IMG tag that points to a remote site.
Sounds better than a future of seeing butthole-shaped chocolates !
/me likes chocolate!
... not necessarily the color, more the substance. xD
This is now part of the 1.1 core? I used this alternative approach for a FreshRSS extension: https://github.com/FreshRSS/Extensions/tree/master/xExtension-ImageProxy It relegates the actual task of proxying to elsewhere.
Its not part of the core, that was just a suggestion / thought on my part while I was making some minor updates to the proxy emanuele wrote. Nice to see there is an API proxy ( https://images.weserv.nl ) that could be used.
Made a package out of it .... https://github.com/Spuds/Elk_Image_Cache ... thats for ElkArte 1.1 only
any plans to release it for 1.0.x?
I'm wanting to do just that, just need to find some time.
First pass at this one for 1.0 ... not tested very much, so you know the drill at this point.
I cannot find the settings in the addon for 1.0 version O:-)
They should be under the main configuration tab in the ACP ... assuming the package works O:-)
I don't see it :) I also checked /index.php?action=admin;area=manage_imagecache put redirects me to main page of ACP
Ah ... I probably attached the wrong file !
I'm working on an update (that will help with performance in some conditions) so will repost it soon.
Lets try this one.
I made a few changes which I need to also put in the version for 1.1
- Added a max size value, so it will not attempt to cache images over X MB in size. This will not always work, but when it does its nice.
- Moved the fetch of the image out of parsebbc and moved it to when its requested from the server. This prevents the page from blocking
- The filename was being leaked in the headers, so I fixed that
- Disable browser cache if the fetch of the image failed. This way if at a later time it does get the image it will show in the browser, and at that point proper cache headers are also set.
The addon installs without errors, but I still don't see the settings for this addon
LOL .. well I removed it again until I can figure out whats going on ... It works fine on my local but I must have set something manual like for that to have worked or its some php version issue error. Anything in your site or server error log?
I use php 5.6, in elkarte error log nothing that could be related to this addon. PHP error log is empty, apache error log displays some errors like
[Thu Mar 02 06:26:26.030611 2017] [proxy_fcgi:error] [pid 4387:tid 140574824826624] [client 66.249.64.252:61046] AH01071: Got error 'Primary script unknownn'
tried also with php 5.4 but no errors
Once more we try :-X
This works for me, thank you Spuds! 8) 8)
htpps is always green now in Chrome, here an example with cached images sharkracingclub.it/index.php?topic=3065
but here I get an error, reply #32
sharkracingclub.it/index.php?topic=333.msg4491#msg4491
I also have some errors in elkarte log
Ah progress! ... I'll take a look at those errors in the AM, well my AM :D Looking quickly at them they look like ones I should hide anyway.
The addon attempts to get the size of the remote file before it attempts to cache it. It does this by requesting only the headers from the remote server and then looks for a filesize header which may or may not be present.
Some servers do not allow this, some servers don't set the header anyway. The check the size is really just a goodie that I believe can be improved, need to think about it some. But first step will be to prevent filling the error log up !!
OK before I run off for a bit ... replace the ImageCache.class.php file in your subs directory with this one .. it should stop those errors.
Just one question, this addon should create a folder to store cached images, am I right? Which is the path to the cached mages? I cannot find it on my forum
It saves them in your standard cache directory ... they will be all start with img_cache_XXXXXXXXX If you want to clear them use the function under routine maintenance -> clear image cache, don't delete them on your own, or if you do, run the task as well since the db needs to be updated.
I see its still tossing an error on that one page .. so I've attached a new one for test.
It is getting better at avery update, thanks Spuds!
I did not deleted the files manually, I saw about the maintenance task
This update solved the error on that page.
before this update images from the first post of this topic were not displayed, now they are ;)
https://www.sharkracingclub.it/index.php?topic=252.msg3342#msg3342
What about these images?
https://www.sharkracingclub.it/index.php?topic=333.msg4498#msg4498
They are available on the external host, but not cached on my forum.
I like the feature
Cache all [img]'s, not just ones needed for HTTPS sites. I can use it toghether with
Remove cached images that have not been accessed in: set to 0, so in the future I won't have missing files if I cache them forever. Is this a good idea?
Regarding the option enabled
Cache all [img]'s, not just ones needed for HTTPS sites. what do you think about the idea to skip images hosted on own domain? I have some topics where images are stored on my ftp and there is no reason to cache them. I already changed all links inside posts from http to https.
here an example
https://www.sharkracingclub.it/index.php?topic=959.0
Another thing I noticed is that cached images are displayed at 640px of width even if they are smaller.
this is a signature a user have on my forum
[url=http://postimg.org/image/6kl2kbzzl/][img width=180 height=100]http://s24.postimg.org/6kl2kbzzl/999682_10201258103565272_849895635_n.jpg
and here how it is displayed:
https://www.sharkracingclub.it/index.php?topic=2310.msg22525#msg22525
The image in the quote http://immagini.p2pforum.it/out.php/i306886_Immagine.JPG is a dead link on my end.
This one http://www.auto-motor.at/Auto/Autos-Neuwagen/Automarken-Automodelle-Neuigkeiten/Audi-News/Audi-A4-Cabrio/Audi-A4-Cabrio-Front_high.jpg is kind of interesting. It redirects 3 times before it finally lands on the image, but ElkArtes default behavior is to give up after 2 redirects . To fix that you would have to make an edit to Package.subs.php and change fetch_web_data to look like
function fetch_web_data($url, $post_data = '', $keep_alive = false, $redirection_level = 3)
so $redirection_level = 3 instead of the 2. Unfortunately the way the addon calls that function I can't change it on the fly. There is no harm in changing it permanently to 3 anyway, so just do that.
The last one which is http://www.motorstars.org/images/gif%20e%20jpeg/rent_ferrari_360modena_spider.jpg I'm going to guess that the system has it in the db, but did not find the file in the cache. I should add a check/reset for that. That image may show up if you run the clear image cache function (there is an error in the scheduled task function which could have caused this, see new attachment). If it does not then its something else.
That is what it would do. Do note that the images in the cache are held to be no larger than 640x480, mainly to save size and to make sure the saved image is just a basic copy and not the real deal. So the downside is really just file storage space and of course url/data requests to your site vs somewhere else. Be fun to do this but write the file to a CDN.
Yes thats a good idea and I should have done that from the start DOH!
I'm going to guess that you are using image magic. In version 3.0.0+ it changed the default behavior of bestfit which used to do nothing if the image was smaller, but now it instead scales it up. This has been accounted for in ElkArte 1.0.10 ;) so once thats out it should fix that issue for you.
Spuds we all make mistakes, my biggest mistake with forums was that in the first 3 years I disabled attachments in smf and used external hosts, so after years I am having lots of missing images. I was able to recover a good part of images from imageshack when they started to delete images, that's why I have images uploaded into a local ftp folder, p2pforum was closed years ago and one member used to upload all his images to that forum, those images are gone. Now I also removed all p2pforum bad links :(
this is a rare case, I uploaded that image as attachment, so I'll leave it as it is now. If I will find more images I will make the edit.
I also uploaded as attachment, but I can make other tests. I'll let you know
Yes, I use Imagemagick 3.4.3 since it's mandatory to rotate correctly attachments from smartphones. Glad to hear it will be solved on next version of Elkarte.
Thanks for detailed explanation and for your help!
Thanks :D
Here is the latest version for testing. I added for testing.
- Don't cache if its from the site already
- Fixed the scheduled task code so its 1.0.x compat
- Added a clear db entry function if the file is not found, but was a valid request
- The error log fixes from above as well.
I installed the latest version and the images from own domain are not cached anymore. Today I will install it on the other forum and see how is playing. No errors for now in elkarte log.
Thanks for all your testing on this :D
@Spuds do you have a version of this for 1.1.x?
I was going to look into creating one but it seems you've done most of the work, so I don't mind testing and correcting any issues I find.
The version in this repo https://github.com/Spuds/Elk_Image_Cache should be the version for 1.1 ... I don't think I've pulled in the changes for the fixes that were found for the 1.0 branch so there are probably a couple of bugs.
Thanks
@Spuds the v10 is working nicely on 1.0.x. How's the 1.1 branch doing these days? I might upgrade this weekend.
Cool, glad its working :) .
We are running the 1.1 version on this site (the latest in the above repo) so except for those pesky bugs it should be good to go.
The problem here is likely related to how the resizing images deal with the max width/height.
They are probably resizing the image to the max irrespective of the fact the image is smaller.
The problem of huge images was also present in the image proxy add-on that
@Spuds made for 1.0.x but was solved for 1.0.x
@radu81 Do tell! I actually encountered this problem in the 1.0.x add-on, so I'd love to have a solution without needing to take the time to do any coding myself. :)
I think that is the correct version of the addon for 1.0.x. I just did a test on my forum with the same image you used here and it works fine:
(Link-5206)
Do you have any special image resizing settings? I'm using 1.0.10 with, afaik, the default.
Edit: oh, atm I don't have ImageMagick enabled. That does tend to result in superior image processing, doesn't it? Is that automatic within Elk or how does that work?
Edit 2: ah yes, that is slightly better. I'll still have to dive into the code because I want working GIFs and I don't want that link but that can probably wait until upgrading to 1.1.
Also there should be some kind of "empty cache" button.
If IM works as expected, then my suspects are right.
I have ImageMagick enabled.
There is an option for that in Maintenance -> Routine -> Empty the image cache. This is referred to 1.0.x addon
@radu81 Thanks! I didn't realize there were more settings related to the add-on outside of "Image Cache Settings".
The problem is really with ElkArte.
The image cache calls the function resizeImageFile() function which in turn will call function resizeImage() but it does so with the $force_resize flag set to true (no way to override that either, at least from that path).
The Imagick logic prevents $force_resize from scaling an image up, however the GD functions will let it scale up. So that is what we are seeing here.
So the question becomes what is wrong ... In some respects the Imagick is not honoring the force_resize flag, so that really is a bug even though in this case its producing what we want. GD is doing whats its told, although its horrible. function resizeImageFile() should really allow us to pass the force flag value.
Anyway something needs to be changed, just not sure which one !
So my feeling was right. :P
resizeImageFile is used in the attachments (including thumbnails) and avatar resizing.
I think the original intent was to have the max_width/heigh as "stretch the images in order to fit the size" in both cases (at least that's what I get from the history).
The cleanest way would be to have another function that does what we need, but if we are sure we don't mind having avatars or thumbnails smaller than the set size, then I guess is it fine to just have resizeImageFile to scale down only if the size is bigger.
Fifteen, maybe even ten years ago some browsers used rather ugly (faster) resizing algorithms so that it would've probably made sense to do it on the server. At least in ImageMagick… less sure about GD. But seeing how you're basically just making the image larger and uglier I'd say only resize it down. And heck, even resizing down should perhaps be more of a size in bytes kind of thing than size in pixels.
An interesting trick to potentially obtain higher quality JPEG images at lower filesizes is to make them a higher resolution with lower quality compression.
To quote myself from elsewhere a few years ago:
Yeah, I think the scale up is more a relic of when the template was composed of only nested tables and have all the images of the same size was the only way (or at least an easy way... or at least a consistent way across browsers LOL) to ensure some consistency of the whole thing.
I can't speak for the IE4/Netscape 4 days, but from IE5 and up I know for sure that setting a width on an image had consistent results as far as the layout goes. Except for the fact that it was freaking ugly in both IE and
Firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=98971).
Since the flag already existed in resizeImage, I added the capability to pass a "force" flag to the resizeImageFile function and then in turn to the resizeImage function. While I was at it made imagick respect the flag as well, so now GD and Imagick act the same to fix that inconsistency.
All these graphics functions need a facelift in 2.0, scale up could certainly be improved.
Anyway that fixes the reported image cache issue, at least on my local. In testing I also noticed that if you uploaded an avatar, the image cache was creating a proxy for that, so I fixed that on my local as well.
Just to note, the proxy does not "work" with animated gifs, it will result in single frame of an animated gif. That could be fixed using imagick not sure with GD. You would have to detect animated, resized each frame (if needed), coalesce and save. If you are an animated gif fan PR's are welcome ;)
@Spuds I'm not but I do have a vested interest in them. :P
Where should I look when I do that? The resizeImageFile function?
Just thinking out loud, I'd probably add a new function to the image cache class and drop the dependency on resizeImageFile Right now some of those graphic functions have an inflexible flow, at least resizeImageFile does.
So I'd add a custom handler to the fetch the image (using the good parts of resizeImageFile, and call it fetchImageFile) in the cache class Then you can control what resize function to call, a new resizeGifImage or the existing resizeImage. resizeGifImage could be added to graphic.subs so its available to other addons as well.
Have not looked to see if GD can handle animated gifs, if not its fine IMO to just have an imagick function.
Just some quick thoughts.
Or doNothingToImage(). :P (Do GIFs have security concerns?)
Do nothing, if sized right, can be an option.
That's what I mean, yes, which applies to the vast majority of images — GIF, PNG, and JPEG alike, but most notably GIF for obvious reasons. :P
Yup we are on the same page. To be cautious, if the image is below the size limit and we just want to save it, we can run the image through checkImageContents() It is possible for a gif to hide meta data that could get executed on a poorly configured server, and then if you could find the file.
I have to say that the checks in that function were just about the last thing I was thinking of when asking about security concerns. That must be one very poorly configured server. :P
Sadly yes :-X ... and those type of checks are found in other net software (Not just ElkArte and its crazy Uncle Buck)
(http://33sm.net16.net/smileys/dunno03.gif)
Quick test of proxy updates and 1.1.1 fixes (scale up issue)
The proxy still doesn't catch avatars used on board index.
I just noticed and not investigated, thought I'm not sure why because they should use the avatar function I fixed a while ago... I guess I didn't fix it entirely. xD
I put up a fix here.
Darn avatars !
We should remove them along with attachments! xD
I like it :D LOL
Remember to also remove messages 😜
This topic was difficult to find,
@Spuds what do you think about moving it to add-ons board and add it to addons.elkarte.net ?
I am using it with Elkarte 1.1.7 and PHP 7.4, I have some errors in elkarte log:
Type of error: General
Notice: Trying to access array offset on value of type bool
index.php?action=mentions;sa=fetch;api=json;lastsent=0&_=1619502095109
File: /sources/subs/ImageCache.integrate.php
Line: 174
173: // No need if its already on this site (like uploaded avatars)
174: if ($parseImg['host'] === $parseBoard['host'])
175: {
176: return false;
177: }
Looks like just the 1.0 version is on the addon site .... someone is lazy!
Thanks for the bug report, I'll have a look.
Moving this topic to addons
Try this php7.4 update
I tried, but the image is not displayed
here is what I see with the latest version:
(Link-7026) and with the previous version:
(Link-7028) I did the test with same image. Finding an http image is not an easy task :)
I post the same image here. (here works fine)
(http://www.regione.lazio.it/binary/rl_main/tbl_news/notizia_nuovi_servizi_digitali_certificato_vaccinalex.jpg)
Strange, that image works fine on my local.
- You can try toggling it off/on on the core features page
- Maybe you are using an older browser? I did update the headers that are sent for inline images but what I'm using is the norm these days.
- Perhaps it failed to download the image and for some reason its not displaying the default icon instead?
Any clues in the development console ? or a link to where I can take a quick look as well?
ETA: Oh FFS ... I have a comma where a period should be. I'll upload a new one in a bit
Give this one a try
This works, thanks! 8)
if I really have to be picky: if you open in a new tab the image you see on forum, the full path to image is displayed. You can see it on also on this forum. IIRC I reported something similar for attachments and was solved.
P.S. For the others: if you uninstall and install a new version of this add-on, don't forget to enable the add-on on ACP -> Configuration -> Core features. When I uninstalled and installed the new version took me some time to understand why the menu ACP -> Configuration -> Image cache settings was missing.
I have improved the right click and save behavior. It will now save as the original filename and not the long name. For example the image posted above currently is
_var_customers_webs_elkarte_community_cache_img_cache_597563c0eada13027d38a842a2addab3
and now it will be
notizia_nuovi_servizi_digitali_certificato_vaccinalex.jpg
That of course is the cached version not the original and prevents the cached name from showing.
As for opening the image in a new tab I see
https://www.elkarte.net/community/imagecache.php?image=http://www.regione.lazio.it/binary/rl_main/tbl_news/notizia_nuovi_servizi_digitali_certificato_vaccinalex.jpg&hash=597563c0eada13027d38a842a2addab3
I'm not seeing the cached name, was there something else you were doing?
Also If we want to go all JS on this, I can fetch the cached image, async, and set the src as a data url. Then when you open the image in a new tab all you would see is
data:image/jpeg;base64,/9j/4AAQSkZJ..... I have that on my local as well. That is nicer than seeing he imagecache.php stuff (Just called in the background via JS instead) All depends on what everyone would like.
I was referring to this:
(Link-7031) The original name would be fine for me, or whatever else, but not the full path.
The images embedded from http are a thing from the past, they are very rare nowadays.
What browser are you using? Is that with open image or open image in new tab ? I'm not seeing that on mine.
ETA: Oh I see that in Fx now
This should not reveal the cached name
Thanks Spuds! that works without showing the full path.
I am using Firefox 88.