Are you using XFES/Elasticsearch and why or why not?

 

With all the faults I found in it so far, I can't believe that I lived without it for so long.

 

It helps tremendously in not getting duplicate threads when necessary (a new thread on the same topic just a week later, not the same topic discussed 1-2 years later, as the latter would be preferable with the time between topics where things may need to be discussed again).

The main issue is having hosting that actually offers ElasticSearch (or, hosting it separately on something like Bonsai). ES wants plenty of memory so you tend to be in decent VPS territory from the start.

 

If it were feasible, I’d bake that into the core - it’d do XF wonders to have it as standard but it immediately reduces the hosting choices considerably.

ES wants plenty of memory so you tend to be in decent VPS territory from the start.

I got it to run on 2GB RAM, which is low-tier VPS territory.

Given how many people are running on XF not on a VPS, that’s not low tier. In any case, ES eats RAM like Smarties, check how much RAM it’s already using - I remember having to fight with it on Docker just to not fall over when Docker wouldn’t immediately give it 4GB just to start up (that was a few years ago now)

Docker just to not fall over when Docker wouldn’t immediately give it 4GB

Funny that you mention that. I tried my hardest to get it running in Docker, and wanted the latest version (8.x). And, like you said, it just would never start.

 

However, it was hard to configure because it came with security settings and wanted to run over HTTPS on a local connection, which didn't make much sense (as I blocked incoming 9200/9300 ports anyway, so it's communicating locally).

 

So, I did more research before giving up and found that Elasticsearch 7 doesn't require the certificates to be registered. Though, for security, I did update the username "elastic"'s password to something more hardened, just in case.

 

I am running Elasticsearch 7.10.2 in Docker and it's taking half the RAM.

 

[ATTACH type=full" alt="1730987096822.webp]378[/ATTACH]

 

The server instance isn't that busy, so I didn't want to upgrade to 4 GB RAM until necessary. It has 0 effect on responsiveness with the forums running. It's constantly at 60-70% usage, but no problems so far.

 

[ATTACH type=full" alt="1730987197518.webp]379[/ATTACH]

 

A 1 vCPU, 2 GB RAM, 55 GB, 2 TB bandwidth with Plesk Admin ($12.95 after 1st month) and auto-backups ($2/mo) comes to a total of $24.95/mo beginning the 2nd month (1st month is $12).

 

You can save $12.95 by choosing a free control panel like CyberPanel FastPanel, or Webmin and $2 if you remember to manually backup or set up automated backup FTP transfers to your computer.

 

So, you could hypothetically pay as little as $10 for an instance that would run Elasticsearch just fine with multiple forums on it. I, on the other hand, would choose to use AMD High Performance, which is 1 vCPU, 2 GB RAM, 50 GB NVMe (faster file read speeds), and 3 TB bandwidth for a base price of $12 (add $2 backups and $12.95 if you choose Plesk).

 

Compare that to GoDaddy 1 vCPU, 2 GB RAM, 40 GB NVMe with cPanel or Plesk for $36.98/mo (although you get 1 additional IP which is like $2 or $4 at Vultr) and 7 snapshots for daily backups for a week (which would be like $5 at Vultr).

that’s not low tier

I would consider my proposition low-tier and cost-saving compared to one of the largest hosts. You might get something similar to GoDaddy (with a reseller or lesser-known/lower-trust service) for $20-25, but you'd still be essentially paying double.

 

If you want to try out Vultr for a month, I believe it's free if you install the lowest tier (which is only 1 GB RAM, so you couldn't test out Elasticsearch), but it could get you a feel for Plesk or the other panels if you're a cPanel user for free, feel free to use my promo code

 

https://www.vultr.com/?ref=9643485-9J

 

If you feel Vultr is right for you after a month and make a $100 deposit, you'll get a $300 credit to your account (I'll get $100*). With $400 deposited, and the bare bones of what I suggested, you could run "free" for 60 days (as the $300 credit expires if you don't use it, but you could put it to use by trying out a lot of different instance configurations for what is right for you with it) and then your $100 deposited would be about a year of hosting paid. *If you just pay $10 to test it out, I'll get $10.

 

* $10 for $10 promo link:

https://www.vultr.com/?ref=7990634

 

Vultr also has a nice built in firewall that would've prevented the recent Cyberpanel attacks if configured, and, free DNS management that is awesome and updates quickly once changes are made (like within seconds if you need to enter a TXT record for Amazon or Google verification). In addition to that, they have Object Storage that is S3 compatible, so you can use that with Xenforo's native support with 1 TB starting at $5.

 

That said, try the Elasticsearch 7.10.2 image on your current VPS. It should boot right up without issue and you shouldn't notice any performance issues whatsoever if you have less than 4GB (2GB is required as I did test it to try and limit it to 1GB and 1.5GB).

 

Just to add, if you didn't know, Xenforo Cloud uses Vultr. So, it's not a service I'd turn my head to vs. Linode or Amazon S3.

Edited November 7, 2024Nov 7 by frm

I’m perfectly fine organising a VPS myself (I was a happy Linode customer for nearly 17 years until the Akamai acquisition started screwing things up) but do not underestimate how many people are running XF on not a VPS. (Heck, it wasn’t that long ago I was running a site that needed a 16GB instance just for its ES, but that wasn’t a forum. I’ve spent time running sites on AWS where the hosting bill is in 5 figures per month.)

 

If more people were on a VPS the requirements wouldn’t need to be as low as they are for PHP versions because it’d be easier for people to be more current - but the surprising majority just aren’t on a VPS.

If more people were on a VPS the requirements wouldn’t need to be as low as they are for PHP versions because it’d be easier for people to be more current - but the surprising majority just aren’t on a VPS.

I can't argue that.

 

I'm just merely pointing out that Elasticsearch 8 requires a much more expensive VPS as it requires 4 GB, or 8 GB total memory (you might be able to hack it down to 2 GB, but that'd still be a 4 GB total instance which is getting into the mid-tier zone). In contrast, you can get Elasticsearch 7 to boot in Docker on a 2 GB instance only using 1 GB and see no performance issues on a VPS running several small boards.

 

So, there isn't an excuse not to try out Vultr (or Linode or S3) and learn how to manage your instances. It's a lot easier (and cheaper than VPS providers) if you install a control panel like cPanel. Just need to learn how to snapshot instances for quick recovery and backup as you typically would from your cPanel installation on shared hosting (which, hopefully, people do at the minimum).

 

If you're going to manage a forum, you might as well get a little bit more IT education to manage it as efficiently and as cheaply as possible. With a control panel, you don't need CLI (command-line) to control things, but it would give you the opportunity to learn that and break away from using a control panel, if you so chose.

I don’t need an excuse myself, but I think you overestimate the average technical skill of a forum admin. I know a number of admins who would otherwise fail to install XF on regular hosting - and I don’t mean for technical reasons, but things like “how do I put the database configuration in” (or even “what’s a database”)

 

Let me put it another way: I work for a digital agency. We literally build websites as a day job. Of the 4 of us who actively get in and do the technical stuff, I’d credit only two of us with the skill to get an unmanaged VPS up in reasonable time. After all it’s not for nothing that services like Cloudways exist to make it easier to manage. Or Laravel Forge for that matter.

 

The number of people who want to get involved in that is much lower than we want to admit and the number of admins who couldn’t care less and just want it to work without having to get in and manage it is honestly higher than you’d imagine.

 

There is a reason why Jcink and ProBoards still get new sites and new users in spite of the service offering being, charitably saying, behind the times. It’s also one reason Invision Community’s cloud offering in particular has traction, but also services like Circle.

 

The great myth of Web 1.0 that Web 2,0 demonstrated is that as much as people want control over the data and the choice to customise it, they will very often trade it out for the convenience of not having to manage it - it’s why aside from the footfall that FB Groups remain popular - because it requires no cost and no technical management effort.

All valid points.

Invision Community

 

I just wanted to say, as my work on ForumDiscovery goes on and I try to find Invision boards, I'm finding it difficult to get boards that a more generalized audience would want. It seems like a majority of them are for companies to manage users/bugs/tickets, etc., but I haven't gotten access to the full list to go through yet ($250); that's just from Google scraping. Examples would be like Sony (not confirmed, just throwing a corporation out to emphasize its user base) to get Playstation or A* camera series feedback. You'd expect these corporations to have a capable IT team to run a board, yet they still pick convenience.

 

Oh, a PRIME example would be LinusTechTips - he can do anything, and many in his company can too, yet they run Invision.

SquareSpace, Fiverr and Guild Wars 2 all have Invision Community sites. These three entities are the three big examples I put out: they all absolutely have the in-house ability to build out such services themselves (and indeed ArenaNet did for the longest time), but they don’t, they outsource it.

 

I don’t spend much time on the first two but the third I have, and there’s not really a lot that’s unique about it. I imagine getting a modern system that they don’t have to put the effort into maintaining goes a long way. I don’t know if they rolled the SSO with the main ANet services or whether they got Invision to do it for them (since the bigger cloud plans get dev too), or even if it’s self hosted or not (hard to tell, and I don’t care enough to figure it out)

 

From the various vague hints Matt M has dropped over the last few years it is clear that their target is bigger corps who just want it primarily SaaS and not have to manage their own infrastructure because it’s cheaper to outsource that.

 

It reminds me of when I used to work for a company that did Moodle hosting - this is something you could do yourself for sure (and I used to work with unis that didn’t just host Moodle, they had devs on staff doing Moodle customisation) and they still turned it over to us to manage the hosting because it was cheaper and easier than them doing it.

 

For the hobbyist arena, IC doesn’t fly much any more owing to its price tag, which is very much in the prosumer tier if we’re being charitable, but the whole aim they sell it as is “we run the site so you can get on running your community”.

ES is great when it works. It may be my setup (although done correctly, as far as I know) my ES connection fails every undefined amount of time. Then I need to log in, restart the ES and then it works again. Annoying.

ES is great when it works. It may be my setup (although done correctly, as far as I know) my ES connection fails every undefined amount of time. Then I need to log in, restart the ES and then it works again. Annoying.

How much memory is it using? Are there any signs in the log that it's the system killing it due to memory consumption? (this is a common issue with ES, as above)

How much memory is it using? Are there any signs in the log that it's the system killing it due to memory consumption? (this is a common issue with ES, as above)

I've got 16gb memory, so I doubt it's due that. And last time I checked, there was nothing in the logs either. It's quite weird.

I've got 16gb memory, so I doubt it's due that. And last time I checked, there was nothing in the logs either. It's quite weird.

This likely isn't the issue, but sometimes even the best of us can overlook a simple setting (like I did with not checking an auto-backup checkbox).

 

Are you running it in Docker and has your system ever rebooted that you know of, manual or automatic (what is uptime)? I don't know how it's set up outside of Plesk (the docker-compose.yml configuration), but there is this checkbox...

[ATTACH type=full" alt="1731005005061.webp]380[/ATTACH]

 

"Automatic start after system reboot"

 

Could be your instance rebooting from cluster maintenance to you doing a manual reboot to a system upgrade rebooting it automatically, but it happens so quickly you don't notice, thus knocking it offline if it's not set to start after a reboot.

Edited November 7, 2024Nov 7 by frm

This likely isn't the issue, but sometimes even the best of us can overlook a simple setting (like I did with not checking an auto-backup checkbox).

 

Are you running it in Docker and has your system ever rebooted that you know of, manual or automatic (what is uptime)? I don't know how it's set up outside of Plesk (the docker-compose.yml configuration), but there is this checkbox...

[ATTACH type=full" alt="1731005005061.webp]380[/ATTACH]

 

"Automatic start after system reboot"

 

Could be your instance rebooting from cluster maintenance to you doing a manual reboot to a system upgrade rebooting it automatically, but it happens so quickly you don't notice, thus knocking it offline if it's not set to start after a reboot.

My system has no reboot that I know of.

 

A simple restart usually does the trick though. I just can't figure out why it stops working in the first place. There's no consistency either. Sometimes it takes days, sometimes weeks.

My system has no reboot that I know of.

 

A simple restart usually does the trick though. I just can't figure out why it stops working in the first place. There's no consistency either. Sometimes it takes days, sometimes weeks.

What version of ES are you using and in Docker or a standalone installation?

 

I'd try pulling the Docker 7.10.2 version. Easiest configuration and have had no problems.

 

Quite odd to fail on 16GB RAM multiple times while ES7 given only 1GB has more uptime.

What version of ES are you using and in Docker or a standalone installation?

 

I'd try pulling the Docker 7.10.2 version. Easiest configuration and have had no problems.

 

Quite odd to fail on 16GB RAM multiple times while ES7 given only 1GB has more uptime.

Standalone, version 10-something. The latest in any case.

 

And in any case, my VPS is a bit over spec'ed for what I need, but I figured, the more space/memory you have, the better.

And in any case, my VPS is a bit over spec'ed for what I need, but I figured, the more space/memory you have, the better.

Ah, the "my truck is bigger than your truck" way of thinking. 😅

 

I like to use the most minimal settings possible and then scale up when needed. It's quite easy to monitor what is needed and when and then scale up to support that.

 

Though, I would love to learn more about AWS and Cloud in general to not scale up permanently, until absolutely needed, and instead launch a new instance to load balance instead, destroying it when it's no longer needed. I haven't gotten around to testing that, nor have enough traffic to try as 2 vCPU/2 GB has no problem struggling with what I have, and have only ever gotten to the point of requiring 4 vCPU/8 GB RAM (but also used that instance for personal use as a VPN amongst other things, which I no longer do as I'm incorporating finally and that makes it kind of messy with finances). I would probably do it on 1 vCPU/1 GB RAM to test and learn at first before requiring true minimum spec requirements to load balance with.

Ah, the "my truck is bigger than your truck" way of thinking. 😅

 

I like to use the most minimal settings possible and then scale up when needed. It's quite easy to monitor what is needed and when and then scale up to support that.

 

Though, I would love to learn more about AWS and Cloud in general to not scale up permanently, until absolutely needed, and instead launch a new instance to load balance instead, destroying it when it's no longer needed. I haven't gotten around to testing that, nor have enough traffic to try as 2 vCPU/2 GB has no problem struggling with what I have, and have only ever gotten to the point of requiring 4 vCPU/8 GB RAM. I would probably do it on 1 vCPU/1 GB RAM to test and learn at first before requiring true minimum spec requirements to load balance with.

Yes. Sadly that is a bit true. I’m also desperately afraid of running out of resources one day. Waking up, to see my server being offline or overwhelmed. I know that isn’t likely to happen overnight (unless a technical failure) but I’m honestly not fussed about it being over-spec’ed. It’s costing me a little over 40USD a month, so.. 😅

Waking up, to see my server being offline or overwhelmed.

That's a worry of mine too, as to why I want to learn load balancing.

 

I think it's even possible to see where you're getting the most traffic (like US: East or West coast, Canada, UK, or Aussie, etc.) and load balancing judges based on that to pop an instance up in that location so that the processing speeds aren't only better, but response times too. Someone with more knowledge would have to correct me on that, but if that's true, that is one of the most awesome aspects I find with it.

That's a worry of mine too, as to why I want to learn load balancing.

 

I think it's even possible to see where you're getting the most traffic (like US: East or West coast, Canada, UK, or Aussie, etc.) and load balancing judges based on that to pop an instance up in that location so that the processing speeds aren't only better, but response times too. Someone with more knowledge would have to correct me on that, but if that's true, that is one of the most awesome aspects I find with it.

And for the same reason, I wouldn't like to find out the limits. Instead, I'm much more comfortable with a server that I know can take it. I do monitor it sometimes, but I don't need to worry about it this way and wonder when we'll hit our limits.

And for the same reason, I wouldn't like to find out the limits. Instead, I'm much more comfortable with a server that I know can take it. I do monitor it sometimes, but I don't need to worry about it this way and wonder when we'll hit our limits.

Probably for the best then as even little things like that can negatively impact your mental health and then have an effect on your overall life. So, it's probably worth it, and understandable, for you to pay the extra to relieve that worry.

 

I don't have a concern over money, but I didn't get to where I am by overpaying or other unscrupulous spending. I'd rather buy an extra share of a high dividend ETF.

 

Forums are also secondary/thirdary(?) sources to me too, so that also factors into my decision to do it the way I do.

Probably for the best then as even little things like that can negatively impact your mental health and then have an effect on your overall life. So, it's probably worth it, and understandable, for you to pay the extra to relieve that worry.

 

I don't have a concern over money, but I didn't get to where I am by overpaying or other unscrupulous spending. I'd rather buy an extra share of a high dividend ETF.

 

Forums are also secondary/thirdary(?) sources to me too, so that also factors into my decision to do it the way I do.

Aye, when something technical happens, I do tend to stress a bit about it. There's something shameful for me personally to see my server offline. It shows an incapability. I know that's a bit exaggerated, but you know what I mean. I'm always relieved when it's all fixed and working perfectly. Luckily this doesn't happen too often, and I know there's always some people willing to help when I ask.

There's something shameful for me personally to see my server offline. It shows an incapability. I know that's a bit exaggerated, but you know what I mean.

No. It's completely understandable, considering you run an admin forum.

 

I run general and niche topics with low user bases that if it went offline, I wouldn’t care as much till I could get it back online for the good Ole apology/welcome back email.

No. It's completely understandable, considering you run an admin forum.

 

I run general and niche topics with low user bases that if it went offline, I wouldn’t care as much till I could get it back online for the good Ole apology/welcome back email.

And for 99% people that is sufficient too. Anyone can fuck up sometimes. And often, it's not even by our doings.