Announcement

Collapse
No announcement yet.

website outage

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    The outage was reminiscent of the social distancing during the pandemic.

    Comment


    • #17
      Another test post...

      EDIT: And the error is gone so it was related to my disabling of Tapatalk. Tapatalk is currently working for the moment but it's gonna get disabled again shortly so don't get your hopes up.

      Comment


      • #18
        Thanks for your work on this BFM and JL. Really appreciated.

        Comment


        • #19
          Originally posted by BigFatMeanie View Post
          Another test post...

          EDIT: And the error is gone so it was related to my disabling of Tapatalk. Tapatalk is currently working for the moment but it's gonna get disabled again shortly so don't get your hopes up.
          Awesome! Thanks for all your work in getting us back up BFM!

          Comment


          • #20
            Here is some additional context/post-mortem about what has been going on (sorry, after 25 years in the corporate world it's ingrained in me. I reflexively do a post-mortem review):

            Sometime late Sunday night or early Monday morning, we started seeing lots of errors with the site. The error messages were based on the VBulletin code trying to load modified files/URL names that didn't exist. The modified filenames in many cases had the string "pharma" in them, which is indicative of a hack where they replace site content with links to scam sites - viagra and stuff. If I had nothing better to do I could go do a bunch of forensic analysis on the site and DB and see if i could pinpoint exactly when it happened, but... I don't have the time.

            VBulletin generates all of it' content and URLs based on information in the database. The fact that the errors had the "pharma" strings in them but there was nothing wrong with the files on disk indicates that it was the DB that was hacked, not necessarily the actual filesystem on the host server. The most common type of DB attack is a SQL injection. Tapatalk is notorious for being a vector for SQL injections. The Dude had the hosting company do a restore with the DB on Monday morning which mostly got us up and running, although there were still some errors. Again, indicative that we had suffered a DB attack.

            On Monday evening the hosting company suggested we repair and optimize some specific tables in the DB to clear away the final few errors, which I did. I also started a process to scan the filesystem for log files and other things of interest that might tell us more about what went on. The filesystem scan took a long time and the site was running slower than cold tar so I terminated it after about 20 minutes. Less than two hours after I repaired the DB and terminated my filesystem scan, the hosting company completely locked out the site and sent The Dude an email saying that we were consuming too many resources with DB deadlocks and it was because we had been hacked again. They provided no actual evidence of us consuming too many resources, and no actual evidence of a second hack. It's possible that we were still seeing side-effects from the original attack but the hosting company wouldn't give us access to the raw SQL logs because it's a shared server and wouldn't provide any other info. Yet they demanded we take "corrective action" to fix our issues before they would re-enable the site. We spent all day yesterday going back and forth with them to try and understand what corrective actions they expected us to take and why they concluded that we were hacked and our site was the guilty party badly behaving on the shared host even after they performed the DB restore from the original potential hack.

            Generally, this hosting company has been pretty good to work with - performance has been good and the price is good and they've had good support, but I'm pretty annoyed with how they handled this most recent incident. On the one hand I'm glad they proactively shut off bad-acting sites on a shared webhost. It sucks when a bad tenant on the shared host impacts the innocent. On the other hand, it was very frustrating because we were guilty until proven innocent and they provided zero actual evidence that our site was the guilty culprit, although we weren't really in a very strong position to argue because we had obviously suffered from some type of issue the day before. In an attempt to show them that we took appropriate corrective action, I disabled Tapatalk by logging into the shared host and trashing the Tapatalk files directly on the server. It still took them another 24 hours before they finally enabled the site this afternoon.

            So, where did we end up? Can we conclusively prove that Tapatalk is the culprit? No. At least not without spending significantly more time than I can spend and probably not without access to raw server and DB logs, which the hosting company won't grant us. But Tapatalk is a known, even notorious, attack vector. It's also a known vector for bypassing the sign-up controls to create spam accounts. At the moment things appear to be stable but we're going to leave Tapatalk disabled until The Dude figures out whether he wants to continue running the Tapatalk plugin or not.

            The version of VBulletin we run actually works pretty well for mobile browsers. All the default templates are "responsive design" which means the site generally looks just fine on a mobile device. So in the short term we recommend those of you who are mobile-only use the browser on your mobile device to look at the site. What you won't get without Tapatalk are push notifications/alerts and things like image upload may be less convenient. Those things are definitely nice to have but the lack of them is a first-world problem that you'll have to live with for the moment. Sorry.
            Last edited by BigFatMeanie; 07-21-2021, 06:12 PM.

            Comment


            • #21
              Thanks, BFM.

              Comment


              • #22
                Ok, so no more Tapatalk… cool. There were a lot things that bugged me about the Tapatalk app anyway. Good work BFM!
                "If there is one thing I am, it's always right." -Ted Nugent.
                "I honestly believe saying someone is a smart lawyer is damning with faint praise. The smartest people become engineers and scientists." -SU.
                "Yet I still see wisdom in that which Uncle Ted posts." -creek.
                GIVE 'EM HELL, BRIGHAM!

                Comment


                • #23
                  Thanks for that excellent summary, BFM.

                  This has been a huge PITA. Hosting company was terribly slow in responding.

                  I am torn on Tapatalk. It is convenient in some ways, but in some ways I hate it. It have never worked consistently for me. It is often slow to load and many times when I see there is a new post, I go to the thread and Tapatalk won't show it - it gets out of synch somehow. They haven't done even an incremental update in ages, which is probably why it is vulnerable to attacks.

                  I have been using a mobile browser (safari on iOS) to browse the site since it came back online. I swear it behaves differently now that Tapatalk is turned off. Before I would get the full version of the site on my phone (not the mobile version) and I would get these annoying "Do you want to browse this forum in Tapatalk?" plugins. I swear, the Tapatalk plugin disabled the mobile vbulletin interface for me. So I suggest we try using the mobile version on browsers for now.

                  Thanks to BFM for the tech help.
                  "There is no creature more arrogant than a self-righteous libertarian on the web, am I right? Those folks are just intolerable."
                  "It's no secret that the great American pastime is no longer baseball. Now it's sanctimony." -- Guy Periwinkle, The Nix.
                  "Juilliardk N I ibuprofen Hyu I U unhurt u" - creekster

                  Comment


                  • #24
                    By the way, to ease the transition to the mobile app, you might want to add a shortcut to the site on your home screen - in place of the tapatalk icon. For iOS, this is how you do it:

                    https://ios.gadgethacks.com/how-to/t...bpage%20chosen.

                    Very easy to set up. Seems to work well.
                    "There is no creature more arrogant than a self-righteous libertarian on the web, am I right? Those folks are just intolerable."
                    "It's no secret that the great American pastime is no longer baseball. Now it's sanctimony." -- Guy Periwinkle, The Nix.
                    "Juilliardk N I ibuprofen Hyu I U unhurt u" - creekster

                    Comment


                    • #25
                      Interesting... all this time I thought BFM was a lawyer

                      Comment


                      • #26
                        Originally posted by Katy Lied View Post
                        Interesting... all this time I thought BFM was a lawyer
                        Super mean post!
                        "There is no creature more arrogant than a self-righteous libertarian on the web, am I right? Those folks are just intolerable."
                        "It's no secret that the great American pastime is no longer baseball. Now it's sanctimony." -- Guy Periwinkle, The Nix.
                        "Juilliardk N I ibuprofen Hyu I U unhurt u" - creekster

                        Comment


                        • #27
                          Originally posted by Jeff Lebowski View Post

                          Super mean post!


                          KL taking the gloves off.

                          Comment


                          • #28
                            Oof - that really hurts. You could have pulled the punch a bit by saying you thought I was in sales. But lawyer? That's beyond the pale.

                            Comment


                            • #29
                              I'm insulted for you, BFM.
                              “Every player dreams of being a Yankee, and if they don’t it’s because they never got the chance.” Aroldis Chapman

                              Comment

                              Working...
                              X