Category Archives: Blog

Main blog category, I’m too lazy to manage multiple categories so this blog has only one category.

UniFi Cloud Key: MongoDB Out of Memory Issue

You may find the following issue if you run a Unifi setup:

tail -f /srv/unifi/logs/server.log

...
Wed Jun 27 21:52:34.250 [initandlisten] ERROR: mmap private failed with out of memory. You are using a 32-bit build and probably need to upgrade to 64

After googling for it you may find a Ubiquiti staff post a prune script on their forum.

But you may find that script can only be executed while the MongoDB is running. However no one mentioned how to solve it when you can’t start your MongoDB. Here’s the solution, actually you don’t even need to repair your database in this situation:

Make sure unifi service is stopped:

systemctl stop unifi

Download the prune script from Ubiquity support

wget https://ubnt.zendesk.com/hc/article_attachments/115024095828/mongo_prune_js.js

Start a new SSH session, run MongoDB without --journal, all others parameters are copied from the unifi service:

mongod --dbpath /usr/lib/unifi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --noprealloc --nohttpinterface --smallfiles --bind_ip 127.0.0.1

Run the prune script:

mongo --port 27117 < mongo_prune_js.js

You should get the similar output:

MongoDB shell version: 2.4.10
connecting to: 127.0.0.1:27117/test
[dryrun] pruning data older than 7 days (1541581969480)... 
switched to db ace
[dryrun] pruning 12404 entries (total 12404) from alarm... 
[dryrun] pruning 16036 entries (total 16127) from event... 
[dryrun] pruning 76 entries (total 77) from guest... 
[dryrun] pruning 24941 entries (total 25070) from rogue... 
[dryrun] pruning 365 entries (total 379) from user... 
[dryrun] pruning 0 entries (total 10) from voucher... 
switched to db ace_stat
[dryrun] pruning 0 entries (total 313) from stat_5minutes... 
[dryrun] pruning 21717 entries (total 22058) from stat_archive... 
[dryrun] pruning 715 entries (total 736) from stat_daily... 
[dryrun] pruning 3655 entries (total 5681) from stat_dpi... 
[dryrun] pruning 15583 entries (total 16050) from stat_hourly... 
[dryrun] pruning 372 entries (total 382) from stat_life... 
[dryrun] pruning 0 entries (total 0) from stat_minute... 
[dryrun] pruning 56 entries (total 56) from stat_monthly... 
bye

Then edit the prune script and rerun the prune script with dryrun=false:

MongoDB shell version: 2.4.10
connecting to: 127.0.0.1:27117/test
pruning data older than 7 days (1541582296632)... 
switched to db ace
pruning 12404 entries (total 12404) from alarm... 
pruning 16036 entries (total 16127) from event... 
pruning 76 entries (total 77) from guest... 
pruning 24941 entries (total 25070) from rogue... 
pruning 365 entries (total 379) from user... 
pruning 0 entries (total 10) from voucher... 
{ "ok" : 1 }
{ "ok" : 1 }
switched to db ace_stat
pruning 0 entries (total 313) from stat_5minutes... 
pruning 21717 entries (total 22058) from stat_archive... 
pruning 715 entries (total 736) from stat_daily... 
pruning 3655 entries (total 5681) from stat_dpi... 
pruning 15583 entries (total 16050) from stat_hourly... 
pruning 372 entries (total 382) from stat_life... 
pruning 0 entries (total 0) from stat_minute... 
pruning 56 entries (total 56) from stat_monthly... 
{ "ok" : 1 }
{ "ok" : 1 }
bye

Start the unifi service

systemctl start unifi

The root cause of this issue is that Cloud Key is currently running on ARMv7, a 32-bit based custom Debian system. so MongoDB cannot handle data larger than 2 GB. I haven’t tried the Cloud Key 2 and 2 Plus I hope they’re ARMv8 based. At the moment you can limit data retention as a workaround.

How to Solve Zammad `BackgroundJobSearchIndex` Errors

1339 failing background jobs.
Failed to run background job #1 ‘BackgroundJobSearchIndex’ 10 time(s) with 228 attempt(s).

Run zammad run rails c then:

items = SearchIndexBackend.search('preferences.notification_sound.enabled:*', 3000, 'User')
items.each {|item|
  next if !item[:id]
  user = User.find_by(id: item[:id])
  next if !user
  next if !user.preferences
  next if !user.preferences[:notification_sound]
  next if !user.preferences[:notification_sound][:enabled]
  if user.preferences[:notification_sound][:enabled] == 'true'
    user.preferences[:notification_sound][:enabled] = true
    user.save!
    next
  end
  next if user.preferences[:notification_sound][:enabled] != 'false'
  user.preferences[:notification_sound][:enabled] = false
  user.save!
  next
}

Delayed::Job.all.each {|job|
  Delayed::Worker.new.run(job)
} 

Further reading: Failing background jobs.;Failed to run background job #1 ‘BackgroundJobSearchIndex’ 10 time(s)

Adding Email Aliases to Office 365 Groups via PowerShell

This is a “hidden” feature that can only be added via PowerShell at the moment. And web admin portal won’t reflect the changes you made via PowerShell. That means you can only view alias for Office 365 Group from the following PowerShell cmdlet.

First log into your admin account with PowerShell

Set-UnifiedGroup –Identity marketing@example.com –EmailAddresses @{Add="hi@example.com"}

Check if it works

Get-UnifiedGroup –Identity marketing@example.com | FL EmailAddresses

EmailAddresses : {smtp:hi@example.com, SPO:SPO_fa5eb50c-147e-4715-b64b-76af8be79767@SPO_f0712c15-1102-49c4-945a-7eda01ce10ff, smtp:marketing@example.onmicrosoft.com, SMTP:marketing@example.com}

Connect to Exchange Online PowerShell

Connect to Exchange Online PowerShell – Microsoft Docs

Set-ExecutionPolicy RemoteSigned # Windows only
$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
Import-PSSession $Session -DisableNameChecking
Remove-PSSession $Session # logout

UniFi Security Gateway (USG) Decrypt Error (Inform Error, aka “Adopting” Loop) for v4.4.28 Firmware

This is a known issue (bug) with USG 4.4.28.* firmware confirmed on official beta forum, the error would be like:

Sep 26 22:38:55 main-router mcad:  ace_reporter.process_inform_response(): Failed to get the decrypted data from custom alert response#012
Sep 26 22:38:55 main-router mcad:  ace_reporter.reporter_fail(): Decrypt Error (http://192.168.1.10:8080/inform)
Sep 26 22:39:10 main-router mcad:  mcagent_data.data_decrypt(): header too small. size=0, should be=40
Sep 26 22:39:10 main-router mcad:  ace_reporter.process_inform_response(): Failed to get the decrypted data from custom alert response#012
Sep 26 22:39:10 main-router mcad:  ace_reporter.reporter_fail(): Decrypt Error (http://192.168.1.10:8080/inform)
Sep 26 22:54:31 main-router mcad:  mcagent_data.data_decrypt(): header too small. size=0, should be=40
Sep 26 22:54:31 main-router mcad:  ace_reporter.process_inform_response(): Failed to get the decrypted data from custom alert response#012

Solutions (try them in the following order):

  • Disable period speed test
  • Reboot the controller
  • Force provision the USG
  • Try official unifi-util patch
  • Reset USG and readopt it
  • Upgrade/downgrade to a previously working version

Mine got everything back to work after resetting the USG.

Some tips:

You can run info to get some basic info what’s going on with your USG:

user@main-router:~$ info

Model:       UniFi-Gateway-4
Version:     4.4.29.5124212
MAC Address: 78:8a:20:7c:ba:1d
IP Address:  11.22.33.44
Hostname:    main-router
Uptime:      192 seconds

Status:      Connected (http://unifi:8080/inform)

How to Rebuild a RAID 1 Setup (WD My Book Duo) on macOS, the Right Way

This guide will talk about rebuilding a failed RAID 1 disk with WD My Book Duo on macOS, it should also works on Thunderbolt Duo or other RAID 1 setup.

Many other guides only tell you how to replace the whole two disks without restoring / rebuilding data for this common situation:

In a common case, your RAID 1 setup could fail with only one defective disk, while the other is online. If you see this status on your macOS:

  1. Buy a brand new disk, same size and model as your failed disk.
  2. Reboot your mac in recovery mode, this step is very important to make sure your disks are not used by macOS.
  3. In recovery boot, open Disk Utility.
  4. Rename your RAID set, for example if your RAID set is External-Raid, rename it to something like External-Raid-Rebuild or just some names different than your original, this is the most important step to make sure your RAID set wouldn’t be using after entering macOS system.
  5. Restart your system in normal mode.
  6. Login to your system, then open Disk Utility, congrats, you will see the new disk is now rebuilding.

Other notes:

  • Restraint editing/adding/removing files on your RAID set while rebuilding.
  • The rebuild process could take hours depends on your disk size (10-20 hours for a 4 TB + 4 TB RAID 1 setup).
  • After the rebuild complete, you can rename your RAID volume name back to original.

This is not the only way to rebuild your RAID with a failed disk. According to WD documentation, you can power on your My Book Duo without connecting to macOS (remove the thunderbolt cable), then the My Book Duo should rebuild it automatically. However, it’s really hard to know when the rebuild process will finish, there’s no special indicator status for this situation, so I prefer rebuilding in macOS. This could be the best method for me.